When things don't work as they used to...

    Once upon a time I did set up a script (something like “yum –assumeno update” but to just filter out package names) on my Fedora machines to automatically send me an report about the updates that need to be applied nightly and since Fedora is an development distribution the emails should be daily (or at least weekly), but for some reason last couple weeks have been rather quiet…

    So digging in the internet a little and what do I found out – Fedora 31 EOL was 24 of November 2020… Another thing that happened on the midst of chaos that the year has been so far… So I have to upgrade several PC’s to the next release… I know Fedora 32 is not the latest version to upgrade to, but I like the stability of being 1 release behind the latest version as it has most likely been fixed from the bugs that sometimes creep into the latest releases. Also I don’t run the latest and greatest hardware so I don’t have the need for the latest kernel drivers so I usually don’t need the bleeding edge software...

    It has been a while since I had to dist upgrade my fedora machines (I usually do it one or twice a year as the EOL time nears, but seems that in the midst of all the chaos that is year 2020 I forgot to do another upgrade cycle until it already was past EOL for some time. Oh well – time to plan one night for downloading a lot of updates… I usually don’t have any issues with upgrades as the process for doing it is pretty much the same as it was 10 years ago. Yes it has gotten simpler and easier in years as the yum/dnf framework has evolved and I think the last mishap of dist upgrade going wrong was with Fedora 19 and that upgrade was long time ago. I just love the simplicity of the process in the midst of things. Windows people would say to that that the system is slow and cluttered and you should re-install windows regularly to keep the clutter and legacy software away, but this is Linux – you don't get all sorts of extra dll’s and who knows what… Yes in time some software is obsoleted and replaced with new, but still… One of the laptops I use regularly has been upgrades since Fedora 7 and now its running 32 and most likely will go to run even newer versions…


    Enough of rambling – the procedure I used to upgrade from F31 to F32 or to do a dist upgrade on my machines is this:

“dnf clean all”

“dnf –releasever=32 update”

“dnf –releasever=32 –downloadonly –skip-broken upgrade“

    This can be done remotely over ssh or from graphical environment and it just cleans out the F31 repository data and downloads the F32 meta-data and packages (I have slow internet so it took a while). I usually don’t have million repositories enabled so I don’t have a lot of package conflicts that need resolving. But now I recommend to switch to the terminal to do the actual upgrade so “Ctrl+Alt+F2” (yes I have done the upgrade over ssh and from the graphical environment, but I have had mishaps where you have 3000 packages upgraded and just when it is time to start uninstalling the previous version packages some script gives an error and the upgrade crashes and you are a left with 7000 installed packages just because some dependency is missing in the graphical environment).

“dnf –releasever=32 --skip-broken upgrade”

    And usually that’s all that is needed to do dist upgrade. Well not this time – this time I was left with packages that have an F32 version available but for some reason would not upgrade and the message was cryptic:


WARNING Modular dependency problems:

Problem 1: conflicting requests

- nothing provides module(platform:f31) needed by module gimp:2.10:3120191106095052:f636be4b-0.x86_64

Problem 2: conflicting requests

- nothing provides module(platform:f31) needed by module libgit2:0.27:3120190407181414:f636be4b-0.x86_64

    So it seems that Gimp and libgit2 need something that was in F31 and is no more in F32… Great… Google was also not particularly helpful as there were bug reports about this message but no real answer. After digging some more I found out that dnf has some new features where packages can have some sort of modules that they require in “/etc/dnf/modules.d/” and well after upgrade the version they were searching was not there any more… Again after some searching that the removal of package was not enough to remove and the only way would be to manually edit each offending module to add “state=disabled” to the file.

    Well… As I don’t have anything that is this module specific I found an easier way:

“dnf module reset '*'”


No more annoying modules :D

And when I now run “dnf upgrade” I end up with

Upgraded:

apache-commons-codec-1.13-2.fc32.noarch apache-commons-io-1:2.6-8.fc32.noarch apache-commons-lang3-3.11-1.fc32.noarch apache-commons-logging-1.2-20.fc32.noarch apache-commons-net-3.6-8.fc32.noarch hawtjni-runtime-1.17-2.fc32.noarch jansi-1.18-3.fc32.noarch jansi-native-1.8-2.fc32.x86_64 jsch-0.1.54-11.fc32.noarch jzlib-1.1.3-12.fc32.noarch slf4j-1.7.30-2.fc32.noarch xalan-j2-2.7.2-2.fc32.noarch xerces-j2-2.12.0-4.fc32.noarch xml-commons-apis-1.4.01-29.fc32.noarch xml-commons-resolver-1.2-29.fc32.noarch

    So… I don’t know why those modules are good for but I don’t need them and resetting them let me to finally upgrade the system…



Until next time I guess...

SIL reloaded...

When I started this project it was meant to be just tips and tricks from random projects I encounter in my travels through the IT world, but like always, life makes it own corrections… Sometimes they are for better, other times they are are for worse.

    The 2019 started like usual with with all sorts of interesting projects and new ideas and new tech to play with. It was fun while it lasted… But all was not well by the summer time. I had a pet project that I was doing with couple of people and at one point I got played. The project died and I had invested so much time to it that the rest of projects had slipped away and taken over by other people. This happens from time to time with joggling multiple projects… No biggie I thought then – boy was I mistaken… By the beginning of summer it was clear that the pet project was dead for me and I was dismissed from it even tho I had put in 100% of my time and energy. So I thought I would spend a nice summer with my family any by autumn another project would come along as it had happened from years before. Little did I know how much time it had took and how much people had I alienated because of it… Yes there were few small projects that took few days or even up to a week but nothing special – nothing to look forward to… Even small insignificant projects wanted NDA so nothing to write about – so I put this project into a dormant state – there was just nothing interesting I could write about that I thought would be interesting….

    And then the 2020 happened – things could not go worse I thought. Well I was mistaken again. Who would have thought that a small insignificant corona virus that has raised it’s head for decade or to would have so big impact on the world. It was not the first time a flu like virus has raised it’s ugly head in the midst of the regular flu season… Who would have thought that it would turn the world upside down and throw it to chaos and disarray. Yes – it is better than living in the dark ages where the plague roamed freely and there was no real cure for it, but still – moving away from your comfort zone is always hard, but still… Being used to a certain standard of living and getting it taken away so easily and with all the restrictions on travel, shopping… I know people in large metropolis's are used to it to wear a mask day to day bases, but to the rest of the world it is not normal. It just rubs against the grain… Yes there are good things that have came from it like most IT conscious companies have had to create remote working capabilities to their employees so people can work remotely and without the hassle of getting to an office exactly the same time as the rest of 90% of people. But still the infrastructure project have withered away and everyone wants a web shop or a e-commerce site to dropshipping from china. And if that was not enough that airmail from china was 20 days before – it has gone up to 90...120 days if it even arrives. The USA and china trade war is also creating some friction between China an the rest of the world and to sum it up – we live in the interesting times.

    What will it mean to this site? Well… in short – I don’t have something interesting lined up for some time now so the best I can do is try to tinker with my home lab and hope it is enough. Maybe something interesting or useful comes up form time to time but no fancy new hardware or licensed software for the time being – lets see what the future brings and take it a day at a time. The project is not dead but just shifted to lower priority.


PS! My writing skills have also deteriorated over the past two years (have not written for public consumption for few years) so I’ll try to improve myself again :)

WSUS broken from start...

Today (well 5 days ago actually) I hit a problem... with Windows Server 2019 and WSUS. Maybe it is just a luck thing, or may be it is just an issue with Linux expert trying to fiddle in the windows world, but my goal was to set up an WSUS server and the first thing after adding a WSUS role to it - well the initial configuration failed to complete.

The WSUS content directory is not accessible.
System.Net.WebException: The remote server returned an error: (503) Server Unavailable.
   at System.Net.HttpWebRequest.GetResponse()
   at Microsoft.UpdateServices.Internal.HealthMonitoring.HmtWebServices.CheckContentDirWebAccess(EventLoggingType type, HealthEventLogger logger)

To start from the beginning - working on one project I was asked to setup WSUS (Windows Server Update Services) because the Windows updates were wreaking havoc and people were pretty pissed off on the management about it.
Right...
Linux guy starting to administer Windows...
That's like asking for trouble...
As I was the only one who had even heard of it, I said "How hard can it be"...
Big mistake...

I'm not in any way certified to administer windows - I poke it with a long stick from as far as I can, but I take this "experiment" as a learning experience, so I try it at least. On a positive note - the server manager with its role based "installation" feature is kind of nice try to make things as simple as possible (if it works). Like windows loves to do, the next.. next.. next... and we are installing. Great I thought - if it really is that simple, then in about some time later we should have a working update server and all the update fuss can be done like once a month or as needed or... (I have never used it so I really don't know how its supposed to work - all I know that when WSUS is added to the domain and all PC's report to it no updates will be installed if they are not "approved" in there".

And then the problems started... The error above... The first thing I saw was that the initial configuration (script?) failed to complete. Well this is great... Fresh clean install of Windows Server 2019 - only role installed was WSUS and it failed. I had a bad feeling it could not go that smoothly from the start, but I hoped for the best. What can I do now - I guess google will have a quick solution for it?

Well - yes and no - after searching for hours I finally found a thing that was wrong in my installation. It seems that some kind of connection limit is set to 0 so IIS won't accept any connections (even from the initial configuration script?).
So to remedy the problem:
1) open  IIS manager
2) navigate to wsus administration
3) on the right under browse website go advanced settings
4) under limits if max connections is 0 change to 2

That at least fixed the problem for me, so that the initial configuration script would setup the WSUS service and allowed me to select the software collections to sync.

As I'm writing it postmortem from notes scribbled on a peace of paper I don't have the original link where I found the solution, but it seems like it is a common problem that has not been fixed for a long time and it may reoccur with updates.

To be continued...

Installing Linux in 2019

It is January of 2019 Lets see how easy it has become to install Linux on a PC. Does it still require complicated hands on hacking to get it up and running? Well it depends on a flavor you choose, but most of the mainstream distributions wont need more than few clicks and couple of text boxes to fill. I could say that it takes even less of an effort than the first boot of an sysprepped stock windows 10 next, next, next, no, no no, no… setup. So lets see how easy it really is.

To begin with we need an installation media from which we can install the distribution. My go-to flavor for desktop/graphical environment is Fedora (specifically KDE spin) and for servers CentOS . I have used other distributions like Debian, Ubuntu, SUSE Linux Enterprise Server, Oracle Linux and Red Hat Enterprise Linux, but Oracle, SUSE and Red Hat are subscription based services so are less used on services that do not require to be certified on specific distribution. With Debian/Ubuntu… I can use them, but have never mastered the software packaging so I don’t use them much…

So today’s subject is installing basic CentOS machine called bazaar (well I have installed it and it is installed on physical hardware so to get the screenshots I’m just creating an virtual machine under KVM). This will become headless “server” for RPM cache to begin with and possibly acquire some more roles in the future.

So why do I need an rpm cache? Well – I have 4 Fedora machines, 3 CentOS machines and about 40 CentOS VM (no they are not online 24/7 – I boot them up/patch them as needed) for different projects and to play with. I patch them regularly and well my internet connection is not great (12Mbit down/1Mbit up) so when I have to upgrade Fedora 28 to Fedora 29 (which means downloading about 1..2GB of rpm-s for 4 times… taking an hour or two each time… well I’m tiered of waiting). I’m planning to do this upgrade in a week or two so I need an rpm caching server. This is not an issue with Fedora only – CentOS also releases major updates that may be as bit as 1GB (depends on how many packages are installed) and doing it 40 times… well… it takes a lot of time just to download everything again and again.

So how powerful the hardware needs to be? The answer is – I don’t know yet. I had an ASUS Eee Box EB1012P lying around with 4GB RAM, 250GB 2.5’’ WD SATA drive and a gigabit LAN adapter. The CPU is not fast, RAM is not fast, HDD is not fast, but it has 2 core 4 thread 13W TDP CPU. So average power consumption is low - it should stay around 5..20W so it is perfect for 24/7 operation. Yes an SoC like an Raspberry Pi would be more power efficient but I have had bad experience with SD cards dying on heavy IO and I don’t currently have a plan to start making backups of the rpm cache so Eee it is...

To start with I downloaded an CentOS 7 IOS and transferred it to an USB stick. Boot the machine and pressing F8 select the removable media as a boot device. The screen should look like this:

If the Install option is not highlighted then just press the up arrow on the keyboard an select it and press enter. The screen should look similar to this:

It should boot and ask for an language selection:

I like the language to stay English so it is easier to search for problems when they occur but it is up to the administrator to use the language they want. When done click continue and you should be arriving at a screen like this:

I’m hoping, that setting the timezone and keyboard layout is self explanatory so I wont go into that. Also leaving software installation source as local media is recommended (there is possibility to add external repositories, but relying on experience, it is easier to add them after the installation has completed). Since it will become headless “server” minimal install is sufficient.

On a production server I would leave kdump enabled as it will create kernel dumps that can help debug hardware issues, but when creating VM’s for testing I usually disable it as VM’s dont have direct hardware connected to them. I also like to disable the security policy as the policies are not included with CentOS and are available only on official Red Hat Enterprise Linux. This does not mean that CentOS is less secure than RHEL – it just means it is not certified under the example policies. Under network configuration… well it should be self explanatory… I’ll use dhcp in this example, but servers should be configured with static IP.

The hing that needs a little attention is the installation destination:

This VM has a 20GB virtual disk attached to it and since it has no partitions on it it is automatically selected and it should be good enough for basic installation (it is easier to make changes afterwards to change partitioning as it uses LVM by default than try to make a custom partitioning beforehand). So all that is left is click done and then “Begin Installation”. We are on the final stretch…

This should be the screen we are on now. Setting root password is like setting local administrator password in windows. The user root is the superuser that can do anything in the machine so its password should be strong. The user creation tab is optional but recommended as root user should be used only on an emergency's and not for day to day operation.

This screen is similar to setting the root user password with exception of adding username to it AND “Make this user administrator” option. Checking this box grants this use an option to run commands as a root without knowing the root password using "sudo" command. If this is done all that is left to do is wait for the installation to complete and reboot the machine when the option appears.

Now, if everything went well we should be greeted with a boot screen like this:

That means that the installation was successful and we have a brand new Linux machine available.

This should be sufficient for an example on how to install Linux in 2019 and in the near future we’ll configure it.

The Beginning...

It has been an decade since I last blogged about IT, but like smart people say, it is never too late to start (again). This time I at least don't have any limitations on what I can write and what not.

This time I come prepared - I at least have an idea on what I should write and not just ramble about random errors and how to fix them. I also keep this as a reminder/go to guide for quick fixes I have found/made so it is not just for You to read, but also for me to use as a scratch pad or a wiki so I would have a place to keep my notes to myself. You never know when you need an solution for a problem you solved 6 months ago.

Most likely the contents of this blog will be about me building up my test lab setup from scratch using off the shelf components, maybe some eBay/Amazon acquisitions or just hardware I think I need or want to play with. The same goes for software - as I'm a Linux user, most of the stuff will be about Linux. But I'm not limiting myself to just one platform - it is always good to educate yourself with different things and so there will most certainly be some other platform notes.

I'm also working networking and my current setup is not adequate for complex network testing, so network rebuild will also be on the road map. I don't know what hardware I will get for the time being, but there will be some hardware upgrades in the future for sure.

I also do some debugging of software so there may be some topics that touch some debugging of applications and/or fixes how I resolved them. I cant show everything I do as work related stuff is off limits but there should be plenty of things I can write about.

What I’m not sure is how much electronics posts should I write or if I should write at all… I occasionally do basic electronics and if there is a micro controller involved the lines get a bit blurry… So well see what comes of this…

In short – there are a lot of topics to cover and I try not to blab about some random topics that have no reference with things mentioned above. I’ll try to keep this blog as clean as I can from unrelated things as I can and if something should creep in – well I’m sorry…

Anyways – if any of the topics interests you – feel free to browse along – otherwise thank you for visiting :)