Tag: sysadmin

VMs not booting after October 2023 Windows Updates

Booting a VM and you get the error “Failed to power on with Error ‘Incorrect Function'”? Yeah, this hit us too. Looks like it hits VMs with secure boot enabled.

Quick fix: Move, delete or rename the .mrt & .rct files associated with the .vhdx file that the OS of the non-booting VM normally boots from. It’ll then boot normally.

Alternatively you can uninstall the updates kb5031364 and/or kb5031361.

Implement but never move

I really appreciate well-written implementation guides for server-client software, but what really gets me excited is seeing migration guides for when you need to decommission that legacy OS and move onto something supported and current. Even the bad migration guides are great to have, but some that I’ve come across are works of art – numbered-list instructions that are clear and concise, incorrect assumptions busy sysadmins have during the migration are highlighted then corrected, screenshots where required, mid-migration tests to ensure things are going as smoothly as they seem… ahh, dreamy.

One of my personal pet peeves, however, is purchased server based software products that don’t come with any migration guidance or instructions.

I’m spending a lot of time at the moment moving niche (read: there’s only a couple of options and they all suck) and poorly made software from older server OS’s to newer ones, and whilst some of them are fine and can be figured out right away, most of them have… issues. Typically you can just install $dumpsterFire fresh to the new server, dump the existing database and throw it over the fence to the new server (yes, some of this terrible software rEqUiReS tHe DaTaBaSe Is oN tHe SaMe SeRvEr), maybe I’ll need to update some config files and point the new install at the existing database. One CNAME change later for the clients and you’re golden. Simple stuff.

Sometimes, however, you need to perform some forbidden magical incantations, and the kick to the teeth in many cases is that nobody in $org will write down what they are publicly. So you gotta pick the phone up and get some overworked and underpaid $tech at $org to walk you through the process, shooting down the constant stream of bugs and errors that occur along the way due to the shoddy quality of $dumpsterFire with bullets of solidified experience (that’ll no doubt be lost when $tech has had enough and leaves for greener pastures.)

Or, even worse than that… the $org requires payment for a migration because migrations aren’t considered “support” and are “optional”. Yes, it is support. And no, it isn’t optional.

Dear organisations that explicitly hide their migration guides to force an already-paying customer to pay you yet again to migrate your horrible software from one server to another (immoral), that INSIST that you “must TeamViewer in to do this” (in business hours only!?), that there’s no possible way anyone else can do it (false and stupid), that require DA/root accounts (you don’t), that have to be installed on a Domain Controller (I’m crying), that MUST have unmitigated 24/7 access by installing un-licensed teamviewer (Oh no you won’t)… There’s even one software solution here which “requires” a physical server. In 2023. The software won’t work in a virtual machine. (Oh, wait, spoiler: it works fine in a VM and has been working fine for over a decade)

To those organisations I say: fuck you. Do better.

Because I’m writing this all down, I’m recording my screen when you connect to solve unhelpful errors in your hidden log files, I’m fixing your stupid permission requirements and immediately uninstalling any additional or third party crap that isn’t a business requirement.

And I’m publishing those guides online for free.

Duct Tape

Back when I did end user support as my primary job I’d often ask myself “if there were no helpdesk tickets, what would I work on right now?”

The list I’d come up with was generally fairly short and changed each time I found myself pondering it, typically because the needs of the environment changed quite frequently, but it would also contain a few regulars – patch this, upgrade that, move that thing from there to there. Stuff that wouldn’t have an immediately beneficial impact but should probably happen. These are all good things to do and I encourage them to be part of the daily workload of a team.

The dynamic entries in the list were always something to do with making something better, getting rid of an annoyance or frustration. These are the things that do actually have immediate ROI and I would argue they should have time spent on them even during high ticket load periods but often find themselves being ignored because… Well, it ain’t broken. It’s just not great.

The issue is, as hinted at, this list changes often. Sure you can keep notes, an ideas board or submit a ticket, but when you do finally get an hour to look at the list, what’s on it doesn’t seem that critical.

I just read an older post on rachelbythebay.com that summarised this and explains what it means in five words:

Look for the duct tape

Read it (and whilst you’re there, if you don’t already consume that content take a look at some other posts – very valuable information and opinions contained within)

Essentially, find the things people have whipped up a quick workaround for and are using and fighting with now, whether that’s within your team or another. Spend some time making that thing better, or resolve the problem at source if possible. You’ll not only make that person or team happy, you’ll also actively solve a now problem that will probably directly impact success, however that’s measured.