Tag: tech support

Interruptions in tech support

I recently read some posts on the damaging effects of interruptions and wanted to explore this in the context of my current job – sysadmin across multiple k-12/ks1-5 educational environments – and offer some thoughts on how to change things.

First off, the core educational environment itself is pretty much built around interruptions. You generally have a single teacher in a room of a few dozen kids. I’m simplifying here, but the teachers essentially have particular targets to hit each lesson (“teach this thing”, “make sure the group produces this result”, etc) and these lessons are typically somewhere around an hour long, though from what I’ve seen these lessons are shrinking in length to squeeze more different material into a day/week. Teachers have to hop between different targets every hour (or less) as the group they teach changes (the students have to similarly hop, but across entire subject ranges. Is there any surprise that, combined with the usual suspects (advertising, media, social medias’ infinite scroll, etc), young people have shorter attention spans?) Often these differing targets are in the same subject (Science, or English, or Mathematics, and so on) however different classes are at different points in the curriculum and even if some of those classes are at the same point in the curriculum (the students are of the same age) they’re often grouped by capability – some classes need more time and effort and revision of material than others, whilst the top groups delve into subjects at a deeper level.

A teacher has an hour to achieve a goal before moving on to the next goal with a different group. During this hour, they will typically outline the thing to be done or learned, then work with the class to help them get there. This naturally results in questions (interruptions) continuously. It’s in the teachers best interest to answer these questions well to ensure that individuals maintain pace with the group, but paradoxically the more questions that are asked the slower the group can move forward, as spending time explaining material again for the benefit of one means that those who do understand it already are not really learning for the time period it takes to get the one who asked caught up. Thus in my uneducated opinion, the priority for the teacher is to get the effectiveness of the first explanation of any material perfected (as closely as possible given the ability of the class) ensuring that most of the time most of the class (if not all) comprehends it to a suitable degree on the first attempt. This minimises interruptions and enhances learning for all, as the class learning velocity is kept high.

This problem is outside the scope of what I want to talk about but it is relevant, because one side effect of this we-must-acknowledge-interruptions behaviour is that I feel it becomes habit. Being interrupted a lot becomes the norm, and it is my opinion that this encourages the teaching individual to also be the one interrupting others more.

Let’s go back to the classroom. You have one hour to teach a group of 33 twelve year olds about some algebra. You fire up your laptop and switch the projector on to find that no matter how you try you can’t get an image to appear on the board. This unplanned interruption has immediately taken up brain power and, critically, time, even if you have backup plans for each and every lesson. As you have chosen to use the technology resource you have clearly decided that it is important, so we can assume that without that resource your teaching and the students learning will be less effective. You are after a fix for this problem quickly. The quality of your teaching is likely somewhat diminished and the longer this goes on the worse this gets.

So you do what anyone in your situation does – you try and get it fixed. This is where tech support comes in. You make a phone call or send an email to that tech you like, or if you’re a hero, log a ticket on the ticketing system. The hero did the right thing – logging a ticket. No complaints from me there. (Pro tip for techs reading this: make every avenue of communication a ticket generating event!)

However emailing an individual or phoning interrupts the support techs. This is often warranted and is always understandable, and your job is a constant stream of interruptions so one more won’t hurt, right? This is where, unfortunately, tech support suffer. We operate in the opposite universe. Your tech problem, as a teacher, is probably one of your biggest current problems. And we totally get that. Promise. We really do. But… it probably isn’t our biggest problem and is almost certainly not something we’re going to want to deal with right away.

An interruption to us does not progress anything, it in fact stops everything. If we are in the middle of resolving another problem, being forced to stop for a reason (whether it’s to deal with a major issue or respond to a phone ringing or being pulled out of the moment by a name being called) causes us to disconnect from “the flow” and at best mentally change gears, at worse slow the brain CPU way the heck down. This is jarring and as evidenced by the many articles published about interruptions can in fact be damaging, both mentally and economically.

The more this happens the less effective overall an individual or a team can be. I argue that an ops team lead (Helpdesk Manager, IT Director, etc) should put work into minimising interruptions for the betterment of the team and yourself, regardless of your vocation or the objectives of the team. Here’s what I propose you can do to help within your tech support team regardless of work environment… Though you will likely need managerial sign off on some or all of these. It’s worth noting that I didn’t come up with these, they’re merely an amalgamation of things I’ve learned and tested which have worked for me. I am assuming you’re part of a team instead of solo hero, too, though if you are solo then interruption reduction could potentially save you from burnout. Some of these might help.

Everything should (automatically) log a ticket

Make it easy to log tickets. Saying “no ticket no fix!” feels good but doesn’t actually achieve anything for the organisation, except to piss off someone with an issue (and let’s hope that person isn’t a C-Level.) Let people call, let people email, let people visit, let people knock on the door. You can more than likely automatically create tickets in your ticketing system that come through to email, and if you can’t, get a better ticketing system. Get a generic “techsupport@” inbox set up and configure your helpdesk to add everything sent to it as a ticket. Bonus points of you reply right away (automatically of course) telling the requester that their ticket has been logged. At least they know it’s in the queue instead of sitting unread in some mystery inbox.

Ten CC’s of Triage, STAT!

Yikes, you now have a flood of tickets. This looks bad. And hey, maybe it is, but at least now you know it’s bad instead of it just feeling bad. Maybe management would be surprised to see that you’ve got 143% more tickets than you last reported, because everything is a ticket now. No more invisible work.

Anyway, I digress. Triaging tickets is essentially reviewing a ticket and deciding if it’s a priority or not. There are many ways to judge this, but typically you want to take into account how urgently this needs to be fixed and how much of an impact it would have if it didn’t get fixed.

A PCI Compliance audit deadline of two hours ago is pretty urgent – it’s something that needs to have happened already. But… the banks aren’t going to block all transactions immediately. Business doesn’t stop because you’re a few hours late on submitting a self eval form (this is not an endorsement to delay PCI compliance!)

A projector in a classroom has a pretty high impact – that’s 30+ students (plus one pissed off teacher) per hour per day. That adds up quickly. And yes, it’s pretty urgent too, but worst case the teacher can still teach like they did in 1985. Right?

Triage is important for two main reasons:

  1. It helps you and your team decide on what to work on without having to decide on what to work on
    Generally, you work on the thing with the smallest SLA time left, and if you have negative SLA counting down then you should probably get on that ASAP. Assign the Priority (the urgency and the impact) an SLA (say, high impact high urgency = 4 hour SLA, low impact low urgency = 14 day SLA – whatever you decide will be unique to you and the organisation and likely requires understanding managements expectations as well as your teams abilities)
  2. Reporting, evidence, promotions, wage increase, redundancy protection, efficiency gains, areas of focus and more!
    Having all this triage data to look back on will highlight where things are working and where they’re not, but bigger picture style. You can use the data and the knowledge that you’ve gained to more closely align the team with the objectives *and reality* of the organisation. Perhaps when triaging your tickets you could also categorise them – hardware, software, licensing, accounts? Or go one level deeper – projector, laptop, desktop, TV, printer. Office, Web browser, MIS. After time passes you can really easily see where your team spends most of its time and where resources (like money) can be allocatied to reduce load in those areas.

Triaging tickets is generally not a full time job, so you could make this person…

Hero Assignment: Interrupt Shield

Designate an “interupt hero” – this can be one person or more and ideally they would triage tickets in the queue too. If you’re lucky enough to be able to do this, put them in another room and leave everyone else in the relative peace and quiet of the main office. Make everyone else go to this smaller interrupt room for direct in person support.

They deal with all phone calls and in person visits and group emails. They also deal with any quick support calls (when not on a phonecall) that don’t require them to leave their desk and that are interruptable, for example password changes, group membership updates, etc. You should rotate this person very regularly so you’re not moving the “interrupt problem” to one person all the time, as that’s a recipe for disaster. Let them do a day at a time, or two days if you use a hero team, rotating half the team out every day so handover and “current events” knowledge transfer (what were yesterdays big issues? What’s on the radar today?) can occur more naturally.

Redirect everyones incoming calls to the interrupt hero team. This further reduces interruptions to the main body, though you may want to make exceptions for VIPs. Your interrupt hero team should be able to forward calls to you, but generally they will be able to deal with most quick things (and push to the focus team via a ticket if not, of course)

This has two advantages. First off, the toil is reduced for the team at large – your senior sysadmin isn’t doing the third password change this week for little Joe – and secondly (and most relevantly) the rest of your team can focus for long periods of time on their work. Better work is done by most of the team. Better work is better results (whether those results are money/profit or a higher quality of education.)

Don’t overload on active tickets

Don’t assign anyone (including yourself) too many tickets. There is no perfect number – it’s different depending on role, personality, type of work, and many more factors. But try and get a ballpark. I would suggest starting with no more than 10 non-pending tickets (for clarity, I define a pending ticket as one which is waiting for something outside the control of any member of the team) and only one should be worked on actively at a time. You can’t install a printer whilst also diagnosing wireless issues. Pick one. Focus.

Low remaining (or negative/expired) SLA first

Plucking tickets from the queue because they look easy isn’t productive, long term. But it’s fine, short term. Don’t sweat it, manager. But keep an eye on it.

Tickets should be worked based on their SLA, accounting for the available suitable resources (don’t put the guy who sucks at DNS problems as the lead on critical DNS problems – train them? Yes. Rely on them? No.)

Sometimes it’s nice to jump 70% down the queue and grab a few easy tickets. It feels like a break.

Just don’t make it a habit.

Having repeat logs of the same call that has gone unanswered for three months is – you guessed it – yet another interruption. Old (the definition of which is, essentially, defined by your SLA – expired SLA? Ticket is old!) tickets should not exist (with very very few exceptions. Like budget availability.)

Don’t judge people on their call closure rates

How many tickets I’ve closed vs the other techs means absolutely nothing. Get honest feedback from the source – ask the staff and, yes, the students, to rate the support. Then pay it attention. This feedback is so valuable. Whenever a ticket is closed, automatically send a followup email asking for feedback, and make it quick. You want the emotional immediate feedback, not the “dwell on it for three days and [calm down/forget]” feedback.

  1. Was the support: [] Amazing! [] Ok! [] Shit!
  2. Optional comments here:

Negative feedback is the most valuable information you can possibly obtain

I argue that getting positive feedback means absolutely nothing, except as a “look how great we are, CEO!” line in the yearly review meeting. The feedback you really want, the feedback with actual damn value, is the negative feedback. And it’s all well and good getting it. But make sure to pay attention to it, don’t let it sit in the database doing nothing. Analyse it. Understand it. This is the closest you’ll get to dipping your hand in the metaphorical technological currents of your organisation. You’ll be able to feel the effect of the teams work, and when something goes against the current, you can fix it immediately.

This makes better, happier staff and students. Better happier staff and students equals better results. Which equals… you guessed it, fewer interruptions.


Interruptions aren’t going away, but you can reduce them. And when you do, that firefighting feeling will decrease, work quality and rate will improve and as a direct result the stability of the environment should improve, too. And when the stability of the environment improves, the interruptions decrease.

At least, that’s my experience.

There’s plenty more about this on the internet at large, much better explained and with additional steps than what I have listed here, however I feel that these steps are the big hitters that might just give you enough time to step back and take a real close look at the bigger picture without worrying about where the next fire will start. And it’ll take time. But it will work.

Error b8bb2b3e on HP Office Jet Pro 8710

TL;DR: Try disabling “SNMP Status” for the port on the Windows print server in Print Management. No guarantee this will work everywhere but it appears to have worked for me. Give it a go, let me know!

Turn the SNMP Status Enabled box off

Printer says… what, exactly?

Feel free to skip the rest of this post. It’s just the sequence of events that took me to the supposed conclusion/solution.

I arrived at a customer site to find a stack of issues, but one in particular caused some confusion to my smooth-brained self. A HP Office Jet Pro 8710 printer, which worked up until last Wednesday, had a black screen with an error on it stating:

There is a problem with the printer. Turn the printer off, then on.

I of course tried a reboot. The printer switched on, booted up and appeared to be fine. But after 5 to 10 seconds it would make a clicking noise, the screen would flash blue, then go black with the same error. I hate printers.

It looked like there was some white text on the blue screen so I recorded it with the phone in slow motion and picked up an error code: b8bb2b3e

Googling this took me to one relevant result, a post on the HP forum with no replies. Great. I hate printers (I may say this several times) and I hate an error code with no official documentation anywhere on the internet. So, what is one to do in such a situation?

Well, poke about, push buttons, and hope you figure out a pattern, obviously!

My immediate thought was something PrintNightmare related, however as the post on the HP forum was made in April I moved that away from the forefront of my mind for now and started looking elsewhere. As luck would have it, I noticed that the error occurred just as the Wireless light finished flashing (because yes, this printer is connected to a domain network wirelessly. Literally the definition of evil.)

I plugged in a network cable that I pinched from a nearby desktop and the printer stopped crashing. Great, progress! This building had a new wireless network installed a couple of months ago so I began poking at that but nothing had changed in the last couple of weeks. Instead, I moved across to the Print server to change the ports over to the new IP address the device has from the wired network assuming it was wireless related and… pop! Blue screen with b8bb2b3e again. My head flipped back around to PrintNightmare patch issues or something up with the server. I didn’t really have a direction of travel from this point as Event Viewer had nothing of interest in it as far as I could see, but I decided to send the printer a Test Print just before it connected. I’ve seen before issues with printers where it’ll process an existing queue of work and then crash out or error, indicating that the core technology (network, server, printer) is functioning but some feature is causing an issue, and that’s exactly what I saw here. The device connected to the wired network, sent out the test print page, then immediately crashed out again. Historically I’ve fixed this (rare) occurrence by removing and re-adding the printer to the print server but I decided to poke around some more and try to nail down which setting was causing this (if, indeed, any at all.)

Luckily it didn’t take too many attempts to switch off SNMP Status Enabled, reboot the printer, and not see any crashes. The printer is still working a few hours later (back on the wireless… grr. We’ll run some cable into this particular office soon) and I have since checked Windows Updates installation date – the printer stopped working right after updates were installed on the server. The server was up to date prior to this month’s (July 2021) patch release so it does look like something in the most recent set of updates caused this. And yes, I have ensured the server is not vulnerable to PrintNightmare by checking for those registry keys, so it’s not that.

If I unveil the exact cause I will update this post.

Ditching GLPI

The recent bad experience with GLPI we had at work was the final nail in the coffin and, after patching the issue, I quickly began looking for alternative ticket management systems. We have wanted a new helpdesk for a long time – the support we provide has evolved over the years since GLPI was first introduced and with the covid-19 pandemic this support has seen yet another shift in the way in which we work. Not only are we doing things differently now, some unrecognised or unrealised issues have surfaced which we all wish to resolve or automate away.

We struggled through the worst of the lockdown but quickly identified some limitations with our existing way of working, namely that whilst our helpdesk did fine enough when it came to tracking tickets, it should do more for us. We spent a lot of time on the management of tickets and attempting to contact people just to get them to perform simple tasks. Why do we have to fight our helpdesk to achieve a goal, and why can’t we have a system that would let us run these simple tasks (read: scripts) ourselves in the background, invisibly to the end user?

I did some reading and some thinking post-GLPI-exploit and realised that we are essentially an MSP. We provide support for departments within a school, each of which has different objectives, priorities, demands and tools. Plus, we support several primary schools on top of that, and they are their own beasts entirely with unique networks, hardware and software, let alone processes and requirements.

As we emerge from total lockdown to a lesser version, we are also going to need to do our normal jobs but much more efficiently. With a “remote first” approach to minimise risk to our end users (both staff and students, but also guardians and members of the public) I quickly decided to look into RMM tools instead of basic ticketing systems.

Given that we also had issues in other areas (namely monitoring and alerting, in that what we have is barebones and actually broke a month ago) I was looking out for a solution that would kill as many birds with as few stones as possible.

The requirements were that it had:

  • A ticketing system
  • Monitoring and alerting
  • Patching, installing and scripting capabilities
  • Easy remote support
  • A good price (hey, we’re a school and don’t pass the cost on to the end user, we don’t have a lot of money!)

I found a list of RMM tools and their features on the /r/MSP subreddit and went through each one, checking videos, documentation and feature lists working out which would obviously not work, which might work, and which would absolutely work. I narrowed down the options to four potentials. In the end, only two of them had a “per technician” pricing model – the “per monitored device/agent” pricing model would end up costing us tens of thousands – so it came down to a war between the two. These were:

To be honest they were a close match. I preferred the look and feel of Atera although SyncroMSP was more feature rich. We didn’t really need all the features SyncroMSP boasted though, and Atera was cheaper (we’re on the cheapest plan) so ended up going with that.

Although we haven’t rolled out the agent to every device yet (we’re still mostly closed and until we have the agent installed have no way of deploying software out to machines not on our network) we have started using it heavily. So far, zero problems and we are all liking it. It has already enabled us to preempt some problems that would become tickets, solving them before they ever affect an end user or get reported. I’m looking forward to diving into it more, I am especially excited about the recently announced Chocolatey support, which seems to work wonderfully.

It’s early days yet, hopefully we can become much more efficient and provide a better service. Only time will tell!

Videos not loading in Google Drive

TL;DR: Check that Third Party Cookies are enabled in Chrome and/or G Suite Device Management user settings.

We use Google Drive a lot at work. We are also beginning to use video as a method of training or recording information that you would typically write down. We do still write this stuff down, but sometimes a video is easier to create and consume alongside a document.

We’ve had an issue recently where, seemingly randomly, some people wouldn’t be able to play any videos through Google Drive. They’d open the video and the youtube video player would load, but the video itself wouldn’t start.

Right clicking and downloading the video worked, but that’s not really a suitable workaround to the issue.

We weren’t really able to find anything online about this issue either, but we did find out what caused it so if someone else searches, here’s what we did to resolve the issue.

(more…)

Tech Support Management: Projects, Respect, and PlayStation 4’s

I am incredibly privileged to work with a team of technical support folks who not only get along very well, but actively support each other on a daily basis. As the manager of this team, it is my duty to ensure they are being supported in every way possible by the workplace itself as well as the other teams within the organisation. This is a very delicate balancing act that I have yet to master.

Often, this ‘support’ means answering questions, taking responsibility for issues or working behind the scenes to enable the team to do their job effectively. A lot of the work I do (that isn’t GDPR related, and there’s a lot of that) is aimed at improving the quality of the work we do as a team, which in turn improves the quality of work the rest of the organisation is able to achieve.

Unfortunately, this kind of ‘enabling’ work that I or my team do is often invisible to both the organisation ah large as well as the team itself. It can take many meetings and a long time to get anywhere in management as everything on the organisation level often happens too quickly to react to, or takes such a long time that any progress that is made is barely noticeable to anyone not involved. Either way, this work is often isolated into individual projects, each of which requires lots of careful thought and consideration before any work can be started.

The first step in any project is to figure out if it is even worth it. Whatever the idea or project is, you have got to make sure you are doing it for the right reasons. These reasons are varied, it could be cost saving or efficiency improvements, or the introduction of something new, such as a product, feature or resource.

Understanding the purpose is key to progressing into the planning stages. These plans could outline the single task required for the implementation of smaller projects or building a fully fledged project plan for larger endeavours, such as to introduce new technology or rebuild some core infrastructure. Whatever it is, planning is important, takes time, and happens invisibly to anyone not directly involved.

Working in the public sector, money is often the deciding factor when it comes to projects or general progress and change. Producing an effective budget is something that I am constantly improving on. It sucks up quite a lot of my time, however at the end of the day all that everyone else sees is a number that they can spend up to. It’s my job to make sure these numbers stay within certain bounds and do not change too much, and to also keep an eye on the state of things to see if any cost savings can be made. Doing this is quite time consuming and can often be invisible to the casual observer.

So there’s an idea for a project that you believe will be good for the organisation, those teams that it directly affects know that it will be great to do, and with luck you know that the money will be there (as it’s always worth getting buy-in from the finance team first!) The next step is trying to convince everyone else in the decision making chain that your idea or project is worth both the time to bother actually implementing as well as the money you want to spend on it. Working towards this “yes” from each person often requires background tasks, sometimes these involve the team in ways they don’t recognise, too…

Quite often I’ll ask my team to perform a string of seemingly unrelated jobs or to focus on a certain type of issue in order to affect the outcome of some discussion about a project I know will occur in the future. There’s an element of “keeping certain people happy” which unfortunately ties in to this, too, but I try very hard to treat everyone as equals, and certainly avoid treating anyone as inferior or less important. As anyone who has worked in a technical support role will know, this is a very difficult thing to do. That one customer who always brings you snacks and tasty treats? It’ll be near impossible for your team to not give them preferential treatment. That other customer who is constantly rude and abusive? Incredibly hard to work with and support effectively.

But that’s my job. Give the treats to the team, take the abuse so it doesn’t hit my colleagues. And get all the invisible management/planning/budgeting stuff done to enable the continual improvement of the organisation as a whole. Oh, and again, there’s a whole boatload of GDPR stuff to deal with, too.

It is difficult, tiring work. I am by no means perfect of course, but I like to think I’m generally successful in doing my invisible work. I always strive to improve and do better for the team and the rest of the organisation.

Unfortunately the team doesn’t often see what I get up to day to day. I’m often out in meetings or deep into spreadsheets, only to occasionally request work from them, seemingly for random reasons or for no reason at all. I’ve been on the other side of this and I know it can be quite frustrating. Communication with the team is key here, and I am always looking for ways to foster this within the team, but especially when it comes to myself communicating with the team members about what I’m up to and where all our work is heading.

Having their respect and support is important to me and I encourage open criticism, however it is very hard for me to measure how well I am doing with the team. It’s one thing to complete projects, but knowing that a team of people are happy with me (or not) is something I have yet to really figure out. It’s something I worry about quite a bit, actually.

So it came as a shock to me on the last day of work in 2018 when I arrived to find that the team had secretly set up a treasure hunt in the workplace filled with riddles and puzzles.

The team had come together and planned this months ago, contributing towards a gift, then working on the clues over the last couple of weeks and setting it all up without any indication to me that it was happening. All out of hours, I should add! After the dozens of clues and puzzles, the final clue led me to a box with a wrapped up Christmas present in it.

That present? It was a PlayStation 4 with four games. Yeah.

It blew me away. It was the last thing I ever expected anyone – let alone the team at work – to gift me. Despite my initial refusal to accept it, I now have it connected up to the 11-year-old 720p 40″ TV (which is now due an upgrade to at least a 1080p model thanks to the PS4) and have been enjoying it every single day of the holiday period.

The four games are:

  • The Last of Us (remastered)
  • Uncharted 4
  • Red Dead Redemption 2
  • Shadow of the Colossus (remastered)

To top it off, my partner has known about this for months and got me God of War for Christmas. She somehow kept it secret the whole time. I had no idea this was happening, even up until holding the thing in my hands. It still blows my mind that anyone would do this for me.

Clearly, I must be doing something right!

I’ve had a PS3 since release, but it died around six months ago. Not that I played any games on it anymore, it was a glorified Netflix box. Normally my gaming was done on a PC, however that too had suffered from age. The newest component is at least 4 years old so it’s not able to run modern games too well, but the PSU has also failed and I haven’t yet replaced it.

I used to play video games a lot, however since moving out I’ve been rather preoccupied with all the renovating and cleaning. I still dabble, though. I’ve been playing a little bit of Borderlands 1 & 2 and The Witcher 3, which I have been running on a borrowed laptop. Since the PS3 died, Netflix was being run through Firefox on Linux Mint which I installed on my partner’s low-to-mid-end-at-release laptop that’s ageing quite rapidly.

As you can probably guess, this has now changed. I’m still a PCMR at heart, but the ease of playing on a console and the fact that it’s the only reliable access to gaming I have, PLUS the fact that it’s the most modern bit of technology I own, means that I am now back on the gaming scene. There are a lot of current gen games I’ve missed out on, plus some really awesome looking PlayStation exclusives that I can now play.

I suspect, though, that this gaming will be infrequent once the holiday period is over. I will still make time for it of course!

In fact, I plan on writing reviews of the games I play on here. They may not all be modern games, nor will they be overly professional or likely to contain anything new, but I have plans for a “media review” feature for this site that has now opened up quite significantly in scope with the addition of modern video games. There’s no better time to start than now!

The first game review will be Uncharted 4 as I have already completed it.

If any of the team from work eventually come across this: thank you. Now get back to work 😁