I’ll move on from GLPI eventually and start working on some interesting technical stuff, but today is not that day.
We ditched GLPI after we got hit by an accidental SQLi from HaveIBeenPwned – in short, version 9.4.5 is vulnerable to an SQL Injection flaw. You can exploit it by sending it an email (say, to helpdesk@company.tld) and once the email gets automatically turned into a ticket and assigned, the SQL will be executed. This affected us because the obscenely simple execution string was included in the header of the haveibeenpwned email notification.
I’ve just been poking around GLPI again (we have kept it around for non-end-user stuff, isolated and kept out of reach) and noticed that there was a “telemetry” scheduled task in the list of Automatic Actions which got me curious.
GLPI have decided to publish some of their telemetry data, which is nice of them. But it shows that there’s still a significant number of users running 9.4.5 and older.
Of the installs that report telemetry in the last year (and only those installs on 9.2 and above do this), 14,313 are on a version at or below 9.4.5, whilst 26,985 are on 9.4.6 and above. Over 34% of GLPI installs are potentially* vulnerable to this painfully simple exploit but over 12% of installs absolutely are still vulnerable as they’re on 9.4.5 exactly.
*Potentially, but I suspect only 9.4.5 is vulnerable – they fixed it by accident in 9.4.6 here which looks like a response to an issue that appeared in 9.4.5.
We learned a lesson with the GLPI issue – keep your software up to date. Though to be fair to us, it was up to date according to their website at the time. There was a newer version available (9.4.6) but that wasn’t advertised anywhere.
I hope these out of date installs get updated. We know there’s a lot of malicious activity out there, but at the same time… accidents can happen.
The recent bad experience with GLPI we had at work was the final nail in the coffin and, after patching the issue, I quickly began looking for alternative ticket management systems. We have wanted a new helpdesk for a long time – the support we provide has evolved over the years since GLPI was first introduced and with the covid-19 pandemic this support has seen yet another shift in the way in which we work. Not only are we doing things differently now, some unrecognised or unrealised issues have surfaced which we all wish to resolve or automate away.
We struggled through the worst of the lockdown but quickly identified some limitations with our existing way of working, namely that whilst our helpdesk did fine enough when it came to tracking tickets, it should do more for us. We spent a lot of time on the management of tickets and attempting to contact people just to get them to perform simple tasks. Why do we have to fight our helpdesk to achieve a goal, and why can’t we have a system that would let us run these simple tasks (read: scripts) ourselves in the background, invisibly to the end user?
I did some reading and some thinking post-GLPI-exploit and realised that we are essentially an MSP. We provide support for departments within a school, each of which has different objectives, priorities, demands and tools. Plus, we support several primary schools on top of that, and they are their own beasts entirely with unique networks, hardware and software, let alone processes and requirements.
As we emerge from total lockdown to a lesser version, we are also going to need to do our normal jobs but much more efficiently. With a “remote first” approach to minimise risk to our end users (both staff and students, but also guardians and members of the public) I quickly decided to look into RMM tools instead of basic ticketing systems.
Given that we also had issues in other areas (namely monitoring and alerting, in that what we have is barebones and actually broke a month ago) I was looking out for a solution that would kill as many birds with as few stones as possible.
The requirements were that it had:
A ticketing system
Monitoring and alerting
Patching, installing and scripting capabilities
Easy remote support
A good price (hey, we’re a school and don’t pass the cost on to the end user, we don’t have a lot of money!)
I found a list of RMM tools and their features on the /r/MSP subreddit and went through each one, checking videos, documentation and feature lists working out which would obviously not work, which might work, and which would absolutely work. I narrowed down the options to four potentials. In the end, only two of them had a “per technician” pricing model – the “per monitored device/agent” pricing model would end up costing us tens of thousands – so it came down to a war between the two. These were:
To be honest they were a close match. I preferred the look and feel of Atera although SyncroMSP was more feature rich. We didn’t really need all the features SyncroMSP boasted though, and Atera was cheaper (we’re on the cheapest plan) so ended up going with that.
Although we haven’t rolled out the agent to every device yet (we’re still mostly closed and until we have the agent installed have no way of deploying software out to machines not on our network) we have started using it heavily. So far, zero problems and we are all liking it. It has already enabled us to preempt some problems that would become tickets, solving them before they ever affect an end user or get reported. I’m looking forward to diving into it more, I am especially excited about the recently announced Chocolatey support, which seems to work wonderfully.
It’s early days yet, hopefully we can become much more efficient and provide a better service. Only time will tell!
I should say before we get started, the fault for this lies entirely with GLPI, I place no blame at the feet of haveibeenpwned.com or Troy Hunt for this issue. It’s all good fun! Concerning? Oh, for sure. You can’t help but laugh though. Obligatory XKCD.
On GLPI 9.4.5, creating a call (via the standard interface or email, etc) that contains the basic SQL injection string ';-- " will be logged normally with no abnormal behaviour, however if a Technician assigns themselves to that call via the quick “Assign to me” button, the SQL query will be executed. In the case of the example string given above, all existing calls, open or closed, will be updated to have their descriptions deleted and replaced with any text that appears before the aforementioned malicious string. You can of course modify this to perform other SQL queries.
This is fixed on 9.4.6, however at the time of writing the GLPI Download page still links 9.4.5 as the latest update available, you need to go to github releases page to see 9.4.6. [2020-06-03 update:] now available directly from the GLPI download page.
9.4.6 was released before I found this exploit, however the GLPI website still showed 9.4.5 as the latest version. As far as we were concerned, we were on the latest version. Credit goes to whoever submitted it first, however at the time I had no knowledge of this already being known and resolved. Here’s a video showcasing the issue:
Low quality to minimise filesize
The Long Version – how I found it, or how haveibeenpwned pwned our helpdesk
GLPI
We use GLPI as our technical support ticketing system at work. There are better solutions out there and we’re investigating others, however GLPI has served us well since 2009ish. It is a web based, self hosted PHP/MySQL application.
haveibeenpwned.com
A website that catalogues and monitors data dumps for email addresses. It collects leaked or stolen databases, analyses them, pulls out any email addresses and can be searched by anyone for free to see if your email address has been included in a breach. You can also subscribe to receive alerts if your email address, or an email address on a domain that you own, is included in any future breaches.
haveibeenpwned.com pwned GLPI
Around late April we upgraded from an ancient version to the latest of GLPI – 9.4.5. All was well until we received an email from haveibeenpwned to our helpdesk support address, which automatically got logged as a support ticket. This email alerted us to some compromised accounts on our domain which were included in the latest Wishbone data dump.
Spoiler: that header isn’t an image, it’s text!
I rushed to get the HIBP report generated to see who’s data on our domain had been compromised by clicking a link in this email-turned-support-ticket. We got the report in a second email, which created a second ticket. I grabbed the data, deleted the second ticket (as we still had the original open) and perused the data. After doing the necessary work alerting any users to the breach of their data I went back to the original HIBP ticket, and realising I hadn’t assigned it to myself did so and promptly solved it. All is well, time to move on?
Not quite. I and the other techs quickly noticed that every single ticket description had been deleted and replaced with partial header data from the HIBP email.
This immediately stunk of some kind of SQL Injection flaw and my mind raced as to what the cause was. I had a suspicion I knew… Unfortunately we were in the middle of business hours and due to Covid-19 are fully remote – we need a working helpdesk, and I don’t have the priviledge of working on potential security issues in the day job. We restored from a backup taken on the previous evening (not too much data was lost, thankfully) and carried on with our day supporting our users.
Understanding the flaw
As soon as work ended, I grabbed an ubuntu .iso and built me a webserver VM. I had a feeling I knew what the cause of this SQLi was (check the header of the email shown above, you don’t need long to figure out where the ‘malicious’ code is!) but wasn’t sure how it got executed – the email was parsed correctly and tickets weren’t affected when the email came in, it wasn’t until around the time I deleted the second ticket and closed the first call that problems arose.
After building the VM with PHP and MySQL, I hopped onto the GLPI website and grabbed the latest version from their site, which is shown as 9.4.5.
The “Download” button took you to the 9.4.5 archive
After setting it up and adding some test calls, I forwarded our original HIBP email to a temporary account I linked this test GLPI install to. Once the email was pulled in I went through the same steps as I had done earlier in the day:
Generate the report (which I didn’t do again via the link in the email, I just forwarded the original email to my test email account creating a second ticket in my install of GLPI)
Delete the second ticket
Assign the first ticket to myself
Close it
I checked my test tickets I loaded in there beforehand and lo and behold, they had all been wiped and replaced with the same content from the HIBP email!
I restored the VM to an earlier snapshot and went through the process again, pausing to check the other tickets at each step. I quickly discovered that the issue only occurs when you assign yourself to the ticket using the handy “Assciate myself” button.
Adding yourself as a watcher also triggers the query
Making it malicious
The email data already wipes the content of all tickets, but as it stands it leaves a lot of junk data behind. I wanted to minimise the data required to exploit the flaw yet retain the same behaviour.
Another restore of GLPI followed, with more tests trying to determine the minimum amount of data needed to execute the flaw. I spent some time cutting down the email from HIBP and quickly found that the opening lines of the HIBP email were indeed the culprit – I managed to shrink the exploit down to six characters (';-- " – the space and double-quote at the end appear to be required though this could do with more testing) to achieve the same kind of malicious behaviour, in this case deleting all content of the descriptions for every ticket in the database. If you log the malicious call with this string as the title (or leave the title field blank – GLPI will then automatically add the contents of the description to the title, in this case the malicious string we have identified gets added as the title) the title on all other calls gets wiped too, however if you do include a non-malicious title in the malicious ticket the original titles on the other calls do not get modified.
Success! This is a pretty severe issue, and although it does require some user interaction you can easily hide this exploit in an innocent looking support call. GLPI supports HTML emails, which get rendered (almost) normally within the interface. Simply hiding the text in an attribute or the <head> or something will keep it invisible to the tech. You’ve just gotta wait for them to assign it to themselves.
In the end, this isn’t a zero-click flaw but it is easily hidden. If you hide the exploit and it doesn’t work out the first time (a tech doesn’t assign it to themselves) you can easily try again with another ticket until it works. Odds are the techs aren’t going to read through the raw HTML of each ticket looking for problems.
Reporting it – late to the party
I hopped over to GLPI’s github page to check for an existing issue and log my own if one didn’t exist when what do I see but 9.4.6! I check the changelog and find this:
Looks like this may have already been fixed!
Well, darn. I downloaded and installed the update and can confirm the issue no longer exists. Congrats to whoever spotted it first! Edit 2020-06-04: Twitter user @thetaphi took a look at this and found that it was spotted and/or accidentally fixed by a developer whilst fixing a separate issue.
As it is already solved I don’t really want to dig through the code and find the offending line or develop the exploit further. Edit 2020-06-04: some people have taken a look after @troyhunttweeted about this issue. It is interesting (concerning?) that something this simple got through to release, especially when you consider the way to initiate the exploit is by assigning yourself the call. Why does the call description get parsed at all here?
Either way – if you’re running GLPI, make sure you’re on the latest release. Or look for alternative software.
