A global IT outage has caused chaos at airports, banks, railways andbusinesses around the world as a wide range of services were taken offline and millions of people were affected.
In one of the most widespread IT crashes ever to hit companies and institutions globally, air transport ground to a halt, hospitals were affected and large numbers of workers were unable to access their computers. In the UK Sky News was taken off air temporarily and the NHS GP booking system was down.
Microsoft’s Windows service was at the centre of the outage, with experts linking the problem to a software update from cybersecurity firm Crowdstrike that has affected computer systems around the world. Experts said the outage could take days from which to recover because every PC may have to be fixed manually.
Overnight, Microsoft confirmed it was investigating an issue with its services and apps, with the organisation’s service health website warning of “service degradation” that meant users may not be able to access many of the company’s most popular services, used by millions of business and people around the world.
Among the affected firms are Ryanair, Europe’s largest airline, which said on its website: “Potential disruptions across the network (Fri 19 July) due to a global third party system outage … We advise passengers to arrive at the airport three hours in advance of their flight to avoid any disruptions.”
Having half of the world depend on a corporate proprietary single company is the stupidest thing ever. They will learn nothing with this, sadly
Reminds me of when Canada lost internet to 12 million of it’s 33 million people because one company messed up doing maintenance.
There will be no consequences for those who made this choice because going with the biggest suppliers is never wrong: they in theory have the highest reliability, and even if they don’t, then it’s not just your problem but everyone else’s too, can’t blame those responsible when the outage is akin to an “act of God”
Are you suggesting lower cost and some convenience in exchange for incomprehensible risk is somehow a bad deal?
It’s great to have alternatives. If it was all linux, and linux got hit, then it’d be the entire world in danger. Too bad M$ is just not good enough for it’s second most popular position.
Well, we got to see roughly something play out with the xz thing. In which case only redhat were going to be impacted because they were the only ones to patch ssh that way.
Most examples I can think of only end of affecting one slice or another of the Linux ecosystem. So a Linux based heterogenous market would likely be more diverse than this.
Of course, this was a relative nothing burger for companies that used windows but not crowdstrike. Including my own company. Well except a whole lot fewer emails from clients today compared to typical Fridays…
Agreed on both counts. This happened because Microsoft made adoption easy. And this will be fixed within a day. None of the fundamentals have shifted. Even though it’s stupid, this isn’t going to fundamentally shake anything up.
Windows PC running Crowdstrike.
Shhh
The OS getting fully bricked because of a third party software update is still very much a OS level fuck up.
Depends. Since this is security software it probably has a kernel driver component. I think in linux a 3rd party kernel module could do the same. But the community would not accept closed source security software, especially not in the kernel.
They even have a version for Linux, which is a kernel module.
Has that been impacted by this?
No it has not. Validated on Ubuntu 16.04, 18.04, 20.04, 22.04 running CrowdStrike Sensor
Except crowdstrike literally doesn’t work like that on Linux.
My Debian system was bricked when it “upgraded” to systemd.
Required attaching a monitor to a normally headless server to fix. (Turns out systemd treats fstab differently and can hang booting if USB drive isn’t attached.)
Steam, a 3rd party program, has nuked the home directory of users who didn’t really do anything wrong.
Programs have huge abilities to bork systems, be it Windows or Linux…
I’ve seen RHEL completely crap itself due to a 3rd party update. Wasn’t that long ago fairly certain it was a McAfee update that took down a bunch of our Linux boxes. It happens.
Got hit with this in the middle of work. We only have one customer using CrowdStrike, and only staff PCs, no infrastructure. But this one is REAL bad, caused by turning your PC on, and cannot be patched - each affected PC needs to be manually fixed. Would not be surprised to see Linux usage go up after this.
Honest question, since I’ve been seeing these sorts of anecdotes all over the Internet: why the fuck didn’t your IT group catch this with a simple patch management process?
Updates for CrowdStike are pushed out automatically outside of any OS patching.
You can setup n-1/n-2 version policies to keep your production agent versions behind pre-prod, but other posts have mentioned that it got pushed out to all versions at once. Like a signature update vs an agent update that follows the policies.
Wow… That’s completely insane. Terrible approach for a software company. Thank you for explaining.
Everyone shitting on windows, yet this thing exists on Linux as well… I also started to dislike windows, yet this is not the time to be against windows users, this is to go against Cloudstrike together for even letting this happen.
I agree. I also think part of the blame can be placed on the system administrators who failed to make a recovery plan for circumstances like these – it’s not good to blindly place your trust in software that can be remotely updated.
In Linux, this type of scenario could be prevented by configuring servers to make copy-on-write snapshots before every software upgrade (e.g. with BTRFS or LVM), and automatically switching back to the last good snapshot if a kernel panic or other error is detected. Do you know if something similar can be achieved under Windows?
Sadly, I don’t know. I’m way worse with computers than I want to be, just careful about where I get my information.
It’s the time to go against proprietary monopolizing software
Exactly, the blame here is entirely on Crowdstrike. they could just as easily have made similar mistake in an update for the Linux agent that would crash the system and bring down half the planet.
I will say, the problem MIGHT have been easier to fix or work around on the Linux systems.
Citation needed, my NUC running Fedora made it through this without a hitch
… my work uses Crowdstrike
I didn’t see any issues rise up yesterday. Is today gonna be a bad day?
The second I read this post my phone started blowing up. Good luck brother.
I made an announcement on our Teams channel, and its blowing the fuck up… today is going to be a bad day :(
Forbes has posted a fix, but it requires a human to boot into safe mode/recovery
This occurred overnight around 5am UTC/1am EDT. CS checks in once an hour, so some machines escaped the bad update. If your machines were totally off overnight, consider yourself lucky
What amazes me is that so many big companies still use windows in critical core infrastructure.
Windows endpoints is one thing, but anyone using windows servers and MSSQL for mission critical application stacks need to be hit with the modernization hammer.
And then on top of that, they do not have a test rollout of any changes in a test environment, before rolling it out in the production stack.
Good luck to all the engineers in the trenches, having to fix the mistakes of their leadership.
There are many, many, many specialized enterprise applications out there that are windows only.
not when it comes to server software. In that regard, linux is infront.
Way too many. It’s not the 90’s or early 2000s anymore.
I’ve not used crowdstrike, but looks like a part of the pitch is “cloud managed”, which often implies that the vendor takes care of everything, including updates. Particularly since they market it as a security solution, they weld likely emphasize that they can update rapidly enough to keep up with security attacks that move very quickly because they don’t care about “risk”.
deleted by creator
Windows Server OSes running CrowdStrike affected too
What does the issue do?
My first company I worked for used crowdstrike. Does it think the computer is infected and locking them down?
They pushed a driver update. That update is broken and causes the bootup sequence to fail.
Boot loop
deleted by creator
