CrowdStrike effectively bricked windows, Mac and Linux today.
Windows machines won’t boot, and Mac and Linux work is abandoned because all their users are on twitter making memes.
Incredible work.
Come on, it was right in their name. CrowdStrike. They were threatening us all this time.
We formed a crowd, then BAM, they striked.
We should have seen this coming!!!
Clown strike
Crowdstrikeout
I wish my Windows work machine wouldn’t boot. Everything worked fine for us. :-(
Could be worse. I was the only member of my entire team who didn’t get stuck in a boot loop, meaning I had to do their work as well as my own… Can’t even blame being on Linux as my work computer is Windows 11, I got ‘lucky’; I just got a couple of BSODs and the system restarted just fine.
You’re a much more honest person than I am. I’d have just claimed mine was BSODing too.
Since it’s [email protected], it would be illegal to be dishonest.
I’ll see myself out.
Funny, mine did a couple BSODs then restarted just fine, at first. Then a fist shaped hole appeared in the monitor and it wouldn’t turn on again.
Weird bug.
Lol why is it always the monitor to get beat. It only has one job, just to show you what the computer is outputting lol
why is it always the monitor to get beat
Because it’s within arm’s reach and the developers aren’t.
Same where I’m at. 😞
Imagine this happening during open heart surgery and all the monitors go blue!
Good lord I would hope critical surgical computers like that aren’t networked externally… Somehow I’m guessing I’m wrong.
Fear not, that’s why we deploy extra security software to these critical systems. It’s called Crowdsource or something.
GroupStrike?
CohortAttack?
PeerOffense?
Gang… Wait no
UnionBuster
StrikeThree
Groupon
deleted by creator
Maybe not everywhere, but all of ours are air gapped.
Good, they absolutely should be.
I’d heard some hospitals were affected. They cancelled appointments and non-critical surgeries.
I’m guessing it was mostly their “behind the desk” computers that got affected, not the computers used to control the important stuff. The computers in patients’ rooms may have been affected as well, but (at least in the US) those are usually just used to record information about medicine given and other details about the patient, nothing critical that can’t be done manually.
It did impact emergency services
I’ve heard anecdotally that some 911 services were down in my area, but I can’t speak to how wide that was.
Good News! Unless something has changed since I worked in healthcare IT, those systems are far too old to be impacted!
I’m half-joking. I don’t know what that kind of equipment runs, but I would guess something embedded. The nuke-med stuff was mostly linux and various lab analyzers were also something embedded though they interface with all sorts of things (which can very well be windows). Pharmaceutical dispensers ran various linux-like OS’s (though I couldn’t even tell you the names anymore). Some medical records stuff was also proprietary, but Windows was replacing most of it near the end of my time.
One place we had ran their keycard system all on a windows 3.1 box still. I don’t doubt some modern systems also are running on Windows which has interesting implications for getting into/out of places.
That said, a lot of that stuff doesn’t touch the outside internet at all unless someone has done something horribly wrong. Medical records systems often do, though (including for billing and insurance stuff).
Security through obsolescence the healthcare way!
I was just watching this show called Connections and the first episode was about a power blackout and it showed how the lights went out during a birth.
Great show it went on about what do you do if the power stays off permanently and how we aren’t well prepared for that and how to start a civilization after you kill some farmers and steal their land but non of their tools work without power either and if you know how to mount an old-school plow to oxen
Is there a good eli5 on what crowdstrike is, why it is so massively used, why it seems to be so heavily associated with Microsoft and what the hell happened?
Crowdstrike is a cybersecurity company that makes security software for Windows. It apparently operates at the kernel-level, so it’s running in the critical path of the OS. So if their software crashes, it takes Windows down with it.
This is very popular software. Many large entities including fortune 500 companies, transport authorities, hospitals etc. use this software.
They pushed a bad update which caused their software to crash, which took Windows down with it on an extremely large number of machines worldwide.
Hilariously bad.
Honestly it is kind of hilarious, with how many people I have had make fun of me for using Linux, and now here I am laughing from my Linux desktop lol
Sure, this time it only affected Windows computers, but Crowdstrike has also broken Linux installs this year:
Sure but I’m not using their software. I’m careful about what I allow to have kernel access on my machines.
So, do all windows machines use this, or do you have to add this software?
It’s separate software; CrowdStrike is independent from Microsoft and it isn’t a default component of Windows.
It’s interesting that Microsoft is getting a lot of flack from this.
Yeah, this isn’t really the fault of windows.
Windows normalized running third party software as kernel level code.
Third parties love their trojans just being treated as normal way of life.
“Anti-cheats” instead of not being imbeciles while designing protocols for multiplayer, “anti-viruses” which need to run kernel-level and download databases with executable code, video drivers which just can’t be packaged with Windows.
One thing I’ve realized is that large parts of social structure are dependent on cheating. We all want to cheat, so we all agree to a system where cheating is possible, but pretend it’s not happening until someone gets caught and then just behave as if nothing happened.
One necessary part of someone’s upbringing is honesty. There’s an amazingly deep moment in LOTR where Eomer says that Rohirrim don’t lie, so they are not easily deceived.
This is not a poetic device. This is how it works. Ponzi schemes usually target people who think they are smarter and more cunning and will gain something from them. And rigged security systems work because most of participants think they are the ones who may at some point abuse those systems, but most of them are the ones becoming eventually victims of such abuse.
If there is any software you want running at kernel though it is your AV. Not saying Spotify has a reason for running at kernel though… But running AV at kernel in theory is a better way to protect the machine and you.
It seems to be an enterprise product, meaning normal users might not have been affected. I wouldn’t personnaly be able to confirm since I usually have 1-2 month uptime on my windows machine.
Their computers may not be affected, but their everyday lives might be. Some of the affected services include 911, stoplights, banks, hospitals, and a whole other smorgasbord of stuff.
It’s a general security solution. They run on Mac and Linux as well. It just happened that crowdstrike only released the broken update for windows.
They make security software for every OS. My company has it running on our Macs, and Linux servers as well. It just happened to only break windows because that’s what they released the update for.
I was puzzled since my work continued on as usual. I guess my company doesn’t use it.
This is very popular software.
if that’s a “good” argument for you, then i’ve already heared that, and it nearly never really fits. here is another one for you that is an argument as generic as yours: “maybe try eating poo, trillions of flies cannot be wrong, poo is VERY popular food, much more popular than any human food !!! (as in mass per day as well as in its number of consumers)”
I wasn’t making a case for adopting this software. Just pointing out that it is widely used, which is why it had such a wide effect.
I think you’ll find most corporations would jump off a bridge if they saw their competitors jump.
Gonna try my best here:
Crowdstrike is an anti-virus program that everyone in the corporate world uses for their windows machines. They released a update that made the program fail badly enough that windows crashes. When it crashes like this, it tries to restart in case it fixes the issue, but here it doesn’t, and computers get stuck in a loop of restarting.
Because anti-virus programs are there to prevent bad things from happening, you can’t just automatically disable the program when it crashes. This means a lot of computers cannot start properly, which means you also cannot tell the computers to fix the problem remotely like you usually would.
The end result is a bunch of low level techs are spending their weekends manually going to each computer individually, and swapping out the bad update file so the computer can boot. It’s a massive failure on crowdstrikes part, and a good reason you shouldn’t outsource all your IT like people have been doing.
It’s also a strong indicator that companies are not doing enough to protect their own infrastructure. Production servers shouldn’t have third party software that auto-updates without going through a test environment. It’s one thing to push emergency updates if there is a timely concern or vulnerability, but routine maintenance should go through testing before being promoted to prod.
It’s because this got pushed as a virus definition update and not a client update bypassing even customer staging rules that should prevent issues like this. Makes it a little more understandable because you’d want to be protected against current threats. But, yeah should still hit testing first if possible.
If a company disguises a software update as a virus definition update, that be a huge scandal and no serious company should ever work with them again…are you sure that’s what happened?
It wasn’t a virus definitions update. It was a driver update. The driver is used to identify and block threats incoming from wifi and wired internet.
The “Outage” section of the Wikipedia article goes into more detail: https://en.wikipedia.org/wiki/2024_CrowdStrike_incident#Outage
it’s a glorified anti virus and does a few other things on top.
cloudstrikecrowdstrike should be sued into hellCrowdstrike*
Cloud Strife*
Counter Stri… no not that.
“the bomb has been planted” - the intern that pushed the update at crowd strike or whatever
Clown Stripe*
Better rebrand to Clownstrife I guess.
Clownstrife
Funniest thing I’ve read all week, lol
Time to rebrand as CloudShrike to prevent future fuckups.
Microsoft*
Both
If someone hands a toddler a gun and they shoot someone, who’s fault is it?
The only thing that can stop a toddler with a gun is a good toddler with a gun
What if that someone is another toddler that found the gun in the street, and it got in the street because it fell off a truck? Your Honor, what if the toddler had murderous intent because they were denied a sucker?
Is this implying that a publicly-traded corporation whose software is installed on millions of computers around the world has the same level of agency and responsibility as a preschooler?
I mean, yes, Microsoft bears responsibility for blindly accepting whatever deployment package CrowdStrike gave it and immediately yeeting it out to 100% of customers via Windows Update without any kind of validation or incremental rollout, and should probably be sued for it. That still doesn’t negate the complete and catastrophic failures at every step of the development process on the part of CrowdStrike. It takes a lot of people to fuck up this bad.
Windows didn’t do anything, this was an update applied by the Crowdstrike agent.
That’s the impressive part of all this. Microsoft didn’t do it. CloudStrike did it.
Microsoft left something in a state that allowed CloudStrike to fuck up enough to brick systems.
It’s why we spend a lot of time reviewing security analysis of our own software - if there’s a way to fuck everything up, it better not because we enabled it to get fucked.
Is this implying that a publicly-traded corporation whose software is installed on millions of computers around the world has the same level of agency and responsibility as a preschooler?
When it comes to IT reliability and security, kinda, yeah.
Windows AV and MDM is a bit of a horror show in the corporate space. I worked somewhere where developers weren’t allowed to use WSL because it was blocked by McAfee. We also had 3 different MDMs running and they were slow as balls even though they were modern 8 core laptops.
It’s not Microsoft’s fault a third party company wrote a kernel module that crashes the OS.
Unlike the mobile world where apps are severely limited and sandboxed, the desktop is completely the opposite. Microsoft has tried many times to limit what programs can do, but encountered a lot of resistance and ultimately had to let it go.
Lol, they only bricked specific machines running their product. Everyone else was fine.
This was a business problem, not a user problem.
They make software for both of them also though, IMO they’re at fault for sure but so should be Microsoft for making a trash operating system.
Not saying Windows isn’t trash, but considering what CrowdStrike’s software is, they could have bricked Mac or Linux just as hard. The CrowdStrike agent has pretty broad access to modify and block execution of system files. Nuke a few of the wrong files, and any OS is going to grind to a halt.
Probably would have been worse if this was on Linux. That’s like 90% of the internet.
Good thing is the kind of people making decisions based on buzzword-bongo filled PR campaigns like Crowdstrike’s are already forcing their IT to use Windows anyway.
It was a serious ask to install it from auditors that don’t know what the fuck they are doing. On all ec2 machines for “reasons” I’m sure there are companies out there pushed into doing this because they don’t have people willing to die on those hills.
Oddly, one of CrowdStrike’s selling point is that it provides pretty good EDR for Linux and Mac. If you want crap EDR, which pushes you towards Windows, Microsoft Defender for Endpoint is the ticket.
I’m sure the three people that use Linux and Crowdstrike together would have been very upset. 🤣
Yes and no. Linux is inherently more diverse. All the different distros doing things in different ways, sometimes with different components. It’s not as much of a monoculture as Windows. There isn’t a Linux that 90% is.
Not only that, with Linux you can/have to manually update. Which means I hear the news before running the update
Mmm depends. I have some automatic updates on my servers: https://wiki.debian.org/UnattendedUpgrades
That’s… Not great. I didn’t actually think about what all these wild AV systems could do, but that’s incredibly broad access.
Maybe I’m just old, but it always strikes me as odd that you’d spend so much money on that much intrusive power that on a good day slows your machines down and on a bad day this happens.
I get that Users are stupid. But maybe you shouldn’t let users install anything. And maybe your machines shouldn’t have access to things that can give them malware. Some times, you don’t need everything connected to a network.
I didn’t actually think about what all these wild AV systems could do, but that’s incredibly broad access.
Always has been. I’ve clean Symantec A/V off way too many systems in my time, post BSOD. That crap came pre-loaded on so many systems, and then borked them. The problem is, that in order to actually protect system from malware, the A/V has to have full, kernel level access. So, when it goes sideways, it usually takes the system down. I’ve seen BSODs caused by just about every vendor’s A/V or EDR product. Shit happens. Everyone makes mistakes, but when that mistake is in A/V or EDR, it usually means a BSOD.
Maybe I’m just old, but it always strikes me as odd that you’d spend so much money on that much intrusive power that on a good day slows your machines down and on a bad day this happens.
I get that Users are stupid. But maybe you shouldn’t let users install anything. And maybe your machines shouldn’t have access to things that can give them malware. Some times, you don’t need everything connected to a network.It’s tough. The Internet and access to networks provides some pretty good advantages to users. But, it also means users making mistakes and executing malware. And much of the malware now is targeted at user level access; so, you can’t even prevent malware by denying local admin/root. Ransomware and infostealers don’t need it. A/V ends up being a bit of a backstop to some of that. Sure, it mostly is a waste of resources and can break stuff when things go bad. But, it can also catch ransomware or alert network defenders to infostealers. And either of those can result in a really, really bad day. A ransomed network is a nightmare. And credentials being stolen and not known about can lead to all kinds of bad stuff. If A/V catches or alerts you to just one or two of those events and lets you take action early, it may pay for itself (even with this sort of FUBAR situation) several times over.
Fair. Thanks for that counterpoint.
When a kernel fails to boot in Linux it rollback to a previous working version so there is a chance it might recover from CrowdStrike update.
That really depends on the distro, most of them, no.
I’d say the issue isn’t that Windows is a trash OS, but everyone using the exact same trash OS and same trash security program.
It was a kernel module, there’s nothing Microsoft could have done
Except not make an OS so shitty and vulnerable that it needs millions of hours and billions of dollars pumped into keeping it from being hacked in a split second. But yes nothing besides that one minor thing.
Companies use the same software on Linux and Mac systems, and it’s a kernel module there as well; this could have happened to any OS that companies are using it on, it just happened to happen on Windows.
Giving kernel access to outside software is always a risk; these companies chose to take that risk
They make plastic pocket protectors too but that doesn’t mean we need to use them. Sounds you missed my point
The reason Windows gets attacked so much to begin with is because of its market share, not necessarily how vulnerable it is. If you want to cast as wide a net as possible while wasting as little effort as possible, why the fuck would you target the lesser used OSes? In the same vein, why would you invest so much in protecting OSes that aren’t as big of targets as Windows? Your comment sounds as ill informed as those people that think Macs can’t get viruses.
It’s well known that windows is chocked full of security issues and ALWAYS has been. Never once implied other OSes are perfect, just better. You sound ill informed yourself, and simping for a corporation
Why do I get the feeling that you’re a flat earther?
My guess would be you have instincts opposite reality
Given that I’m not even close to that, and it has zero relationship to talking about how shitty windows is.
The impacted Channel File in this event is 291 and will have a filename that starts with “
C-00000291-” and ends with a.sysextension. Although Channel Files end with the SYS extension, they are not kernel drivers.
As a career QA, i just do not understand how this got through? Do they not use their own software? Do they not have a UAT program?
Heads will roll for this
From what I’ve read, it sounds like the update file that was causing the problems was entirely filled with zeros; the patched file was the same size but had data in it.
My entirely speculative theory is that the update file that they intended to deploy was okay (and possibly passed internal testing), but when it was being deployed to customers there was some error which caused the file to be written incorrectly (or somehow a blank dummy file was used). Meaning the original update could have been through testing but wasn’t what actually ended up being deployed to customers.
I also assume that it’s very difficult for them to conduct UAT given that a core part of their protection comes from being able to fix possible security issues before they are exploited. If they did extensive UAT prior to deploying updates, it would both slow down the speed with which they can fix possible issues (and therefore allow more time for malicious actors to exploit them), but also provide time for malicious parties to update their attacks in response to the upcoming changes, which may become public knowledge when they are released for UAT.
There’s also just an issue of scale; they apparently regularly release several updates like this per day, so I’m not sure how UAT testing could even be conducted at that pace. Granted I’ve only ever personally involved with UAT for applications that had quarterly (major) updates, so there might be ways to get it done several times a day that I’m not aware of.
None of that is to take away from the fact that this was an enormous removed up, and that whatever processes they have in place are clearly not sufficient. I completely agree that whatever they do for testing these updates has failed in a monumental way. My work was relatively unaffected by this, but I imagine there are lots of angry customers who are rightly demanding answers for how exactly this happened, and how they intend to avoid something like this happening again.
If only there were a way to check a file’s integrity.
If I were part of CS management, I’d look into recently fired personnel.
I’d be looking to see who made the most shorting the stock.
A corrupt file wouldn’t be nulled. I’ve never ran across a file with all zeroes that wasn’t intentional.
Can an OS be bricked?:
A brick (or bricked device) is a mobile device, game console, router, computer or other electronic device that is no longer functional due to corrupted firmware, a hardware problem, or other damage.[1] The term analogizes the device to a brick’s modern technological usefulness.[2]
Edit: you may click the tiny down arrow if you think it can’t. ;)
🤓
It’s just a setup to the punchline, not a legitimate assessment of the situation.
Colloquially, I’d use it to mean “requires physical access to fix.”
no longer functional due to corrupted firmware, a hardware problem, or other damage.
Other damage being an invalid driver
I wouldn’t consider something to be bricked if it can be fixed by simply restoring a backup or reinstalling the OS.
soft bricked is a subset of bricked
https://en.wikipedia.org/wiki/Brick_(electronics)#Soft_brick
rm -rf /
:) the hardware is still more capable than a brick:
Plug in favourite installation media stick, push reset button (if there is one), reinstall.
No, unless its an embedded OS and it has no external ports.
You can absolutely start writing garbage to bios and brick the mobo firmware.
and the one I was replying to was asking about an OS being bricked, not about the bios or firmware.
AND even then you can reflash the bios, its time consuming and costly but you can.
Yes, time consuming. But it’s still working hardware, not a now useless, now unrepairable paper weight…
Which is the definiton of “bricked” although people nowadays start to use the term inflationary.
AND even then you can reflash the bios, its time consuming and costly but you can.
then nothing can be bricked because on paper you can desolder the rom chip and put another one in place.
If you want to be stupidly pedantic about shit, then nothing is anything.
I agree with the sentiment but …
then nothing can be bricked because on paper you can desolder the rom chip and put another one in place.
Companies already put serial numbers in components and configure them so only specific ones work together, requiring OEM tools to pair them.
It’s imaginable that someone makes something similar with e-fuzes instead.well, you started this pedantic shit. So deal with it.
Nah. The person you responded to asked a facetious question. You started being pedantic.Everyone know what it means when someone says something is bricked.
it’s not costly or time consuming maybe in the past but now you plug in a memory stick and hit a button then wait 5 minutes for a light to stop flashing
brick my tiny down arrow
effectively
Ummm. Yes?
What’s the criteria for a Windows machine to be affected? I use Windows but haven’t had any issues today.
This is specifically caused by an update for CrowdStrike’s Falcon antivirus software, which is designed for large organizations. This won’t affect personal computers unless they’ve specifically chosen to install Falcon.
be a windows based machine protected by crowdstrike as a security service, and received said botched update
protected
Um, about that…
i means cops exist to protect and serve, whether they actually do that is a different story
cops exist to protect and serve
Supreme Court says otherwise. It’s just a slogan, not an actual mission statement
ACAB - now with CrowdStrike!
(taps head) Can’t infect a system that won’t turn on.
Hey! Its VERY secure, nobody can get into it!
“This is a nice little server you’ve got here. Be a shame if something happened to it.”
You need to be a Crowdstike customer
CrowdStrike effectively bricked windows, Mac and Linux today.
Oh so its not windows only
The joke is Mac and Linux users, who aren’t actually effected, are incapacitated due to being busy gloating on social media.
Alr
They’re not wrong.
Let your plans be dark and impenetrable as night. And when you strike fall like a thunderbolt.
on twitter?!?
