Still, at least it is the summer holidays in the UK now, so lots of parents off to look after the kids - less urgency to sort their kit out.

It is affecting global payment systems, being on holiday is irrelevant.

Any company that uses CrowdStrike is affected. That includes anyone relying on a SaaS service where that provider uses it.

Yes, but servers could/should have snap shots so you can revert, or admins maybe able to get access for manual intervention (appreciate not always). Some servers will require care. This is where a company, you would hope, would be directing most of their resource to right now.

EUC is a big problem, especially with WFH., In the old days, if this happened we could walk into a floor of users, hit a few banks of desks at the same time and get through it a lot quicker. Can't do that over the phone, and with offices now utilising a lot of hot desks, you may go to 2 banks on Monday morning and find 20 non working users, but on Tuesday it maybe a a scattering of 3 over the same 2 banks - so you are almost doing a 1to1 fix, not enmass

As a lot of people will now be taking time off, there is going to be less stress for the IT teams as quite a few of their users will be away coimpared to everyone being, or coming in and demanding to be fixed NOW.

"as quite a few of their users will be away"

So IT support will be getting calls for a few weeks when WFH users return and find that their PC has been frozen while on vacation. The lucky users shut the thing off and will miss the FUBAR ClownStrike update altogether.

It could have been worse. In the old days of CRTs, users could return to a BSOD burned in to the phosphor.

If the computer was turned off while on holiday, the faulty update would be skipped! Another good reason to unplug during vacation.

> If the computer was turned off while on holiday, the faulty update would be skipped! Another good reason to unplug during vacation.

It seems to me to be another very good reason to not invite strangers to manage your critical infrastructure over a network.

I run a script triggered by logoff, sceen lock or disconnect from rdp from schedule task that changes the gateway so my system stays only on the local net when not in use.

When logged back on, another script that puts the gateway back.

Anything I want my system to connect to while logged off is provided by adding the specific route to the script.

Maybe I was not so paranoid after all.

Am seeing a lot of 'holier than thou' posts on Twitter from Mac and Linux users castigating anyone with a Windows PC and laughing.

This sort of thing can happen to anyone and it's only a matter of time before it happens to you, regardless of what flavor OS you chose.

Stop being a d*ck.

Try running Checkpoint on Ubuntu. Periodic desktop freezes, and an unbootable machine after an update. Only fixable as a) I'm a dev, and can get into grub to boot an old kernel, and b) drive is encrypted with LUKs, so i just needed the normal unlock password to mount the root partition to enable the grub menu.

We have 1 client running crowdshit, their production systems have been down all day.

The client was extremely happy with their IT and had no complaints about any of their computers. They were forced to accept another IT team taking over due to corporate politics.

The other IT team forced us to remove the Microsoft antivirus and install crowdshit. Users immediately started complaining that their computers were constantly crashing and grinding to a halt. The other IT team confirmed that it was a known issue with crowdshit that they have with all of their systems. Their solution was "when your computer gets so slow it is unusable, reboot".

They literally will not give us the codes required to uninstall it, even temporarily for testing, because they know the operations manager will order us to "get that shit off of every computer" and we will happily comply.

The other IT team fucked up big time with this incident. They messaged the users directly, taking full responsibility for the issue and stating that they were working on a fix. Of course they forgot the part where they are meant to follow up with instructions for how to fix the computers. They also forgot to transfer some of the responsibility on to us by like telling us about it or something.

We are probably the only IT company who spent today looking at a list of offline computers and laughing our arses off!

I wish I could listen in on the "why are we STILL down???" phone call the tech-literate manager who knows the fix and what time it was announced will be making on Monday...

I'm seeing this stupid shit far too often. Giving control of all the company systems to an outside vendor and getting rid of the in-house IT staff.

You cannot fix that kind of fucking stupid.

(same poster)

It's actually an in-house team replacing the outside vendor (us) that used to have "sole jurisdiction" over one part of the business...

The 2 IT teams issue is due to the migration having been on hold half way through due to "capacity issues" limiting their ability to handle the increased support demands since crowdstrike was installed on the machines. Their IT team is several times bigger than our company.

I have not personally been involved in that side of things, I discovered the reason they were citing capacity issues for a client that used to send 2-3 support tickets a week only today when the subject came up...

It affected just one of my clients as well. Their parent company had insisted CrowdStrike was installed. A little bit of serendipity helped them. They're only a small company and I'm a lone IT support engineer. So they don't have automated tools for updates/installs - the users had to install CrowdStrike manually. I've been on extended holidays for 3 weeks and only 1/3rd of them had installed it. If I'd not been off the grid, I'd have been hassling the others to install and therefore the impact would have been bigger.

Still they're having to pay me for a call out on Monday to rebuild one of the desktops which they just can't get into recovery mode/remember Bitlocker key.

Sounds very familiar...

While I understand the sentiment of your comment - people shouldn’t gloat about this stuff - I’m afraid (actually I’m not) it is almost ALWAYS Windows that’s fucked up by this sort of thing. It’s NEVER any other OS (well it might be, but the percentage is so low it’s not even statistical noise)

The fact that anyone feels the need to run stuff like Cloudstrike just to keep the OS up and running is a very long-standing joke - there is something deeply wrong with an OS that needs this stuff to keep it going

While I understand the sentiment of your comment - people shouldn’t gloat about this stuff - I’m afraid (actually I’m not) it is almost ALWAYS Windows that’s fucked up by this sort of thing. It’s NEVER any other OS (well it might be, but the percentage is so low it’s not even statistical noise)

I have a system where pacman said "hold my beer"..... Apparently it doesn't have a dependency tree for libraries? To avoid removing ones that other packages rely on. And pacman-static doesn't help when the server no longer finds the NIC. IE it knows the hardware is there, but won't assign an interface to it.

Ah, the crux of the issue with Crowdstrike...why do you need it at all? Maybe some CxO in Mahagony Row fell for the marketing? Maybe some faceless "cybersecurity" auditor recommended it as best-of-breed? Maybe some insurance underwriter demanded it?

The way I see it, most of this garbage should already be part of the OS, not some add-on. BUT NOOOO...we have to "let the market determine what's best". If Microsoft hadn't been forced by legal decree to write in hooks for 3rd-party "security products" we'd have a smaller pool of idiots to blame for this kind of cock-up. A pool of one: Microsoft. Certainly, that'd sharpen up peoples' principals knowing that trusting Windows means trusting Microsoft, and only Microsoft, with your crown jewels.

The way it stands now, people will throw whatever 3rd-party product that promises to make up for the lack of Windows security into their infrastructure and sleep well at night knowing that all that money they spent doing so isn't going to Microsoft, but to a vendor that is "smarter" and "more agile". If that were the case, why aren't these vendors PUBLISHING safer, more more secure OSes? See the problem?

I'm not advocating for the demise of Windows but simply pointing out the rather obvious fact that a Zero-Trust Architecture means you don't trust anything, INCLUDING ALL OF YOUR SOFTWARE AND SERVICES VENDORS.

I'm hoping that this is a wake-up call to those who think that writing a check means that they don't have to think about security anymore.

"The way I see it, most of this garbage should already be part of the OS, not some add-on."

You want the same people who wrote the 'Flaky OS' to also write the 'defense against the arts' software as well !!!???

They cannot write an OS without so many holes/issues/bugs *but* apparently *can* write the 'defense against the arts' software that protects those holes/issues/bugs.

If they were that good, why not 'fix' the OS in the first place !!!???

Going 3rd party is reasonable [Different set of eyes/minds etc] *BUT* do not trust any claims 100%, all software vendors lie.

Not because they are bad *but* that is the industry we have allowed to grow, we accept these lies everyday, pay good money for them & come back for more !!!

---------------------------------------------------------------------------------------

"I'm not advocating for the demise of Windows but simply pointing out the rather obvious fact that a Zero-Trust Architecture means you don't trust anything, INCLUDING ALL OF YOUR SOFTWARE AND SERVICES VENDORS."

Correct, you test for everything you can and have protection in depth.

i.e. Don't trust the claims of *anyone* and have backups/roll-back images/etc to allow you to recover from *failed* recoveries by the 'defense of the arts' software.

THIS SHOULD BE THE PRIMARY LESSON THAT HAS BEEN TAUGHT BY THIS FIASCO !!!

:)

Same organisation doesn't mean the same people. Microsoft in particular have REALLY high variance in the quality of the software they ship.

Does it goes from bad to worse?

In the UK: Cyber Essentials compliance.

That's a joke in itself. To get CE+ compliance you have to install the latest browser. When we had the scan done we failed initially because Google released a Chrome update the day before the test and we hadn't packaged the replacement up yet. We also had to remove IE from all the machines being scanned because their scan detected it was "out of date" despite the fact that MS had stopped updating it & were forcing everyone onto Chrome.

But we could pass because a) the testers let us choose which laptops to test and 2) we could run the test as many times as we liked until it passed, and then give them the pass results only.

Your CE+ assessors clearly didn't have a clue. You have 14 days to install updates from release. The assessor has to choose the test devices. You are not allowed to run the tests - the assessor must do this.

Oh we ARE going to gloat, BECAUSE there is something deeply wrong with an OS that needs this stuff to keep it going and we've been warning about it for years.

And this WILL keep happening.

Gloat away, it won't change the penetration of the OS in the corporate world, or even the home one...

I rekon a few old PC will get replaced by one or more android device.

Luck not judgment is why other stuff is not affected.

Othe operating systems also need security solutions but there is this occult that believe all non Windows operating systems are invulnerable. They are not, particularly as the attack vectors are increasingly moving to the applications.

Complacency just because this group thinks they are somehow invulnerable is what leads to catastrophe. Everyone needs to sharpen up on this and infosec teams need to start listening more to technical teams and not sales people or Gartner.

Infosec Teams are the cause of some crazy risks because all that matters is ticks in boxes.

The entire debacle should be a huge wake up call (on top off the other recent attacks) to the tech sector. Sadly that is unlikely to happen, no lessons have been learnt from previous fiascos. That CrowdStrike are likely to escape without being sued out of existence is even more depressing.

You will gloat until this becomes part of systemd.

(and then I will totally gloat, running bsd and Devuan - until stupid stuff hits my systems, but since I no longer run Debian Sid chances are slim)

Coz we are all clamouring for more Systemd here!

Systemd IS M$.

Systemd, the reason I dumped Linux for anything serious. Offends every software engineering principle. If you look at the sources, it's a real mess of a trainwreck. Looked the one module, network.c, from memory and it's a thousands of lines of C module, pulling in upards of 100 header files. Absolute garbage, and big business depends on this ?. Originally from a Solaris and VMS environment, when an os was written by engineers, for engineers. FreeBSD for several years now, and never a serious issue at all.

When you consider that crowdshite is an enterprise class employee spyware program, looks lie karma finally caught up with them. Serves them right, hit hard in the pocket, is the only thing they understand...

That's not my experience of Linux.

Twice now updates have nuked my system - replacing a fully functional graphics card driver with one that doesn't work.

It's not just Windows that is improved to death.

About the best I can say about Linux, is that a bad update generally only takes down one specific type of hardware or configuration at a time, and the rest are ok. So at least [1]a bug in the 5.15 kernel 's amdgpu driver only took down twenty machines last week, instead of the whole estate. Although I do deserve some of the blame for not testing every possible hardware combo.

[1] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2068738

phuzz,

Blame is exactly what you should accept !!!

Why the hell do you roll out something you do not test ???

You get away with this once/twice maybe more if you are *very* lucky ...

*BUT* you will be bitten eventually ...

at the worse time and it *is* ALL your fault.

Every IT techie I have ever worked with thinks they are the *BEST* and *infallible*.

I lack that ego and can make mistakes, like everyone else, *but* not because I think I cannot make them !!!

:)

I wasn't thinking I was "*BEST* and *infallible*", I was thinking that I've never seen a Linux kernel update (and this was a mainstream release, not a beta/canary/unstable release) fuck a computer hard enough to stop it booting. Turns out kernel devs are just as fallible as the rest of us.

I guess I'm lucky, I dont remember a Linux update doimg any damage within one version . Major version update sometimes are a headache but I have never seen a yum / apt security update stop a physical pc from booting

Whereas I've seen it (very rarely) on RHEL6 boxes where an improperly installed kernel update prevented the machines from booting - booting into the previous kernel and reinstalling the new one resolved it 100% of the time but that machine is then down until someone intervenes.

Yea but in case you hadn't noticed Windows is pretty ubiquitous in the world of corporate IT. That isn't going to change any time soon..

And it’s not going to change until Governments mandate something that enables competition. Sadly there are too many brown envelopes.

This has nothing to do with brown envelopes. Linux fails despite being being free because the lack of cost is not enough to offset the missing functionality in software that businesses need, and therefore are willing to buy software that saves them much more money than the purchase or license cost.

[1]This is a comment I made on this subject literally ten years ago. It's still basically correct today.

[1] https://forums.theregister.com/forum/6/2014/08/19/munich_dumping_linux_for_windows/#c_2275198

Naturally it's Windows that gets hit with this kind of thing the most often, the vast majority of companies use it.

And you're right, this could have happened to any OS that Crowdstrike supported... well, in this case I'm not sure "supports" is the right term!

My home PC is a Mac and I'm not one of the annoying fanboi's who insists that Macs are safe from everything including a direct hit from a nuclear missile because . I'm well that I'm just one lazy click away from days of pain so, I just brought forward my quarterly "off-site" backup (it's in a plastic bag in a jam jar in the shed) and updated the reminder to do it monthly instead of quarterly. I've also made sure that the written copy of the disc encryption key is where I thought it should be and, just to be safe, done a separate backup of my password manager to an encrypted stick and hidden it somewhere. Tomorrow I'll be getting my fallback MBA out of the loft and getting the files up to date.

It is a heck of a lot easier to recover from a time machine backup though.

My shed backup has a Time Machine backup on it and an rsync copy of my files (photos, docs, music, etc). I have a number of other Time Machine backups running to Apple devices and a NAS and also some stashed removables, but I also run rsync copies as well. This comes from a hard lesson the first time I tried to restore a brand new Mac from Time Machine because the OS's were so far apart that the new Mac required the old Mac to be updated before it continued. I couldn't do this because the old Mac wasn't compatible with the latest OS. I can understand that settings and apps from an older OS might not be compatible but it wouldn't let me bring anything across. I cabled the macs together and copied across my files but if my old Mac had died completely I would probably have lost data, although I assume there would have been some way to access the Time Machine files. Since then my personal backup policy has been Time Machine dailies plus Time machine and rsync monthlies to removeable drives. I also keep the Mac OS up to date (I had a bee in my bonnet about something in the above situation and refused to update), although I always wait a week after it comes out.

I'm not slagging Time Machine off - it's easy to set up, reliable and invisible to the user once running and rolling files back to earlier states is easy and the UI is oddly satisfying.

I'm sure there's a pithy saying for this in IT circles, but if you've never tried to recover from a backup then you can't really be sure it's a backup.

"You don't test backups - you test restores"

Reminds me of the time I was asked to look at a small offices IT system, mid 90’s, they’d been using zip discs to back up documents. Well 1 zip disc with a year/month/date directory structure and each one contained a shortcut to my computer…

I had this issue at work recently as the person most likely to solve computer problems. I actually did manage to convince an old Mac to go to a newer - not current, it’s about 12 years old - version and then restore everything. It required fully wiping and reformatting it, then updating the “to” Mac.

It was certainly harder than I wanted and even expected, given how easy TM is to set up.

I’ve never had to do the same to Windows; I don’t imagine that’s great fun either.

You can actually write your own timemachine in shell -- I did this in 1996.

Basically, it's just an ongoing series of datestamped folders, each with the entire filetree underneath them but hardlinks for unchanged files rather than copies.

You use find to walk your entire current filetree* and at each node look at the most recent datestamped copy. If it doesn't exist in the prior run, you use cp / mkdir to copy this node into the new filetree; if it's a file and exists but has changed, you copy it into the new filetree. If it exists AND hasn't changed, you use ln to create another catalogue entry for it in the new filetree.

That's it.

.

* (Put some filters on the find for handling OS "special files" if you're doing whole-machine replication.)

The nice thing about doing it manually is you can then create custom schedules for people with special needs. Eg, graphics/video artists on big projects wanting, say, 30min backups intraday, then 4hourly for yesterday, then daily for a week, then weekly for 2 months, then collapse to more-normal. This becomes a simple case of 30min TMs and a culling script to run daily.

I have vague recollections of rsync having an option to do this built-in, too. That is, hard-linking unchanged "copies" rather than re-creating the file as a new copy. So that could be worth looking at if you already have an rsync process set up.

I don't know about other snapshot systems, but Windows let you specify a shadow copy schedule per folder if you wanted to

Or you could insert a ZFS filesystem or similar and take snapshots on a schedule. Can be as simple as two disks in a mirror if that's all you need.