As someone with some levels of AWS certification, that's terrifying. Mostly because I understood most of that gobbledygook explanation. Pass the mind bleach.

Not until you explain to mere mortals what exactly a "DNS Planner" is, and why it's needed in simple terms. It'll help if you assume (accurately) I know nothing about hyperscale or cloud operations.

I think it's part of their Route 53 service. I think it's to do with load balancing and DNS routing for latency. Read: "Voodoo".

Hmmm, I though route 53 was the bus service from Wolverhampton to Bilston, via Wednesfield. So I'm afraid there's no mind bleach for you, sir!

One thing missing in that description was any attempt to apply rate limiting to, well, any part of it.

So a huge pile of machines basically all try to come up at once, without the staggering that limiting would cause (or inflict, depending upin p.o.v.) and start getting into a mess.

Is this genuinely a surprise to anybody? Isn't everyone charged with engineering a system supposed to be asking "what happens if it all switches on at once?" no matter what the cause might be? From checking whether recovery from a power failure[1] means the hard drives[2] can be allowed to spin up all at once[3], to whether you can serve netboot images fast enough to prevent watchdog reboots or even how many DNS leases can you serve out before you are swamped just handling renewals because you KNOW you set the lease period way shorter than the DNS designers ever expected[4]

[1] or Lady Florence pushing the Big Red Switch on Opening Day, not realising this one isn't a dummy

[2] or the dynamos, each racing to be the master frequency the rest have to sync to

[3] even in your home lab, can the circuit take that strain

[4] I *think* I understand what was going on here, allowing machine identities to move around as hardware becomes available to handle requests for user operations (please, if anyone can correct that understanding, do so) but is that how people normally do load balancing? Not my area at all, but this really feels like a misuse of DNS.

really feels like a misuse of DNS.

I also got the impression low TTL DNS records were being used to route traffic which I would think at this scale really is playing with fire.

One of the advantages of IPv6 is that you have such a large address space that each end point could be assigned a permanent address along with any number of ephemeral addresses which could be assigned to processes, applications etc and follow them as they migrate around the cluster and hopefully push the routing out to the network and dynamic routing processes.

I suspect much of this stuff is a dark art simply because many of its practitioners have avoided en light enment. ;)

In my career (of distant memory) we'd have a layer of boxes behind a single IP address that distributed "work" one layer down to the servers.

PHB decided that machines left on overnight is bad.

Everyone switches machine off, goes home.

Gets to work. Powers up. Machine needs to download and install an update.

3,000 machines and nearly 24 hours later when no one has done a days work, it's ITs fault.

That's the funniest thing I've read in a while, superb!

You have to hand it to the ignorant PHBs who will gladly shout the techies down, implement some dictat or we get fired, we do as commanded and still get a bollocking 'cos PHB is desperately trying to keep the C-suite from chewing his arse to pieces over whatever his latest cock-up was!

I wanna know when my fucking bed will start working again!

You have a special bed for fornicating ?

Maybe you can use your sleeping bed for the time being ? Or the couch ...

https://www.msn.com/en-us/news/other/this-weeks-aws-crash-made-smart-beds-overheat-get-stuck-in-wrong-position/ar-AA1P0rN1

You have a special bed for fornicating ?

Maybe you can use your sleeping bed for the time being ? Or the couch ...

Or a sock.

I have to question what state humanity is in when we have to have a f**king bed connected to the world's biggest network 24/7 or it won't work!

So when DNS was fixed there were so many services trying to restart many of them failed.

The report explains what the race condition was, but not why the Enactor was running so slowly in the first place, which was technically the cause of the problem. I wonder whether they know - a challenge of work at this scale is that you can have problems that only happen with production workloads and it's hard to reproduce that and properly isolate the cause.

A cascade of FFS! causes:

1. The enactor should not have been running slow.

2. Nothing to monitor its status.

3. The various boxen had no idea what to do about it before jumping in feet-first, cuz nobody had thought through Murphy's Law.

4. Someone who knows what they are doing should have been retained with sweeteners, not driven out by bullying manglement.

I expect there are several more.

As soon as you have more than one or two operations happening at the same time, being (re)triggered (by external sources), or just operating at scale, then dealing with all of the permutations and combination is rather complex - to say the least. Even supposedly simple systems can be mind boggling sometimes.

At this point earlier articles about a brain drain in AWS make a lot of sense. Experienced developers/network admins would understand these complexities, and probably not roll out "quick changes" to anything without a thorough review (including in-house experience about parts of the system that are more prone to issues...etc). Juniors are more likely to feel the pressure to deliver (and kowtow to a PHB), and roll something out without understanding the implications. Worse, if someone was cutting and pasting simplistic code generated from AI...