AWS is fed up with tech that wasn’t built for clouds because it has a big 'blast radius' when things go awry
- Reference: 1607663897
- News link: https://www.theregister.co.uk/2020/12/11/aws_infrastructure/
- Source link:
That’s The Register ’s takeaway from today’s “Infrastructure Keynote” at the cloud giant’s elongated re:invent conference, which featured veep for global infrastructure leadership Peter DeSantis revealing a little about how AWS keeps itself running.
Among the nuggets he revealed was that AWS has designed its own uninterruptible power supplies (UPS) and that there’s now one in each of its racks. AWS decided on that approach because the UPSes it needs are so big they need a dedicated room to handle the sheer quantity of lead-acid batteries. The need for that room creates more risk and makes for a larger “blast radius” in the event of failure or disaster.
Amazon’s cloudy Macs cost $25.99 a day. 77 days of usage would buy you your own Mac [1]READ MORE
AWS is all about small blast radii, DeSantis explained, and in the past the company therefore wrote its own UPS firmware for third-party products.
“Software you don’t own in your infrastructure is a risk,” DeSantis said, outlining a scenario in which notifying a vendor of a firmware problem in a device commences a process of attempting to replicate the issue, followed by developing a fix and then deployment.
“It can take a year to fix an issue,” he said. And that’s many months too slow for AWS given a bug can mean downtime for customers.
This approach has also seen AWS design its own software to manage switch gears, the devices that cut over from mains power to UPS in the event of an outage.
Amazon’s home-grown Arm processor, the Graviton 2, was developed for similar reasons.
Software you don’t own in your infrastructure is a risk
DeSantis said the reason that commercial UPSes and switch gears don’t meet its needs is that they’re designed for the many scenarios in which they’ll be put to work, rather than Amazon’s requirements. The same logic goes into developing CPUs, he said, arguing that the likes of Intel and AMD design products that will sell well by making them general-purpose devices.
The result is processors that pack in features to make them suitable for more tasks. When raw power was needed, multi-core CPUs were the answer. When utilisation rates of CPUS became an issue, simultaneous multithreading came along. None of that tech ever left mainstream CPUs, DeSantis argued, and the result is architectures ripe for side-channel attacks and which delivers variable performance (which is why the HPC crowd turn off SMT).
AWS would rather processors designed for the cloud. Hence its investment in Graviton, the many-core architecture and extra-large caches as they allow better per-core performance without the need for other trickery. The architecture is designed from the ground up for microservices, which AWS sees as the dominant wave of software development.
“Graviton 2 delivers 2.5-3 times better performance/watt than any other CPU in our cloud,” DeSantis said.
In conversation with The Register he added that such performance is only possible thanks to AWS’ Nitro silicon, to which the cloud colossus offloads virtualisation and networking chores.
DeSantis declined to tell The Register what’s inside a Nitro device but did say it is not correct to characterise them as SmartNICs.
“SmartNIC is a subset of its functionality,” he said. “It is very specialised hardware for us, really deeply formed for AWS. DeSantis allowed that there are “some similarities, logically, but it is more specialised.”
And then in his keynote he showed one of the devices and said it connected to AWS’ new [2]Mac instances over Thunderbolt.
[3]
An AWS Nitro card, at bottom left. Click to enlarge
Much of DeSantis’ talk was dedicated to AWS’ green credentials – it has just ordered a stack more renewable energy – and not-so-subtle digs at the language cloud rivals use when describing the physical separation of availability zones. AWS, he said, is perfectly clear that its data centres are a disaster-proof distance from one another, but less than a millisecond of latency apart.
That’s a configuration that he said delivers what cloud apps need: enough distance to be safe, but not so network-challenged that stateless apps will struggle.
DeSantis also talked up AWS’ newly-revealed plans for a second Australian region. ®
Get our [4]Tech Resources
[1] https://www.theregister.com/2020/12/02/aws_ec2_mac_pricing/
[2] https://www.theregister.com/2020/12/01/bare_metal_macsasaservice_come_to/
[3] https://regmedia.co.uk/2020/12/11/aws_nitro.jpg
[4] https://whitepapers.theregister.com/
Re: chip looks like the Annapurna Labs
Nitro refers to a card, not an individual processor (the ambiguous term 'silicon' was used by the article author, not AWS's executive). Most of the chips on the card are designed by Annapurna, which Amazon bought in 2015, with this application in mind.
https://community.cadence.com/cadence_blogs_8/b/breakfast-bytes/posts/the-aws-nitro-project
"Among the nuggets he revealed was that AWS has designed its own uninterruptible power supplies (UPS) and that there’s now one in each of its racks. AWS decided on that approach because the UPSes it needs are so big they need a dedicated room to handle the sheer quantity of lead-acid batteries. "
I'm having trouble reconciling that the UPSes Amazon requires "need a dedicated room" but there is also "one in each of its racks".
I think the one in each rack is the alternative solution to having a big room full of UPSs.
Instead of one big bang on major failure, you'd get a small bang. (Which would probably cascade around the server room, but hey.)
That's how I read it as well. Instead of one big explosion that blows the walls down, it is like a string of firecrackers.
the top-of-rackl UPS needs to have enough power for about 30 seconds worth of load - they might have as much as 2 minutes. Buffers can be flushed and checkpoints written before everything goes black. The Gensets fire immediately on loss of mains. But if one were to fail and the N+1 also, then yes, part of the DC goes magically silent.
We've seen many occasions when a large DC-scale UPS fails to live up to it's name, and downs a whole DC. Equinix, BA to name ones in my recent memory.
A few servers going down is a lot more tolerable than a whole AZ, and with a larger number of devices UPS failures become a routine problem - much better than a very large one-off problem.
So, plenty of merit, until one of them catches fire or somesuch anyway...
"“Software you don’t own in your infrastructure is a risk,” DeSantis said, outlining a scenario in which notifying a vendor of a firmware problem in a device commences a process of attempting to replicate the issue, followed by developing a fix and then deployment.
"“It can take a year to fix an issue,” he said."
Yep. Now ask me why I don't use or recommend clouds. If I own the software, and indeed the infrastructure, I can fix it today, not next Thursday when the AWS techs get around to it. If they ever get around to it.
Yeah - I know what you're saying - but I suspect that if you found an issue in APC's UPS Firmware, you'd be SOL getting a fix out of them at all... If amazon are talking about a year to get a fix from their vendors with the scale of purchase they make, the rest of us can probably wait until hell freezes over before anything gets done about it from our vendors - the argument isn't about cloud vs on prem - it's about how good vendor support is, and we all know how variable that can be!
I suspect that if I found an issue with APC's firmware, it would have been in acceptance testing, and it wouldn't have made it into production. Twice. There was no third time; I took APC off the approved vendor list.
"“Software you don’t own in your infrastructure is a risk,” DeSantis said, "but only when it comes to us, for you, our AWS service is the best!"
Next time he should engage his brain, before opening his mouth.
That's all groovy, but what if you needed to provide a service to people across the world? To get comparable latency performance you'd have to build and maintain multiple data centres of your own across on several continents.
You would need a damned good crystal ball too, in order to predict the demand for your business's services months in advance so as to commission and build the required capacity.
So, if you have faster-than-light interconnects and and you see the future... then you're right, there's no need for any business to use cloud services.
Let's face it ...
... who really needs the infrastructure you describe? I'll bet you a plugged nickel that all of them own their own infrastructure and wouldn't touch AWS (or any other so-called "cloud") with a ten foot pole.
Re: Let's face it ...
Anyone who isn't in a position to know their infrastructure requirements sufficiently in advance to commission their own, such as a growing company. Anyone who has only intermittent need for a lot of processing, such as an engineering company.
I'm not saying cloud is suitable for everything, just that it is suitable for some things. Or at least it is the least bad option for some things.
If this were not true then we wouldn't have the likes of Boeing exploring ways mitigating the security concerns of using the cloud. Mitigations include splitting up the data, and performing calculations on encrypted data.
Of course it isn't a one size fits all situation. One step along from building your own infrastructure company might be to rent space in a data centre where the power and physical security is managed by another party, but only your staff install the servers and hold the keys.
"there's no need for any business to use cloud services."
Quite true. We ran for years without such a thing existing but I don't suppose that's what you meant.
I see cloud as a useful option for smoothing out the bumps in a long term internal infrastructure growth strategy and providing capacity to cover short term events.
It's when 'cloud available = company operating ' that I get a bit twitchy.
Somebody has just won a bet that they could get a senior exec to publicly give a good reason as to why using cloud products for mission critical isn't smart.
I'd give this a '2' on the Ratner* scale
*If you don't know Ratner, search for 'ratner prawn' for a level '10'
I half waited that he would say they developed software defined UPS.
"Software you don’t own in your infrastructure is a risk"
Oh, the irony.
AWS, he said, is perfectly clear that its data centres are a disaster-proof distance from one another, but less than a millisecond of latency apart.
And a hurricane that traipses thru Herndon and Sterling and the oh, at least 9 datacenters that sit next to each other cheek to jowl is what exactly? Sure, the other cluster for us-east-1 are in Manassas off of 234-bypass which is about 24 miles apart as the crow flies.. Then there is the tiny matter of the big power-lines that follow VA RT7 west of Lessburg. Topple a couple of those with a brick of explosive and also the ones that run North/South across I-66 and you've wiped out Manassas to some extent. The power for Micron Technologies (VA rt28) has major power infra but I think that comes in from points further south.
US-east-1 is grossly over-built and over-concentrated. It really needs to be diffused, bigly.
I think you have the makings of a strategy video game, like Sim City but with more Godzillas!
Define disaster. Hurricanes, earthquakes, wildfires, sure. War, meteorite strikes, zombie plagues... not forgetting squirrels, whose suicidal attacks on America's power grid has caused more outages than terrorist action has to date. Effects of disasters can cascade, such as a tsunami taking out the primary and back up power supplies to the pumps that cool a nuclear reactor. Or, all of your storage suppliers have their factories located in the same flood plain (and they still do).
The point is, 100.0000000% resiliency can only ever be an ideal.
there are 4 DC that are 1 minute walk from each other. That's disaster-proof? Sure, a raging inferno resulting in total outage of a single facility won't take down the region or even an entire AZ depending on the service. We also get 'twisters' thru the river corridor though generally nothing big enough to be more of an inconvenience on secondary roads and local power outage.
The interesting exercise is the bidding war that erupts when 30-odd DC all need diesel shipped in and there aren't enough trucks to make it happen even if the Dulles fuel dump has the fuel reserves stockpiled. Logistics can be a cruel mistress.
There's no good reason why they couldn't have put some facilities out toward Winchester, or a new region down by Richmond/Petersburg or Lynchburg..
Dropping a nuke on DC or NYC is a worthless act. Put one on Dulles airport and Hoboken NJ and you do some useful damage.
Maybe that would make for a better video game... Instead of building and maintaining like Sim City, you get to bomb a continent with nukes... and squirrels!
High score is for causing the most disruption with the least bombardment - as you say, taking out a single fuel depo rather than multiple data centres.
put a point 2 miles due north of runway 1L. draw a circle with radius 3 miles. That's "half the internet" within the boundary.
Strange terminology
What are all these "switch gears" turning?
In 40 years of producing control systems for power distribution I've only ever heard it called 'switchgear".
Re: Strange terminology
Well how do *you* level the output of your hamsters if not with gears?
chip looks like the Annapurna Labs
this is general purpose mid range ARM stuff that is cheap hence why they are putting it into a NIC yes it can run linux and do offload for the main general purpose processor.
really this is general purpose processor being pushed into places to replace application specific integrated circuits that used to do offload, basically the complete opposite of what the exec is talking about...
dont let the details get in the way of a good story for the executive...