AWS Tokyo outage takes down banks, share traders, and telcos
(2021/09/02)
- Reference: 1630569311
- News link: https://www.theregister.co.uk/2021/09/02/aws_ap_northeast_1_outage/
- Source link:
The AP-NORTHEAST-1 region of Amazon Web Services, located in Tokyo, has endured six hours of sub-optimal performance.
The cloud colossus's status report states that AWS Direct Connect hybrid cloud networking service had trouble connecting to resources in the region due to "failures in core networking devices".
As of 15:30pm Pacific Time on September 1st — 7:30am in Tokyo — AWS said it was working to restore the services.
[1]
Two hours later, the company had determined that virtual private network traffic was still flowing but could not offer an estimated time for service restoration.
[2]
[3]
By 21:00 Pacific Time — lunchtime in Tokyo — users were advised that signs of partial recovery had become evident.
The incident was closed at 21:42 — more than six hours after packet loss was first observed.
[4]
AWS's all-clear announcement update expanded a little on the cause of the incident, stating it was caused by "the loss of several core networking devices that are used to connect Direct Connect network traffic to all Availability Zones in the AP-NORTHEAST-1 Region".
[5]In 2006, Amazon debuted EC2. 15 years on, HashiCorp says firms blowing their cloud budgets is all part of the fun
[6]'Worst' AWS service ever? Cloud giant introduces Redis-compatible MemoryDB – to mixed response
[7]Splunk spots malware targeting Windows Server on AWS to mine Monero
Just what AWS means by "loss" was not explained. The Register will assume they broke, rather than disappearing behind a couch.
Whatever happened, "The issue has been resolved and the service is operating normally."
The incident again demonstrated that when AWS goes down, so do its customers; Japanese media report disruptions to banks, payment systems, share traders, and others beside.
Users were, of course, not very happy. But in this instance some found an additional reason for anger.
It was very considerate of them to send the AP-NORTHEAST-1 notice in Japanese [8]pic.twitter.com/wjgDUomSZD — James Harvey (@jmhredsox) [9]August 7, 2021
®
Get our [10]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YTCg5Z4u-tpNtzOyZndV0gAAABU&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YTCg5Z4u-tpNtzOyZndV0gAAABU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YTCg5Z4u-tpNtzOyZndV0gAAABU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YTCg5Z4u-tpNtzOyZndV0gAAABU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://www.theregister.com/2021/08/24/ec2_at_15/
[6] https://www.theregister.com/2021/08/23/aws_memorydb_reaction/
[7] https://www.theregister.com/2021/08/10/crypto_botnet_targets_windows_on_aws/
[8] https://t.co/wjgDUomSZD
[9] https://twitter.com/jmhredsox/status/1423990378009559045?ref_src=twsrc%5Etfw
[10] https://whitepapers.theregister.com/
The cloud colossus's status report states that AWS Direct Connect hybrid cloud networking service had trouble connecting to resources in the region due to "failures in core networking devices".
As of 15:30pm Pacific Time on September 1st — 7:30am in Tokyo — AWS said it was working to restore the services.
[1]
Two hours later, the company had determined that virtual private network traffic was still flowing but could not offer an estimated time for service restoration.
[2]
[3]
By 21:00 Pacific Time — lunchtime in Tokyo — users were advised that signs of partial recovery had become evident.
The incident was closed at 21:42 — more than six hours after packet loss was first observed.
[4]
AWS's all-clear announcement update expanded a little on the cause of the incident, stating it was caused by "the loss of several core networking devices that are used to connect Direct Connect network traffic to all Availability Zones in the AP-NORTHEAST-1 Region".
[5]In 2006, Amazon debuted EC2. 15 years on, HashiCorp says firms blowing their cloud budgets is all part of the fun
[6]'Worst' AWS service ever? Cloud giant introduces Redis-compatible MemoryDB – to mixed response
[7]Splunk spots malware targeting Windows Server on AWS to mine Monero
Just what AWS means by "loss" was not explained. The Register will assume they broke, rather than disappearing behind a couch.
Whatever happened, "The issue has been resolved and the service is operating normally."
The incident again demonstrated that when AWS goes down, so do its customers; Japanese media report disruptions to banks, payment systems, share traders, and others beside.
Users were, of course, not very happy. But in this instance some found an additional reason for anger.
It was very considerate of them to send the AP-NORTHEAST-1 notice in Japanese [8]pic.twitter.com/wjgDUomSZD — James Harvey (@jmhredsox) [9]August 7, 2021
®
Get our [10]Tech Resources
[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YTCg5Z4u-tpNtzOyZndV0gAAABU&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YTCg5Z4u-tpNtzOyZndV0gAAABU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YTCg5Z4u-tpNtzOyZndV0gAAABU&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YTCg5Z4u-tpNtzOyZndV0gAAABU&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://www.theregister.com/2021/08/24/ec2_at_15/
[6] https://www.theregister.com/2021/08/23/aws_memorydb_reaction/
[7] https://www.theregister.com/2021/08/10/crypto_botnet_targets_windows_on_aws/
[8] https://t.co/wjgDUomSZD
[9] https://twitter.com/jmhredsox/status/1423990378009559045?ref_src=twsrc%5Etfw
[10] https://whitepapers.theregister.com/
Dan 55
Don't forget governments moving to cloud either.
It is surely inevitable now that regulators intervene to mitigate the concentration risk of AWS and Azure. It's insane that essentially our entire financial services infrastructure runs from half a dozen cloud regions around the globe.