Google Cloud's US-East load balancers are lousy with latency
(2023/03/20)
- Reference: 1679293868
- News link: https://www.theregister.co.uk/2023/03/20/google_cloud_us_east_incident/
- Source link:
Google Cloud is having a nasty not-quite outage in its US-East region.
At the thoroughly indecent time of 05:56 AM on Sunday, Pacific time, Google [1]reported an incident it called "Cloud L2 overload in us-east4 causing harm in us-east4 & nearby regions" and ascribed the cause to "an issue with Cloud Load Balancing."
Google Cloud has three such regions: South Carolina hosts east1, North Virginia is home to east4, and Ohio is the site for east5. Confusingly, Google Cloud does not have an east2 or east3 region.
[2]
At 07:56 the incident was renamed "Google Engineers are investigating high latency and elevated errors in US-East."
[3]
[4]
The issue has persisted ever since and kept that moniker, with Google listing South Carolina (us-east1) and Northern Virginia (us-east4) as the currently affected locations.
Mitigation work had commenced by 11:41 AM, but at 21:34 Google's 11th status update on the incident still reported a diagnosis of "Higher latency and possibly elevated errors on customers' global L7 Load Balancers during US peak hours (~05:00 to ~19:00)."
[5]
That won't have been a colossal issue on Sunday, US time. But as The Register writes this story the clock has just tipped past 01:00 AM on the US East Coast. As workers arrive at their desks on Monday, greater inconvenience is to be expected.
[6]WFH? Google Cloud's offices like a 'ghost town' before new policy
[7]Google: You get crypto, you get crypto, almost everyone gets email crypto!
[8]Taking notes from AWS, Google prepares custom Arm server chips of its own
[9]Big three cloud giants tighten grip as overall spending slows
Google's offered a workaround for customers – move their backends to another region. "Customers moving workloads to us-west, europe and asia may see improvements," the ads and search giant advises.
But it has no workarounds at this time "for customers who are required to host in us-east regions." Nor can searchzilla offer a time at which the incident will ebb.
"We do not have an ETA for mitigation at this point" customers are being told.
All of which adds up to an interesting choice for some Google Cloud customers as they arrive at work on Monday: wait out this incident and hope it doesn't degrade apps or annoy customers, or make a hasty move to another region. Because fast moves made under pressure always go well.
[10]
Hit the comments to let us know which option you chose. ®
Get our [11]Tech Resources
[1] https://status.cloud.google.com/incidents/bhM8RvvupuzK36xUpx5Q
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZBg81GvAhsObVvSRoFdNjgAAAM0&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZBg81GvAhsObVvSRoFdNjgAAAM0&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZBg81GvAhsObVvSRoFdNjgAAAM0&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZBg81GvAhsObVvSRoFdNjgAAAM0&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2023/03/07/wfh_google_clouds_offices_empty/
[7] https://www.theregister.com/2023/03/01/google_client_side_encryption/
[8] https://www.theregister.com/2023/02/14/google_prepares_its_own_custom/
[9] https://www.theregister.com/2023/02/07/big_three_cloud_market/
[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZBg81GvAhsObVvSRoFdNjgAAAM0&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[11] https://whitepapers.theregister.com/
At the thoroughly indecent time of 05:56 AM on Sunday, Pacific time, Google [1]reported an incident it called "Cloud L2 overload in us-east4 causing harm in us-east4 & nearby regions" and ascribed the cause to "an issue with Cloud Load Balancing."
Google Cloud has three such regions: South Carolina hosts east1, North Virginia is home to east4, and Ohio is the site for east5. Confusingly, Google Cloud does not have an east2 or east3 region.
[2]
At 07:56 the incident was renamed "Google Engineers are investigating high latency and elevated errors in US-East."
[3]
[4]
The issue has persisted ever since and kept that moniker, with Google listing South Carolina (us-east1) and Northern Virginia (us-east4) as the currently affected locations.
Mitigation work had commenced by 11:41 AM, but at 21:34 Google's 11th status update on the incident still reported a diagnosis of "Higher latency and possibly elevated errors on customers' global L7 Load Balancers during US peak hours (~05:00 to ~19:00)."
[5]
That won't have been a colossal issue on Sunday, US time. But as The Register writes this story the clock has just tipped past 01:00 AM on the US East Coast. As workers arrive at their desks on Monday, greater inconvenience is to be expected.
[6]WFH? Google Cloud's offices like a 'ghost town' before new policy
[7]Google: You get crypto, you get crypto, almost everyone gets email crypto!
[8]Taking notes from AWS, Google prepares custom Arm server chips of its own
[9]Big three cloud giants tighten grip as overall spending slows
Google's offered a workaround for customers – move their backends to another region. "Customers moving workloads to us-west, europe and asia may see improvements," the ads and search giant advises.
But it has no workarounds at this time "for customers who are required to host in us-east regions." Nor can searchzilla offer a time at which the incident will ebb.
"We do not have an ETA for mitigation at this point" customers are being told.
All of which adds up to an interesting choice for some Google Cloud customers as they arrive at work on Monday: wait out this incident and hope it doesn't degrade apps or annoy customers, or make a hasty move to another region. Because fast moves made under pressure always go well.
[10]
Hit the comments to let us know which option you chose. ®
Get our [11]Tech Resources
[1] https://status.cloud.google.com/incidents/bhM8RvvupuzK36xUpx5Q
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZBg81GvAhsObVvSRoFdNjgAAAM0&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZBg81GvAhsObVvSRoFdNjgAAAM0&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZBg81GvAhsObVvSRoFdNjgAAAM0&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZBg81GvAhsObVvSRoFdNjgAAAM0&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://www.theregister.com/2023/03/07/wfh_google_clouds_offices_empty/
[7] https://www.theregister.com/2023/03/01/google_client_side_encryption/
[8] https://www.theregister.com/2023/02/14/google_prepares_its_own_custom/
[9] https://www.theregister.com/2023/02/07/big_three_cloud_market/
[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZBg81GvAhsObVvSRoFdNjgAAAM0&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[11] https://whitepapers.theregister.com/
Isn't that where it rains right now?
Anonymous Coward
Might explain why its clouds have problems :)
Anonymous Coward
I'm being horribly naive, but isn't one of the selling points of all this Cloudiness the ease, almost trivial ease, of creating/starting/stopping/migrating all these machines and their data?
Ah yes, but that's what the Marketing says. The reality is that it's simply someone else's computer and you're stuck with whatever they do in reality..
make a hasty move to another region
I'm being horribly naive, but isn't one of the selling points of all this Cloudiness the ease, almost trivial ease, of creating/starting/stopping/migrating all these machines and their data?
Aren't you supposed to be able to pretty much select a location on a dropdown, click Ok and sit back whilst the Mighty Machinery Leaps to Your Bidding and seamlessly migrates the instances, finishing with a flourish by directing traffic to the new location?
And if any of that is true, at what point do SLAs indicate that Google should do that move automatically to work around issues in their own systems?