Worldwide outage problems (Internet)

[northwest]

Some of you may be experiencing email or website issues today:
There are widespread service provider outages and this is affecting multiple providers. Currently Google, AWS, Spotify, CloudFlare, Azure, Shopify, and others are having problems or are down for these customers. Because some websites rely upon these 3rd-party services it is having a widespread impact;
More here:

Google Cloud and Cloudflare hit by widespread service outages

Google Cloud and Cloudflare are investigating ongoing outages impacting access to sites and various services across multiple regions.

www.bleepingcomputer.com

Chat GPT is being widely impacted too.

 

[mart]

Ok, but do we know why this is happening?

 

[OJCar]

Ok, but do we know why this is happening?

Google Cloud = Single Point of Failure

 

[Erie Fred]

Pinky & The Brain are at it again.....

 

[jmp49]

Ok, but do we know why this is happening?

Google pushed some bad code out without the flags to stop it wrecking things with no error handling, and it replicated everywhere with no checks and balances because they missed the flag, and this caused a major outage, then while recovering the extra traffic overloaded the ones that survived & everything else dependent on them had to recover. Because google has their tendrils in most of the internet now, it impacted a wide audience.

They found it after someone pushed a policy update with blank fields and it caused a null pointer exception. Oh gosh, lots of "agile" coding happening there...

Google Cloud Service Health

status.cloud.google.com

 

[Erie Fred]

Google pushed some bad code out without the flags to stop it wrecking things with no error handling, and it replicated everywhere with no checks and balances because they missed the flag, and this caused a major outage, then while recovering the extra traffic overloaded the ones that survived & everything else dependent on them had to recover. Because google has their tendrils in most of the internet now, it impacted a wide audience.

They found it after someone pushed a policy update with blank fields and it caused a null pointer exception. Oh gosh, lots of "agile" coding happening there...

Google Cloud Service Health

status.cloud.google.com

Lovely....Just %$#@! lovely....now the hackers have a new tool at their disposal.

 

[jmp49]

No, its been fixed and made safe and pushed out to the cloud, so the potential for any zero day via it has been fixed and wont be exploitable, the window for that was tiny and between the code being pushed and the bad policy arrived and while the SRE (site reliability engineers) were investigating. And I imagine several people are having a bad week in the aftermath as it will have cost them millions in lost profits.

The agile barb was because they didn't test it properly in staging because someone had forgot to flag it.
Whats also interesting to me is if you can choose to miss a flag off in Critique (their inhouse equivalent of jira tracking) and bypass the staged release process, your deployment process is broken and needs fixing, or some automation that add's it had broken, or some people got maximum power to do what they want. I expect the very same to have come up in a meeting or two there, and reading between the lines on their post incident review, it has.
Peer review doesnt catch or stop this because coders in general are so under resourced and innundated with review requests permanently, they are not given time to review things properly, and also do their own job, so a lot of the time you basically skim through something at best, and basically trust that XYZ knows his stuff and its ok.

The staged release is relevant because if it was staged to hit each region at different times, the SRE's would have spotted it when it affected a smaller amount of the world in the earlier regions and quietly reverted it. Or maybe it would have been fine until the new policy got pushed out global with the blank fields, and it woudnt have made a bit of difference. IDK but they say it would have been caught in staging if it was flagged correctly so I'll go with them.
I've been involved with global releases of cloud software and it hits certain servers and the SRE's monitor it, and you wait for the customers angry phone calls to alert you, and if all is good after a hour or two, to the next region it rolls out.
Modern cloud infrastructure and software is riddled with process and issues like this.

Hats off to Google for being so transparent about it though, most companies try to deny and lie their way through things imho.

 

[gaz1]

youtube is down has been playing up most of the day

i can access the site no problem but forget about playing those videos as they wont play

some have played when i needed them too but most wont and hasnt since 5pm for me as ive been on and off youtube most of the day

 

[Seadog]

Really? I've just been watching footage with no problem.

 

[Robotstar5]

Watched few today with no problems.

 

[Hood]

It was screwed up for me at the workshop for a while, click a video and it said there was an issue and to try back later.

 

[Morrisman]

Google pushed some bad code out without the flags to stop it wrecking things with no error handling, and it replicated everywhere with no checks and balances because they missed the flag, and this caused a major outage, then while recovering the extra traffic overloaded the ones that survived & everything else dependent on them had to recover. Because google has their tendrils in most of the internet now, it impacted a wide audience.

They found it after someone pushed a policy update with blank fields and it caused a null pointer exception. Oh gosh, lots of "agile" coding happening there...

Google Cloud Service Health

status.cloud.google.com

A few months ago I uploaded all the pics on my iPad to the Cloud, to free up space for an update. Now they appear all jumbled up and in random order every time I look in my Photos page. And I can’t work out how to simply get them out of the cloud and back full time and whole on my iPad.

 

[brightspark]

been watching it all nite .still am

 

[gaz1]

It was screwed up for me at the workshop for a while, click a video and it said there was an issue and to try back later.

ive had that

even a suggestion to restart the pc but on a new start up and its not the pc thats for definate

ive also had a constant round circle arrow just going round and around not even playing the vid

ive closed it off and tried at a later time and so far its been the same