News: 1633946704

  ARM Give a man a fire and he's warm for a day, but set fire to him and he's warm for the rest of his life (Terry Pratchett, Jingo)

The planet survived six hours without Facebook. Let's make it longer next time

(2021/10/11)


Opinion At the time of writing, it has been exactly 100 hours since Facebook, Instagram and WhatsApp crept back out of the void onto the internet*. They'd been gone for six hours – or [1]seven billion dollars , if you measure out your life by Zuck's net worth, which we don't recommend.

The time for hot takes has passed. We are now in the brief window for sober analysis before everyone forgets it ever happened. Next Thursday, by current standards.

There are three angles that matter. One is the pure technical side – how a giant corporation, built out of the most resilient networking technologies ever created, just vanished. One is what it teaches us about Facebook's importance to our daily lives. The last is what it tells us about Facebook itself, where it goes next, and whether its strip-mining of societal values for profit will continue.

[2]

The technical side is [3]simply put . Someone, most likely a hapless sysadmin, tried to check backbone availability, but a malformed command shut down access to the company's DNS servers instead.

[4]

[5]

It did this by cancelling the routes into Facebook that the internet knew about. The safety checks to stop that happening were broken, there was no Plan B and, as a bonus, most if not all of Facebook's internal tools, comms, virtual and physical access systems shut down too. Hilarity ensued.

Did it matter?

[6]

Yes and no. There are lots and lots of small businesses that rely on Facebook pages to talk to their customers. They had six hours of uncertainty.

WhatsApp has been widely adopted as a team channel by all sorts of people who had to wait to plot, plan or parcel out work.

Families dealing with sickness or crises remotely suffered.

[7]

You don't turn a multibillion-user service off without pain, and most of that was felt by people with the least heard of voices. But think what would happen if Google, Microsoft or Amazon had a total service outage like that. Even the [8]partial ones we've seen have created universal concern that was absent here.

[9]Facebook rendered spineless by buggy audit code that missed catastrophic network config error

[10]We have some sad news about Facebook. It has returned to the internet after six-hour mega outage

[11]Things that are not PogChamp: Amazon's Twitch has its source code, streamer payout data leaked

[12]Facebook takes bold stance on privacy – of its ads: Independent transparency research blocked

[13]Sorry, Mr Zuckerberg isn't in London that day. Or that one. Nope. I'd give up if I were you

[14]Cloud load balancer snafu leads to 3D printer user printing on a stranger's kit

Instead of concern, there was a sigh of relief all around, and a surge of schadenfreude among that hapless admin's peers. That's unusual and noteworthy.

Most sysadmins and other ops types have huge sympathy when an internal fumble by one of the clan causes public conniptions. There but for the grace of God is the rule – and the #HugOps tag is deployed in fraternal and sororal support. This time, not so much. Normally, everyone in the business recognises that work is work, but the social network seems like social death for network admins.

This failure of respect amplifies the technical aspects of the outage. The raw components from which Facebook is built, the technologies and the people in the engine room, can do much better when allowed. You can build secure backdoors that bypass your main networks. You can build watchdogs that fall back to a known safe state when things stop happening.

You can't foresee everything, but you can assume systemic failures whatever the cause. Yet a single point of failure took everything down. Efficiency had won over redundancy. The worst case that never happens, happened. Too much security is never enough – until it's just too much.

And these are managerial decisions.

When we look at how Facebook's business model is constructed, we see the same thinking. Concerns of malfeasance from the outside world must be locked out, no matter how badly the internal systems are behaving. Maximal efficiency, a single-minded approach to gathering data and selling it – allegedly without nuance or safeguard – is the one true way.

[15]Whistleblowers are the enemy. The media is to be ignored, rebuffed or treated like idiots, and what the hell do regulators or [16]governments know anyway? Best practice is what Facebook says it is, not what everyone else decides.

This is hubris, characteristic of a company that is too powerful to care. State telcos had it. IBM in the 1960s had it and [17]Microsoft in the 1990s – neither are by any means cured. And Apple, Google, Amazon all have it too.

But where Facebook is uniquely vulnerable is that if it goes away, it doesn't much matter, even in the medium term. It is a social network, and nothing but, and users can rebuild that social interaction on a different platform in weeks. Others stand ready to take the advertising and analytics spend.

Unlike Microsoft, Amazon and Google, unlike the telcos and IBM, Facebook provides no essential services to business or state. Quite the opposite – it threatens, and it has money, but it has no leverage. A loss of confidence at the top, or a sufficiently puissant legal or regulatory challenge, and it could vanish as quickly as its BGP routes, an irrelevant Yahoo ! for the 2020s.

Facebook embodies the single point of failure. It has no friends it cannot buy, just enemies it cannot pay off. Six hours on a Facebook-free planet felt like six thousand hours too few, and now we know it. Does Facebook? ®

Down and out in Menlo Park...

*The firm later gave some [18]Facebook and Instagram users another little breather on Friday from around noon Pacific Time (19:00 UTC) to around 13:00 PT (20:00 UTC).

Get our [19]Tech Resources



[1] https://www.bloomberg.com/news/articles/2021-10-04/zuckerberg-loses-7-billion-in-hours-as-facebook-plunges

[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YWRfwDETdokQLzYRM2t97AAAAMo&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0

[3] https://www.theregister.com/2021/10/06/facebook_outage_explained_in_detail/

[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YWRfwDETdokQLzYRM2t97AAAAMo&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YWRfwDETdokQLzYRM2t97AAAAMo&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YWRfwDETdokQLzYRM2t97AAAAMo&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YWRfwDETdokQLzYRM2t97AAAAMo&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[8] https://www.theregister.com/2021/09/28/aws_east_brownout/

[9] https://www.theregister.com/2021/10/06/facebook_outage_explained_in_detail/

[10] https://www.theregister.com/2021/10/04/facebook_outage_fixed/

[11] https://www.theregister.com/2021/10/06/twitch_data_leak/

[12] https://www.theregister.com/2021/08/04/facebook_research_lockdown/

[13] https://www.theregister.com/2018/11/15/facebook_mark_zuckerberg_international_committee_decline/

[14] https://www.theregister.com/2021/08/20/3d_printer_spaghetti_detectives_cloud_config/

[15] https://www.theregister.com/2021/10/05/facebook_whisteblower_congress/

[16] https://www.theregister.com/2018/11/15/facebook_mark_zuckerberg_international_committee_decline/

[17] https://www.theregister.com/1998/10/20/microsoft_on_trial/

[18] https://www.theregister.com/2021/10/08/facebook_instagram_outage/

[19] https://whitepapers.theregister.com/



SlippyBuckfast

Here here. The sooner facebook goes titsup the better imho. Nothing good has ever come out of it. It's negative by design. If you've not read it, "Ten Arguments For Deleting Your Social Media Accounts Right Now" by Jaron Lanier is worth a read.

Chris G

As much as I would like to see Feacebook shrivel and die horribly, I can't get rid of the feeling that the likes of Google have something waiting in the wings, ready to throw onto the stage so that the show goes on.

If google took over the reins of the zuck empire, it would be an even worse disaster than keeping FB.

Nature and marketing abhor a vacuum.

Not necessarily.

Steve Button

I'm sure Google and others would love to step in and replace FB, but would people not want something that doesn't have all the horrors that Facebook has? A generally nicer place that's not so confrontational all the time? I guess that would have fewer eyeballs on it, and therefore would not be as popular (survival of the fittest, and all that?)

I would sign up for NiceBook, but I deleted my FaceBook account years ago.

Sorry that handle is already taken.

Hear, hear. Both to your comment and by way of correction.

Jaron Lanier's book

alain williams

If you've not read it, "Ten Arguments For Deleting Your Social Media Accounts Right Now" by Jaron Lanier is worth a read.

Here is a [1]summary of what he says.

[1] https://thinktheology.co.uk/blog/article/ten_arguments_for_deleting_your_social_media_accounts_right_now

Anonymous Coward

There are three angles that matter. One is the pure technical side – how a giant corporation, built out of the most resilient networking technologies ever created, just vanished.

A future story for "who me" without a doubt. Lack of change control, or a wobbly finger on a keyboard - there are many ways it could have happened.

One is what it teaches us about Facebook's importance to our daily lives.

Absolutely none at all, assuming you actually have a meaningful life to start with.

The last is what it tells us about Facebook itself, where it goes next, and whether its strip-mining of societal values for profit will continue.

Who cares?

And as for those who built their business on someone elses platform over which they have no control, and I assume, pay not a penny? Well, that's a risk of doing business that way - like someone who runs a business from home, using the cheapest possible broadband service with no backup plan.

But I do feel for those "Families dealing with sickness or crises remotely suffered." Just because it is cheap and effective does not mean it is guaranteed to be reliable. Perhaps one should raise a glass as to how reliable and resilient the internet normally is, 99.9999999% of the time. A beer to those who thought up these protocols and technologies 40-50 years ago.

It will take a while

Roger Kynaston

I have got Whatsapp and use it. I have two main groups who I talk to. I have been banging the drum for an alternative platform for a while and all I get is silence.

I am coming to the view that they know that the communications provided are not important so it doesnt' matter if you don't go to the trouble of having a fallback/standby/alternative.

They are reconciled to the Zuck hoovering up all their data and do their best to stop their offspring from being bullied or whatever (or don't know if said malfeasance is happening?)

Final meandering question: Would it be possible to create an antisocial media that wasn't antisocial?

Re: It will take a while

John Robson

Final meandering question: Would it be possible to create an antisocial media that wasn't antisocial?

Yes - but you need to figure a way to pay for it, or to have it truly distributed (and that would need it to be a very reliable "just plug it in" appliance - and then you need to support that as well...

Re: It will take a while

Doctor Syntax

"Yes - but you need to figure a way to pay for it, or to have it truly distributed"

Like Diaspora?

The actual problem is that these networks connect people and some people are antisocial so how do you keep those out?

Re: It will take a while

WolfFan

Hmm… how _do_ you keep them out? Hmmm… Hunt them down. Kill them all. By this axe I rule!

Why are you looking at me that way? I’ve got a little list, you want to be on it?

Re: It will take a while

ThatOne

It would be terribly difficult bordering to impossible, because you can lead the user to water but you can't make him drink. The biggest problem with building any "social media" is the herd effect: Unless everybody uses it, nobody will.

For this reason we can only have one single big actor at any time, instead of several competing ones. And for the same reason it keeps getting bigger no matter what happens. If Google and Apple despite their billions didn't manage to compete with Facebook, it's because the market had already decided Facebook was the place to be and to be seen.

Let's take a real life example, WhatsApp since we're talking about it: You could use Signal, which is better in most ways - but , it isn't what the cool kids use, so nobody would be seen dead using it. If you don't believe me, ask a young one about switching to Signal: He/she/it will sprout a long list of mostly fallacious reasons he can't, which mostly boil down to the fact his peers expect him to be on Whatapp, because that's where the cool kids are. Inertia does the rest. WhatsApp would indeed need to become very uncool for the masses to start looking for an alternative.

Re: It will take a while

DavidYorkshire

Another problem is the number of clubs and societies which use it as their only / main way to communicate. Fifteen years ago these would have had their own website and email list, or used something fairly non-offensive like a Yahoo group to communicate. Now they just have a Facebook page.

Re: It will take a while

werdsmith

For this reason we can only have one single big actor at any time, instead of several competing ones.

Not if they used a common api. We can use any mail client with smtp or imap. We can write our own. We can use any browser with a number of different web servers. There are a number of different search providers.

It is not beyond human capability to devise a standard by which a user of any social media platform could choose to share their content with a user of any other, without having to insist on having an account on both. So a chav who insists on using the sewers of faecebook or whatscrap could share their posts with someone using any other social media, and vice versa for the fundamental communication. Much like SMS works across any phone on any network. This is much more in tune with what the internet was like before scum started trying to fence off as much as possible for their corporate greed and before the likes of chavbook attempted to become the de facto internet.

Re: It will take a while

Doctor Syntax

"it isn't what the cool kids use"

And all the better for that.

Anonymous Coward

Back in the day, Novell had a tool which allowed you to run the same command across multiple servers in your NDS tree. One day it disappeared from Novell's site and all mention of it was purged. I asked my contacts about it as I found it a useful tool.

It turned out that one customer had used the tool to run a diagnostic command intended for a single Netware server across every server in their NDS Tree. It deleted every bit of the NDS tree from every server. (for you youngsters: NDS Tree == Active Directory) I'm guessing I know the answer to the question "Did you have a backup?"

It sounds like some poor Facebook Network Admin had a similar moment - but on a much bigger scale!

find users who cut cat tail

> One is what it teaches us about Facebook's importance to our daily lives.

I read about the outage in the news the following day. Not surprising that I wouldn't notice – but apparently no one else cared enough to mention it either.

Without Facebook...

Pete4000uk

....how would I keep up with old friends I've not seen since 1992? I'll have to go out and make 'real' friends and spend time with them.

Re: Without Facebook...

Def

But it's not about just keeping up with old friends you never see anymore.

I keep up with current friends who live all around the world. Not so much through what they post on Facebook (that's obviously part of it), but more through Messenger and Whatsapp. Yes, other messaging apps exist, but most people I know use the above.

I also use Facebook to see what events are happening locally, which friends are going, and organise meeting up with them there. (Or in the case of my local pub knowing when to avoid the awful live music events they put on most weekends.)

Re: Without Facebook...

tiggity

I don't like how many companies, groups use FB for disseminating information.

Especially given that FB normally only allows a non FB member to see the last few posts, so (as someone without a FB account) I often do not see lots of local events.... But, I'll just keep missing out (if something happening locally that is "right up my street", invariably one of my friends tells me about it (not by social media!), so as long as I have a critical mass of FB using friends I do not miss out).

What I really hate is companies that have their own websites but still put urgent / important news on FB but not on their website (e.g. stares at Trent Barton buses who in the last UK bad snow spell cancelled services due to the heavy snow but the info was not on their web site, just on FB & Twitter)

Re: Without Facebook...

DavidYorkshire

Railway companies tend to do that as well - and use social media as the only way to contact them for a quick answer (they could have a chat function on their websites, but don't).

Often the only option on the website is an enquiry form, which might get a response in a couple of weeks. Maybe.

Re: Without Facebook...

Doctor Syntax

"how would I keep up with old friends I've not seen since 1992?"

Email. Phone. Snail mail.

If Facebook didn't exist it wouldn't have to be invented. The only thing wrong is that somebody did.

Re: Without Facebook...

werdsmith

"how would I keep in touch with old friends.......?"

That puerile excuse for supporting scum again. It's really not difficult, if they are really friends they will stay in touch. The Faecebook excuse is just laziness.

Re: Without Facebook...

Skiver

That's a bit rude. While there is nothing anyone can say that will change your perspective, but it's a lot more than just friends. As I wrote earlier, for those of us who have family members who live in different parts of the country or world, social media provides the ability to keep in touch with them. I rather enjoy enjoy staying in touch with family members.

It isn't laziness. It's practicality. It would simply be infeasible to stay in touch with more than a few family members.

And to be clear, this is not commentary on FB as a company or even a defense of FB in general. It's an argument against the "social media bad" point of view.

Re: Without Facebook...

Doctor Syntax

Email. Phone. Snail mail.

Re: Without Facebook...

Sixtiesplastictrektableware

Plenty people use it for legitimate emotional connectivity. Lots of community building, too. I always forget all of the dedicated businesses running off the thing.

The best evil always has a thin veneer of good to it.

Re: Without Facebook...

Anonymous Coward

>> ...how would I keep up with old friends I've not seen since 1992?

If you've not seen them for 30 years, can they and you really be friends with each other? Besides, you would have made those friends long before evil shit like facebook existed. How did that happen and why can't you make contact with those people like you did all those years ago?

>> I'll have to go out and make 'real' friends and spend time with them.

Shock! Horror! What is the world coming to?

Re: Without Facebook...

Skiver

While this is undoubtedly true for some, for many of us social media allows us to stay in touch with family and friends no matter where they live. I have family that lives all over the U.S. and even one family member in South America. It would be extremely difficult to stay in touch with all of them on a regular basis without social media.

Anonymous Coward

I deleted my account years ago and would party in the streets the day Facebook's IP address was permanently scorched from the earth. I think social media is the most dangerous technology we've invented since the bomb; perhaps more so, and similarly it's a pandora's box that can never be closed. Other services could replace it and maybe they will, but the masters of this world have seen the immense value of having such a vast portion of human consciousness hard wired into a single feed to allow it to fail. Rather than seeing the beginning of the end, I fear we're just seeing the end of the beginning, with the metaverse on the horizon, Facebook casts its eye ever outwards toward the entire digital landscape. As Orwell said - the choice for mankind lies between freedom and happiness and for the great bulk of mankind, happiness is better.

Loyd Grossman accent ... Who would run a company like this ?

Andy The Hat

A question I ask myself a lot is, given Zuck is now worth a bit more than £3.50, why is he still there? Either

a) he can never have enough money

b) he can never have enough power

c) he's an android developed purely to make money and power for it's mysterious operator

or

d) all the above.

Maybe I'm missing something?

Re: Loyd Grossman accent ... Who would run a company like this ?

Def

Or maybe he just enjoys working at the company he helped create? It's a crazy thought, I know, but some people actually like what they do.

Re: Loyd Grossman accent ... Who would run a company like this ?

DavidYorkshire

Indeed - many of the founders of the previous generation of really big IT companies stayed in charge for years, and one or two (e.g. Michael Dell) still are.

Re: Loyd Grossman accent ... Who would run a company like this ?

Chronos

Have you seen the man speak? He has an emotion deficit that Spock would have envied. The problem with Facebook and, indeed, most social notworks is a lack of empathy, both on the part of the companies running them and the users using them. It's like giving a man a chainsaw: Give any old munchkin a mechnism to destroy people with a pithy put-down and they're going to do it.

Why do we feel comfortable gesturing at other drivers when driving? Because the mobile box insulates us and virtualises interaction. It's the same with sitting behind a keyboard. I'm probably doing it right now, although I am always aware on ElReg that other commentards are at least as smart as I.

What is really dangerous, though, is the blurring of the lines between fact and opinion. Facts are becoming "number of likes/upvotes/thumbs up" with the proliferation of mob validation. This, in my opinion is driven by the social networks and real experts are being lost due to the low SNR. Covid has been an excellent heads-up. Whether we hear that warning us up to us.

Re: Loyd Grossman accent ... Who would run a company like this ?

werdsmith

Why do we feel comfortable gesturing at other drivers when driving?

Because we want to demonstrate to them how pathetic we are?

I don't think I could ever be comfortable doing that.

Re: Loyd Grossman accent ... Who would run a company like this ?

Anonymous Coward

Because when someone pulls up behind you in the overtaking lane and flashes his (it's almost always a "he") lights because your doing 70 and he was doing 90 a few seconds ago, ignoring him usually makes him more aggressive. A quick wave to say "I note your displeasure but I'm here legitimately and you'll just have to wait while I finish my manoveur" is only courteous.

:-)

Doctor Syntax

"Pity the lonely, unloved Facebook network admin"

Pity? Unloved? No, we applaud and love him or her and wish the rest of them would follow suit.

Which is pretty well the point being made a couple of paragraphs above.

Anonymous Coward

I deleted my account sometime back in 2012. Life goes on just fine thanks.

Life goes on...

Steve Davies 3

Never felt the need/urge/whatever to sign up to any of these networks in the first place. The same for GMail.

When incidents like these happen I just get on with life while people all around me get the heebies aka withdrawal symptoms and wonder if the world has indeed ended but no one was able to tell them because Zuckbook went TITSUP.

Suck on this Zuck --> see icon

Re: Life goes on...

Doctor Syntax

Last time I looked Gmail was a mail service provider. OK, it's Google so I suppose if you access it via the web you'll get ads but why do that - pop.gmail.com port 995 works perfectly well.

Efficiency or hubris?

Doctor Syntax

"Efficiency had won over redundancy."

Zuck has made his position clear - everyone should use Facebook for communication with each other. Perhaps this was taken literally which mean there was no communication network other than Facebook's. No back-channel to manage the infrastructure. If you think you're all the communication the world needs why would there be?

Re: Efficiency or hubris?

Anonymous Coward

Except it was widely reported some time back that Zuck himself is in fact a Signal user.

Signal groups work as well as Whatsapp groups IME and video calling is ok too. One wonders whether that was the medium used to organize the recovery.

The worst case that never happens, happened

Howard Sway

The worst case that never happens, always happens eventually.

Unfortunately that also happens to society too. Time to press the off switch permanently. And let whoever replaces it know that if they facilitate the undermining of democracy, knowingly profit from violence and organised disinformation or behave in ruthlessly anticompetitive behaviour then the off switch will be activated there too.

Rant!

ThatOne

> There are lots and lots of small businesses that rely on Facebook pages to talk to their customers

That's one of the unforgivable crimes in my eyes. Force me to use Facebook to communicate with you, just because you're lazy? No, I'd rather not do business with you, as you're clearly lazy, selfish and generally unreliable.

Setting up a one-page web site with some kind of contact form is something everybody can do, most of the time for free (everybody has a computer-obsessed nephew). Proof is everybody (including tiny mom and pop stores) already managed to do it just fine all those years before the great Zuckerisation.

Re: Rant!

werdsmith

Yes, if they rely on faecebook then the problem is entirely of their own making.

Lack of interoperability = single point of failure

LDS

I've no compassion for business using Facebook to host their web pages, also I do not make any business with them since every time I land on such pages I'm asked to access them with a Facebook account I don't have nor I'm going to have.

Nor I have any compassion with people dependant Whatsapp and similar services designed as silos without any interoperability to work as cages to keep people locked in. I still use phone calls, SMS and emails because I'm not bound to a single provider and I can reach anyone from anything because of INTEROPERABILITY.

My internet provider is different from my mobile one to reduce the risk a single outage will cut me from everything. Why should I use a single provider for web hosting and messaging - no matter how big?

If people need to learn this the this the hard way, this outages are only welcome. I didn't notice any disruption - all of my communication means kept on working.

werdsmith

Take a stand you spineless wimps. Delete your faecebook accounts and do the world a huge favour. It's really not that important and you will live (better) without it.

Doctor Syntax

"Delete your faecebook accounts"

Sorry, won't can't.

John70

Never felt a thing. Anything I do doesn't involve Facebook.

I wonder how many lives were saved?

adam 40

Probably a few dozen teenage girls didn't commit suicide during this 6-hour outage, and who knows, might survive to this day...

Lucas' Law: Good will always win, because evil hires the _stupid_
engineers.