BBC makes switch to AWS, serverless for new website architecture, but observers note HTML is no longer clean
- Reference: 1604498469
- News link: https://www.theregister.co.uk/2020/11/04/bbc_embraces_aws_serverless/
- Source link:
"Until recently much of the BBC website was written in PHP and hosted on two datacentres near London," Matthew Clark, head of architecture, said in a [2]recent post . "Almost every part has been rebuilt on the cloud."
PHP runs fine in the cloud, but this is not a matter of lift and shift. Instead, the BBC team devised a new architecture based on serverless computing. It also endeavoured to combine what used to be several sites – such as News, Sport, and so on – into one, though Clark said the World Service, iPlayer video, and the radio site BBC Sounds remain separate.
The rest have been combined into a new thing called WebCore. "By focussing on creating one site, rather than several, we're seeing significant improvements in performance, reliability, and SEO," said Clark.
Web traffic initially hits a Global Traffic Manager (GTM), an in-house solution based on the Nginx web server and running partly on-premises (showing that the BBC has not entirely ditched its data centres) and partly on AWS. GTM handles "tens of thousands of requests a second," said Clark. A second layer on AWS handles caching and routing, before hitting functions running on AWS Lambda, which perform server-side rendering (SSR) of dynamic content using React, a JavaScript framework.
Server-side rendering means that the browser gets an HTML page ready to view but increases the burden on the server – though caching mitigates this. Walmart engineer Alex Grigoryan, who also oversaw a migration to SSR, [3]tested SSR vs client-side rendering (CSR) and said: "When we did A/B tests on SSR vs CSR... our numbers showed better engagement from the customer with rendering early," though he noted increased server load as a major disadvantage.
In the BBC's case Lambda is used, which is able to auto-scale on demand. "About 2,000 lambdas run every second to create the BBC website; a number that we expect to grow," said Clark. He added that Lambda scales better than VMs on the AWS Elastic Compute Cloud (EC2), saying that "our traffic levels can rocket in an instant; Lambda can handle this in a way that EC2 auto-scaling cannot."
Another aspect of the BBC site is the logic that goes into requesting content, which Clark calls the "business layer". Content is provided to the web rendering layer via a REST API, and a solution called Fast Agnostic Business Layer "allows different teams to create their own business logic," he said, so that different requirements are met while still sharing the same system for things like access control and caching. Clark didn't say much about how the content itself is stored, though he promised to return to this topic in future posts.
The WebCore platform uses CI/CD to enable rapid iteration, and Clark showed an example monthly report showing 110 releases or around three per day. Builds take around 3.5 minutes, and the average time from a pull request (when new code is merged) to running it in production was one day and 23 minutes, in this particular month. On average 67 per cent of pull requests were actually merged into the code.
[4]
A small section of the HTML delivered for a news article today on the BBC site. A news aggregator says it is much harder to parse than before
Great work? Comments on Hacker News show that opinions vary. "Running a site the size of the BBC on Lambda is nothing short of an exuberant waste of a government-subsidized budget, it's absolutely crazy. Lambda VM time has a massive markup compared to regular compute... IMHO this is the epitome of serverless gone wrong," [5]said one .
Another [6]comment from John Leach, who runs a headline aggregation site called News Sniffer, said that the generated HTML is not clean. "I run the News Sniffer project which has to parse BBC News pages and I knew about this rollout a few weeks ago when the HTML all changed format completely and my parsers broke. As a side note, the new HTML is way more complicated and much harder to parse than before – I know the aim isn't to help parsing for content, but I was still saddened to see how it's ended up."
There is also curiosity about unanswered questions. What is the cost impact of moving from on-premises to AWS? What is the cost impact of Lambda versus using EC2? Why, if the caching and content delivery network is working as expected, are 2,000 lamdbas a second required?
We have asked the BBC for more details. ®
Get our [7]Tech Resources
[1] https://www.alexa.com/topsites/countries/GB
[2] https://www.bbc.co.uk/blogs/internet/entries/8673fe2a-e876-45fc-9a5f-203c049c9f9c
[3] https://medium.com/walmartglobaltech/the-benefits-of-server-side-rendering-over-client-side-rendering-5d07ff2cefe8
[4] https://regmedia.co.uk/2020/11/04/bbchtml.png
[5] https://news.ycombinator.com/item?id=24986384
[6] https://news.ycombinator.com/item?id=24984710
[7] https://whitepapers.theregister.com/
Re: Does this change explain
I wondered that. Was a bit strange having pages flipping from 'normal' font to large-print for no apparent reason. I guess once it's stabilised, it can be the first site i give a negative zoom override to.
Re: Does this change explain
I think the new font is the BBC's very own new 'Reith' font. It was, at one point available to download in case you were going to create any content for the BBC. It is horrible. I can't understand why on earth they thought they needed their own font, let alone one as ugly as this. They were planning to move everything over to 'Reith' in the future.
Re: Does this change explain
They created the Reith font because they currently pay significant royalties for using existing fonts.
Whilst they paid handsomely for the creation of Reith, their ROI is I think 3-4 years, they save millions after that. I actually like the Reith font, they've done a good job imo.
Server side rendering is as it should be
The current trend of building the page with javascript in the browser is wrong. It just complicates things for the browser and means that the browser has to run javascript from all manner of places.
Server side rendering means that for the browser it is KISS.
Re: Server side rendering is as it should be
I've heard it all now.
Server Side Rendering sounds like a move towards the browser being nothing more than a remote viewing client for some sort of content displaying application running on someone else's server. If that's the way it's going, what the hell is the point of doing it in HTML / CSS / Javascript in the first place (other than legacy / internia)? You may just as well have a native application running on that server, stream something akin to a video of it to the client, and ditch the complex / expensive to render HTML / CSS / resource fetching stuff altogether to save a ton of cycles and network bandwidth.
There's even a bunch of technologies out there that make this easy - RDP, VNC, Waylandpipe, even (if you squint a bit) X Protocol to name but a few.
Re: remote viewing client
I'd see this as more of a step back to the Good (?) Old Days of static, hand-coded HTML. The server served HTML to the browser, the browser rendered it and everything was good. None of this "you must enable JavaScript to load this page" nonsense.
If all that the JS is doing is generating HTML, I'd much rather it did it somewhere else so that when I look at the page source I've got some chance of working out what's going on.
Re: Server side rendering is as it should be
Is that what it means? I thought it was a case of the client side just being told where to get additional content from. No Javascript required.
Lambda makes a lot of sense when you need to scale up and down quickly. We are talking milli-seconds. Try doing that on a VM.
Perhaps, but it's easy to imagine that the BBC's traffic is highly predictable. 0700-0900, 1200-1300, 1700-1900, 2100-2300 is my guess.
It also ties them into AWS, so far as I can tell. Politically, helping enrich Amazon at license payer's expense without the option of upping sticks and moving elsewhere could be a bad idea. Especially if it turns out they've done it in a needlessly expensive way.
Yes, easy to imagine I am sure, but I guess the BBC would have done their research. It seems to me like they are not daft...
Is Poe's law at work here?
If performance is an issue React is your problem.
The new pages seem to be mainly inline Javascript. Though they do render quickly, doing this on the server isn't the solution, doing less in Javascript is as a [1]recent comparison . Also, ditch the inlining so that the browser can decide whether it wants to use cached versions.
It's also difficult to see why, in a largely non-transactional site, you'd be wanting to get the extra CPU power on demand. Even with content updating reasonably frequently, you're still largely serving static content. But the move is presumably justified by being able to offload all the skilled and expensive people from the own data centre in the hope that Amazon won't ever want to take advantage of the situation to raise prices.
[1] https://gomakethings.com/just-how-much-faster-is-vanilla-js-than-frameworks/
Re: If performance is an issue React is your problem.
Just throwing a wild theory out there,
With all content going online the BBC may just decide to reposition itself as one of the biggest independent content producers (including 3rd party commissions) and has near future plans for the full BBC content to be available via Amazon globally? That would bring the running cost down substantially.
Like I said just a wild theory.
"Running a site the size of the BBC on Lambda is nothing short of an exuberant waste of a government-subsidized budget, it's absolutely crazy. Lambda VM time has a massive markup compared to regular compute...
government-subsidized? Surely the bulk of the funding is the Licence Fee?
In fact, the licence fee subsidises the government:
Part of the fee also contributes towards Freeview and Freesat, and towards the UK broadband rollout, funding local TV channels and S4C, the Welsh language TV channel, as agreed with the government as part of the 2010 licence fee settlement.
And the World Service. And now free licences for (some) elderly.
Having said that, I'd be rather more impressed if their content management worked rather better so you didn't get glaring stupidities like articles about a photograph turning up on the Red Button service where you can't see the photo and articles that mention a place turning up in the regional news for that place even if the mention is irrelevant. Wrapping some HTML or MHEG around the content is rather less impressive.
Gubbermint funded
To some, no make that a lot of people, the BBC is the propaganda arm of the UK Gubbermint and should be despised as much as the Bozo's in Westminster. It does not matter that this view is wrong. Apart from the World Service everythibng else comes from the license fee or selling programmes.
Propaganda
"propaganda arm of the UK " government.
Yes, have a read of the comments on Daily Telegraph stories about the BBC. Fleck and spittle comes to mind.
I sometimes wonder if they think the BBC's 'Woke, Left wing' staff eat babies for breakfast.
WebCore?
Those little golden birdies....
Does this change explain
why half the news site pages use a sensible, clean, sans-serif font, while the others use an ugly, larger serif font in two shades of grey that make it difficult to tell whether one has viewed a page or not? There's been a definite feeling the last month or two of 'left hand doesn't know what right hand is doing'.