British Library pushes the cloud button, says legacy IT estate cause of hefty rebuild
- Reference: 1710163806
- News link: https://www.theregister.co.uk/2024/03/11/british_library_slaps_the_cloud/
- Source link:
Rhysida broke into the British Library in [1]October last year , making off with 600GB worth of data and, crucially, destroying many of its servers which are now in the process of being replaced.
The institution says in a new report looking into the incident that many of its systems can't be restored due to their age. They will either no longer work on the fresh infrastructure or they simply can't get any vendor support after going end of life (EOL).
[2]
It also highlights the "historically complex network topology" that ultimately afforded the [3]Rhysida affiliate wider access to, and opportunities to compromise, its network and systems than they would normally expect with more typical corporate targets.
[4]
[5]
Those legacy systems were also reliant on less secure, manual extract, transform, and load (ETL) data processes, rather than an encapsulated, end-to-end workflow as is typical in modern environments. As a result, a greater volume of staff and customer data was in transit on the network.
"There is a clear lesson in ensuring the attack vector is reduced as much as possible by keeping infrastructure and applications current, with increased levels of lifecycle investment in technology infrastructure and security," [6]the report [PDF] states.
[7]
"The Library responded as quickly as it could in the circumstances, and followed the necessary steps to limit the attack, but still suffered very significant damage."
As for why the British Library was running systems so old they can no longer be restored, it says The Legal Deposit Libraries (Non-Print Works) Regulations, introduced in 2013, had a big part to play.
These regs meant the library was required to make a number of key investments, using money taken from core Library funds, into mandatory services such as web archiving, digital preservation systems, and viewing applications. This depleted Library funds that otherwise may have been used to modernize its IT estate.
[8]
The management of yesteryear also has responsbility for the British Library's "unusually diverse and complex technology estate" – one that was formed around "very different collections and organizational cultures brought together by the 1972 British Library Act."
What's the damage?
The disruption caused [9]Rhysida's attack , which resulted in nearly all Library services being pulled offline until the incident was contained, including on-site Wi-Fi access and payment terminals, is still being felt today.
The British Library is proud that it was able to remain open throughout the entirety of the incident, but many of its core services remain disrupted.
Its research services, for example, remain incomplete even after the January return of the Library's online catalog search functionality. It was substantially restricted in the two months immediately after the attack too.
The Library is home to unique pieces and texts and is relied upon by researchers of all kinds for various projects, and also usually offers online access to various resources such as research journals. Electronic access to these is still offline five months after the attack.
Library Reading Rooms, which can be booked out by members to examine works on-site (British Library doesn't allow books to leave the premises) are also available fully but access to its physical collection is reduced by around 50 percent. Content held in its sprawling 44-acre Boston Spa archive vaults is also unavailable still.
Loans of works to other institutions are continuing, but with restrictions, the report adds. And access to collections for staff is limited, and that's having knock-on effects on other Library functions, although it didn't specify what these are.
When in doubt, head to the cloud
The British Library now has a renewed focus on [10]cloud-based technologies and is expected to rely on them more so than ever before, starting in the next 18 months.
The current email, finance, HR, payroll, and physical security systems are currently cloud-based and were largely unaffected by [11]the attack . The Library acknowledges the fact that cloud doesn't solve all its security risks, and introduces new ones in the process, but it believes they will ultimately be easier to manage in the long run.
Speaking of managing all of that, it seems as though the Library may have its work cut out for it. By its own admission, the tech team was "overstretched" prior to the attack and there was no mention of this being rectified between then and now.
Staff shortages were a big concern at the time. Now, there is currently a belief that the team may not be sufficient in size to meet the demands of the rebuild program, and the report alludes to a potential issue with the way the Library [12]pays for its talent .
"The need to grow cybersecurity capacity and cloud engineering capabilities will be particularly acute and will be difficult to remediate without reconsideration of how the Library remunerates high-demand IT skills," it says.
This IT overhaul is being bankrolled with funding that was originally slated for dissemination over the course of seven years between 2023 and 2030, but a significant chunk is set to be brought forward as part of a revised three-year budget.
The six-month period following the attack, which will conclude next month, was designated as a time to implement interim solutions to recover systems as fully as possible. The following 18 months is where all the IT upgrades are expected to take place.
The Library's seven-year funding period was originally devised in 2022 when it lost its Cyber Essentials Plus certification. It originally passed in 2019 but a criteria change saw the Library fall short of the government-backed program's standards.
For those wanting to read the Library's full account of how the attack played out, its report goes into extensive detail about how it all unfolded. Investigators aren't, however, certain about all aspects of the attack since the cybercrims' server destruction was comprehensive enough to wipe away many of its digital tracks.
[13]Korean eggheads crack Rhysida ransomware and release free decryptor tool
[14]British Library: Finances remain healthy as ransomware recovery continues
[15]UK government woefully unprepared for 'catastrophic' ransomware attack
[16]Hershey phishes! Crooks snarf chocolate lovers' creds
The British Library posts, via the report, a list of things it learned from the incident that will inform its future approach to IT and cybersecurity, which will be broadly applicable across the cultural and technical aspects of the organization.
"Investment, boldness, and relentless focus are all needed to ensure that we are as secure as we can be against this threat, as the cost of investing in prevention is outweighed by the risk of failing to prevent," it says.
"Although the security measures we had in place on 28 October 2023 were extensive and had been accredited and stress-tested, with the benefit of hindsight there is much we wish we had understood better or had prioritized differently." ®
Get our [17]Tech Resources
[1] https://www.theregister.com/2023/10/31/british_library_it_outage/
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cybercrime&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Ze84rEHegN1th4caYXY@JwAAAVY&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://www.theregister.com/2023/11/20/rhysida_claims_british_library_ransomware/
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cybercrime&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Ze84rEHegN1th4caYXY@JwAAAVY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cybercrime&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Ze84rEHegN1th4caYXY@JwAAAVY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.bl.uk/home/british-library-cyber-incident-review-8-march-2024.pdf
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cybercrime&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Ze84rEHegN1th4caYXY@JwAAAVY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/cybercrime&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Ze84rEHegN1th4caYXY@JwAAAVY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[9] https://www.theregister.com/2023/11/20/rhysida_claims_british_library_ransomware/
[10] https://www.theregister.com/2023/09/11/cloud_costs_feature/
[11] https://www.theregister.com/2023/10/31/british_library_it_outage/
[12] https://www.theregister.com/2023/05/31/uk_tech_roles_sees_sustained/
[13] https://www.theregister.com/2024/02/13/rhysida_ransomware_decrypted/
[14] https://www.theregister.com/2024/01/08/british_library_finances_remain_healthy/
[15] https://www.theregister.com/2023/12/14/uk_jcnss_ransomware_report/
[16] https://www.theregister.com/2023/12/04/hershey_phishes_data_breach/
[17] https://whitepapers.theregister.com/
Somebody else's computer
So now they're following it up with more stupidity.
There is nothing dumber than putting your critical infrastructure under the control of somebody else. You're now a paypig, exploitable for as much money as your new corporate overlord wants to extract from you.
I'm gonna laugh so hard when one of the 'cloud' providers gets taken out by the same ransomware gangs.
The cloud: a defence against bean counters.
"...many of its systems can't be restored due to their age. They will either no longer work on the fresh infrastructure or they simply can't get any vendor support after going end of life (EOL)."
That, however, won't be allowed to happen in the cloud. They won't be allowed to leave systems rotting ("because, funds...") and then find they can't support them when they need to restore.That problem will have to be dealt with when the vendor or cloud provider pulls support.
Re: Somebody else's computer
" when one of the 'cloud' providers gets taken out "
More likely, the cloud client gets taken out via a wide open browser on its user base (actually, very likely indeed -- remember "somebody clicked on a link"?). The general assumption that the cloud is "more secure" is a gross misunderstanding. The cloud provider's infrastructure may be well secured, but a client's security effectively remains its own responsibility. There may be some support tools to help, but they (and other controls) have to be deployed with understanding to achieve adequate security. The big snag is that once you've sacked your IT staff because you've gone into the cloud, you've got nobody left who can optimise those controls. (Yes, that's an extreme scenario, but it does exemplify a real trend).
So now they're paying two ransoms? One from the ransomware gang and another (legitimate) ransom from a cloud (feudal lord) provider. Surely having ones own server is still cheaper because regardless someone has to maintain the server physical or virtual.
I'll leave this link here.
https://world.hey.com/dhh/the-big-cloud-exit-faq-20274010
>Surely having ones own server is still cheaper because regardless someone has to maintain the server physical or virtual.
That *depends*. A lot.
An organization buying say 100, or 1000+, servers is a big CapEx outlay in one go - even if you can buy them in sufficient quantities - then you have to rack, power, and get the other stuff sorted around backups. Whereas, provisioning the same number of cloud servers is a smaller OpEx outlay and the other services are already sorted. You just provision and pay.
Over time, the OpEx total will probably be higher than the CapEx but its often easier to get approval for a lower operational spend than it is to buy the servers - and much quicker.
You make a good point however you missed some key costs mainly related to staff. You're going to need someone to secure that cloud, properly cost that cloud, determine which cloud is best (instant access/archived), decide on the optimum connection, upgrade all your software to use said cloud, test the software. These costs aren't free and sure you will have existing staff that could potentially do it but you rather risk that or bring in experts for some of them? As it stands there are 3 main cloud players with no incentive in the long term to keep it cheap as once people move to a cloud it's nearly impossible cost wise to move to another especially for a business like this with this amount of data. Even coming out of the cloud can be extortionate.
With all that in mind I would counter that on premises in this case would be the wiser option. I'm not saying you do but I have no idea why some people have this idea that you can just cloud it and it all just magically works and is secure.
But cloud is just pushing that CapEx expense onto someone else, who will charge the cost of money plus some on top of everything else for the customers OpEx subscription price.
Just having this problem with a client, because they failed to purchase new kit the last two plus years they now have a much larger CapEx to pay out, ie. They are looking at having to purchase 400 new servers this year rather than 100…
> So now they're paying two ransoms?
Three, software licensing for cloud verses on-prem licensing…
Legacy is here to stay... Live with it...
"Legacy software" or more generall "Legacy IT" is usually discussed as if it were some anomalous hangover from the bad ol' days, from old, undesirable and badly regarded projects, or from when things "weren't done properly."
Well, that's clearly wrong... Legacy is and always will be a fact of every project. If not now, then sooner or later.
It's unfortunate that Legacy software is most often mentioned in the same statements that tells us about the bright new future ahead, with no more Legacy software to hold us back, to drag down performance and to remove the need for software maintenance.
Let's help inject some realism into these discussions and remind everyone that focussing on the good news and the bright future that lies ahead is no way to run projects that we know will have problems. We know that. That's life...
I hope they don't imagine a problem-free follow-up and then forget any lessons learned... we don't learn from the good news, so relish the bad news...
Re: Legacy is here to stay... Live with it...
This.
A Gartner researcher I once knew cheerfully reminded people that projects currently in delivery are 'legacy' by definition. He followed that up by saying "Another definition of legacy software: the programmer is dead or should be".
Because of 'agendas', Manglement conveniently forget the 'ilities' - usability, security, scalability, availability/reliability, manageability/maintainability, recoverability, and so on. All significant parts of total cost of ownership - ignore at your peril.
Re: Legacy is here to stay... Live with it...
Securility, surely.
The cloud may enable them to simplify their systems, which can help security. Guess what? So does investing in new on prem systems. The Cloud isn't inherently more secure than on prem (on the contrary, it may be less secure). What enables you to secure it is the systems analysis you should do when moving to any new system, on prem or cloud.
First question to ask their new cloud overlord:
When you make backups of our data, are those backups immutable?
Lest they end up losing everything a second time. Yes, TietoEvry, I'm looking at you!
"destroying many of its servers which are now in the process of being replaced."
How does ransomware 'destroy' a server such that it needs to be replaced?
Legacy IT overwhelming factor delaying ransom recover efforts :o
> The British Library says legacy IT is the overwhelming factor delaying efforts to recover from the Rhysida ransomware attack in late 2023.
Wha', has the software atrophied (worn out) from being stored too long in the janitors cupboard.