Consolidating databases has significant storage benefits – and therefore everyone should be doing it
- Reference: 1600106414
- News link: https://www.theregister.co.uk/2020/09/14/storage_consolidation_debate_for_motion_mon/
- Source link:
During the week you can cast your vote using the embedded poll, choosing whether you're in favor or against the motion. The final score will be announced on Friday, revealing whether the for or against argument was most popular. It's up to our writers to convince you to vote for their side.
For this debate, the motion is: [1]Consolidating databases has significant storage benefits, therefore everyone should be doing it.
Earlier today, Chris Mellor [2]argued against the motion. And now, arguing FOR the motion, is DAVE CARTWRIGHT...
Consolidating databases has significant storage benefits, therefore everyone should be doing it
Storage presents us with a major problem: it constantly gets smaller, faster and cheaper.
What’s that? You see smaller, faster and cheaper as a good thing? You’ll be telling me next that you think it’s great that CPUs are getting faster, with the price per CPU cycle heading constantly downhill. (And just to be clear: that’s not great either).
Some of us learned about technology in the days when you had to be mindful of how you used it. When you had to consider the complexity of your algorithm – because if you didn’t write an efficient algorithm, you could be claiming your pension. And you made darned sure that you stored as few copies of your data as you could, because you didn’t have much storage; part of this limitation was the technical limits of the hardware, but most of it was the sheer cost of the stuff.
If you have limitations, you become mindful of those limitations. And if you don’t start with that mindfulness you soon acquire it, because your program run never completes, or runs out of memory, or fills the disk up. Sadly, these days the technology is so fast, cheap and forgiving that you can use it inefficiently and it’ll save your bacon through raw speed and size … most of the time, anyway.
This leads to the problem of not being efficient in the way you use storage. One of the classic reasons is that, if you need to do some development work on a database you’ve not used before, It’s easier to take a copy of a database to work on than to go through access management procedures to get access to the existing copy. Does that copy ever get removed? No, generally not. It does, however, get increasingly out of date and irrelevant – and when you decide that the live version has crossed that watershed of being too different from your copy, it’s time for a new copy … but of course you daren’t remove the old one for fear of losing a table, or a view, or a stored procedure that you “might need one day”.
And if you’re cluttering up your storage with multiple copies of the same thing – each of which could be tens or hundreds of gigabytes – that’s not a patch on the sin of failing to consolidate databases that are inherently different but whose contents overlap. Although some of your databases will have no data in common (the HR database tends to be pretty exclusive, for instance) you’ll often have pockets of the same data in different systems – customer data, product data, pricing data, the list goes on.
Leaving aside the data protection nightmares of understanding what data you have and maintaining its accuracy, you’re also wasting storage. And by doing the “yet another copy” across all those databases (and don’t kid yourself you won’t), you’re using space that you mostly won’t ever claim back.
Consolidating databases makes the data easier to maintain, easier to keep in line with regulatory and data protection requirements, and all that fun stuff. And the exercise can improve performance – not only will you be making sure you index stuff properly and write queries to access fewer data stores. But it also saves you a boatload of storage – which means it also saves you the money you’d otherwise have spent on that extra space.
Oh, and if you’re thinking to yourself: “Hey, my SAN kit de-deduplicates the data before it hits the disk, so I don’t have to worry about consolidating and de-duping it myself” … yes, if you have on-premises SAN. But these days the data’s probably in the cloud, and it’s the service provider that’s making the storage savings, not you.
Consolidate your data. Save yourself space.
Oh, and save yourself money. And make your systems better and more efficient. Why wouldn’t you? ®
Cast your vote below. You can track the progress of the debate [3]right here .
JavaScript Disabled Please Enable JavaScript to use this feature.
Get our [4]Tech Resources
[1] https://www.theregister.com/Debates/2020/09/14/database_storage_consolidation
[2] https://www.theregister.co.uk/2020/09/14/storage_consolidation_debate_against_motion_mon/
[3] https://www.theregister.com/Debates/2020/09/14/database_storage_consolidation
[4] https://whitepapers.theregister.com/
Re: Hack once, read many
My counter argument would be that if you have a single, well-maintained source of data, it is a lot easier to secure than the clusterf*** you can end up with in the scenario described in that argument (lots of "we don't know what it's for or if it's useful" copies of your databases everywhere).
In a hypothetical situation and hear me out on this, having everything in one database makes queries a bitch especially when every person and their dog are doing it. It also leaves you wide open to user errors if you don't set the permissions right which I know can be an issue with multiple databases and yes I have heard of backups but it's just too risky. Divide and conquer I say. There is a reason we don't put our eggs in one basket. Best case is also local duplication for anything that doesn't require real time access. This is just my opinion on the matter.
Err, no.
I deal with a couple of legacy databases, Oracle RDB, and a hierarchical database that originated on UNIVACs.
Neither of those is going to be on the table for consolidation.
Funny story, a developer brought me a SQL-Server backup, and asked for a Linux recovery. I downloaded the RPMs, initialized it, and the developer loaded the backup.
My next surprise was a call from management on an emergency SQL-Server conversion to Linux - turns out my newly-installed database was 100x faster than the production VM farm.
There are a few reasons for consolidation, but there are many reasons to refrain. Packing your favorite bowl or cup in your attic chest of porcelain means that you will constantly dis/reassemble the contents, and things will likely get broken. Certain architectural aspects become brittle and very difficult to change.
Everything in one database?
Why, that would be SAP, wouldn't it? That always works out well.
In seriousness though I'd be picking apart the question rather than trying to answer it. Merging databases into a single one is likely to fall on its arse at the first piece of software you come across that can only interact with its existing DBMS. Normalising individual databases and nominating the "source of truth" for any shared/copied data would likely be far more efficient for getting data integrity sorted. Either of these actions could be considered to be "consolidation".
Returning to my first point though, I've only worked at one place that claimed to run everything from one database (yes, SAP) and I know for a fact that they had at least three (yes, SAP plus other software for all the things SAP couldn't do which was anything and everything to do with the internet retail side) — and likely hundreds more, because those were just the ones I came to know about and I didn't work in an IT capacity there. It's a cute idea, but most places choose the right tool for the job rather than dictating the tool in advance.
Consolidating databases saves you also OS overhead if you use virtualisation and Microsoft licenses as well if that is the OS of choice as well as resources for Antivirus programs as CPU cycles and RAM bytes.
Hack once, read many
The primary benefit (for the bad people) of consolidating all the enterprise databases into one is that it makes the job of a hacker easier. They only have to break in to one database to steal all your confidential data.