News: 1718965567

  ARM Give a man a fire and he's warm for a day, but set fire to him and he's warm for the rest of his life (Terry Pratchett, Jingo)

Guess how much stored data is ever used or accessed

(2024/06/21)


Interview NetApp's Chief Technology Evangelist, Matt Watts, is worried about sustainability and data wastage, even as his employer withdraws third-party support from BlueXP classification.

In a 2023 report, just before the IT world became obsessed with AI, Watts wrote a foreword to a [1]report [PDF] that made clear just how bad the data wastage situation was getting. That report noted that 41 percent of data currently stored by UK organizations was unused and unwanted.

But "40 percent is low, very low," Watts tells The Reg . "What we have seen coming back is that it can be between 70 to 80 percent in some cases."

[2]

That's a lot of redundant data sitting on powered servers.

[3]

[4]

Until recently, NetApp's BlueXP classification tool was heterogeneous. "It doesn't just look at NetApp storage," says Watts, "it looks across any storage. It even looks into the cloud providers, S3 buckets, all those sorts of things.

"It then gives us back an entire suite of metadata to tell us who owns it, what the file is, permissions, and it also tells us the last access times as well."

[5]

Watts describes it as "a deeper dive into actually pulling out what the reality of the situation is, not the gut feel, but what actually does it look like."

Watts estimates that 15 to 20 percent of power consumed by datacenters is storage, with fluctuations depending on what the hard drives are doing. [6]According to the UK's National Grid , approximately 2.5 percent of the UK's electricity consumption is accounted for by the country's 400 to 600 commercial datacenters. That figure is expected to rise to around six percent by 2030.

"The biggest challenge with managing data more effectively," says Watts, "is ownership. And we have had this problem since I came into the IT industry 30-plus years ago, which is who actually owns the data.

[7]

"You can give people as much knowledge as you possibly can about what the data is. But you go out into an organization and say to a group, 'Well, it's your data because you created it,' and they'll say 'No no no no – it's IT's. They own the data.'

"And then you'll go and see IT … and IT will say 'no no no no – we're just the custodians of the data.'"

[8]Now NetApp sued by its own veep over claims of broken sales commission promises

[9]NetApp ditches 8% of staff as customers put away wallets

[10]RIP: NetApp Advanced Technology Group shuffles off this mortal coil

[11]NetApp reorg: Plans to 'scale public cloud', grow storage systems biz as company sheds hundreds of staffers

It's a challenge that many administrators will recognize, and one that makes NetApp's decision to make BlueXP classification (formerly Cloud Data Sense) focused on NetApp storage systems all the more vexing for data estates not running on the company's technology.

"Initially the product was heterogeneous," explains Watts. "So we added in the capability for it to support S3, it'll scan any NFS mount point, SMB, all of those sorts of things. And it was a chargeable piece of software.

"What we saw was a lot of partners were looking to try and build services around it … So we took a decision, as part of the May launch, that we would use it as a way of better differentiating ourselves."

That differentiation has meant [12]yanking support for the likes of Google Cloud Storage, Amazon S3, OneDrive, and so on. BlueXP classification is now available as a core capability within BlueXP at no extra charge but without many legacy features.

Considering the increasing concern around data wastage and dark data squatting in datacenters, would the heterogeneous functionality make a comeback? Watts is noncommittal: "I don't think we would rule anything out. We are going to continue to allow our partners to request to be able to use it in a heterogeneous nature.

"But going forward, the initial plan is no cost and included for all customers … longer term, we could change that. But right now, we think that that creates maximum value for people who are thinking of using NetApp or currently use NetApp." ®

Get our [13]Tech Resources



[1] https://www.netapp.com/pdf.html?item=/media/83312-data_waste_research_report_final_for_submission.pdf

[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZnWjln2I2zNRPMmDxkQk9wAAAMg&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0

[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZnWjln2I2zNRPMmDxkQk9wAAAMg&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZnWjln2I2zNRPMmDxkQk9wAAAMg&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZnWjln2I2zNRPMmDxkQk9wAAAMg&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[6] https://www.nationalgrideso.com/document/246446/download

[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/storage&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZnWjln2I2zNRPMmDxkQk9wAAAMg&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[8] https://www.theregister.com/2023/08/23/netapp_accused_of_denying_commissions/

[9] https://www.theregister.com/2023/01/31/netapp_layoffs_8_percent/

[10] https://www.theregister.com/2020/08/28/rip_netapp_advanced_technology_group/

[11] https://www.theregister.com/2020/08/27/netapp/

[12] https://docs.netapp.com/us-en/bluexp-classification/reference-free-paid.html#supported-data-sources

[13] https://whitepapers.theregister.com/



So, up to 80% of stored data is unused

Pascal Monett

Not a problem for the NSA. It loves trawling through all that unused data to try and find something under the paranoia du jour .

Unused data is not useless data, and it'll be a cold day in Hell before your average manager authorizes the complete and irrevocable distruction of data he hasn't looked at since at least a decade.

Just in case, you understand.

Re: So, up to 80% of stored data is unused

Snake

It's true, probably over 80% of the data we have stored here is 'unused'. But they serve as historical data backup, yes probably unused for the rest of time but, in business, you never really know when some crazy customer from 20 years ago asks for something and you have the data to reference it all...

(and it is actually not that crazy, it has happened twice to me in the past 7 months. So who really knows??)

Re: So, up to 80% of stored data is unused

Doctor Syntax

There are also such things as legal requirements to hold data for 6 or 7 years or whatever. The Horizon & Covid enquiries in the UK are showing the significance of stored records on one hand (especially for those witnesses who were able to produce a paper trail) and the significance of having deleted Whatsapp messages on the other.

Re: So, up to 80% of stored data is unused

tiggity

Legal requirements definitely a big thing.

Some customers just love to hoard data though.

We provide tools (with lots of flexibility) so customers can purge data that is no longer required for regulatory / legal reasons - those tools generally sit around metaphorically gathering dust.

Which 80% is going to be unused ?

alain williams

It is probably cheaper for organisations to keep more data than to work out what 20% they are going to need again. Even without understanding that requirements may change in the future.

It is like backups: I would love to only have to do a backup just before I have some sort of failure.

Backups! Unused data should be 90% plus

Chris Evans

I follow the rule that there should be three copies of current data, so 66% should be backups and then when you take into account historical copies I'd expect over 90% was unused. I'm sure many organisations have a lot of 'Unwanted' old data and many won't have a system in place to cull old data.

Re: Backups! Unused data should be 90% plus

ravenviz

I’ve not been convinced about cloud storage model for personal media so far, but this is what has been sold: get your stuff when you want it. Turns out this could be a problem. Most of the stuff that is stored online is crap anyway, I know most of my own local (and doubly backed up) archive is, but that’s my problem to solve!

jokerscrowbar

Not just the data. You never know when version 5.0 of a program might actually be the optimum version because v.31.5 is a subscription only, All your data belong to us POS.

"What surprises you most about mankind?"

God answered:

"That they get bored of being children, are in a rush to grow up, and then long
to be children again. That they lose their health to make money and then lose
their money to restore their health. That by thinking anxiously about the
future, they forget the present, such that they live neither for the present
nor the future. That they live as if they will never die, and they die as if
they had never lived."
-- Jim Brown, published by Reata Strickland as
An Interview with God ISBN 0743229576. (Variation of it was misattributed
to https://en.wikiquote.org/wiki/Tenzin_Gyatso,_14th_Dalai_Lama )