AI Hard Drive Shortage Makes Archiving the Internet Harder (404media.co)
- Reference: 0183161948
- News link: https://hardware.slashdot.org/story/26/05/08/1619212/ai-hard-drive-shortage-makes-archiving-the-internet-harder
- Source link: https://www.404media.co/the-ai-hard-drive-shortage-is-making-it-more-expensive-and-harder-to-archive-the-internet/
> Skyrocketing hard drive and storage costs caused by the AI data center boom are making it more expensive and more difficult for digital archivists, academics, Wikipedia, and hobby data hoarders to save data and archive the internet. Specific drives favored by some high profile organizations like the Internet Archive have [1]become far more expensive or are difficult to find at all , archivists said. Over the last several months, prices for both consumer level and enterprise solid state drives, hard drives, and other types of storage have skyrocketed. As an example, a 2TB external Samsung SSD I purchased last fall for $159 now costs $575. PC Part Picker, a website that tracks the average price of different types of drives, shows a universal increase in storage prices starting in about October of last year. Prices of many of the drives it tracks have doubled or increased by more than 150 percent, and at some stores SSDs and hard drives are [2]simply sold out . There is now even a secondary market for some SSDs, with people scalping them on eBay and elsewhere.
>
> Brewster Kahle, founder of the Internet Archive and the Wayback Machine, the most important archiving projects in the history of the internet, told 404 Media that the skyrocketing costs of storage is "a very real issue costing us time and money." "We have found that the preferred 28-30TB drives are just not available or at very high price," Kahle said. "We gather over 100 terabytes of new materials each day, and we have over 210 Petabytes of materials already archived on machines that need continuous upgrades and maintenance, so we need to constantly get new hard drives." "We are fortunate to have an active community that donates to the Archive, and we are also looking for help from hard drive manufacturers in these difficult times. We are always looking for more help," he added. "So far we have ways to work around these shortages, but it is a very real issue causing us time and money."
>
> The Wikimedia Foundation, which runs Wikipedia and various other projects, including Wikimedia Commons, an open repository of royalty free media, told 404 Media that the cost of storage has become a concern for the foundation's projects as well. "With over 65 million articles on Wikipedia alone, access to server and storage capacity is vital to us. We've certainly seen price increases since the end of 2025. These price increases are of concern to us, as with every other player in the industry. We see the primary impact in the purchase of memory and hard drives but also in terms of lead times on server deliveries and our capacity to place future orders," a Wikimedia Foundation spokesperson told us. "The Wikimedia Foundation is a non-profit, and as such how we allocate budget is very carefully considered. We maintain our own data centers to serve our users from all over the world. We're putting workarounds in place where we can, mainly involving being smart with how we prioritize investment in hardware, building in flexibility as well as extending the life of existing hardware where possible."
>
> Western Digital, one of the largest manufacturers of hard drives and other storage systems, said that it has [3]essentially sold out of its 2026 inventory to enterprise clients, many of which run data centers. Micron, which made RAM and SSDs under the brand name Crucial, has [4]exited the consumer market altogether because "AI-driven growth in the data center has led to a surge in demand for memory and storage. Micron has made the difficult decision to exit the Crucial consumer business in order to improve supply and support for our larger, strategic customers in faster-growing segments."
[1] https://www.404media.co/the-ai-hard-drive-shortage-is-making-it-more-expensive-and-harder-to-archive-the-internet/
[2] https://www.reddit.com/r/DataHoarder/comments/1sp196m/hard_drive_shelf_at_micro_center/?ref=404media.co
[3] https://hardware.slashdot.org/story/26/02/16/188251/western-digital-is-sold-out-of-hard-drives-for-2026
[4] https://hardware.slashdot.org/story/25/12/03/2118251/after-nearly-30-years-crucial-will-stop-selling-ram-to-consumers
We must really struggle with storing data (Score:2)
Because I'm sure this can't be the first time I see this headline.
I hope they triage what they archive (Score:1)
And they do not use resources to save AI slop. But again, who am I to say what it will be of interest in the future.
Re: Sure it's not the other way round? (Score:2)
It would be useful if they were required to make any training data they didn't pay for publicly accessible...
But they aren't
Re: (Score:1)
> It would be useful if they were required to make any training data they didn't pay for publicly accessible...
Isn't such data already publicly available? That's how the AI companies got it?
Re: Sure it's not the other way round? (Score:2)
It is when they get it, but I'm not sure you understand the purpose of an archive
Re: (Score:1)
> It is when they get it, but I'm not sure you understand the purpose of an archive
It seems you do not understand the words you wrote: "make any training data they didn't pay for publicly accessible".
Did you perhaps really mean that AI companies should somehow participate in Internet archiving, which again, is something a little more involved that just making files publicly accessible.
Re: (Score:3)
No, I think he understood the words he wrote. AI companies do NOT make the data publicly available, they insert a slop-generator between the data and the user that means the original data cannot be retrieved.
You think you're being clever, but you're both pretending to misunderstand what's been said, and saying something very stupid and false. You are why Slashdot sucks sometimes. Knock it off.
Re: (Score:1)
I'm pretending nothing. asking for a redundant copy is not the same as asking for an archive. The latter is more complex than the latter. Ask for the latter is insufficient.
You may be correct the original author meant an archive, but that is not what he wrote. I simply pointed that out in a non confrontation manner.
Re: (Score:1)
> ... asking for a redundant copy is not the same as asking for an archive The latter is more complex than the latter. Ask for the latter is insufficient.
Typo. Should be "The latter is more complex than the former. Ask for the former is insufficient."
Blow the dust off the rust (Score:2)
I have about 20 1TB WD black spinning rust hard drives sitting in a box. I stopped using them years ago when SSD prices finally became reasonable. I can't believe I may have to resort to these again if any of my 1TB or larger SSDs fail in the near future. Ugh. The IT world is a fucking mess.
Re: (Score:2)
Heh. I too have a big stack of spinning drives. I really don't think GPU, RAM or SSD prices are going down soon - even if the AI companies fail there will be a backlog of consumers wanting them. Maybe the AI companies will sell them all for cheap leading to a massive glut, but I suspect data centers will be kept intact instead and a big push will come to do all your computing in the cloud rather than buy your own hardware.
Fuck that and fuck the people pushing that.
Re: (Score:2)
I installed a 28TB hard drive in my gaming PC to back up to. Man is that thing loud when you're used to a quiet PC with SSDs.
I checked prices a few weeks ago and discovered I couldn't even afford to replace that PC if it broke. It would cost as much as a small car.
All so the Internet can create funny cat videos.
This whole AI thing is ridiculous (Score:2)
IMO they are pricing in AGI, if they don't get it or if they aren't predicting inference computing costs correctly, there could be a huge rollback. Then we'll have an oversupply of components instead of a shortage. The amount of spend is ludicrous and unrealistic for future needs
Re:This whole AI thing is ridiculous (Score:5, Insightful)
"Then we'll have an oversupply of components instead of a shortage."
No, there isn't a free market. Corporate interests want ALL of supply so you don't get any. That changes only when corporations say it changes, and specifically when VC funding dictates those changes. You assume brainpower not demonstrated to exist, you compete against privilege and you are guaranteed to lose.
"The amount of spend is ludicrous and unrealistic for future needs"
Because billionaires compete to own the future without knowledge or understanding of what the future is. They're trying to buy up all supply on speculation. Musk rents out capacity he bought on speculation, what matters is that he owns the capacity not that he uses it.
One breakthrough on AI computing requirements could make everything known today obsolete, but we can take for granted that billionaires will know that before anyone else. It's a rigged game. Billionaires will own ALL computing power yet demand that the public pay for all the electrical infrastructure necessary to power it. We share the costs and they make the profits. The Sam Walton way.
Re: (Score:2)
> No, there isn't a free market. Corporate interests want ALL of supply so you don't get any.
That is still a free market, the problem is you're seeing it through a lens of someone who hasn't got enough money to play in that market. Their dollar is worth as much as yours, the only problem is they have more of said dollars.
Re: (Score:2)
If the big company cancels the order, the next smaller one is already waiting for their supply.
Re: (Score:2)
When the AI bubble bursts, the government will run a "Cash for Clankers" program where they buy all the old AI hardware and crush it so it can't wreck the market for new parts.
Re: (Score:2)
You would think that they would be preemptive and control the risks, seeing as this could create a problem for them in the future. I guess they'll pass out AI stimulus recession checks when crap hits the fan
Re: (Score:2)
> IMO they are pricing in AGI, if they don't get it or if they aren't predicting inference computing costs correctly, there could be a huge rollback. Then we'll have an oversupply of components instead of a shortage. The amount of spend is ludicrous and unrealistic for future needs
We are in an economic mania right now. Governments, corporations, startups, you name it, are all afraid of being left behind. They are buying up memory, disks, computing capacity because, well, if they don't, someone else--one of their competitors--will.
Supply will be expanded and built out while demand remains high.
How long will this take? That's the trillion dollar question. It could be months or it could be years, but at some point, demand and supply will come back into closer to equilibrium. Whether tha
Re: (Score:2)
They are not going to get AGI. We are _very_ far removed from that and we reliably know LLMs cannot do it. The LLM industry will collapse catastrophically, there is no other way this can go. The only question is when.
Tape drives (Score:1)
This is not a new problem.
Our industry has a long history of moving data to slower storage to free up faster storage.
So the Internet Archive should simply move less-used stuff to tape drives, freeing up hard drives for popular stuff.
50TB:
https://newsroom.ibm.com/2023-08-29-Fujifilm-and-IBM-Develop-50TB-Native-Tape-Storage-System,-Featuring-Worlds-Highest-Data-Storage-Tape-Capacity-1
Re: (Score:2)
That is not even the problem. Scarcity is artificial and could disappear tomorrow, that's the problem. The industry aligns with the interests of very few extremely rich people, tape drives don't solve that.
Re: (Score:2)
Also those tape drives appear to cost as much as a car. So not really good solution for regular users.
Re: (Score:3)
How is that supposed to work? I click on a link to an archived site and then wait for the robot in the tape library to grab a particular tape and dump the contents?
Re: (Score:2)
I remember years ago reading a post on some Microsoft techie's blog where he answered a question about why Windows did something in a weird way and it was apparently because it would break otherwise if you tried to do that thing on a system where the file you were trying to access was on actually a tape drive and had to be retrieved very slowly.
Re: (Score:1)
In our age of entitlement when people are used to instant results, I can see this might be hard for some, but do you really need that book in 5 seconds or would 5 minutes do ?
In practice, I've had bad tape experiences (Score:1)
> So the Internet Archive should simply move less-used stuff to tape drives, freeing up hard drives for popular stuff.
That would be counterproductive. Have you ever worked at a company that did that? I have. It seemed that about 50% of requests for some old files that were moved to tape result in "files not retrievable".
I know tapes are supposed to be ultra reliable. Back in the day I used QIC-80s on my PC, a consumer grade tape system. They worked great. 20 years after I made those tapes I was about to through out the last PC I had that I could plug in the old tape drive. The tapes had been store in my garage which get
Re: (Score:2)
If keeping drives in your garage performed better than a "well regarded company that specialized in that", I would stop using that company.
I won't forget (Score:5, Interesting)
Dear AI companies, I won't forget how you fucked our entire industry for a long time. I won't forget how you forced scarcity and high prices. I won't forget that I need to double-check everything that your half-assed solution pukes out. I won't forget that you set everything on fire so you could try to profit.
Re: (Score:1)
I'm sure that Sam Altman is cowering in fear knowing that he just offended some random guy on Slashdot!
Re: (Score:2)
You reap what you sow.
Re: (Score:2)
You would think, but ... in the late 19-th and early 20-th Centuries coal companies sowed rape and pillage throughout Lackawanna and Luzerne Valleys in Pa. What did they reap? A couple governorships a couple Senators & judges by the score ... rivers so polluted even catfish can't live and land so decimated it burns night and day. Be damned careful what you sow.
Re: (Score:2)
Totally get it, but assuming the bubble bursts and most of the LLM companies just end being sold for pennies to Google and Microsoft, or go bust, what can you possibly do about it?
It's not even as if you can boycott them NOW while they're functioning entities. You can decide you want to, but then Google ensures you can't do a search without AI, your boss refuses to let you code Java or PHP without AI, you're basically fucked.
It's Big Tech we need to rally against. More self hosting. More ad blocking. etc. S
Re: (Score:2)
And? It's one thing to not forget, but what are you actually goanna do about it? What have you done about it right now? AMD, Intel, NVIDIA, SK Hynix, Samsung, Google, Amazon, everyone is all in on it.
You want to impress us? Pledge never to buy another PC. Just remembering shit is meaningless.
Re: (Score:2)
It might be crazy to think that I make technology purchase decisions, not just for myself but for more than one company.
You're right, it might be hard to avoid AMD, Intel, SK Hynix and some other hardware vendors. But I can absolutely choose which software solutions are going to be purchased.
Stockpiling (Score:3)
My home storage setup is currently is two 8 20TB drive arrays - one live, one a remote backup.
I was buying drives to add another stripe when the pricing started to ramp up - I try to buy them over time to get different drives from different lots. Now I wish I'd just bought a bunch.
This time last year they were $369, sometimes cheaper. The most recent one I bought was $500. The cheapest I see them right now is $769.
I think I'll be waiting on that new stripe, but at least I have four spares to keep the existing system running.
Our archive is also struggling (Score:2)
I've spent most of the past decade working (for free) on an archiving project for a nonprofit organization. This is a labor of love for me: it's a chance to use a lifetime of technical skills to help preserve the past for the future. I've put in every spare minute that I can, and have given up most other things in my life to do so. I have to: there isn't anyone else with the requisite skill set to do this work for free, and the organization certainly can't afford to pay anybody.
The AI companies have c
Tape is not dead (Score:2)
Buy a LTO library
Re: (Score:2)
Kind of difficult to use as the storage for a website like the Internet Archive.
AI gate keepers (Score:2)
2025 is the year that personal computing is no longer personal.
You are not going to be permitted to handle your own affairs offline. Everything as a service because AI will be in everything and the equipment must be in the cloud. Average consumers and gamers cannot compete with trillions in hype investment. All you can do is wait for the bubble to pop, and hope your retirement or national economy is not popped with it.
AI made "archiving the Internet" silly (Score:2)
It's harder to archive the Internet.
So what? Nothing of value will be lost.
The clamied price is nonsense (Score:2)
A current 2TB Samsung SSD is more around $300.
The extreme observed price for a specific old model comes from old hardware sometimes being more expensive because some people need that specific model as a replacement part. Looks like somebody did not do their homework.
slop (Score:2)
No dispute ... *.ai generates strings one-billion times faster than humans. And AI grows AI so it's exponential . But archived slop is still slop. In deep time, when acetic acid emitting microbes grow over the SSD / HD we still can say "no information was harmed during this process". So we wasted money and time on what ...?
SSDs as investment (Score:4, Interesting)
> a 2TB external Samsung SSD I purchased last fall for $159 now costs $575
I can confirm - I've got a brand new Samsung 870 EVO 2TB drive sitting on my desk that I bought in 10-2022 for $176, and have never got around to installing.
Same drive is now $676 on Amazon!
Re: (Score:2)
It's $800 at Best Buy and over $1000 directly from Samsung.
Re: (Score:2)
I remember when I could buy 8TB SSDs for $499 (Canadian). It wasn't that long ago.
Now an 8TB hard drive costs almost that much.
Re: (Score:2)
Yep. And that is because old, out of manufacture hardware sometimes gets expensive. A current Samsung 2TB is more like $300.
What a fail on your part.