Cloudflare Says It Blocked 416 Billion AI Scraping Requests In 5 Months
- Reference: 0180299713
- News link: https://tech.slashdot.org/story/25/12/05/1940241/cloudflare-says-it-blocked-416-billion-ai-scraping-requests-in-5-months
- Source link:
> "The business model of the internet has always been to generate content that drive traffic and then sell either things, subscriptions, or ads, [Cloudflare CEO Matthew Prince] [2]told Wired . "What I think people don't realize, though, is that AI is a platform shift. The business model of the internet is about to change dramatically. I don't know what it's going to change to, but it's what I'm spending almost every waking hour thinking about."
>
> While Cloudflare blocks almost all AI crawlers, there's one particular bot it cannot block without affecting its customers' online presence -- Google. The search giant combined its search and AI crawler into one, meaning users who opt out of Google's AI crawler won't be indexed in Google search results. "You can't opt out of one without opting out of both, which is a real challenge -- it's crazy," Prince continued. "It shouldn't be that you can use your monopoly position of yesterday in order to leverage and have a monopoly position in the market of tomorrow."
[1] https://www.tomshardware.com/tech-industry/big-tech/cloudflare-says-it-has-fended-off-416-billion-ai-bot-scrape-requests-in-five-months-ceo-warns-of-dramatic-shift-for-internet-business-model
[2] https://www.wired.com/story/big-interview-event-matthew-prince-cloudflare/
Prediction (Score:2)
Google will get fined in the EU in 2026.
AI scrapers should be illegal (Score:3)
And if the owners of AI programs want data they can request it from the owners of the website and can only slurp up data they were given specific permission to consume
Re: (Score:2)
Agreed. If someone wants a movie or music or software, they can request it from the owners and can only use the item they were specifically given permission to use. Otherwise, they have to pay for it.
Just block Google entirely (Score:2)
Yeah, the internet will shit itself for a while but a lesson needs to be taught.
Is this only Google? (Score:3)
> The search giant combined its search and AI crawler into one, meaning users who opt out of Google's AI crawler won't be indexed in Google search results
I can think of another company which has a search engine and is also one of the "big boys" when it comes to AI. Do Microsoft do their own AI crawling or is it sufficiently separated from Bing's that Cloudflare can tell them apart (so far).
A second approach is that Google's search engine is their original business. How tolerant would their user-base be if Cloudflare blocked their crawler, worldwide. That war would hurt the sites depending on Google for their traffic, but it would also hurt Google as well. The DOJ was already going after the company (not sure if that's still a "thing" under Trump) but their machinations take years anyway, as [1]thegarbz points out [slashdot.org], the EU more likely to get involved and they sometimes move faster.
[1] https://tech.slashdot.org/comments.pl?sid=23861891&cid=65837993
416e9, really? (Score:2)
Were all of those really AI bots?
What about all those times that I went to a web site, got a stupid captcha, said fuck it, and went to a different site?
Re: (Score:2)
One of the browsers I use cannot handle Cloudflare Capchas, I'm not sure if it's down to browser sniffing or some incompatibility, but at that point I either say fuck it or copy the link into another browser.
I'll bet some of those (Score:2)
Aren't scraping requests. I keep getting blocked by cloudflare and it sucks, it happens probably 1 out of 50 sites on multiple machines. They still can't recognize between power users and AI.
Why not create a standard to deliver to AI (Score:1)
I think there could be HUGE benifit, both for AI and Web site owners, if there could be a way of tellling/redirect AI Bots or Search Bots to a data index, incremental even, to crawl. Skipping all the javascript and html bullshit. Sites could deliver RAW data for bots to digest, with pointers to the "human-web" - that is, disallow scraping, but giving the same or part of information to use for search and AI indexing... Like checking the box "I AM a robot", and then get a robot friendly datastream...paid or
fuck this guy (Score:1)
The internet didn't always have a business model. Started out as an academic wonderland. Still the best internet ever created and the newer versions have only gotten worse.
Re: (Score:3)
Oh you sweet summer child. The internet had a business model the second it was accessible outside of ARPANET.
Also no. I'm not sure if you stole your dad's Slashdot account, but no the best internet ever created was not preserved in academic wonderland. The internet was far more useful in the early 2010s than at any time in the past and long after it was commercialised. Is it going downhill? Yep. But does that mean it was better in its infancy? Hell no. It was fucking useless back then. There's far more info
Re: (Score:2)
And 2010 coincides with the ascendency of the smartphone as the primary device most people use to access the internet. It's not the sole reason, but it does play a big part.
Re: (Score:2)
> There's far more information available now than there ever was.
You didn't have gopher?
The Internet was, and still is, more than http/https. But that's only for people who know their way around it. The rest of you are like the fat kid on a boy scout camp-out when the bear (Google) arrives. As long as you keep it fed, we'll be safe.
Re: (Score:2)
I disagree, I am glad that there are ways to make money with or on the internet, and that there are many useful services available, either paid, or paid for by ads. Sure, there's a lot of slop, nefarious data harvesting, or downright fraud, but with that comes a lot of good as well. Well beyond the things at our disposal back when it was still largely a thing of academia.