Cloudflare Flips AI Scraping Model With Pay-Per-Crawl System For Publishers (cloudflare.com)
- Reference: 0178245910
- News link: https://tech.slashdot.org/story/25/07/01/1745245/cloudflare-flips-ai-scraping-model-with-pay-per-crawl-system-for-publishers
- Source link: https://blog.cloudflare.com/introducing-pay-per-crawl/
When an AI crawler requests paid content, it either presents payment intent via request headers for successful access or receives a "402 Payment Required" response with pricing information. Cloudflare acts as the merchant of record and handles the underlying technical infrastructure. The company aggregates billing events, charges crawlers, and distributes earnings to publishers.
Alongside Pay Per Crawl, Cloudflare has [2]switched to blocking AI crawlers by default for its customers, becoming the first major internet infrastructure provider to require explicit permission for AI access. The company handles traffic for 20% of the web and more than one million customers have already activated its AI-blocking tools since their September 2024 launch, it wrote in a blog post.
[1] https://blog.cloudflare.com/introducing-pay-per-crawl/
[2] https://www.cloudflare.com/ru-ru/press-releases/2025/cloudflare-just-changed-how-ai-crawlers-scrape-the-internet-at-large/
I prefer my method (Score:1)
Someone already developed an AI Tar Pit code for website. When it detects bot traffic or robots.txt violations, you just keep creating more and more dummy pages in an infinite chain and then slow down the response time for the server to like 750ms. It traps the bots there forever without using much bandwidth.
Re: I prefer my method (Score:2)
Why not both though?
Re: (Score:2)
That's what CF was already doing.
So when are the lawsuits coming? (Score:2)
No doubt Meta[stasize], Google, OpenAI and all other major AI shops will whine about having to pay for anything and conjure up some reasoning why this system is illegal because reasons and sue Cloudflare to tie them up in litigation - so my question is: when is that happening?
Re: (Score:2)
I actually think that's the way to go. There needs to be a law about whether AI is allowed to learn from content on the same terms as a living individual, or not. Then there also need to be technical means to enact whatever policy is legislated, which is here this Cloudflare technology could fit in.
Re: (Score:2)
will whine about having to pay for anything
You mean like people on here and elsewhere who brag about stealing music/movies/software because they don't want to pay?
If it's okay for you to steal someone else's work, why is not acceptable for these companies to scrape available content?
Re: (Score:2)
> will whine about having to pay for anything You mean like people on here and elsewhere who brag about stealing music/movies/software because they don't want to pay?
"brag" != "whine"
> If it's okay for you to steal someone else's work
WTF here said that in this discussion. Provide citations.
> why is not acceptable for these companies to scrape available content?
Comparing apples to oranges.
Multi-billion$ AI companies scrape content, then repeatedly sell access to services that use that content at scale without compensation to the creators, without whose content those companies would have nothing to offer in the first place.
Quite different than some individual "stealing" a song for their own use (sure there's some level of deprivation of funding to the creator, but they're not making money of
Interesting but too late... (Score:2)
I'd imagine most of the AI crawlers out there have already ingested most of the freely available content. I don't imagine many places that have new content who haven't already struck up some agreement with the scrapers. For instance Ars Technica has.
Had this existed three years ago, it might've been interesting.
That said, there's something missing. There's two ways a crawler can work. Either they request content, get told a price and have to reconnect and agree to the price OR they can declare in ad
but muh profits! (Score:2)
how is big tech supposed to profit from stealing your content if they have to pay for it?
So AI only crawls spam content for AI from now on (Score:1)
Former it was like: AI might hurt itself when it gets more and more ai generated input
Now its like: AI will only get ai generated content at all
Re: (Score:2)
But not the spam behind cloudflare...
The great internet paywall begins (Score:5, Interesting)
All because you couldn't behave, now we have the equivalent of the TSA for the internet. Expect browsers like Pale Moon, Ladybird and Seamonkey to get put on the wrong side of the wall again. Expect adblocker users to get hit by this soon too, as now they have the technology.
Re: (Score:3)
I did see a lot more Cloudflare prompts lately on sites which had no issue with Pale Moon before.