Cloudflare Launches Content Signals Policy To Fight AI Crawlers and Scrapers
(Wednesday September 24, 2025 @05:30PM (BeauHD)
from the power-to-the-publishers dept.)
- Reference: 0179520938
- News link: https://tech.slashdot.org/story/25/09/24/1953230/cloudflare-launches-content-signals-policy-to-fight-ai-crawlers-and-scrapers
- Source link:
[1]BrianFagioli shares a report from NERDS.xyz:
> Cloudflare has [2]unveiled the [3]Content Signals Policy , a free addition to its managed robots.txt service that aims to give website owners and publishers more control over how their content is accessed and reused by AI companies. The idea is pretty simple: robots.txt already lets site operators specify which crawlers can enter and where. Cloudflare's new policy [4]adds a layer that signals how the data may be used once accessed , with plain-language terms for search, AI input, and AI training. "Yes" means allowed, "no" means not allowed, and no signal means no preference.
>
> Matthew Prince, Cloudflare's co-founder and CEO, said: "The Internet cannot wait for a solution, while in the meantime, creators' original content is used for profit by other companies. To ensure the web remains open and thriving, we're giving website owners a better way to express how companies are allowed to use their content." Cloudflare says more than 3.8 million domains already use its robots.txt tools to signal they don't want their content used for AI training. Now, the Content Signals Policy makes those preferences clearer and potentially enforceable.
Further reading: [5]Cloudflare Flips AI Scraping Model With Pay-Per-Crawl System For Publishers
[1] https://slashdot.org/~BrianFagioli
[2] https://blog.cloudflare.com/content-signals-policy/
[3] https://contentsignals.org/
[4] https://nerds.xyz/2025/09/cloudflare-content-signals-policy-ai-crawlers/
[5] https://tech.slashdot.org/story/25/07/01/1745245/cloudflare-flips-ai-scraping-model-with-pay-per-crawl-system-for-publishers
> Cloudflare has [2]unveiled the [3]Content Signals Policy , a free addition to its managed robots.txt service that aims to give website owners and publishers more control over how their content is accessed and reused by AI companies. The idea is pretty simple: robots.txt already lets site operators specify which crawlers can enter and where. Cloudflare's new policy [4]adds a layer that signals how the data may be used once accessed , with plain-language terms for search, AI input, and AI training. "Yes" means allowed, "no" means not allowed, and no signal means no preference.
>
> Matthew Prince, Cloudflare's co-founder and CEO, said: "The Internet cannot wait for a solution, while in the meantime, creators' original content is used for profit by other companies. To ensure the web remains open and thriving, we're giving website owners a better way to express how companies are allowed to use their content." Cloudflare says more than 3.8 million domains already use its robots.txt tools to signal they don't want their content used for AI training. Now, the Content Signals Policy makes those preferences clearer and potentially enforceable.
Further reading: [5]Cloudflare Flips AI Scraping Model With Pay-Per-Crawl System For Publishers
[1] https://slashdot.org/~BrianFagioli
[2] https://blog.cloudflare.com/content-signals-policy/
[3] https://contentsignals.org/
[4] https://nerds.xyz/2025/09/cloudflare-content-signals-policy-ai-crawlers/
[5] https://tech.slashdot.org/story/25/07/01/1745245/cloudflare-flips-ai-scraping-model-with-pay-per-crawl-system-for-publishers