Cloudflare debuts one-click nuke of web-scraping AI
(2024/07/03)
- Reference: 1720035852
- News link: https://www.theregister.co.uk/2024/07/03/cloudflare_ai_blocks/
- Source link:
Cloudflare on Wednesday offered web hosting customers a way to block AI bots from scraping website content and using the data without permission to train machine learning models.
It did so based on customer loathing of AI bots and, "to help preserve a safe internet for content creators," it said in a [1]statement .
"We hear clearly that customers don’t want AI bots visiting their websites, and especially those that do so dishonestly. To help, we’ve added a brand new one-click to block all AI bots."
[2]
There's already a somewhat effective method to block bots that's widely available to website owners, the [3]robots.txt file . When placed in a website's root directory, automated web crawlers are expected to notice and comply with directives in the file that tell them to stay out.
[4]
[5]
Given the [6]widespread [7]belief [8]that [9]generative AI [10]is [11]based on theft , and the many lawsuits attempting to hold AI companies accountable, firms trafficking in laundered content have graciously allowed web publishers to opt-out of the pilfering.
Last August, OpenAI published guidance about [12]how to block its GPTbot crawler using a robots.txt directive, presumably aware of concern about having content scraped and used for AI training without consent. Google took [13]similar steps the following month. Also in September last year Cloudflare began offering a way [14]to block rule-respecting AI bots, and 85 percent of customers - it's claimed - enabled this block.
[15]BT bets big on AI with ServiceNow to cut legacy baggage
[16]Figma pulls AI design tool for seemingly plagiarizing Apple's Weather app
[17]EFF wants FTC to treat lying chatbots as 'unfair and deceptive' in eyes of the law
[18]YouTube confirms it'll pull AI fakes in 48 hours if a complaint's upheld
Now the network services biz aims to provide a more robust barrier to bot entry. The internet is "now flooded with these AI bots," it said, which visit about 39 percent of the top one million web properties served by Cloudflare.
The problem is that robots.txt, like the [19]Do Not Track header implemented in browsers fifteen years ago to declare a preference for privacy, can be ignored, generally without consequences.
[20]
And recent reports suggest AI bots do just that. Amazon last week [21]said it was looking into [22]evidence that bots working on behalf of AI search outfit Perplexity, an AWS client, had crawled websites, including news sites, and reproduced their content without suitable credit or permission.
Amazon cloud customers are supposed to obey robots.txt, and Perplexity was accused of not doing that. Aravind Srinivas, CEO of the AI upstart, denied his biz was underhandedly ignoring the file, though conceded third-party bots used by Perplexity were the ones observed scraping pages against the wishes of webmasters.
Spoofed
"Sadly, we’ve observed bot operators attempt to appear as though they are a real browser by using a spoofed user agent," Cloudflare said. "We’ve monitored this activity over time, and we’re proud to say that our global machine learning model has always recognized this activity as a bot, even when operators lie about their user agent."
Cloudflare said its machine-learning scoring system rated the disguised Perplexity bot below 30 consistently over a period from June 14 through June 27, indicating that it's " [23]likely automated ."
This bot detection approach relies on digital fingerprinting, a technique commonly used to track people online and deny privacy. Crawlers, like individual internet users, often stand out from the crowd based on technical details that can be read through network interactions.
[24]
These bot tend to use the same tools and frameworks for automating website visits. And with a network that sees an average of 57 million requests per second, Cloudflare has ample data to determine which of these fingerprints can be trusted.
So this is what it's come to: machine learning models defending against bots foraging to feed AI models, available even for free tier customers. All customers have to do is click the Block AI Scrapers and Crawlers toggle button in the Security -> Bots menu for a given website.
"We fear that some AI companies intent on circumventing rules to access content will persistently adapt to evade bot detection," Cloudflare said. "We will continue to keep watch and add more bot blocks to our AI Scrapers and Crawlers rule and evolve our machine learning models to help keep the Internet a place where content creators can thrive and keep full control over which models their content is used to train or run inference on." ®
Get our [25]Tech Resources
[1] https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZoXKAYTlyQ@Dkl@jj4U53QAAAIY&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://developers.google.com/search/docs/crawling-indexing/robots/create-robots-txt
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZoXKAYTlyQ@Dkl@jj4U53QAAAIY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZoXKAYTlyQ@Dkl@jj4U53QAAAIY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.newyorker.com/culture/infinite-scroll/is-ai-art-stealing-from-artists
[7] https://juliabausenhardt.com/how-ai-is-stealing-your-art/
[8] https://www.commentary.org/articles/james-meigs/ai-openai-chatgpt-copyright-theft/
[9] https://arxiv.org/abs/2401.06178
[10] https://www.washingtonpost.com/opinions/2023/10/19/ai-large-language-writers-stealing/
[11] https://www.economist.com/business/2024/04/14/generative-ai-is-a-marvel-is-it-also-built-on-theft
[12] https://www.theregister.com/2023/08/08/openai_scraping_software/
[13] https://blog.google/technology/ai/an-update-on-web-publisher-controls/
[14] https://blog.cloudflare.com/ai-bots
[15] https://www.theregister.com/2024/07/03/bt_servicenow_savings/
[16] https://www.theregister.com/2024/07/03/figma_plagiarizes_apple/
[17] https://www.theregister.com/2024/07/02/eff_ftc_chatbot/
[18] https://www.theregister.com/2024/07/02/youtube_deepfake_privacy_rules/
[19] https://en.wikipedia.org/w/index.php?title=Do_Not_Track&oldid=1231982930
[20] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZoXKAYTlyQ@Dkl@jj4U53QAAAIY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[21] https://www.msn.com/en-us/news/technology/amazon-probes-perplexity-ai-for-alleged-content-scraping/ar-BB1peoMo?ocid=BingNewsVerp
[22] https://rknight.me/blog/perplexity-ai-is-lying-about-its-user-agent/
[23] https://developers.cloudflare.com/bots/concepts/bot-score/
[24] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZoXKAYTlyQ@Dkl@jj4U53QAAAIY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[25] https://whitepapers.theregister.com/
It did so based on customer loathing of AI bots and, "to help preserve a safe internet for content creators," it said in a [1]statement .
"We hear clearly that customers don’t want AI bots visiting their websites, and especially those that do so dishonestly. To help, we’ve added a brand new one-click to block all AI bots."
[2]
There's already a somewhat effective method to block bots that's widely available to website owners, the [3]robots.txt file . When placed in a website's root directory, automated web crawlers are expected to notice and comply with directives in the file that tell them to stay out.
[4]
[5]
Given the [6]widespread [7]belief [8]that [9]generative AI [10]is [11]based on theft , and the many lawsuits attempting to hold AI companies accountable, firms trafficking in laundered content have graciously allowed web publishers to opt-out of the pilfering.
Last August, OpenAI published guidance about [12]how to block its GPTbot crawler using a robots.txt directive, presumably aware of concern about having content scraped and used for AI training without consent. Google took [13]similar steps the following month. Also in September last year Cloudflare began offering a way [14]to block rule-respecting AI bots, and 85 percent of customers - it's claimed - enabled this block.
[15]BT bets big on AI with ServiceNow to cut legacy baggage
[16]Figma pulls AI design tool for seemingly plagiarizing Apple's Weather app
[17]EFF wants FTC to treat lying chatbots as 'unfair and deceptive' in eyes of the law
[18]YouTube confirms it'll pull AI fakes in 48 hours if a complaint's upheld
Now the network services biz aims to provide a more robust barrier to bot entry. The internet is "now flooded with these AI bots," it said, which visit about 39 percent of the top one million web properties served by Cloudflare.
The problem is that robots.txt, like the [19]Do Not Track header implemented in browsers fifteen years ago to declare a preference for privacy, can be ignored, generally without consequences.
[20]
And recent reports suggest AI bots do just that. Amazon last week [21]said it was looking into [22]evidence that bots working on behalf of AI search outfit Perplexity, an AWS client, had crawled websites, including news sites, and reproduced their content without suitable credit or permission.
Amazon cloud customers are supposed to obey robots.txt, and Perplexity was accused of not doing that. Aravind Srinivas, CEO of the AI upstart, denied his biz was underhandedly ignoring the file, though conceded third-party bots used by Perplexity were the ones observed scraping pages against the wishes of webmasters.
Spoofed
"Sadly, we’ve observed bot operators attempt to appear as though they are a real browser by using a spoofed user agent," Cloudflare said. "We’ve monitored this activity over time, and we’re proud to say that our global machine learning model has always recognized this activity as a bot, even when operators lie about their user agent."
Cloudflare said its machine-learning scoring system rated the disguised Perplexity bot below 30 consistently over a period from June 14 through June 27, indicating that it's " [23]likely automated ."
This bot detection approach relies on digital fingerprinting, a technique commonly used to track people online and deny privacy. Crawlers, like individual internet users, often stand out from the crowd based on technical details that can be read through network interactions.
[24]
These bot tend to use the same tools and frameworks for automating website visits. And with a network that sees an average of 57 million requests per second, Cloudflare has ample data to determine which of these fingerprints can be trusted.
So this is what it's come to: machine learning models defending against bots foraging to feed AI models, available even for free tier customers. All customers have to do is click the Block AI Scrapers and Crawlers toggle button in the Security -> Bots menu for a given website.
"We fear that some AI companies intent on circumventing rules to access content will persistently adapt to evade bot detection," Cloudflare said. "We will continue to keep watch and add more bot blocks to our AI Scrapers and Crawlers rule and evolve our machine learning models to help keep the Internet a place where content creators can thrive and keep full control over which models their content is used to train or run inference on." ®
Get our [25]Tech Resources
[1] https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZoXKAYTlyQ@Dkl@jj4U53QAAAIY&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://developers.google.com/search/docs/crawling-indexing/robots/create-robots-txt
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZoXKAYTlyQ@Dkl@jj4U53QAAAIY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZoXKAYTlyQ@Dkl@jj4U53QAAAIY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[6] https://www.newyorker.com/culture/infinite-scroll/is-ai-art-stealing-from-artists
[7] https://juliabausenhardt.com/how-ai-is-stealing-your-art/
[8] https://www.commentary.org/articles/james-meigs/ai-openai-chatgpt-copyright-theft/
[9] https://arxiv.org/abs/2401.06178
[10] https://www.washingtonpost.com/opinions/2023/10/19/ai-large-language-writers-stealing/
[11] https://www.economist.com/business/2024/04/14/generative-ai-is-a-marvel-is-it-also-built-on-theft
[12] https://www.theregister.com/2023/08/08/openai_scraping_software/
[13] https://blog.google/technology/ai/an-update-on-web-publisher-controls/
[14] https://blog.cloudflare.com/ai-bots
[15] https://www.theregister.com/2024/07/03/bt_servicenow_savings/
[16] https://www.theregister.com/2024/07/03/figma_plagiarizes_apple/
[17] https://www.theregister.com/2024/07/02/eff_ftc_chatbot/
[18] https://www.theregister.com/2024/07/02/youtube_deepfake_privacy_rules/
[19] https://en.wikipedia.org/w/index.php?title=Do_Not_Track&oldid=1231982930
[20] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZoXKAYTlyQ@Dkl@jj4U53QAAAIY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[21] https://www.msn.com/en-us/news/technology/amazon-probes-perplexity-ai-for-alleged-content-scraping/ar-BB1peoMo?ocid=BingNewsVerp
[22] https://rknight.me/blog/perplexity-ai-is-lying-about-its-user-agent/
[23] https://developers.cloudflare.com/bots/concepts/bot-score/
[24] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_offprem/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZoXKAYTlyQ@Dkl@jj4U53QAAAIY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[25] https://whitepapers.theregister.com/
Whatever
Kevin McMurtrie
Let me know when Cloudflare has a means to detect when people have had enough of thier criminal organization protection.
Does Cloudflare detect when they're serving exact copies of corporate logos on new sites? Does their domain service detect deceptive naming? Do they have AI to prevent recurring abuse from the same customers? Does their abuse contact perform any function? Do they even detect when major DNS relays are blocking Cloudflare domain lookups because of phishing lists? Nope to all.
They did recently offer a service that attempts to sabotage browser developer consoles. I can only assume this is to help their phishers stay obfuscated.
robots.txt
sarusa
robots.txt is as useful as 'do not track' flags. They only work for the people you're not worried about. It assumes the people looking at it have any ethical concerns at all, and nobody involved in LMM 'AI' does. The whole model is based on theft, why would they honor your robots.txt?
hmm
Any goldrush has people selling shovels.