Websites clamp down as creepy AI crawlers sneak around for snippets
- Reference: 1721658486
- News link: https://www.theregister.co.uk/2024/07/22/ai_training_data_shrinks/
- Source link:
The Data Provenance Initiative in their [1]study titled "Consent in Crisis" looked into the domains scanned in three of the most important datasets used for training AI models. Training data usually includes publicly available info from all sorts of websites, but giving the public access to data isn't the same as giving consent for collecting it automatically using a crawler.
Crawling for data, also known as scraping, has been around much longer than generative AI, and websites already had rules on what crawlers could and couldn't do. These rules are contained in the robots.txt standard (basically an honor code for crawlers) as well as websites' terms and conditions.
[2]
The researchers examined the whole datasets – C4, Dolma, and RefinedWeb – as well as their most used domains. The data shows that websites have reacted to the introduction of AI crawlers in 2023.
[3]
[4]
Specifically, OpenAI's GPTBot and Google's Google-Extended crawlers immediately triggered websites to start changing their robots.txt restrictions. Today, between 20 and 33 percent of the top domains have enacted complete restrictions on crawlers, as opposed to just a few percent in early 2023.
Across the whole body of domains, only 1 percent enforced restrictions prior to mid-2023; now 5-7 percent have done so.
[5]
Some websites are also changing their terms of service to completely ban both crawling and using hosted content for generative AI, though the change isn't nearly as drastic as it is with robots.txt.
When it comes to whose crawlers are getting blocked, OpenAI is by far in the lead, having been banned from 25.9 percent of top sites. Anthropic and Common Crawl have been kicked out of 13.3 percent, while crawlers from Google, Meta, and others are restricted at less than 10 percent of domains.
As for what sites are putting up barriers to AI crawlers, it's largely news sites. Among all domains, news publications were by far the most likely to have terms of service (ToS) and robots.txt settings restricting AI crawlers. However, for the top domains specifically, social media platforms and forums (think Facebook and X) were just as likely to restrict crawlers via the terms of service as news publications.
New rules on crawling needed to fix this mess
Although it's clear lots of websites don't want their content being scraped for use in AI, the Data Provenance Initiative says they're not communicating that effectively.
Part of this is down to the restrictions in robots.txt and the ToS not lining up. 34.9 percent of the top training websites make it clear in the ToS that crawling isn't allowed, but fail to mirror that in robots.txt. On the other hand, websites with no ToS at all are surprisingly likely to set up partial or complete blocks on crawlers.
[6]
And when crawling is banned, websites tend to just ban OpenAI, Common Crawl, and Anthropic. The study also found some websites fail to correctly identify and restrict certain crawlers. 4.5 percent of sites banned Anthropic-AI and Claude-Web instead of Anthropic's actual crawler ClaudeBot.
Plus, there are bots for collecting training materials but also those for grabbing up-to-date info, and the distinction might not always be clear to website operators. So while GPTBot is banned on some domains, ChatGPT-User isn't, even though they're both used for crawling.
[7]More big city newspapers drag Microsoft, OpenAI hard in copyright lawsuit
[8]Coders' Copilot code-copying copyright claims crumble against GitHub, Microsoft
[9]Big brains divided over training AI with more AI: Is model collapse inevitable?
[10]Meta won't train AI on Euro posts after all, as watchdogs put their paws down
Obviously, sites locking down their data will negatively impact AI model training, especially since the websites most likely to crack down tend to have the highest quality data. But the team points out that crawlers are used by academia and nonprofits like the Internet Archive, and are getting caught in the crossfire.
The study also brings up the possibility that AI firms might have wasted their time crawling so hard they're getting banned. While almost 40 percent of the top domains used in the three datasets were news-related, over 30 percent of ChatGPT inquiries were for creative writing, compared to about 1 percent that concerned news.
Other common requests were for translation, coding assistance, general information, and sexual roleplay, which was in second place.
The researchers say the traditional structure of robots.txt and ToS aren't capable of accurately defining rules in the age of AI. Part of the problem is that enforcing a total ban is the easiest solution, since robots.txt is mostly useful for blocking specific crawlers rather than communicating certain rules, like what crawlers are allowed to do with collected data.
Until that happens, however, the current trajectory of AI data scraping could affect how the web is structured, which is likely to be less open than it was before. ®
Get our [11]Tech Resources
[1] https://www.dataprovenance.org/consent-in-crisis-paper
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Zp6CGbPVqG3SG6e-c6nkeQAAAJM&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zp6CGbPVqG3SG6e-c6nkeQAAAJM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Zp6CGbPVqG3SG6e-c6nkeQAAAJM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zp6CGbPVqG3SG6e-c6nkeQAAAJM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Zp6CGbPVqG3SG6e-c6nkeQAAAJM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[7] https://www.theregister.com/2024/04/30/newspapers_microsoft_openai/
[8] https://www.theregister.com/2024/07/08/github_copilot_dmca/
[9] https://www.theregister.com/2024/05/09/ai_model_collapse/
[10] https://www.theregister.com/2024/06/14/meta_eu_privacy/
[11] https://whitepapers.theregister.com/
Victim blaming
"Part of this is down to the restrictions in robots.txt and the ToS not lining up. 34.9 percent of the top training websites make it clear in the ToS that crawling isn't allowed, but fail to mirror that in robots.txt"
A classic case of victim blaming. For one, robots.txt can not mirror the ToS anyway, because the former is a technical, and the latter a legal document, designed to express very different things that are only marginally related to each other. Also the robots.txt format does not allow to include instructions for any yet unknown entity (crawler), is not legally binding, and even if it would be could be easily circumvented by simply renaming the user-agent in question.
And finally robots.txt and copyright law have completely opposing defaults. The robots.txt spec essentially says that if the file is missing or any explicit disallow directive is missing, then you're free to crawl whatever url you want. Copyright law on the other side says that unless there's explicit permission, you're not allowed to access or use any copyrighted content. So, of course robots.txt - which is not law and not a contract anyway - can not possibly used to override copyright law (and the ToS), but AI companies act as if it would, and that the lack of explicit disallow directives would give them permission to use any content for training.
Also large internet corporations, like for ex. Google or Facebook, etc. are actively working to make robots.txt useless for blocking AI crawler even at the theoretical/technical level, by simply using the very same user agent that they use for collecting content for other purpose. So, for ex. Facebook is using the very same user agent ("facebookexternalhit") for obvious AI crawling that they've been and are still using for building the link previews in shares - which in the end means that you can't tell to them to not train their AI models with your data but show meaningful link previews in shares.
Similarly Google has Googlebot-Extended, which supposedly is there so you can prevent them using your data for AI training - but that only applies to training for ex. their Bard model, but doesn't affect AI Summaries, which are shown based on what data their regular Google crawlerbot has retrieved.
robots.txt talks about crawlers & not their purpose
It requires that Web Site Owners (WSOs) list the different crawlers and specify rules.
Most WSOs do not care what the crawlers care called they just want to control what the content is used for; so indexing might be fine as might humans use as learning material but feeding to an AI might not be. Also should there be some attribution made whenever derived content is displayed somewhere ?
The [1]Consent in Crisis paper kind of talks about this but, as all to often in such papers, does not provide an easy to read summary.
Maybe [2]Automated Content Access Protocol needs to be revived. But this will be objected to by AIs and others who want to continue their free lunch.
[1] https://www.dataprovenance.org/Consent_in_Crisis.pdf
[2] https://en.wikipedia.org/wiki/Automated_Content_Access_Protocol
Financial and Environmental cost
We found that over 90% of our traffic was Bot related. After adding entries to robots.txt , and sending a % of 429 responses (thanks google for ignoring robots.txt), it's down to a still high, but manageable, 30%. Quite a few bots identify themselves in the agent string as a normal client rather than a bot which makes filtering them quite complex.
Financially we've saved about $5k / month in cloud costs, and all that processing and serving of pointless traffic inevitably has an energy cost as well. The scans kill the content caches as they just pull anything with a link rather than just hitting popular items like real customers do, so our cache hit percentage is now way higher, and we're not training competitors' AI systems with our data any more.
Next step is to detect the bot and serve it junk, though I think I'll struggle to get signoff on that project!
"an honor code for crawlers"
Ah, honor.
The kind of thing the Founding Fathers were sure people had when making important decisions.
Ah, how times have changed.
Ban "em, ban 'em all.
Can you imagine an AI trained on social media drivel. The amount of AI electrons needlessly dying so that Cheryl's and her BFFs "thoughts" can be stamped for posterity on X, TikTok et al and then swallowed like a toxin by some poor unsuspecting AI shouldn't be allowed.