Automation needed to fight army of AI content harvesters stalking the web
- Reference: 1722324125
- News link: https://www.theregister.co.uk/2024/07/30/taming_ai_content_crawlers/
- Source link:
iFixit boss Kyle Wiens complained about the uninvited bot visitations on social media. "I get you're hungry for data. Claude is really smart," the CEO [1]said last Wednesday, referring to Anthropic's [2]family of LLMs fueled by information harvested by ClaudeBot.
"You're not only taking our content without paying, you're tying up our devops resources," Wiens added. "Not cool." And also not in compliance with iFixit's terms of service.
[3]
The publisher repelled some of the traffic Anthropic’s bots created by adding a disallow directive to the website's robots.txt file – the tech industry's [4]agreed-upon mechanism for turning away crawlers.
[5]
[6]
This is not the first time one large tech outfit has sent [7]an excessive amount of network traffic to another.
"The crawling stopped after we added them to our robots.txt," Wiens told The Register . "Now, they hit that file every thirty minutes."
[8]
He added, "Anthropic never replied to me. I'd still be interested in talking to them." This comes as Freelancer.com [9]accused the Claude crawler of visiting its site nearly four million times in as many hours.
Wiens explained that iFixit's mission is to help people repair their devices – which requires information, parts, and tools. "I'd love to deliver that experience through another platform," he said. "I'm a Claude user, and if I ask Claude how to fix my phone and it said, buy this part and these are the instructions – hey, that would be cool."
But that's not happening at the moment. "Right now, [Claude] mangles our instructions and outputs them incorrectly. So people will break their phone if they follow the LLM directions, and it doesn't point you to the part or tool that you need. Not very helpful."
[10]
Wiens provided an example of Claude explaining how to install a display screen on a Google Pixel 6a by opening the phone from the back. "It opens from the front, so this would not work and would cause damage," the CEO explained.
Here's Claude's erroneous advice on the matter:
[11]
Incorrect instructions from Anthropic's Claude about how to repair a Google Pixel screen – Click to enlarge
Asked to comment, a spokesperson for Anthropic – built by ex-OpenAI staff and others with the goal of creating a kinder AI super-lab – pointed The Register to its developer FAQ [12]entry titled: "Does Anthropic crawl data from the web, and how can site owners block the crawler?"
That document states: "As per industry standard, Anthropic uses a variety of data sources for model development, such as publicly available data from the internet gathered via a web crawler." It goes on to note that Anthropic tries to make its crawling transparent and non-intrusive, and respectful of robots.txt directives and anti-circumvention mechanisms like CAPTCHA challenges.
Robots.txt dates back [13]to 1994 and was once largely a set-and-forget technology. The idea being that you list in it instructions for bots: What they can and can't index, and which are welcome and those that aren't, with the hope that bot operators will respect site owners' wishes.
"Technically there are no assumptions baked into the protocol itself, but this is how things were in practice for a long time," said Gavin King, founder of Dark Visitors – a [14]venture that offers content protection services including automatic robots.txt updates and automated blocking of bots that ignore the file.
"There used to be only a few bots that people cared about," he told The Register , "and much of the time the rules were more about optimistically helping them – eg, keeping Googlebot away from content that didn't make sense to surface in search results – or doing general things like setting rate limits for all bots. For simple cases like these, you didn't need to change your file that often."
[15]Meta's AI safety system defeated by the space bar
[16]Google DeepMind's latest models kinda sorta take silver at Math Olympiad
[17]Sam Altman wants a US-led freedom coalition to fight authoritarian AI
[18]Study shock! AI hinders productivity and makes working worse
But the AI age has changed the landscape dramatically. AI firms have proliferated, with many crawling websites to harvest data.
And each of them may operate multiple bots. Just when you think you've stopped one crawler, another one shows up from the same outfit.
For instance, Anthropic previously operated Claude-Web and Anthropic-AI to gather training data from sites. If you've disallowed either or both of them, you might be surprised to see ClaudeBot show up. The Associated Press's [19]robots.txt file shows its ongoing attempts to turn away Anthropic's bots, including the latest ClaudeBot, while [20]Reuters and [21]WiReD's still only target the older crawlers.
"We're witnessing a Cambrian explosion in the ecosystem of artificial agents crawling the internet," declared King.
"For example, OpenAI just launched a new one last week ( [22]OAI-SearchBot ), Meta the week before ( [23]Meta-ExternalAgent ), and Apple last month ( [24]Applebot-Extended )."
Growing numbers of crawlers mean it's difficult for website owners to keep their robots.txt files up to date to counter these new agents. We note that [25]OpenAI and [26]Google last year publishing guidance about how to block their respective crawler bots.
"The pace at which website owners need to update their robots.txt is a direct reflection of the pace at which LLMs and the companies that train and operate them are evolving and competing," said King. "This is a typical cycle for any new technology gaining traction – but this one happens to have this side effect."
Dark Visitors, for what it's worth, provides a programmatic way to automatically update robots.txt entries as new crawlers emerge, and to understand which crawlers have been stopping by and to stop them from accessing pages if they misbehave.
Which, incidentally, appears to be a growing area of interest. Cloudflare recently announced enhancements to its own [27]bot-blockers service to turn away more AI crawlers.
It's always possible that there's crawling going on disguised as normal traffic
While it's long been known that some crawlers ignore robots.txt settings – [28]an allegation leveled at AI search biz Perplexity (it's played down claims of wrongdoing and was [29]hit with a cease-and-desist by publisher Conde Nast) – King observed that most companies respect the rules.
"There's a lot of unfair sentiment and coverage out there about crawlers not following robots.txt rules," he said.
"We have a unique perspective across many websites based on our agent analytics data – another feature that lets Dark Visitors users observe agent behavior on their websites – and this just isn't the case.
"Most of them identify themselves with a proper user agent string, and start following the rules within 24 hours (a reasonable cache delay). Basically all of the big companies do this from what I've seen. At the same time, you don't know what you don't know. It's always possible that there's crawling going on disguised as normal traffic."
One problem is that unscrupulous AI developers can just create crawlers with new names that aren't covered by an existing robots.txt entry specifically and deliberately to maliciously evade the rules.
The main issue that King sees for those of us operating websites is not how to implement blocking – but knowing what to add to their robots.txt file, given the constantly changing bot population. ®
Get our [30]Tech Resources
[1] https://x.com/kwiens/status/1816128302542905620
[2] https://www.theregister.com/2024/06/20/anthropic_claude_35/
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Zqi5x@Ek73oMKIjmCeuukQAAAUY&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[4] https://datatracker.ietf.org/doc/html/rfc9309
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zqi5x@Ek73oMKIjmCeuukQAAAUY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Zqi5x@Ek73oMKIjmCeuukQAAAUY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[7] https://www.theregister.com/2023/06/28/microsofts_github_gmp_project/
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zqi5x@Ek73oMKIjmCeuukQAAAUY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[9] https://www.ft.com/content/07611b74-3d69-4579-9089-f2fc2af61baa
[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Zqi5x@Ek73oMKIjmCeuukQAAAUY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[11] https://regmedia.co.uk/2024/07/29/pixel_claude_converation.jpg
[12] https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
[13] https://www.robotstxt.org/norobots-rfc.txt
[14] https://darkvisitors.com
[15] https://www.theregister.com/2024/07/29/meta_ai_safety/
[16] https://www.theregister.com/2024/07/26/google_deepmind_maths/
[17] https://www.theregister.com/2024/07/25/sam_altman_ai_freedom/
[18] https://www.theregister.com/2024/07/26/ai_hinders_productivity/
[19] https://www.apnews.com/robots.txt
[20] https://www.reuters.com/robots.txt
[21] https://www.wired.com/robots.txt
[22] https://darkvisitors.com/agents/oai-searchbot
[23] https://darkvisitors.com/agents/meta-externalagent
[24] https://darkvisitors.com/agents/applebot-extended
[25] https://www.theregister.com/2023/08/08/openai_scraping_software/
[26] https://blog.google/technology/ai/an-update-on-web-publisher-controls/
[27] https://www.theregister.com/2024/07/03/cloudflare_ai_blocks/
[28] https://rknight.me/blog/perplexity-ai-is-lying-about-its-user-agent/
[29] https://www.theinformation.com/articles/cond-nast-sends-cease-and-desist-letter-to-ai-search-engine-perplexity
[30] https://whitepapers.theregister.com/
Re: Whac-A-Mole
rather than blocking them I now feed nonsense text to those visitors
As a matter of interest: what sort of nonsense do you feed them ? How do you generate it ? Is it always the same or different nonsense with every page hit ?
Re: Whac-A-Mole
I am planning to integrate it with a better nonsense generator API in future (OpenAI). At the moment it's very simple and uses Wikipedia random pages option to get some text, which it then mixes the word order and outputs. It's actually so quick that I don't need to cache any nonsense.
Re: Whac-A-Mole
"I am planning to integrate it with a better nonsense generator API in future (OpenAI)"
Maybe target the output of the actual LLMs that are doing the scraping to promote faster model collapse.
Re: Whac-A-Mole
Forgot to mention - anyone looking for a good Robots.txt file for their website and they are lazy like me, you can do worse than copy the one from New York Times:
https://www.nytimes.com/robots.txt
You'll want to change the comments at the top and remove all of the GoogleBot restrictions and sitemap mentions but it's a good list for blocking ML scrapers and no doubt they will keep it up-to-date.
Re: Whac-A-Mole
This might have been a follow-up to a suggestion I made. What I had in mind was feeding a steam of randomly selected words from the contents of /usr/share/dict or its equivalent with the added refinement of occasionality selecting a pair or triplet of words from the stream and subsequently throwing them in at intervals so the stream would become a stream of meaningless word associations padded out with random words.
Removal pain incentive?
If an AI company were to crawl a site's contents against the robots.txt policy, I presume that site could then press legally for the removal of their content from the AI. I suspect this would be non-trivial and expensive for the AI company question. Perhaps it needs a test case to sharpen the issue.
Re: Removal pain incentive?
A couple of copyright infringement law suits will probably define "fair use" and set fees but nothing is going to stop what has been happening for years anyway.
Whitelist?
As far as I can tell, you can easily make your robots.txt a "whitelist" - allow a list of known good user-agents, and "Disallow: /" for anything else. While that might mean a bit of extra work keeping up with wanted crawling, if the problem is that the number of malicious/problematic crawlers is increasing much faster than the good ones, does that not largely solve the problem (barring the ones that just ignore robots.txt entirely)? Not a web developer, so correct me if I'm wrong.
When does scraping-for-machine-learning become a DoS attack?
Because that's what it sounds like to me.
This is exactly the principle of the NHS handing data to ML companies.
It's not their data, it's our data.
Let's start with some simple basic principles.
1) All data has value.
2) You want to use it for anything other than what it was collected for you need to pay the supplier. And "Selling it on" is not a legitimate basic use.
3) You need to have the mechanisms in place to pay them.
4) And they need to have the mechanisms in place to pay the people they got it from (at some level this will get down to accounts of individual people, but this does not need some kind of universal ID card to do this, just an account number. It doesn't have to be the same account number for all data sources).
I guarantee Big Tech, many of whom have built $Bn businesses on not paying a cent for data will whine like a b**ch at this. But that's how a real free market works.
Whac-A-Mole
For my own custom coded websites it's not that hard. I've added code that tracks visitors visiting too many pages too quickly - those IP addresses get sin binned. Thanks to an idea in El Reg comments, rather than blocking them I now feed nonsense text to those visitors. It's a bit trickier with the CMS websites, which I rely on robots.txt for. I want to allow search engines, the wayback machine, archive.is but what is to stop these scofflaws scraping content from the wayback machine? I don't want to spend my time keeping up with all of their tricks but I'm not quite ready to surrender.