Reddit hopes robots.txt tweak will do the trick in scaring off AI training data scrapers
(2024/06/27)
- Reference: 1719473352
- News link: https://www.theregister.co.uk/2024/06/27/reddit_ai_info/
- Source link:
For many Reddit has become the go to repository of community and crowdsourced knowledge, a fact that has no doubt made it a prime target for AI startups desperate for training data.
This week, Reddit [1]announced it would be introducing measures to prevent unauthorized scraping by such organizations. These efforts will include an updated robots.txt — a [2]file found on most websites that provides directions to web crawlers on what they can and can't index — "in the coming weeks." If you're curious you can find Reddit's current robots.txt [3]here .
It should be noted that robots.txt can't force scrapers to do anything; the file's contents are more like guidelines or firm requests. Web crawlers can be made to ignore them, so Reddit says it will continue to rate limit and/or block rogue bots – presumably that includes bad ones that ignore robots.txt – from accessing the site.
[4]
Indeed, crawlers that shun robots.txt risk getting blocked entirely, if possible, from sites in general by their administrators.
[5]
[6]
These measures, vague as they are at the moment, appear to be targeted specifically at those accessing Reddit for commercial gain. The site says that "Good faith actors — like researchers and organizations such as the Internet Archive — will continue to have access to Reddit content for non-commercial use."
The announcement comes just weeks after Reddit [7]unveiled a fresh public content policy, which it spun as a way to more transparently communicate how user data is used and protect user privacy.
[8]
"We see more and more commercial entities using unauthorized access or misusing authorized access to collect public data in bulk, including Reddit public content," the site said.
It seems Reddit execs would much rather interested parties pay it for curated access to its crowdsourced hive mind of knowledge, opinion, trolling, and karma farming, as the announcement ends with a sales pitch for its data access plans.
[9]Read AI about it... OpenAI does deal with News Corp
[10]Slack tweaks its principles in response to user outrage at AI slurping
[11]Hey, Reddit. Quick question. All those clicks on my ads. Were they actually real?
[12]Ahead of IPO, Reddit blends advertising into user posts
As we've previously discussed, training large language models, like GPT-4, Gemini, or Claude require a prodigious amount of data. Meta's relatively small Llama3 8B model [13]used some 15 trillion tokens.
Because of this, supplying AI training data used to build these models has become a lucrative business proposition. Last month Scale AI — which sells AI data services including pre-labeled datasets — saw its valuation [14]soar to nearly $14 billion amid a $1 billion funding round led by Nvidia, Amazon, and Meta.
Meanwhile, this week also saw the [15]formation of an AI data trade group called the Dataset Providers Alliance. The group's members include Rightsify, vAIsual, Pixta AI, Datarade, Global Copyright Exchange, Calliope Networks, and Ado.
[16]
Naturally, Reddit is keen to cash in on this demand, having already announced an agreement to sell API access to Google in a deal [17]reportedly worth $60 million a year. The Front Page of the Internet last month [18]reached a similar agreement with OpenAI, though the terms of the deal weren't disclosed.
How useful Reddit's data actually is has, however, been called into question in recent weeks after Google started [19]citing obvious troll posts in its AI generated answers. In one case the search engine suggested adding "non-toxic glue" to pizza sauce to keep the cheese from sticking.
The Register reached out to Reddit for comment on its efforts to block rogue web scrapers and on its future plans. ®
Get our [20]Tech Resources
[1] https://www.redditinc.com/blog/robot-txt-update
[2] https://developers.google.com/search/docs/crawling-indexing/robots/intro
[3] https://www.reddit.com/robots.txt
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Zn04SW1m25pavpQ7kldnqwAAA8A&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zn04SW1m25pavpQ7kldnqwAAA8A&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Zn04SW1m25pavpQ7kldnqwAAA8A&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[7] https://www.redditinc.com/blog/publishing-our-public-content-policy-and-introducing-a-new-community-for-researchers
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zn04SW1m25pavpQ7kldnqwAAA8A&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[9] https://www.theregister.com/2024/05/23/openai_news_corp/
[10] https://www.theregister.com/2024/05/20/slack_ts_and_cs_update/
[11] https://www.theregister.com/2024/05/11/reddit_sued_ad_clicks/
[12] https://www.theregister.com/2024/03/16/reddit_promoted_posts/
[13] https://www.theregister.com/2024/04/19/meta_debuts_llama3_llm/
[14] https://www.theregister.com/2024/05/21/scale_ai_funding/
[15] https://www.prnewswire.com/news-releases/leading-dataset-licensors-unite-to-launch-the-dataset-providers-alliance-dpa-302182603.html
[16] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Zn04SW1m25pavpQ7kldnqwAAA8A&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[17] https://www.theregister.com/2024/02/22/reddit_google_license_ipo_altman/
[18] https://www.theregister.com/2024/05/17/reddit_signs_ai_deal/
[19] https://www.theregister.com/2024/05/31/google_ai_search_update/
[20] https://whitepapers.theregister.com/
This week, Reddit [1]announced it would be introducing measures to prevent unauthorized scraping by such organizations. These efforts will include an updated robots.txt — a [2]file found on most websites that provides directions to web crawlers on what they can and can't index — "in the coming weeks." If you're curious you can find Reddit's current robots.txt [3]here .
It should be noted that robots.txt can't force scrapers to do anything; the file's contents are more like guidelines or firm requests. Web crawlers can be made to ignore them, so Reddit says it will continue to rate limit and/or block rogue bots – presumably that includes bad ones that ignore robots.txt – from accessing the site.
[4]
Indeed, crawlers that shun robots.txt risk getting blocked entirely, if possible, from sites in general by their administrators.
[5]
[6]
These measures, vague as they are at the moment, appear to be targeted specifically at those accessing Reddit for commercial gain. The site says that "Good faith actors — like researchers and organizations such as the Internet Archive — will continue to have access to Reddit content for non-commercial use."
The announcement comes just weeks after Reddit [7]unveiled a fresh public content policy, which it spun as a way to more transparently communicate how user data is used and protect user privacy.
[8]
"We see more and more commercial entities using unauthorized access or misusing authorized access to collect public data in bulk, including Reddit public content," the site said.
It seems Reddit execs would much rather interested parties pay it for curated access to its crowdsourced hive mind of knowledge, opinion, trolling, and karma farming, as the announcement ends with a sales pitch for its data access plans.
[9]Read AI about it... OpenAI does deal with News Corp
[10]Slack tweaks its principles in response to user outrage at AI slurping
[11]Hey, Reddit. Quick question. All those clicks on my ads. Were they actually real?
[12]Ahead of IPO, Reddit blends advertising into user posts
As we've previously discussed, training large language models, like GPT-4, Gemini, or Claude require a prodigious amount of data. Meta's relatively small Llama3 8B model [13]used some 15 trillion tokens.
Because of this, supplying AI training data used to build these models has become a lucrative business proposition. Last month Scale AI — which sells AI data services including pre-labeled datasets — saw its valuation [14]soar to nearly $14 billion amid a $1 billion funding round led by Nvidia, Amazon, and Meta.
Meanwhile, this week also saw the [15]formation of an AI data trade group called the Dataset Providers Alliance. The group's members include Rightsify, vAIsual, Pixta AI, Datarade, Global Copyright Exchange, Calliope Networks, and Ado.
[16]
Naturally, Reddit is keen to cash in on this demand, having already announced an agreement to sell API access to Google in a deal [17]reportedly worth $60 million a year. The Front Page of the Internet last month [18]reached a similar agreement with OpenAI, though the terms of the deal weren't disclosed.
How useful Reddit's data actually is has, however, been called into question in recent weeks after Google started [19]citing obvious troll posts in its AI generated answers. In one case the search engine suggested adding "non-toxic glue" to pizza sauce to keep the cheese from sticking.
The Register reached out to Reddit for comment on its efforts to block rogue web scrapers and on its future plans. ®
Get our [20]Tech Resources
[1] https://www.redditinc.com/blog/robot-txt-update
[2] https://developers.google.com/search/docs/crawling-indexing/robots/intro
[3] https://www.reddit.com/robots.txt
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Zn04SW1m25pavpQ7kldnqwAAA8A&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zn04SW1m25pavpQ7kldnqwAAA8A&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Zn04SW1m25pavpQ7kldnqwAAA8A&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[7] https://www.redditinc.com/blog/publishing-our-public-content-policy-and-introducing-a-new-community-for-researchers
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Zn04SW1m25pavpQ7kldnqwAAA8A&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[9] https://www.theregister.com/2024/05/23/openai_news_corp/
[10] https://www.theregister.com/2024/05/20/slack_ts_and_cs_update/
[11] https://www.theregister.com/2024/05/11/reddit_sued_ad_clicks/
[12] https://www.theregister.com/2024/03/16/reddit_promoted_posts/
[13] https://www.theregister.com/2024/04/19/meta_debuts_llama3_llm/
[14] https://www.theregister.com/2024/05/21/scale_ai_funding/
[15] https://www.prnewswire.com/news-releases/leading-dataset-licensors-unite-to-launch-the-dataset-providers-alliance-dpa-302182603.html
[16] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Zn04SW1m25pavpQ7kldnqwAAA8A&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[17] https://www.theregister.com/2024/02/22/reddit_google_license_ipo_altman/
[18] https://www.theregister.com/2024/05/17/reddit_signs_ai_deal/
[19] https://www.theregister.com/2024/05/31/google_ai_search_update/
[20] https://whitepapers.theregister.com/
Re: Poisoning the honey pot
Paul Crawford
...feed them a pile of gibberish, misinformation and random garbage
Cunning, but some sites already provide that as their core activity?
Re: Poisoning the honey pot
theOtherJT
When dealing with Reddit, how would they tell the difference?
robots.txt is a machine understandable copyright notice
alain williams
There is an assumption that if something is available on the web then it can be downloaded for free and used for whatever purpose.
OK: we know that that is not true, there is plenty of stuff that cannot be used for some purposes: eg not to be sold on.
AI people like to pretend that they have no way of knowing that content cannot be used by them. This is what [1]robots.txt would be perfect for. A tweak that lists allowed/disallowed purposes would mean that operators of web crawlers would no longer be able to claim ignorance of a web site owners wishes; this could make it easier to sue them in court.
The well funded AI crowd would fight tooth & nail to be allowed to break copyright with impunity but some large web sites might be able to win, set precedence, ... that would prolly get ignored unless you had enough money.
[1] https://en.wikipedia.org/wiki/Robots.txt
Poisoning the honey pot
> crawlers that shun robots.txt risk getting blocked entirely
Instead of blocking non-compliant bots, why not feed them a pile of gibberish, misinformation and random garbage.
No, I'm not suggesting they are forwarded to far-right (or left) propaganda sources, just that the pages they do scrape are subtly altered to replace words with others, opposites or something completely irrelevant. Do the same with any numbers, too. And images can be corrupted such that they do not display.
It seems to me that the value of AIs is that they can produce responses that users value. If their core data is corrupted (as punishment for straying past what robots.txt permits) then their use is vastly diminished, and so is their value - monetary value.
That should be enough of a deterrent to force them to play nice.