New Really Simple Licensing spec wants AI crawlers to show a license - or a credit card
(2025/09/11)
- Reference: 1757624087
- News link: https://www.theregister.co.uk/2025/09/11/rsl_content_grabbing_ai_digital_licensing/
- Source link:
Content creation and delivery companies have introduced a digital licensing mechanism in an effort to compensate media makers when AI companies use their work.
The [1]Really Simple Licensing (RSL) standard is intended to provide websites with a programmatic way to present web crawlers with licensing terms, and to gate site access based on license compliance, which may require payment.
RSL attempts to improve upon robots.txt, a voluntary way for websites to declare how bots should interact with site content. Crawlers frequently ignore robots.txt, and some disguise themselves to avoid being blocked. So RSL offers a compliance mechanism that requires bots to present an authorization header as part of the network negotiation process.
[2]
"With RSL, websites can enforce stricter control over their content usage by blocking crawlers that have not obtained a free or paid license from an [3]RSL License Server ," the documentation [4]explains . "When a crawler requests a page that is managed by an RSL license from your website, it must include a valid [5]RSL License Token for the page in the Authorization header using the new proposed License authentication scheme defined in [6]RFC 7235 HTTP Authentication ."
[7]
[8]
RSL is based on Really Simple Syndication (RSS), a popular decentralized web protocol.
"RSL builds directly on the legacy of RSS, providing the missing licensing layer for the AI-first Internet," said Tim O'Reilly, CEO of O'Reilly Media, one of the organizations steering the project, in [9]a statement . "It ensures that the creators and publishers who fuel AI innovation are not just part of the conversation but fairly compensated for the value they create."
[10]
RSL is administered by a newly formed nonprofit, the [11]RSL Collective , a rights collective similar to ASCAP and BMI in the music industry.
Along with O'Reilly Media, RSL and the RSL Collective are backed by Reddit, People Inc., Yahoo, Internet Brands, Ziff Davis, wikiHow, Medium, The Daily Beast, Miso.AI, Raptive, Ranker, and Evolve Media. Fastly, Quora, and Adweek have endorsed the RSL standard but are not participating in the RSL Collective.
The technical aspects of the RSL Standard are overseen by the [12]RSL Technical Steering Committee , staffed by representatives from participating publishing and technology companies.
[13]
AI vendors like Anthropic, Google, Microsoft, OpenAI, and others rely on web crawling scripts or bots to ingest website content, which then gets used for training AI models or other applications like search.
Initially, this was done without permission, sometimes using unauthorized datasets, [14]sparking copyright disputes , or based on assumptions about fair use that continue to be litigated. But following the debut of ChatGPT in November 2022 and the ensuing expectation that search will be subsumed by AI, content owners have become more concerned that their commercial assets are being co-opted without compensation.
Large publishers like the New York Times and social media sites like Reddit have negotiated specific deals that provide AI vendors with access to their content, and have sued in court when model makers have ignored their wishes.
[15]Appeals court blocks Trump bid to ax top copyright official in AI spat
[16]Walmart's bet on AI depends on getting employees to use it
[17]OpenAI reportedly on the hook for $300B Oracle Cloud bill
[18]AI pricing is currently in a state of 'pandemonium' says Gartner
Amid reports that AI-based search services like Google's AI Overviews are [19]starving websites of visitor traffic – a claim [20]Google has denied even as it [21]acknowledges the decline of the web in court – efforts to obtain compensation from AI vendors have led to web tollbooth services like Cloudflare's [22]Pay per crawl .
"RSL complements the Cloudflare announcement by enabling publishers to define rules of how crawlers legally license (including paying per crawl) their content and get unblocked," RSL co-founder Eckart Walther told The Register in an email.
"Most publishers that support RSL also participated in the Cloudflare AI blocking and Pay-per-Crawl announcement, and both approaches can work together to ensure that publishers can assert their content rights and receive fair compensation from AI companies."
"Fair compensation" in this instance will be determined by RSL Collective members.
"The RSL Collective is a nonprofit Collective Rights Organization that is guided by the priorities of its members – it negotiates on behalf of its members, but the definition of 'fair' will be determined by an open, transparent process that needs to be approved by members," explained Walther. "As a nonprofit, the RSL Collective passes all royalties to its members, minus any costs needed to operate the service."
RSL, according to its creators, supports a variety of licensing, usage, and royalty models, including free, attribution, subscription, pay-per-crawl, and pay-per-inference.
Since RSL is simply a standard that defines the criteria for admission, the challenge of separating human visitors from bots and forcing bots to comply with terms has been left to internet service providers, particularly content delivery networks like Fastly, Akamai, or Cloudflare, and custom crawler detection solutions.
"While there are certainly bad actors that will not follow the rules set by publishers and might need to be blocked by solutions developed by companies like Fastly and Cloudflare, we believe that the majority of large AI firms understand, and have publicly spoken, about the need for fair compensation for the publishing industry," said Walther.
"The nonprofit RSL collective exists to make this process dramatically simpler for AI firms by pooling licensing rights for millions of publishers and creators, the same way that large music distributors are able to license large music catalogs through music collective rights organizations." ®
Get our [23]Tech Resources
[1] https://rslstandard.org
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2aMNGcprfVMhPMUteye5ySQAAAE8&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://rslstandard.org/guide/license-servers
[4] https://rslstandard.org/guide/web-crawlers
[5] https://rslstandard.org/api#acquire-license
[6] https://datatracker.ietf.org/doc/html/rfc7235#section-5.1
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aMNGcprfVMhPMUteye5ySQAAAE8&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aMNGcprfVMhPMUteye5ySQAAAE8&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[9] https://rslstandard.org/press/rsl-standard
[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aMNGcprfVMhPMUteye5ySQAAAE8&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[11] https://rslcollective.org
[12] https://rslstandard.org/about
[13] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aMNGcprfVMhPMUteye5ySQAAAE8&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[14] https://www.theregister.com/2025/09/08/anthropic_settles_author_lawsuit/
[15] https://www.theregister.com/2025/09/11/appeals_court_blocks_trump_copyright_official_firing/
[16] https://www.theregister.com/2025/09/11/walmarts_bet_on_ai_depends/
[17] https://www.theregister.com/2025/09/11/openai_reportedly_on_the_hook/
[18] https://www.theregister.com/2025/09/10/ai_software_licensing_immature/
[19] https://www.theregister.com/2025/07/22/google_ai_overviews_suppress_search/
[20] https://blog.google/products/search/ai-search-driving-more-queries-higher-quality-clicks/
[21] https://x.com/jason_kint/status/1964380170741305846
[22] https://www.theregister.com/2025/07/01/cloudflare_creates_ai_crawler_toll/
[23] https://whitepapers.theregister.com/
The [1]Really Simple Licensing (RSL) standard is intended to provide websites with a programmatic way to present web crawlers with licensing terms, and to gate site access based on license compliance, which may require payment.
RSL attempts to improve upon robots.txt, a voluntary way for websites to declare how bots should interact with site content. Crawlers frequently ignore robots.txt, and some disguise themselves to avoid being blocked. So RSL offers a compliance mechanism that requires bots to present an authorization header as part of the network negotiation process.
[2]
"With RSL, websites can enforce stricter control over their content usage by blocking crawlers that have not obtained a free or paid license from an [3]RSL License Server ," the documentation [4]explains . "When a crawler requests a page that is managed by an RSL license from your website, it must include a valid [5]RSL License Token for the page in the Authorization header using the new proposed License authentication scheme defined in [6]RFC 7235 HTTP Authentication ."
[7]
[8]
RSL is based on Really Simple Syndication (RSS), a popular decentralized web protocol.
"RSL builds directly on the legacy of RSS, providing the missing licensing layer for the AI-first Internet," said Tim O'Reilly, CEO of O'Reilly Media, one of the organizations steering the project, in [9]a statement . "It ensures that the creators and publishers who fuel AI innovation are not just part of the conversation but fairly compensated for the value they create."
[10]
RSL is administered by a newly formed nonprofit, the [11]RSL Collective , a rights collective similar to ASCAP and BMI in the music industry.
Along with O'Reilly Media, RSL and the RSL Collective are backed by Reddit, People Inc., Yahoo, Internet Brands, Ziff Davis, wikiHow, Medium, The Daily Beast, Miso.AI, Raptive, Ranker, and Evolve Media. Fastly, Quora, and Adweek have endorsed the RSL standard but are not participating in the RSL Collective.
The technical aspects of the RSL Standard are overseen by the [12]RSL Technical Steering Committee , staffed by representatives from participating publishing and technology companies.
[13]
AI vendors like Anthropic, Google, Microsoft, OpenAI, and others rely on web crawling scripts or bots to ingest website content, which then gets used for training AI models or other applications like search.
Initially, this was done without permission, sometimes using unauthorized datasets, [14]sparking copyright disputes , or based on assumptions about fair use that continue to be litigated. But following the debut of ChatGPT in November 2022 and the ensuing expectation that search will be subsumed by AI, content owners have become more concerned that their commercial assets are being co-opted without compensation.
Large publishers like the New York Times and social media sites like Reddit have negotiated specific deals that provide AI vendors with access to their content, and have sued in court when model makers have ignored their wishes.
[15]Appeals court blocks Trump bid to ax top copyright official in AI spat
[16]Walmart's bet on AI depends on getting employees to use it
[17]OpenAI reportedly on the hook for $300B Oracle Cloud bill
[18]AI pricing is currently in a state of 'pandemonium' says Gartner
Amid reports that AI-based search services like Google's AI Overviews are [19]starving websites of visitor traffic – a claim [20]Google has denied even as it [21]acknowledges the decline of the web in court – efforts to obtain compensation from AI vendors have led to web tollbooth services like Cloudflare's [22]Pay per crawl .
"RSL complements the Cloudflare announcement by enabling publishers to define rules of how crawlers legally license (including paying per crawl) their content and get unblocked," RSL co-founder Eckart Walther told The Register in an email.
"Most publishers that support RSL also participated in the Cloudflare AI blocking and Pay-per-Crawl announcement, and both approaches can work together to ensure that publishers can assert their content rights and receive fair compensation from AI companies."
"Fair compensation" in this instance will be determined by RSL Collective members.
"The RSL Collective is a nonprofit Collective Rights Organization that is guided by the priorities of its members – it negotiates on behalf of its members, but the definition of 'fair' will be determined by an open, transparent process that needs to be approved by members," explained Walther. "As a nonprofit, the RSL Collective passes all royalties to its members, minus any costs needed to operate the service."
RSL, according to its creators, supports a variety of licensing, usage, and royalty models, including free, attribution, subscription, pay-per-crawl, and pay-per-inference.
Since RSL is simply a standard that defines the criteria for admission, the challenge of separating human visitors from bots and forcing bots to comply with terms has been left to internet service providers, particularly content delivery networks like Fastly, Akamai, or Cloudflare, and custom crawler detection solutions.
"While there are certainly bad actors that will not follow the rules set by publishers and might need to be blocked by solutions developed by companies like Fastly and Cloudflare, we believe that the majority of large AI firms understand, and have publicly spoken, about the need for fair compensation for the publishing industry," said Walther.
"The nonprofit RSL collective exists to make this process dramatically simpler for AI firms by pooling licensing rights for millions of publishers and creators, the same way that large music distributors are able to license large music catalogs through music collective rights organizations." ®
Get our [23]Tech Resources
[1] https://rslstandard.org
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2aMNGcprfVMhPMUteye5ySQAAAE8&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://rslstandard.org/guide/license-servers
[4] https://rslstandard.org/guide/web-crawlers
[5] https://rslstandard.org/api#acquire-license
[6] https://datatracker.ietf.org/doc/html/rfc7235#section-5.1
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aMNGcprfVMhPMUteye5ySQAAAE8&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aMNGcprfVMhPMUteye5ySQAAAE8&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[9] https://rslstandard.org/press/rsl-standard
[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aMNGcprfVMhPMUteye5ySQAAAE8&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[11] https://rslcollective.org
[12] https://rslstandard.org/about
[13] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aMNGcprfVMhPMUteye5ySQAAAE8&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[14] https://www.theregister.com/2025/09/08/anthropic_settles_author_lawsuit/
[15] https://www.theregister.com/2025/09/11/appeals_court_blocks_trump_copyright_official_firing/
[16] https://www.theregister.com/2025/09/11/walmarts_bet_on_ai_depends/
[17] https://www.theregister.com/2025/09/11/openai_reportedly_on_the_hook/
[18] https://www.theregister.com/2025/09/10/ai_software_licensing_immature/
[19] https://www.theregister.com/2025/07/22/google_ai_overviews_suppress_search/
[20] https://blog.google/products/search/ai-search-driving-more-queries-higher-quality-clicks/
[21] https://x.com/jason_kint/status/1964380170741305846
[22] https://www.theregister.com/2025/07/01/cloudflare_creates_ai_crawler_toll/
[23] https://whitepapers.theregister.com/
So close, but then....
IGotOut
" it negotiates on behalf of its members, but the definition of 'fair' will be determined by an open, transparent process that needs to be approved by members,"
And it all falls apart here. Why should "members" decide how much I charge? A huge site might go, sure 10c a item is fine, knowing they will get 10s of thousands in revenue. But what about a small artist, that gets their designs ripped off, all for 50c, probably costing more in bandwidth than they make
Nope.
It should be the site owner that gets to choose, end of.
"Large publishers like the New York Times and social media sites like Reddit have negotiated specific deals that provide AI vendors with access to their content..."
This is why 'AI' is dumb.