AI's 'Creepy' Crawlers Criticized by Linux Foundation's IT Infrastructure Director (kernel.org)
- Reference: 0185308120
- News link: https://linux.slashdot.org/story/26/08/30/2235224/ais-creepy-crawlers-criticized-by-linux-foundations-it-infrastructure-director
- Source link: https://people.kernel.org/monsieuricon/creepy-crawlies
> At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html....
>
> [W]hen a source is guaranteed to be LLM-free, like the entire history of kernel commits, it's worth its weight in gold as a source of training data... At the time of writing, linux.git is about 1.48 million commits. Oh, and we have about 922 forks of it on git.kernel.org — but don't worry, it's actually extremely efficient on the backend, since it's mostly the same objects in every fork. Unless, of course, you're a scraper, in which case you have, oh, several BILLION valid URLs you can scrape, only to get 922 duplicates of the same 1.48 million commits — which is exactly what the scrapers are doing. But wait, it's not just commits itself. You can also ask for patches, plain renders, diffs between arbitrary commits — cgit is happy to let you, which was perfect for the times when the Internet was for humans or crawlers who obeyed robots.txt, and is AWFUL right about now, because we can generate 1.2 METRIC BAJILLION valid URLs just for a single fork of linux.git.
>
> Initially, this was the solution — look through the logs, find out which IPs are obvious scraper bots, and fail2ban them. At first, this was easy, because the bots helpfully told you who they were via their user-agent. Then, they wised up and started pretending that they were random vanilla browsers. So, we started banning them by IP — after all, it's easy to figure out that an IP that is trying to grab every possible commit in a 8-year-old abandoned fork of linux is not really some lone Chrome on Windows user who is just furiously clicking every link that comes across their screen. The bots then started fanning out to entire subnets, but this was still meh, because obviously an IP coming from Google Compute is just pretending to be a Firefox user...
>
> And... that's when things turned really, really ugly. Suddenly, the crawlers were coming from millions of random residential or mobile IPs, all pretending to be random modern browsers. An IP like that would make 4-5 requests and then never show up in the logs again... They descended like swarms of locust, hit hard and fast until the system fell over and then moved on to the next target until you recovered. Then, they returned. Rinse. Repeat. They still do that — welcome to the wonderful world of "proxy SDK monetization." It's big business, and [2]your TV is probably doing it ...
>
> Today, git.kernel.org receives about 6M daily requests demanding to see random commits. Of these, 66% are still immediately batted away with the Anubis challenge, but 33% are now solving the math and getting through to the main site — because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge... With a bunch of generous assumptions, legitimate requests are only about 2% of git.kernel.org traffic — everything else are scrapers...
>
> [W]e're turning off features to reduce the number of crawlable URLs and to gate off actions that are expensive for us to run. Expect to lose some functionality, at least when accessing our resources anonymously. Trust me, we hate it just as much as you, but at this point it's a necessity... [W]e promise to still offer all of our data for download to anyone who asks. You just may have to jump through more hoops to get it.
>
> Sorry.
[1] https://people.kernel.org/monsieuricon/creepy-crawlies
[2] https://spur.us/blog/smart-tv-apps-residential-proxy-sdks
Dumb crawlers require dumb solutions (Score:5, Interesting)
I had a problem where AI scrapers were absolutely DETERMINED to fish out every possible query string from a search results page. Almost all of the query strings they tried were invalid due to shitty and dysfunctional string substitution. "&page=100" wouldn't be followed by "&page=101", it would be followed by "&pag&pag=1010" or something even more insanely half-baked, until the query strings were like 100+ characters long. It was the technological equivalent of watching HIV mutate in real time.
But the insane thing was that, aside from page number, they were always requesting info about the same other criteria: filtered by the same user, the same page type, and with no text string. So I just took those particular values and started banning logged-out users who requested that combination of criteria.
I figured I'd need to change my tactics in a couple of days once the botnet got bored of that particular page and moved on to requesting bogus entries for another user.
> MariaDB> select count(*) from ip_bans;
> +----------+
> | count(*) |
> +----------+
> | 671671 |
> +----------+
It hasn't.
I wasn't sure what an Anubis Challenge was... (Score:2)
It sounds a heck of a lot like Bitcoin's mining algorithm. But what really surprised me was where Google's AI tried to offer workarounds.
"How to Kill the Delay
Because Anubis relies entirely on client-side JavaScript execution to generate the cookie, you can use these methods to skip the wait:
Use a Browser Extension: Community-made extensions (like Anubis Bypass) automatically inject the required tokens or pre-calculate hashes instantly.
Pre-Mine with a Script: If you are scraping or scripting, you can write
Long-term solution: (Score:2)
Limit IP an address to viewing X commits a day. After that provide AI friendly instructions for how use git to render the desired page.
I don't expect 100% of people or bots to comply but the ones that don't will get nothing.
Re:Long-term solution: (Score:5, Informative)
> Limit IP an address to viewing X commits a day.
I guess you missed this part:
"Suddenly, the crawlers were coming from millions of random residential or mobile IPs, all pretending to be random modern browsers. An IP like that would make 4-5 requests and then never show up in the logs again"
Re: (Score:2)
Or just require logins, and require a modest microtransaction of $1 to create a login, or explicit administrator approval. Longtime actual users and contributors will be exempt through the administrator provision, and bots are free to create $1 accounts (which are bannable for abuse) thus funding the project. New actual users probably woud not be bothered if the rationale is explained.
If it's really worth it for them to crawl endless random URLs until the sun burns out its fuel, make them pay for the privil
Graduated overkill (Score:3, Interesting)
Perhaps it's time to apply the same methods here that helped deal with spam back in the day: start threatening the ISPs that don't deal with users infected with proxies with having all netblocks belonging to them blocked unconditionally. That works by shifting the complaints from people the ISP doesn't get paid by (the services being scraped) to people it does (it's users who can't access services). We even have the tech needed already: DNS-based RBLs. Condemn providers who host scrapers who don't provide useful user-agent strings or don't comply with robots.txt to the same fate.
We don't need regulations to do this. The services belong to the people who run them, and they've got pretty much free rein to take actions to protect themselves. Is it severe? Yes. Is it disproportionate? Also yes. But then, to avoid it they just have to do one easy thing: not be assholes. If they can't manage that, are we obligated to shoulder the costs of letting them freely be assholes? IMO, no.
Re:Graduated overkill (Score:4, Interesting)
> We don't need regulations to do this.
While I agree 100%, though I'd prefer to see criminal charges against CEOs for the DOS attacks they knowingly commit, with things like changing user agents to ignore blocks as evidence of criminal intent (which it is), while I agree 100%, there would be lawsuits that, in the current climate, have enough of a chance to succeed to result in temporary restraining orders while it takes years to resolve (not too different from when the same tactics were used in the spam wars).
Trying to discourage Layer6/7 functionality? (Score:1)
Trying to stop an application from creating a session to do, whatever? Whole corporations have been chartered to inform the rest of the Internet of their constitutional rights to do so.....
DDoS (Score:2)
Thats a DDoS attack.
Enshittification run amok (Score:2)
and mucking things up on a global scale
Cache/CDN? (Score:2)
I don't understand why they're rendering the same content over and over. Why not render it once and shunt these URLs to a CDN?
Someone else mentioned a $1 paywall and that seems like a good solution too.
And it will only get worse. (Score:2)
AI run locally is fun, AI allowed to interact with the internet is an increasingly spreading cancer. Just wait till it goes parabolic!
Tragedy of the commons (Score:2)
This is why we can't have nice things.