Top Online Sites Debate Cutting Off Google's Crawlers (futurism.com)
- Reference: 0184674244
- News link: https://news.slashdot.org/story/26/07/25/030246/top-online-sites-debate-cutting-off-googles-crawlers
- Source link: https://futurism.com/artificial-intelligence/google-greed-destroying-open-web
> [Some online publications] are now debating whether to cut Google off entirely, as the Wall Street Journal reports, illustrating an increasingly fraught relationship between the tech giant and the publishers that are creating content its AI models are regurgitating. According to the newspaper, prominent outlets including USA Today , Politico , the Economist , People , and Reuters are all reexamining their relationship with Google. Some are debating whether to continue to work with the tech giant at all... Even Reddit executives are reevaluating the company's [2]$60 million-a-year contract that allows Google to train its AI models on user-submitted content on the platform. They've similarly watched as Google's AI features discourage users from navigating to Reddit...
>
> Beyond pondering whether to cut Google off, other publishers have [3]resorted to suing the company , accusing it of illegally rehashing their intellectual property via AI summaries. It's an extremely undesirable position for publishers. By severing ties with the search giant, they could face even steeper declines in traffic. At the same time, there's seemingly little to gain from having Google's AIs crawl their content — and in the long term, it could guarantee their destruction.
Two interesting data points from the article:
"Last month, Cloudflare CEO Matthew Prince noticed that automated bot traffic had [4]overtaken human traffic for the first time in the internet's history."
" USA Today has seen its traffic from US users drop by almost half over the last year."
[1] https://futurism.com/artificial-intelligence/google-greed-destroying-open-web
[2] https://www.reuters.com/technology/reddit-ai-content-licensing-deal-with-google-sources-say-2024-02-22/
[3] https://www.yahoo.com/news/articles/rolling-stone-variety-owner-sues-165956458.html
[4] https://www.forbes.com/sites/josipamajic/2026/06/04/bots-now-outnumber-humans-online-and-the-internet-was-never-built-for-this/
A solution (Score:2)
A hastily thought-up solution would be to let Google crawl headlines and the bare minimum of information. This way, Google won't be able to provide all the information, so people who are really interested will actually visit your website.
Cutting Google off entirely will destroy most of your leads.
Re: (Score:2)
This is what I think too. Basically have www.mynewssite.com and content.mynewssite.com - the latter has the full articles and a no-robots policy, the former headlines, first two paragraphs, and (legitimate) SEO keywords from the article so people doing a search will come from it. The former redirects to the latter after a few seconds so people landing on it get to the right page ultimately.
Until Google ceases being abusive, I think unfortunately we're going to have to structure all new websites this way if
Re:A solution [must consider the money] (Score:2)
Much as I would like to encourage solution-oriented thinking, I don't think that approach offers much hope in these days of clickbait headlines.
On the other hand, I'm not sure it matters because the problems created by AI are getting so bad that the entire notion of websearch is failing. It's like a cesspool down there (which might remind you of old Kiyo fishing out the three one-yen notes, one of the most famous scenes in Japanese literature). It's simply that the value of the "top online sites" is too low
Re: (Score:2)
ChatGPT can search reddit posts within a few minutes of them being posted. I assume they have API access to all newly created posts. Maybe there is a solution like that which works.
Any solution that gets created will be abused by SEO (and SEO is supported and encouraged by Google). Most of us don't get much organic traffic from Google anyway.
Re: (Score:2)
Google stopped doing that (and using keywords and so on) because spammers are not honest. You can't rely one someone reporting correctly how relevant their content is, if everyone wants to be on top of the search result page.
Re: (Score:2)
using [meta] keywords. Slashdot seems to eat HTML.
IP is a farce (Score:2)
Intellectual property is a farce. That's why every IP except attribution has been created with an expiry condition of date, disuse, genericide or simply oblivion.
Feed it nonsense? (Score:2)
If it's the Google bot, maybe give it some alternative page which contains likely search terms but shuffles them and connects them with some sort of repeatable gibberish to poison the model?
Re: (Score:2)
If Google weren't able to filter spam, you would have only spam results. Don't you think the spammer try every trick they can come up with to poison Google to rank them better?
The Economist? (Score:2)
Experts in economics are struggling how to handle the biggest economic paradigm shift in decades? LLMs are so darn useful for society but none of the money is flowing back to you. Well time to think of something, propose something, Bernie Sanders has proposed LLMs are using the collective knowledge of humanity so the people should own 50% of each LLM company. Do you economists like that proposal? Guess what, if you take the Economist out of the LLM training the models might become more left leaning in thei
Bullshit (Score:2)
Google's feature don't discourage people from using Reddit. Reddit's shitty UI keep me from using it. I WAS an OldReddit viewer but they did away with that option as well. News site like USA Today are such an ad laden crap fests that it is no wonder people use any option to NOT have to view it. Don't get me wrong, I'm not defending Gurgle, they suck and we all know it, but they are a convenient scapegoat. I get 2 papers at my house, the local Yuma Sun which is like a High School paper, and the SF Chronicle.
Re: (Score:2)
Old Reddit is still there, but they started requiring accounts. And I fear they will abandon it eventually. Maybe Reddit then finally dies and makes place for something better, as I think they keep old.reddit around because most high quality posters use it, while the new page is mostly the clickbaity design for people with short attention spams wanting to consume instead of creating.
Web Rings (Score:3)
Block Google and bring back web rings, that's what I say.
Re: (Score:2)
While webrings were a good idea in their day, I could only imagine what would happen when one gets polluted with malware/misinformation and inevitably ends up training an AI. Webrings that are popular would be highly sought after for engagement, clicks, and malware.
Re: (Score:2)
Web Rings were a small web method. They neither scale nor are they spam resistant. If you want to network a few "We remember the good web" nerds it works, but if you want to make new sites popular it will result in a lot of business with paid links and so on until you get them to be the same crap as the current web is. There would also be pure spam webrings and webrings that look good until enough pages participate and then replace 90% of the sites with spam.
The old web worked because it was mostly noncomme