Bot invasion increases with Google scraping the way, Cloudflare says
(2025/12/15)
- Reference: 1765822452
- News link: https://www.theregister.co.uk/2025/12/15/cloudflare_report_bot_traffic/
- Source link:
Global internet traffic grew by 19 percent during 2025, while nearly half of traffic now comes from mobile devices. A significant and growing portion also comes from bots, many designed to train AI.
In its [1]2025 Year in Review , Cloudflare notes that internet traffic worldwide grew by almost a fifth, with the increases coming in fits and starts. In fact, traffic was somewhat flat through mid-April and even experienced a mysterious dip before bouncing back sharply to 5 percent growth during May.
Most of the growth occurred after mid-August, however, accelerating rapidly to hit 19 percent by the start of December. This pattern is similar to that seen for the past few years, although growth peaked at about 17 percent last year.
[2]
According to Cloudflare, 43 percent of requests across the interwebs were from mobile devices this year, up from 41 percent in 2024. The balance came from "classic" laptop and desktop type devices. But despite the slight increase, the firm says it believes that mobile device usage has now effectively reached a steady proportion of the traffic.
[3]
[4]
When it comes to the most accessed internet services, Google remains top of the overall list, followed by Facebook, then Apple and Microsoft. For social media sites, X (formerly Twitter) has now slipped into sixth position, behind LinkedIn and Snapchat.
Few will be surprised that OpenAI is the most accessed AI service, with Anthropic in second position, ahead of Perplexity and Google's Gemini. Elon Musk's Grok languishes in ninth place, just ahead of China's DeepSeek.
[5]
AI bots made their presence felt this year, accounting for 4.2 percent of HTML request traffic as they trawl the web for content to be used in training models. In response, Cloudflare called for clear " [6]rules of the road " for AI bot behaviour, such as that companies should publicly disclose information about their AI bots, and that these should have one distinct, openly declared purpose.
Despite this, the highest bot traffic (over a quarter) came from Googlebot, which trawls for both search engine indexing and AI training, as does Microsoft's Bingbot, but on a much smaller scale.
Cloudflare says that 6.2 percent of the traffic that attempted to traverse its global network was automatically mitigated by its systems as potentially malicious, or else was blocked for "customer-defined reasons," which may mean rate-limiting requests or blocking all of the traffic from a given source.
[7]
Cloudflare saw a large increase in mitigated traffic during July, the company says, due to a very large DDoS campaign that targeted a single customer domain.
Meanwhile, the US remains, by far, the largest source of bot traffic, with 40 percent originating there. For comparison, the second largest source was Germany at 6.5 percent. Not all bots are malicious, the firm is keen to point out, and says it maintains a directory of verified bots that includes those used for search engine indexing, security scanning, and site/application monitoring.
[8]Cloudflare broke itself – and a big chunk of the Internet – with a bad database query
[9]Cloudflare blames Friday outage on borked fix for React2shell vuln
[10]Cloudflare suffers second outage in as many months during routine maintenance
[11]Cloudflare fesses up to config change that borked internet access for all
Cloudflare notes a jump in cyberattacks, leading to more than 25 record-breaking DDoS incidents. There were also plenty of outages with less malevolent causes.
One of the most notable incidents during the year was an [12]outage suffered by Cloudflare itself in November, triggered by a [13]change to the permissions of one of the database systems it uses for its operations. This caused the websites of many big brands, including The Register , to be unavailable.
The firm also suffered a [14]second outage earlier in December, this time blamed on a change made to how Cloudflare's Web Application Firewall parses requests.
Cloudflare says that nearly half of outages it observed this year were due to government activity. These were often attempts to prevent students from cheating on academic exams, in countries such as Iraq, Syria, and Sudan, although in Libya and Tanzania, the cause was civil unrest, while in Afghanistan, the Taliban ordered a shutdown under the excuse of preventing immorality.
However, cables being cut were also a major cause of internet disruption, in territories such as the United States, South Africa, Haiti, Pakistan, and Hong Kong.
Despite IPv6 being around for decades and the supply of IPv4 addresses being [15]declared exhausted several times, less than a third of requests from dual-stack capable systems were made over IPv6, Cloudflare claims. This is because of solutions like Network Address Translation (NAT) that have enabled network providers to stretch limited IPv4 resources by hiding lots of nodes behind a single external IPv4 address.
For those interested in seeing more detail, Cloudflare has posted it on a [16]Cloudflare Radar 2025 Year in Review microsite for your viewing pleasure. ®
Get our [17]Tech Resources
[1] https://radar.cloudflare.com/year-in-review/2025/
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2aUCTDlep7AKPD7pP5geG4AAAAAY&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aUCTDlep7AKPD7pP5geG4AAAAAY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aUCTDlep7AKPD7pP5geG4AAAAAY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aUCTDlep7AKPD7pP5geG4AAAAAY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://blog.cloudflare.com/building-a-better-internet-with-responsible-ai-bot-principles/
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aUCTDlep7AKPD7pP5geG4AAAAAY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[8] https://www.theregister.com/2025/11/19/cloudflare_incident_report/
[9] https://www.theregister.com/2025/12/05/react2shell_pocs_exploitation/
[10] https://www.theregister.com/2025/12/05/cloudflare_outage_again/
[11] https://www.theregister.com/2025/07/16/cloudflare_fesses_up_to_config/
[12] https://www.theregister.com/2025/11/18/cloudflare_outage/
[13] https://www.theregister.com/2025/11/19/cloudflare_incident_report/
[14] https://www.theregister.com/2025/12/05/cloudflare_outage_again/
[15] https://www.theregister.com/2019/11/25/ipv4_addresses_gone/
[16] https://radar.cloudflare.com/year-in-review/2025
[17] https://whitepapers.theregister.com/
In its [1]2025 Year in Review , Cloudflare notes that internet traffic worldwide grew by almost a fifth, with the increases coming in fits and starts. In fact, traffic was somewhat flat through mid-April and even experienced a mysterious dip before bouncing back sharply to 5 percent growth during May.
Most of the growth occurred after mid-August, however, accelerating rapidly to hit 19 percent by the start of December. This pattern is similar to that seen for the past few years, although growth peaked at about 17 percent last year.
[2]
According to Cloudflare, 43 percent of requests across the interwebs were from mobile devices this year, up from 41 percent in 2024. The balance came from "classic" laptop and desktop type devices. But despite the slight increase, the firm says it believes that mobile device usage has now effectively reached a steady proportion of the traffic.
[3]
[4]
When it comes to the most accessed internet services, Google remains top of the overall list, followed by Facebook, then Apple and Microsoft. For social media sites, X (formerly Twitter) has now slipped into sixth position, behind LinkedIn and Snapchat.
Few will be surprised that OpenAI is the most accessed AI service, with Anthropic in second position, ahead of Perplexity and Google's Gemini. Elon Musk's Grok languishes in ninth place, just ahead of China's DeepSeek.
[5]
AI bots made their presence felt this year, accounting for 4.2 percent of HTML request traffic as they trawl the web for content to be used in training models. In response, Cloudflare called for clear " [6]rules of the road " for AI bot behaviour, such as that companies should publicly disclose information about their AI bots, and that these should have one distinct, openly declared purpose.
Despite this, the highest bot traffic (over a quarter) came from Googlebot, which trawls for both search engine indexing and AI training, as does Microsoft's Bingbot, but on a much smaller scale.
Cloudflare says that 6.2 percent of the traffic that attempted to traverse its global network was automatically mitigated by its systems as potentially malicious, or else was blocked for "customer-defined reasons," which may mean rate-limiting requests or blocking all of the traffic from a given source.
[7]
Cloudflare saw a large increase in mitigated traffic during July, the company says, due to a very large DDoS campaign that targeted a single customer domain.
Meanwhile, the US remains, by far, the largest source of bot traffic, with 40 percent originating there. For comparison, the second largest source was Germany at 6.5 percent. Not all bots are malicious, the firm is keen to point out, and says it maintains a directory of verified bots that includes those used for search engine indexing, security scanning, and site/application monitoring.
[8]Cloudflare broke itself – and a big chunk of the Internet – with a bad database query
[9]Cloudflare blames Friday outage on borked fix for React2shell vuln
[10]Cloudflare suffers second outage in as many months during routine maintenance
[11]Cloudflare fesses up to config change that borked internet access for all
Cloudflare notes a jump in cyberattacks, leading to more than 25 record-breaking DDoS incidents. There were also plenty of outages with less malevolent causes.
One of the most notable incidents during the year was an [12]outage suffered by Cloudflare itself in November, triggered by a [13]change to the permissions of one of the database systems it uses for its operations. This caused the websites of many big brands, including The Register , to be unavailable.
The firm also suffered a [14]second outage earlier in December, this time blamed on a change made to how Cloudflare's Web Application Firewall parses requests.
Cloudflare says that nearly half of outages it observed this year were due to government activity. These were often attempts to prevent students from cheating on academic exams, in countries such as Iraq, Syria, and Sudan, although in Libya and Tanzania, the cause was civil unrest, while in Afghanistan, the Taliban ordered a shutdown under the excuse of preventing immorality.
However, cables being cut were also a major cause of internet disruption, in territories such as the United States, South Africa, Haiti, Pakistan, and Hong Kong.
Despite IPv6 being around for decades and the supply of IPv4 addresses being [15]declared exhausted several times, less than a third of requests from dual-stack capable systems were made over IPv6, Cloudflare claims. This is because of solutions like Network Address Translation (NAT) that have enabled network providers to stretch limited IPv4 resources by hiding lots of nodes behind a single external IPv4 address.
For those interested in seeing more detail, Cloudflare has posted it on a [16]Cloudflare Radar 2025 Year in Review microsite for your viewing pleasure. ®
Get our [17]Tech Resources
[1] https://radar.cloudflare.com/year-in-review/2025/
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2aUCTDlep7AKPD7pP5geG4AAAAAY&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aUCTDlep7AKPD7pP5geG4AAAAAY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aUCTDlep7AKPD7pP5geG4AAAAAY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44aUCTDlep7AKPD7pP5geG4AAAAAY&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[6] https://blog.cloudflare.com/building-a-better-internet-with-responsible-ai-bot-principles/
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/networks&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33aUCTDlep7AKPD7pP5geG4AAAAAY&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[8] https://www.theregister.com/2025/11/19/cloudflare_incident_report/
[9] https://www.theregister.com/2025/12/05/react2shell_pocs_exploitation/
[10] https://www.theregister.com/2025/12/05/cloudflare_outage_again/
[11] https://www.theregister.com/2025/07/16/cloudflare_fesses_up_to_config/
[12] https://www.theregister.com/2025/11/18/cloudflare_outage/
[13] https://www.theregister.com/2025/11/19/cloudflare_incident_report/
[14] https://www.theregister.com/2025/12/05/cloudflare_outage_again/
[15] https://www.theregister.com/2019/11/25/ipv4_addresses_gone/
[16] https://radar.cloudflare.com/year-in-review/2025
[17] https://whitepapers.theregister.com/
Re: FSCK Google
Dinanziame
I've had to block SIX site scrapers so far this month
I take it you mean scrapers that have nothing to do with Google... Unless you found that Google scrapers are not respecting your robots.txt file? To my knowledge they do, but maybe there's something new?
FSCK Google
you know that it makes sense.
I've had to block SIX site scrapers so far this month. It is a sign of the times, sadly.
Just block the whole IP address range or better still redirect them to a Google Cloud Service then they can suck on their own {you know what}