Google's plan to make User-Agent string even less useful breaks our device detection tech, says NetMarketShare
- Reference: 1604326758
- News link: https://www.theregister.co.uk/2020/11/02/netmarketshare_to_end_browser_stats/
- Source link:
In a [1]notice on its site , the California-based company said: "We are retiring NetMarketShare in its current form. October, 2020 is the last month of data. All billing for existing accounts has been stopped. All outstanding balances are being refunded."
The primary reason given is that "an upcoming change in browsers will break our device detection technology and will cause inaccuracies for a long period of time."
Secondly, it said that bot detection is "increasingly difficult" and that it did not want to accept "increasing levels of inaccuracy." It is promising to re-emerge at some unspecified future date.
[2]
The October 2020 report is to be NetMarketShare's last
The stats were based on data from users of its Net Applications analytics and social bookmarking products, which the company said gives it around 100 million valid sessions per month over thousands of websites. NetMarketShare explained that data gathered from web servers is [3]more accurate than surveys (self-selected) or ISP data (regional bias), though the figures are still vulnerable to bias from the make up of its customer base.
That said, NetMarketShare has reported similar figures to its rival [4]Statcounter . For example, for October 2020 NetMarketShare's browser stats had Chrome at 65.02 per cent, Safari at 18.25 per cent, Firefox at 3.39 per cent, and Edge (2.96 per cent) as its top four, where Statcounter had Chrome (66.12 per cent), Safari (17.24 per cent), Firefox (3.98 per cent), and Samsung (3.18 per cent), with Edge at 2.85 per cent. These are global market share figures on all device types.
Access to reliable statistics is valuable for both business and technical reasons, and is especially important to determine the extent of monopoly in browser, operating system, or device technology.
Guess who has the best stats?
There is no doubting who has the best statistics. Google benefits from the dominance of Chrome plus its 90-plus per cent search engine share, further bolstered by wide use of Google Analytics, which, according to [5]W3Techs , is used "by 84.1 per cent of all the websites whose traffic analysis tool we know."
Aside from bots, the problem faced by NetMarketShare is the declining value of the User-Agent string included in the header data sent by a browser to a web server. It refers to [6]this project from the Web Incubator Community Group (WICG), which includes the proposal that "Browsers should deprecate the User-Agent string over time, initially locking bits of its value, and ramping up over time to lock the entire string to something generic for the device type."
In its place, the group offers Client Hints (UA-CH), a set of name value pairs giving information including browser version, platform, CPU architecture, device model, viewport dimensions and pixel density, and device memory. Google is backing these proposals with [7]information for developers . There are a few other notable features of Client Hints:
Most Client Hints are not supplied by default, but only when the server requests them
Most Client Hints are not sent on the first navigation request when a user visits a site
Client Hints are only ever sent over secure connections (HTTPS)
Client Hints are not available to JavaScript, though some may be surfaced by a userAgentData API. JavaScript developers are expected to "examine other parts of the exposed API surface" to determine compatibility, or to do so in server-side code
As the WICG notes: "Sites that wish to provide market share analytics using UA-CH will need to inspect the Sec-CH-UA header, that is sent by default on every request, and keep a record of it."
[8]
Developers can experiment with Client Hints [9]here
Getting information from the browser has long been a problem since it is untrusted: a browser can send what it likes. Incorrect information is common, especially from browsers with small market share, as developers may otherwise declare a browser incompatible with their site or serve downgraded content. Another issue with providing detailed information is that it enables easier fingerprinting, where sites attempt to identify a user even when cookies are disabled or refused.
The current status of Client Hints is that it is in development in Chrome/Chromium. The latest [10]status report said that "its timeline will be finalized once US-CH ships," though freezing the User-Agent string can take place before Client Hints are available.
In a discussion [11]here , Google Developer Relations guy Yoav Weiss said the intent was to freeze the Chromium browser version in June 2020, but in fact Chrome 86 (the current version) still gives detailed version information.
Weiss said: "We expect this change to improve compatibility, as UA sniffing based on UA-CH is bound to be more reliable than the current status quo... we have other vendors on board with UA freezing, but not necessarily with the UA Client Hints mechanism, that is supposed to replace it.
"That can create a tricky situation, where developers would need to rely on the User-Agent string for some browsers and on UA-CH for others."
Hard for developers, and even harder for those trying to gather market share data. While there are good reasons for browsers to send less detailed information, making it harder for sites to collect browser statistics does happen to play into the hands of Google, which has many other ways to gather this information. ®
Get our [12]Tech Resources
[1] https://netmarketshare.com/
[2] https://regmedia.co.uk/2020/11/02/marketshare.png
[3] https://netmarketshare.com/methodology
[4] https://gs.statcounter.com/browser-market-share
[5] https://w3techs.com/technologies/details/ta-googleanalytics
[6] https://github.com/WICG/ua-client-hints
[7] https://developers.google.com/web/fundamentals/performance/optimizing-content-efficiency/client-hints/
[8] https://regmedia.co.uk/2020/11/02/clienthints2.png
[9] https://user-agent-client-hints.glitch.me/
[10] https://www.chromestatus.com/feature/5704553745874944
[11] https://groups.google.com/a/chromium.org/forum/#!topic/blink-dev/-2JIRNMWJ7s%5B1-25%5D
[12] https://whitepapers.theregister.com/
Re: Original Sin
It was historically used by the server to deal with the various and significant differences between different browsers and devices. The differences are significantly fewer these days and much of the decisions of how to render content are done client-side in javascript, but historically when the server rendered the HTML it needed to know what particular flavour of HTML to send you.
Re: Original Sin
Deep in the mists of time, there were a number of websites that would complain to Firefox and Netscape that their site 'only works with Internet Explorer'
Changing the UA string (pretty simple to do in Firefox) got rid of the complaint and in the vast majority of cases the site worked just fine.
One company I worked for from 1998 had some developers in California (I was sitting in my office in Princeton NJ running Netscape on an SGI Indy) where an internal set of pages written by said developers gave me an error that 'your browser does not support frames'. Quite amusing really as those were invented at Netscape IIRC.
Re: Original Sin
Isn't it still the way the server decides whether to redirect you to the website for mobile?
But also, let us remember this was used by Microsoft to [1]prevent people using Opera from accessing their MSN portal . That was back in 2003, when Microsoft thought that MSN could become a "trusted" part of the web, where everybody would want to go and they would control everything, and the rest of the web would become irrelevant.
[1] https://www.theregister.com/2003/02/06/msn_deliberately_breaks_operas_browser/
Re: Original Sin
Isn't it still the way the server decides whether to redirect you to the website for mobile?
Not if you have a modern "mobile first" web site. Having two different versions of a web site is a PITA for all concerned.
We expect this change to improve compatibility...
Surely if the standards were being followed, there would be no incompatibility? As Claverhouse suggests: the site should not need to know.
Re: We expect this change to improve compatibility...
You are new to IT standards aren't you?
Re: We expect this change to improve compatibility...
Trouble is which version of the standards ... 'the standard' is not so meaningful in a world where they are continually evolving and browsers implement features in different orders over time. See [1]https://caniuse.com/
Listing all of the features supported (maybe partially) is hard as there are so many of them. So, if you are at the bleeding edge browser+version+browscap-database might be what you have to do -- or resort to javascript magic.
Screen/viewport size ... should not be needed, CSS can handle that.
If the browser lies: well, in theory, if things go wrong the will blame the browser - in practice the user will blame the web site.
[1] https://caniuse.com/
Re: We expect this change to improve compatibility...
Trouble is which version of the standards ... 'the standard' is not so meaningful in a world where they are continually evolving ...
The browser should, rather than identifying itself, identify the most recent version of the standard(s) with which it is fully compliant.
... and browsers implement features in different orders over time.
Which they should not do ...
It's not as though anything in the standards should be a surprise to the major browser-writers, they all have representation on the bodies that agree those standards.
Re: Trouble is which version of the standards
Back in the day.. it is Microsoft and IE6 that defined the standards.
Not the standard is whatever Google decides that they would be...
This is clearly wrong but that's life and us plebs have to accept it.
An the browser designers just have to lump it. Google + Chrome controls the internet even if many of us would rather wash our mouths out with soap than use Google for anything.
Certainly time for google to be broken up. I'd go for a million tiny pieces.
Re: We expect this change to improve compatibility...
Relying on the User Agent string was always broken, servers shouldn't be using this before deciding which content should be served. Client hints are definitely a more granular approach and can be disabled by the browser. In general, they shouldn't matter but there are cases where they can help in content negotiation.
Apple probably won't bother with them as is the case with much of the newer stuff that it didn't come up with itself.
Well
I'll be blocking client hints as soon as Firefox or an EFF extension gives me the option.
I suggest a rename...
Not-so-generic Browser Fingerprhints.
The problem for Google is that user-agent switcher extensions and simple editing of UA are possible, making it harder for them to track and label individuals.
As always disguised as "for your benefit".
I change my UA randomly, makes no discernible diff in browsing experience.
Original Sin
I really don't see any justification for the original decision to let browsers declare their information to sites. If one lawfully accesses a site and it works what business is it of the site to see what one is using ?
That let in everything else.