The weird thing about this is that the only company I've seen doing problematic user-agent handling in recent years is Google themselves. They have released several products as Chrome-only, which then turned out to work fine in every other browser if they just pretended to be Chrome through the user agent. Same with their search pages, which on mobile were very bad in every non-Chrome browser purely based on user agent sniffing.
If you have the new Chromium-based Edge ("Edgium") installed: the compatibility list at edge://compat/useragent is really interesting.
Edgium pretends to be Chrome towards Gmail, Google Play, YouTube, and lots of non-Google services; on the other hand, it pretends to be Classic Edge towards many streaming services (HBO Now, DAZN, etc.) because it supports PlayReady DRM, which Chrome doesn't.
Other browsers support DRM too, but with different tradeoffs.
Chrome uses Widevine, but one of Chrome's philosophies is that you should be able to wipe a Chrome install, reinstall Chrome, and have no trace that before/after are the same person. That means no leveraging machine-specific hardware details that would persist across installs. "Software-only DRM", essentially.
Edge on Windows (and Safari on OSX) are able to leverage more hardware-specific functionality --- which from a DRM perspective are considered "more secure", but the tradeoff is a reduction of end-user anonymity (i.e. if private keys baked into a hardware TPM are involved).
Last I checked, Chrome/Firefox were capped at 720p content, Safari/Edge at 1080p, though it looks like Edge is now able to stream 4k.
Last time I used piratebay, I saw a lot of porn and malware/scam ads. I had to find and install a torrent client. Then I had to make sure I was downloading a movie that had enough seeders. And then I couldn't watch the movie until (and if) the download finished.
When I use netflix, I have a much better experience.
I know this is all anecdotal, but last time I used a torrent site, I found the movie immediately and it pulled the whole thing down in under 3 minutes. Could be that it was a newer movie and pretty popular. I do see a lot of older stuff that's not being seeded much anymore.
> When I use netflix, I have a much better experience.
If you're on Linux, you won't be able to stream at 1080p, let alone 4k. Netflix even went out of their way to disable workarounds that users developed.
Try "actual" content videos. You can press ctrl+alt+shift+D to open a debug overlay that shows the playback resolution.
The actual DRM limitations also vary by content (and region) - with some titles I get 720p on Linux while some other titles are limited to SD, while I get 1080p on Windows Edge on those same titles.
They may not have the same DRM restrictions as actual content? It's not that Firefox is not technically capable of rendering the videos, it's that it doesn't give the DRM the control that Netflix and rightholders want to require in exchange.
Some torrent clients support sequential downloading, which will be equivalent to streaming with most video formats. And obviously there's uBlock for the ads.
DRM on streaming and BluRays made it so that any usage outside basic consumption on prescribed devices is better served by illegal means.
You can have a similar experience with a private tracker and a seedbox. The content is curated (no malware), there's a larger selection, the quality is sometimes higher, and peers generally have better connections.
After you pick your torrent, it takes the seedbox a few seconds to download the content. Then you can stream your download using emby, vlc over http, or whatever you prefer.
With torrents you can get a film in minutes. With popcorn you are exposed the entire time you watch the film.
In Germany they monitor peer connections and send a payment demanding an out of court settlement. After two years they escalate to a court appearance in a remote town. If you don't show, you lose and they turn it over to debt collection.
I don't know this technology, but I'd really recommend using a VPN that provides a SOCKS proxy for your Bittorrent connections. Otherwise you're just announcing your IP address related to your torrent activities to the whole world.
I don't live in Europe but I canceled my Netflix the minute I learned that Reed Hastings is taking advantage of the coronavirus situation to increase Netflix's bottom line by limiting streaming quality to everyone in Europe after one phone call with a French politician.
There are plenty of torrent streaming and download clients that work just as well and are just as convenient as Netflix, without needing to rely on a central authority.
You need: 1. Usenet provider (this allows you to download the links) and Usenet indexer (this allows you to search for content you want because files like movies usually have obfuscated filenames). Example of the former is Frugal Usenet, example of the latter is Drunken Slug (free-ish) or DogNZB (paid). One of the many things you need to watch out for with providers is retention, the length of time they keep the files so you can download them. In general, the older content you find on the indexer, the less likely it's still going to be available. The downloading itself can be done in an automated way (for example with Sonarr and the like) or kind-of manual with tools like SABnzbd.
> one of Chrome's philosophies is that you should be able to wipe a Chrome install, reinstall Chrome, and have no trace that before/after are the same person.
Why would that be their philosophy? It sounds like some kind of privacy-motivated idea which seems contrary to Google’s typical philosophy. Or is it more about portability?
I am not sure about Edge specifically, but as someone who tries to use mostly open source software: Digital Rights Management (DRM) requirements often directly conflict with licensing related to open source software.
Not the only browser to support DRM. But the only browser to support PlayReady on Windows, which brings added security compared to what Widevine offers on Windows.
Another popular choice for high quality is Safari on macOS because it implements Apple's FairPlay.
If you still see pirated copies of shows marked with WEB-DL (rather than Webrip), there's a way of decrypting the content directly. I really doubt the methods that are used are public, though.
It allows websites to run arbitrary code blobs which interact with a hardware backdoor in your CPU with a higher permission level than the OS. With the multiple exploits that have been found in intel cpus and the Management Engine itself there is no way you should be letting any website do that.
Simply because Netflix uses Playready DRM for 4k streaming, which is even harder to bypass and requires WinRT API (?) to even able to use the recent version.
Currently only Microsoft itself even try to implement it on their own Chromium-based browser.
There are different kinds of DRM. Streaming websites allow different quality for different kinds of DRM. E.g. they allow best quality only for best protected DRM (which should use encryption all the way from Netflix webserver to your display). There's software DRM (decrypting stream inside proprietary blob) which is considered weaker, so you'll receive acceptable quality in Chrome. I don't know why Chrome did not implement the most secure DRM. Hopefully Microsoft will contribute their patches back.
Which makes sense, given that the list could change anytime (ideally, it wouldn’t even exist) and no developer should rely on Edgium identifying as Chrome or Classic Edge.
I had been thinking recently as I've been using Firefox more that Google maps had got clunky. With a little fiddling prompted by your comment, it turns out Maps sniffs specifically to reduce fluid animations on Firefox (and probably some other browsers).
Thnak you, this had been bugging me for a while. Looks like I'll need to permanently install a UA-switcher extension.
Yesterday I saw a HN comment saying you can add the (?|&)disable_polymer=1 parameter to the end of YouTube URLs to make the site much faster - iirc Polymer is extremely slow on Firefox only. This extension was also linked: https://addons.mozilla.org/en-US/firefox/addon/disable-polym...
Unfortunately there doesn't seem to be any workaround for ReCaptcha on FF. I generally end up opening the website in the GNOME or KDE (Falkon) browser which use something like WebKit/Blink - there it works on the first try every time.
Some advice about ReCaptcha, the audio test is way easier and usually only makes you do it once or twice (as opposed to the 5-10 times you usually have to do when you have disabled tracking). Sometimes it will say you aren't eligible or something, just refresh the page and it will let you try again.
I wonder what will be the point of these captchas if most people started avoiding them by installing extensions. If the end goal is to put significant resource constraint on the scrapper or bots, you can do it quite easily by running something to short the usage up for a few seconds if you think someone is suspicious.
I just use the contact site to send them a note to cut it out. There are better ways to prevent spam. Like my password that I just entered in the latest case. I'll go elsewhere in the next, they have competitors that won't be that different in price and easier to use.
Not the first time. For some reason Google really doesn't like people talking about the games they play with browser detection.
(That's not snark - I really don't get it. They don't appear to mind people talking negatively about a lot of other stuff they get up to. Maybe lingering antitrust fears from the 90's MS suit?)
I mean we are in 2020, none of this should be news, but then a lot of people only discovered them now. They have been doing this since early 2010s.
They have been a hypocrite from the start, but people got too consumed with its free Gmail, RSS Reader along with its Do no Evil they decided to trust them blindly.
The current browser, and Web Tech scenario is pretty much Google's way or the highway. So I am glad Apple kept Safari as the only option on Apple platform. Not allowing them to dictate everything.
I was hitting that too! I wondered what I was doing. I kept getting these weird fields. That fucking sucks. I'd like someone to explain google's point of view about why this is happening? I do override user agent on some systems so random websites work.
I wouldn’t really shame them on this one. Google docs and sheets uses contenteditable under the hood. Contenteditable isn’t specified or standardized anywhere and varies wildly between browsers, and sometimes between browser versions.
Oh wait...THAT might be the problem?? I've been having that issue too! I have to cut the cell content, delete the cell, and then paste the cell content back in in order for the row height to appear the same. It never even occurred to me to switch browsers, I thought it was an issue with Google Sheets.
Maybe. Still not switching away from Firefox though. I have Chrome installed because WebRTC is a lot better and teleconferencing needs have shot up recently, and because I lock my phone away in a Kitchen Safe and use 2FA with Authy the Chrome app, but Firefox is my daily driver :)
Except, that has been pointed out in the parent comments, it's not the application, it's a deliberate bug targeted towards a different browser(hence changing the user agent fixes it).
What versions of Firefox and Chrome are you seeing different behavior in Google Maps? I get the same experience in Chrome 80 and Firefox 74 on OSX Catalina.
Every single Google product is slower on Firefox and it’s hard to not call this malice and artificial. Many people check out Gmail and GMaps on Firefox and go back to Chrome because of their clunkiness on Firefox.
It could also be a passive, malicious de-prioritization of bugfixes for Firefox that would cause the same effect. It seems like this would be a more likely scenario.
Probably a lot of the other way around as well. Google employees likely use Chrome and Docs/Sheets heavily, so they get a rapid high-quality feedback loop with any bugs or performance issues affecting the Chrome devs and people around them.
If there was direct evidence it would warrant its own post, so I think your comment has been made in bad faith since people are talking about their own experiences.
That said; if it's possible to measure firefox/chrome performance (with altered user-agents) it would make for a good blog post.
Huh, I guess "hard not to call" could be interpreted as a disclaimer or qualifier meaning "I can't prove this, and it's only my opinion".
However, I interpreted it differently, to basically mean that because of objective fact, of all the explanations you can think of, only one of them is defensible. In other words, it's similar to saying "admit it, this is the only reasonable conclusion".
Slack, Skype, and Zoom video calls don't work in Firefox, even though WebRTC is an open standard. But Google Hangouts works perfectly.
I'm loath to give Google credit for something that ought to be standard practice, but of the major (key word) free video conferencing options, they seem to be the only one that's Firefox-compatible.
I prefer mu4e (mu for email) although ultimately I'm still using my gmail account just with another interface. I like the idea of ultimately just setting up a mail server but realistically its really really hard to switch everything at once when you have a lot of existing accounts so I'll probably be maintaining a gmail account forever.
I'm trying maps.me because unlike google it does offline walking directions but I haven't used it enough.
I really want to like duck duck go but it feels like google still provides better results.
In fact, just this week I thought about why Google doesn't make its mobile maps website fast again. It is such a pain to use it on older phones and I totally don't get, why it has to be that slow (doesn't matter if chrome or Firefox).
Here in Safari, Gmail is not only 10x buggier than it used to be before the redesign, it also uses at least 10x more client-side resources (CPU, network, ...). A handful of open Gmail tabs single-handedly use more CPU over here than hundreds of other web pages open simultaneously, including plenty of heavyweight app-style pages.
It’s hard to escape the conclusion that Google’s front-end development process is completely incompetent and has no respect for customers’ battery or bandwidth.
> The weird thing about this is that the only company I've seen doing problematic user-agent handling in recent years is Google themselves.
I frequently consume web articles with a combination of newsboat + Lynx, and it's astounding how many websites throw up HTTP 403 messages when I try to open a link. They're obviously sniffing my user agent because if I blank out the string (more accurately, just the 'libwww-FM' part, then the site will show me the correct page.
I'm pretty sure that the webmasters responsible for this are using user agent string blocking as a naive attempt to block bots from scraping their site, but that assumes that the bots that they want to block actually send an accurate user agent string the first place.
> I'm pretty sure that the webmasters responsible for this are using user agent string blocking as a naive attempt to block bots from scraping their site, but that assumes that the bots that they want to block actually send an accurate user agent string the first place.
That is exactly what they are doing, and it works really well.
We blocked user agents with lib in them at reddit for a long time.
Any legit person building a legit bot would know to fake the agent string.
The script kiddies would just go away. It drastically reduced bot traffic when we did that. Obviously some of the malicious bot writers know to fake their agent string too, and we had other mitigations for that.
But sometimes the simplest solutions solve the majority of issues.
> Any legit person building a legit bot would know to fake the agent string.
What, that's totally backwards. Anyone using a bot to do things that might get blocked by publishers fakes the string, legit purposes should really show who / what they are.
It actually is encouraging people to have useful user agents. By default most people end up with a user agent that's something like "libcurl version foo.bar.baz", which isn't actually a description of who or what they are; given the prevalence of curl, it really just tells you that it's a program that uses http.
We only blocked agent strings with "lib" in them. You could change the agent to "WebScraperSupreme.com" and it would have been fine (and in fact some people did do that).
Yes perhaps. But it caused problems for regular users like this fellow. I also have tried various 'download via script' for web pages for offline use. I thought I had a problem on my end, I never realized I could have been getting blocked.
Hard to argue with the economics of that mitigation though. The abuse:legitimate use ratio is probably pretty high. Getting rid of user agent strings will bring back the scaling problems, as they should probably be addressed directly.
> They have released several products as Chrome-only, which then turned out to work fine in every other browser if they just pretended to be Chrome through the user agent.
This seems like a pretty good reason in itself why they might be interested in phasing out User-Agents.
It's the exact opposite. Without User-Agents, sites need to depend on feature detection, and closing the feature discrepancy between Chrome and other browsers is more complicated than just spoofing your UA to get Google to serve you functioning versions of their products.
1. Our Chrome detection relies on the User-Agent header.
2. But people can just lie in the User-Agent header.
3. Let's get rid of it and use something that's harder to lie about.
Closing any feature discrepancy isn't a goal here, as far as I can see. The whole point is to lie to the user that a feature discrepancy exists when it doesn't.
You can make the argument that Google is free to do their browser detection however they want (and therefore doesn't need to solve this problem by eliminating User-Agents), but this is still an obvious example of the User-Agent header causing problems for Google.
I interpreted your parent's comment differently; namely, if Google's developers can't do User-Agent detection, then internally even they will have to improve how they develop (eg. via feature detection), making their products more compatible with other browsers.
Many people assume Google, as an upper-level business decision, purposely makes products work better on Chrome in order to vendor-lock users to the browser. Maybe that's true; or maybe it's developers being lazy and using User-Agent detection. Removing their ability to do so might actually improve cross-browser compatibility of Google products.
You can see Chrome devrels on Twitter expressing disappointment with Chrome-only web sites, saying that they raise the issue internally. Of course we have no visibility into what happens after that, but it's an indicator that you're right.
I’m guessing all of Google’s internal apps are only tested on Chrome, with plenty of Chrome extensions, which means that all of the developers have to use Chrome to make the tools work, and at that point, switching back and forth between different browsers is a pain so none of the other browsers get the love they deserve.
The attitude of “it works on Chrome, I don’t care about anything else” is fairly widespread anyway. Just to stem the tide a little bit I’ve been developing on Firefox and Safari first, and then checking Chrome last.
I got bitten before when I made a browser game, and then noticed that it was all sorts of broken on Edge, even though Edge supposedly had all the features I needed. It turns out that Edge did have all the features I needed, but I had accidentally used a bunch of Chrome features I didn’t need. The easy way out is to turn things off when I detect Edge. The hard way is to find all the broken parts and fix them. So nowadays, I don’t do any web development in Chrome.
At least in my part of the googleverse, we have automated tests running in all the browsers (even ie11).
But I'll admit I will also poke around outside of the tests, and I'll usually only be doing that in chrome, unless I've had a bug report about firefox in particular. And I'll only really open up Safari when I'm testing VoiceOver. ChromeVox just isn't good enough.
Oh, I’m sure there are automated tests. But if you have 50,000 developers using Google Docs in Chrome, they’re gonna submit some high-quality bug reports internally, whenever it breaks.
Considering that there's been an internal and external bug filed about the US states not being in alphabetical order on contacts.google.com for years, making it impossible to type 'new y' to get New York, I don't think raising it as an issue will help much.
I’ve started seeing alphabet employees use this as an excuse: “oh that happened on team x there’s nothing I could’ve done”. On small technical issues the excuse is fine - on large moral issues it does not work.
In large corporations politics are abound. If the Chrome division cannot get other divisions to behave through other means, this is fine.
You can see that they should have fought harder and escalated, but issues like this are probably not the ones most upper-middle management want to potentially damage their career for.
A fair number of websites will still block perfectly working features based on what OS you use.
Some examples I've seen using the latest Firefox on *BSD:
Facebook won't let you publish or edit a Note (not a normal post, the builtin Notes app). I think earlier they wouldn't play videos but they might have fixed that.
Chase Bank won't let you log in. Gives you a mobile-looking UI which tells you to upgrade to the latest Chrome or Firefox.
In these cases if you lie and say you're using Linux or Windows it works flawlessly.
In addition to these, I've also noticed outlook365 (use it for accessing work email) gives a very minimal interface to the point of being nearly unusable with a FreeBSD user agent. Switch to a more popular one and I get full functionality.
It's usually just that the company doesn't want to have to deal with potential support calls that come in about their website not working on x os in x browser, so they do the bare minimum to disable it - then if a user complains about it not working, they can plausibly deny supporting the os/browser configuration.
This takes more effort than letting it just work. People running an OS more obscure than Linux are not the type that are going to call asking for support. We are used to supporting ourselves.
Did I mention it's the same code as a working configuration?
I think it's more likely somebody did not know how to properly parse user-agent and they blocked more than they intended to.
I would guess they have built something into chrome that gets even more data that isn't user-agent based.
UA has a lot of limitations and is fairly easy to work around giving data to for power users. I would imagine Google didn't want to keep playing around with that.
> I would guess they have built something into chrome that gets even more data that isn't user-agent based.
Chrome includes a unique installation id in requests to Google owned domains. They don't need any cookies or user agents to guess who you are and best of all they don't have to share that information with their competition.
Not for a technical reason, but they had a reason: they provided no support or guarantee that Netflix would ever work on Linux + FF (Ubuntu + Chrome was guaranteed) and they didn't want any support calls for something that they wouldn't help people with anyway.
A lot of stuff gets blocked for this reason. The company doesn't want you calling them because HD video doesn't work on Firefox even though you pay for HD quality, they do not test or guarantee Firefox compatibility in the slightest and yet they have to talk to an angry customer now. It makes business sense to redirect people to supported use cases when you know your product probably won't work as intended otherwise.
You don't have to agree with the decision (and you can always cancel your membership if you do) but they had their reasons.
Exactly. This is going to turn into a game of whack-a-mole whereby we need to load the latest firefox extension that tricks websites into thinking we're using Chrome.
Or we could build for Firefox. There's always that.
Facebook also uses the user-agent string to determine which version of a site to send to someone. I installed a user-agent spoofer a while back and messenger.com would fail due to it every few refreshes (as evidenced by JS console).
To get Microsoft Teams to run in Chromium there used to be a user-agent hack to make it pretend to be Chrome. This was superseded by someone packaging up using Electron. And finally this has been superseded by Micrsoft themselves supporting Linux using something that looks and feels like Electron again.
So, basically, Microsoft using user-agent to detect Chrome....
I don't really understand how this will result in any real difference in privacy or homogeneity of the web. Realistically every browser that implements this is gonna offer up all the info the server asks for because asking the user each time is terrible UX.
Additionally this will allow google to further segment out any browser that doesn't implement this because they'll ask for it, get `null` back and respond with sorry we don't support your browser, only now you can't just change your UAS and keep going, now you actually need to change your browser.
And if other browsers do decide to implement it, they'll just lie and claim to be chrome to make sure sites give the best exp... so we're back to where we started.
> I don't really understand how this will result in any real difference in privacy or homogeneity of the web.
It does a little: sites don't passively receive this information all the time, instead they have to actively ask for it. And browsers can say no, much like they can with blocking third party cookies.
In any case I'm not sure privacy is the ultimate goal here: it's intended to replace the awful user agent sniffing people currently have to do with a sensible system where you query for what you actually want, rather than infer it from what's available.
Switching it from passive to active means you can count it towards https://github.com/bslassey/privacy-budget . Yes, sites can ask for all sorts of things, but if they ask for enough that they could plausibly be fingerprinting you then they start seeing their requests denied.
(Disclosure: I work at Google, speaking only for myself)
So why not still send limitied information by default in the User-Agent header, and if they ask for it, send more information in User-Agent header? (keep everything in one spot?)
The problem is that without User Agent sniffing in some circumstances there is no other way of working round a browser bug e.g. There are cases where browsers will report that it supports such feature using one of the feature checks but the implementation is garbage. The only way is to have a work around based on user-agent sniffing.
Sure a lot of developers abuse the feature but I fear this might create another set of problems.
> It does a little: sites don't passively receive this information all the time, instead they have to actively ask for it. And browsers can say no, much like they can with blocking third party cookies.
Lets run through that scenario:
sites that don't need this info still aren't gonna ask for it or use it. sites that want it will get it this way and even if you respond with "no" that's useful to them as well for fingerprinting and as a way to fragment features to chrome only. So, what's changed?
> sites that want it will get it this way and even if you respond with "no" that's useful to them as well for fingerprinting
To an extent, sure. But to follow the model of third party cookies, let's say client hints are used extensively instead of user agent and all cross-domain iframes are blocked from client hint sniffing. All the third party iframe is going to be able to detect is whether user has a client hint capable browser or not. That's a big difference from the whole user agent they get today.
The idea is that this won't be a Chrome-specific API. It's been submitted to standards bodies, but Chrome is the first to implement. For example, Firefox have said they "look forward to learning from other vendors who implement the "GREASE-like UA Strings" proposal and its effects on site compatibility"[1] so they're not dismissing the idea, they're just saying "you first".
If it's well designed, then the system will only be able to query for feature support rather than ask what browser is in use.
I have a feeling Google won't do it that way, because they intentionally gimp most of their apps on non-Google browsers for no reason other than to be dicks.
Also, last year, Vivaldi switched to using a user-agent string identical to Chrome’s because websites refused to work for Vivaldi, but worked fine with a spoofed user-agent string.
https://vivaldi.com/blog/user-agent-changes/
If companies like Google wouldn't abuse the user agent string to block functionality, serve ads, force their users to specific browser then companies like Google wouldn't have to use fake UA strings and then maybe companies like Google wouldn't have to drop their support.
As a web developer, I have very little trouble reading the User-Agent header.
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/71.1.2222.33 Safari/537.36
Sec-CH-UA: "Chrome"; v="74"
Sec-CH-UA-Full-Version: "74.0.3424.124"
Sec-CH-UA-Platform: "macOS"
Sec-CH-UA-Arch: "ARM64"
Sec-CH-Mobile: ?0
Isn't moving this information to a separate Sec-CH-UA headers going to make things _more_ messy? Especially if it's in _addition_ to the frozen User-Agent header?
Aren't we still going to have the issue with needing to fake even the new Sec-CH-UA header?
If we're going to freeze the User-Agent header, that's fine, but don't just move the unfrozen info to a separate header. Now you have 2 problems.
Very little trouble? That user agent says it’s Mozilla, then says it’s Windows 10, then says it’s Apple, then says it’s Gecko, then says it’s Chrome, then even though it said it was Windows it goes on to say it’s Safari.
The individual headers, on the other hand, tell you EXACTLY what the system and browser is.
Well that's quite rude, you don't know me at all so you have no idea how much experience I have. And it's also quite contradictory... if the string is only rarely to be parsed by developers, why would "any web developer worth their salt" bother memorizing every browser's user agent? Experienced web developers don't have anything better to do than memorize a useless string they rarely have to work with?
This is a good idea, and is something I've thought of for a while; the user agent header was a mistake from both a privacy and a UX perspective.
Ideally, web browsers should attempt to treat the content the same no matter what device you are on. There shouldn't be an iOS-web, and a Chrome-web, and a Firefox-web, and an Edge-web; there should just be the web. In which case, a user-agent string that contains the browser and even the OS only encourages differences between browsers. Adding differences to your browser engine shouldn't be considered safe.
Beyond that, the user agent is often a lie to trick servers into not discriminating against certain browsers or OSes. Enough variability is added to the user-agent string that a server can't reliably discriminate, but it still remains useful for some purposes in JavaScript and as a fingerprint for tracking.
Which brings me to privacy. It's not as if there aren't other ways to try and fingerprint a browser, but the user agent is a big mistake for privacy. It'd be one thing if the user-agent just said "Safari" or "Firefox", but there's a lot more information in it beyond that.
If the web should be the same web everywhere, then the privacy trade-off doesn't make much sense.
I agree, but this also is incredibly dependent on the major players (e.g. Google) not going off on their own making changes without agreement from other browsers...
There are still issues today where chrome, edge, and Firefox render slightly differently. I certainly agree user agent isn't terribly necessary, but it's literally the only hook to identify when css or JavaScript needs to change... Or to support people on older browsers (e.g. Firefox ESR). How can I know when I can update my website to newer language versions without metrics confirming my users support the new ES version?
I would argue simplifying the UA, product + major revision, maybe, or information relevant to rendering and JavaScript only
Thinking cynically, it could be a power-move by Google to strengthen their hold on the ecosystem.
Right now when they go out and make their own API changes without consensus (which already happens), it's possible to distinguish the "for Chrome" case and still support the standard. But if there were no User-Agent, and Google wanted to strongarm the whole group into something, and 90% of browsers are Chromium-based, devs will likely just support the Chromium version and everyone else will have no choice but to fall in line.
As far as I can tell, HTTP/2 is such a major improvement that no strong-arming is necessary. Speaking as a consumer of the web, as an individual who runs their own website, and at a developer working at a company with a major web presence.
The web suffers a ton from the “red queen” rule in so many different ways anyway—you have to do a lot of work just to stay in the same place.
But, is it really such an improvement ? Or is it just an improvement for Cloud provider that keep pushing the Kool-Aid ?
I still see a lot of contradicting benchmark and, apart from some Google Apps, personnally, I have not seen a lot of sites actually really leveraging HTTP2 (including push).
But maybe you did put and leverage HTTP2 on your own website ? At your company ? Did you use push ? Do you use it with CDN ?
Yes, unequivocally. It’s amazing, even without push. The websites that use it are faster, and the development process for making apps or sites that load quickly is much more sane. You don’t have to resort to the kind of weird trickery that pervades HTTP/1 apps.
> Or is it just an improvement for Cloud provider that keep pushing the Kool-Aid ?
I don’t see how that makes any sense at all. Could you explain that?
> But maybe you did put and leverage HTTP2 on your own website ? At your company ? Did you use push ? Do you use it with CDN ?
From my parent comment,
> Speaking as a consumer of the web, as an individual who runs their own website, and at a developer working at a company with a major web presence.
My personal web site uses HTTP/2. It serves a combination of static pages and web apps. No push. HTTP/2 was almost zero effort to set up, and instantly improved performance. With HTTP/2, I’ve changed the way I develop web apps, for the better.
My employer’s website uses every technique under the sun, including push and CDNs.
>> Or is it just an improvement for Cloud provider that keep pushing the Kool-Aid ?
> I don’t see how that makes any sense at all. Could you explain that?
I've seen a few CDN having a page loading a grid of image in HTTP/1 at page load, and then load the same stuff with HTTP/2 on a button click. It indeed shows you a nice speed up.
Except, when you block the first HTTP/1 load and start with loading with HTTP/2 first and flush cache between loads, the speedup vanishes. The test is disingenuous, it is not testing HTTP/2 but DNS cache velocity.
So, those type of website makes me rather cautious. And the test, for the small scale workloads I work with, have not been very conclusive.
Do you have serious articles on the matter to recommend ? Preferably not CDN provider trying to sell me there stuff.
> Except, when you block the first HTTP/1 load and start with loading with HTTP/2 first and flush cache between loads, the speedup vanishes. The test is disingenuous, it is not testing HTTP/2 but DNS cache velocity.
The demos I’ve seen use different domain names for the HTTP/1 and HTTP/2 tests. This makes sense, because how else would you make one set of resources load with HTTP/1 and the other with HTTP/2? This deflates your DNS caching theory.
I didn’t rely on tests by CDNs, though. I measured my own website! Accept no substitute! The differences are most dramatic over poor network connections and increase with the number of assets. I had the “privilege” of using a high-RTT, high congestion (high packet loss) satellite connection earlier this year and difference is bigger.
What I like about it is that I feel like I have more freedom from CDNs and complicated tooling. Instead of using a complicated JS/CSS bundling pipeline, I can just use a bunch of <script>/<link>/"@import/import". Instead of relying on a CDN for large assets like JS libraries or fonts, I can just host them on the same server, because it’s less hassle with HTTP/2. If anything, I feel like HTTP/2 makes it easier to make a self-sufficient site.
Finally, HTTP/2 is so dead-simple to set up on your own server, most of the time. It’s a simple config setting.
Maybe web publishers need to let go of this idea of pixel-perfect rendering and identical JavaScript behavior across browsers, and instead just worry about publishing good content. The web is not Adobe InDesign or Photoshop. At its essence it is a system for publishing text and hyperlinks that point to other content. Get the content right and don't worry so much about whether the scrollbar is 2 pixels thick or 3.
I, too, remember how this opinion was repeated as nauseam about a decade ago. It didn’t make much sense then, and even less now.
Nobody today expects identical rendering: people are used to responsive websites, native widgets etc. The problem people are actually experiencing (far less now than in the past) were more serious, such as z-axis ordering differences resulting in backgrounds obscuring content.
For JavaScript, I struggle with how non-“identical behavior” would express itself, except as a blank page and a small red icon in devtools.
If I'm connecting to a site with Lynx, I sure as heck don't want them to try to serve me some skeleton HTML that will be filled in with JS. Because my browser doesn't support JS, or only supports a subset of it.
User Agent being a completely free form field is the real mistake IMO. Having something more structured, like Perl's "use" directive, might have been better.
The problem with services using the user-agent to determine whether or not to allow a client access to a resource outweighs any benefit. I'm in the "it was a mistake to include this in the spec" camp.
One problem with this is that browsers don't behave the same. For example, iOS Safari prevents multiple pop-up windows to be opened by a single user interaction. Each one requires clicking back to the original page and allowing the popup. Now you might say, "Why would you ever want to do that?" But there are always going to be edge cases—in this case it's an integral part of one of the features of autotempest.com. But that's just one example. And the only way we can detect whether that behaviour is going to be blocked is by checking the UA.
I can understand why this is a good thing for privacy. Like many things to do with security on the web though, it's just a shame that bad actors have to ruin so many things for legitimate uses. (The recent story on Safari local storage being another example of that...)
That just makes things harder for those who wants this information. Still you can fingerprint browser by features and API support, but it requires javascript now and up to date library with recent features support check. I mean that it doesn't prevent obtaining this information, it's still available for big players who has big data
At my employer we are using UserAgent to detect the browser so that we can drive SameSite cookie policy for our various sites (e.g. IE11 and Edge, which we still support, doesn't support SameSite: None).
There are a variety of scenarios where this comes up (e.g. we ship a site that is rendered, by another vendor, within an iframe; so we have to set SameSite: None on our application's session cookie so that it's valid within the iframe, thus allowing AJAX calls originating from within the iframe to work based on our current auth scheme.. BUT only within Chrome 70+, Firefox but NOT IE, Safari, etc).
Just providing this as an example of backend applications needing to deal with browser-specific behavior, since most of the examples cited in other comments are about rendering/css/javascript features on the client and how UserAgent drives that.
The User Agent Client Hints API looks like a very early draft. I could not see any proposed timeline for implementation or estimate of when this might become a supported standard.
I would not personally rely on this as a substitute or replacement for User Agent by September (Google Chrome 85).
We are in the same boat. Certain browser/OS combinations don't handle Same-Site correctly, so we are using UA sniffing to work around their limitations by altering Same-Site cookie directives for those browsers. We will likely have to look at some other mechanism for dealing with nonconforming Same-Site behavior.
These days, it feels like the sole use of User-Agent is as a weak defence against web scraping. I've written a couple of scrapers (legitimate ones, for site owners that requested machine-readable versions of their own data!) where the site would reject me if I did a plain `curl`, but as soon as I hit it with -H "User-Agent: [my chrome browser's UA string]", it'd work fine. Kind of silly, when it's such a small deterrent to actually-malicious actors.
(Also kind of silly in that even real browser-fingerprinting setups can be defeated by a sufficiently-motivated attacker using e.g. https://www.npmjs.com/package/puppeteer-extra-plugin-stealth, but I guess sometimes a corporate mandate to block scraping comes down, and you just can't convince them that it's untenable.)
Preventing scraping is an entirely futile effort. I've lost count of the number of times I've had to tell a project manager that if a user can see it in their browser, there is a way to scrape it.
Best I've ever been able to do is implement server-side throttling to force the scrapers to slow down. But I manage some public web applications with data that is very valuable to certain other players in the industry, so they will invest the time and effort to bypass any measures I throw at them.
As a person who scrapes sites (ethically), I think it's impossible or pretty damn near impossible to prevent a motivated actor from scraping your website. However, I've avoided scraping websites because their anti scraping measures made it not worth the effort of figuring out their site. I think it's still worth for do minimal things like minify/obfuscate your client side JS and use some type of one time use request token to restrict replay-ability. The difference between knowing that I can figure it in 30 minutes vs 4 hours vs a few days is going to filter out a lot of people.
Of course, sometimes obfuscating how your website works can make it needlessly more complicated, so it's a trade off.
Checking the user-agent string for scrapers doesn't work anyway. In addition to using dozens of proxies in different IP address blocks, archive.is spoofs its user agents to be the latest Chrome release and updates it often.
Meanwhile, you can still use youtube.com/tv to control playback on your PC from your phone—but only if you spoof your User-Agent to that of the Nintendo Switch [1]. Sounds like they are more interested in phasing out user control than ignoring the header entirely.
Yes. I firmly believe this is an attack on user control.
For example, I believe they REALLY want us to use the youtube app:
- Viewing youtube.com on a new iPad pro, Goolag lies and says "your browser doesn't support 1080p."
- Ok, change to desktop version in app. Goolag once again lies and says "your browser doesn't support full screen." They also lie and say they've redirected you to the "desktop version", and nag you with a persistent banner that you should return to the safety of the mobile website.
- Ok, change to "request desktop version" via user agent. Full functionality. Full screen is DEFINITELY possible with a javascript bookmark. 1080p+ is DEFINITELY possible. Ads blocked in browser.
If I were to use the app, they would have FULL CONTROL.
This feels very ivory tower. It reminds me of the "You should never need to check user agent in JavaScript because you should just feature detect!!". Well in the real world that doesn't work every time.
The same is true for server side applications of user-agent. There are plenty of non-privacy-invading reasons to need an accurate picture of what user agent is visiting.
And a lot of those applications that need it are legacy. Updating them to support these 6 new headers will be a pain.
Chrome will support the legacy apps by maintaining a static-user agent. It just won't be updated when chrome updates. If you want to build NEW functionality that where you need to test support for new browsers, you do that via feature detection.
Interesting. We don't use UA to track customers, but it has been invaluable information for trying specific bugs. Eg, twice in the past 2 months, I've had to fix weird bugs that didn't make sense. The only way I was able to solve them was to look for patterns in which browsers and versions those who reported the bugs were using. Both turned out to be to do different iOS Safari cookie related bugs that only occurred in specific versions. Without logging the UA there would have been no way I would have been able to discover those bugs and create workarounds for those iphone users.
I'm all for preventing tracking, but I can't imagine a time where all browsers behavior so similarly that we won't have to write workarounds for browser bugs and differences. As a developer I can't imagine caring about Edgium vs Chrome, but it's important to know what the underlying engines are.
Why not getting rid of the `User-Agent` completely?
Try browsing the web without any UA header for a week or two, and you'll understand. You get blank pages, strange server errors, and other weird behaviour --- almost always on very old sites, but then again, those also tend to be the sites with the content you want. Using a UA header, even if it's a dummy one, will at least not have that problem.
(I did the above experiment a long time ago - around 2008-2009. I'm not sure whether sites which expect a UA have increased or decreased since then.)
I agree with getting rid of all that new noise, however.
I mean at first this change will cause all of these errors too until the servers migrate from user agent string to device hints. Getting rid of UAs would actually force meaningful update rather than migration.
Getting rid of UAs would actually force meaningful update rather than migration
...and effectively block access to a bunch of existing content on the Internet, still very valuable, whose owners may not have the effort to spare to make any changes.
The browser isn't for "regular" websites, it's for all websites.
And believe it or not, there are crazy JavaScript bugs that are OS-dependent.
I remember when I was writing a library around the audio API, and the ways it behaved on Chrome were different across Macs, Windows and Android. Detecting the OS with the user-agent string was literally the only way to build code that would work.
Now I've never personally come across that for the CPU architecture, but I certainly wouldn't be surprised if there were behavioral differences between 32-bit and 64-bit processors somewhere that affects some JavaScript function or HTML5 call somewhere.
Yeah, aren't we just making things a lot more messy? Especially if we're not planning on removing the User-Agent header?
This pattern keeps repeating itself, freeze "Mozilla/5.0", start changing "Chrome/71.1.2222.33", freeze that, start changing "Sec-CH-UA", etc. Browsers will start needing to fake "Sec-CH-UA" to get websites to work properly, etc.
I can understand including the browser and version (to work around bugs that are not detectable with feature detection), the OS, OK I guess there are also a few OS-specific bugs?
> Blocking known bots and crawlers
Currently, the User-Agent string is often used as a brute-force way to block known bots and crawlers. There's a concern that moving "normal" traffic to expose less entropy by default will also make it easier for bots to hide in the crowd. While there's some truth to that, that's not enough reason for making the crowd be more personally identifiable.
This means that consumers of the Google Ad stream have one less tool to identify bots, and will pay Google for more synthetic traffic, impressions and clicks; this could be a huge revenue boost for Google. A considerable amount of their traffic is synthetic. I doubt this was overlooked.
does this mean there will no longer be a way of determining if the device is primarily touch (basically all of "android", "iphone" and "ipad") or guesstimating screen size ("mobile" is typical for phones in the UA) on the server?
Presumably you'll grab the dimensions (could cache after first load) and then render dynamically based on that. If you're doing some sort of if statement on the server to deliver content based on screen size you're probably doing it wrong. Obviously I can't speak for every mobile user, but for myself, it's infuriating to have a completely different set of functionality on mobile.
> If you're doing some sort of if statement on the server to deliver content based on screen size you're probably doing it wrong. Obviously I can't speak for every mobile user, but for myself, it's infuriating to have a completely different set of functionality on mobile.
there's not a "right" and a "wrong" here; it's about trade-offs.
you're either stripping things down to the lowest common denominator (and leaving nothing but empty space on desktop) or you're wasting a ton of mobile bandwidth by serving both versions on initial load (the most critical first impression).
you frequently cannot simply squeeze all desktop functionality from a 1920px+ screen onto a 320px screen - unless you have very little functionality to begin with. Amazon (or any e-commerce/marketplace site) is a great example where client-side responsiveness alone is far from sufficient.
https://www.walmart.com/ does it okay, but you can see how much their desktop site strips down to use the same codebase for desktop and mobile.
browser feature detection is the way grown up developers have been doing this for several years now. user agent sniffing is dumb because it bundles a ton of assumptions with a high upkeep requirement, all wrapped up in an unreadable regex. It's been bad practice for ages; I'd be surprised if that's how Amazon is doing it still.
the realistic answer to this line of questioning is : "we don't care about the edges because they constitute such a small percentage of the user base."
Can't happen soon enough. As a frequent user of various non-mainstream browsers i'm sick and tired of seeing "your browser isn't supported" messages with download links to chrome/etc. At least in the case of Falkon it has a built in user agent manager, and I can't remember the last time flipping the UA to firefox/whatever actually caused any problems. Although, i've also gotten annoyed at the sanctimoniousness web sites that tell me my browser is to old because the FF version I've got the UA set to isn't the latest.
I log this for coarse statistics about what our user base is running, but that is about it.
The good news: IE use is down over the last year to only about 40%.
The bad news: the growth elsewhere is all Chrome, with less than 1% Firefox or Safari. There’s a tiny sprinkling of Edge, as well, but I forget the numbers on that.
Our users are state and county offices and medical facilities, rather than private individuals, so the users are somewhat captive to whatever their organization mandates.
The only browser detection we do is in client side scripting to detect if the browser can directly display a PDF inline (or not, in the case of IE11)
I would much prefer a new version of the user-agent string. Normalize basic information (like OS and browser versions) without revealing too much (build numbers).
That would let servers still get necessary info without having to run even more javascript. It can just be in querystring format to simply parsing on both client and server.
Any user agent string will eventually be forced down the same path. Web sites use them to deny content. And the browsers will continue to try to match more patterns so their users see the content.
As long as they exist, I can see no escaping this arms race.
Google wants to create a walled-garden net. Goog-net. All ads, shopping, videos, documents, email, locations, articles flowing through THEIR protocols and THEIR servers and THEIR fiber. No possibility of blocking ads they don't want blocked. No URLs. No Agent strings. No user control. Only user consumption.
If they have to allow some small chump players to have a piece of this cake (a la: "Oh trust us, AMP is an 'open' protocol and anyone can host it. it's not just for our own benefit in the end. Trust us.") in order for the entire population to accept their changes bit by bit, so be it in their eyes. They know we would reject an outright takeover.
This is OK...I guess? I mean it's great to get rid of that overloaded carbuncle of user-agent, but that will just lead to a new round of interpreting "hints". shrug
Google is a serial abuser of user-agent already so this is somewhat ironic.
So Google found good way to fingerprint users without user agent and found that a lot of user agents are forged and this stopped working anyway. It's time to switch to API support forging.
User-agent is super useful to human people. But corporate people don't have a use for it. They will get that information via running arbitrary code on your insecure browser anyway. So, because mega-corps now define the web (instead of the w3c) this is life.
But it doesn't have to be. We don't have to follow Google/Apple-Web standards. Anyone that makes and runs websites has a choice. And every person can simply choice not to run unethical browsers.
For what? Honest question. You have to be like a 5th-level user agent wizard to make any sense of user agent strings, since every browser now names every other browser. How do you do anything useful with this in a way that's forward-compatible?
Not sure why you are being downvoted since your statements are correct.
Few advertisers rely on user agent for ad targeting since it can be easily mocked with each HTTP request. It is used for fingerprinting, sure, but from my experience, mostly as a way to identify bot traffic.
It is also true that the advertisers that fingerprint people rely on JS that executes WebGL code in order to get data from the machine.
Finally, you are right that it doesn't make sense that a company like Google dictates these standards since they have a conflict of interests worth almost a trillion dollars.
Unfortunately they are either unethical or have other problems (or most commonly, both); I have made suggestions how to make a better one. See other comment elsewhere they explain
This is insane. You know, no HN post to a google blogspot site works for me because these jerks are the only ones that discrinate on UA?
Google engoneering is Sooooooo disconnected from the rest of the world, I think we need legal regulation to stop them from doing stuipd things like this. Do they have any idea how many things need it?
HNers with a position of power at work, I plead with you: please advocate banning of Chrome at work and replacing it with any one of the webkit based alternatives or firefox. These people are insane. Every month I hear of some ridiculous thing. They took out navbar url parameters, add links to in page words, now this!
For those who think this is good for privacy...it is not! This is the same old sneaky ass evil thing they do. UA can be used to finger print you but it's very easy to set a generic user agent. Actually, if you look at user agents most of them have the latest string for Chrome, IE or firefox so it isn't useful without a whole lot of other details correlated with it. You know what the exception is? Android and iPhone browsers that incluse your device make and model in the UA and apps that includes whole lot more like facebook's apps.
Do you know what a "flexible" api like the one they're talking about allows? More fingerprintable data points! The fact that you even use that api is a privacy issue. Let's sat clienthints allows for 10 different variations of responses from clients,your specific client details might have just 3 things different from the mean and bam, now they can track your specific device. With UA,all versions of a client have the same exact detail and most people need an extension to change it, so it makes it much less easy to finger print.
This is the same ol sneaky bait and switch Google pulls. The content of your UA is not the privacy concern (although it contains too much at times) , it is the fact that it can be correlated with timing info,IP (especially v6),and if they already know your UA they will also use client default http header options to identify and track you without consent.
Any idea on how to identify devices then? We currently check the user agent to to send a new code when an user logs in from a new device. How would you do this without user agent?
gp is likely asking about how servers decide to redirect to the m.* version instead of the desktop version (or in some cases serve a different mobile version under the same domain), in which case, yes, it’s usually user agent sniffing.
UA strings have never been an accurate indication. If you're not using JS, then you probably have no reason to be sniffing the UA string to detect browser features, since most of those features are JS-related anyway.
It's an upgrade for the people who actually need to get an indication of the supported features and APIs of the user's browser. Otherwise, you should be using media queries.
One exception: you might want to user sniff IE and serve a completely different version due to all the CSS problems. (I know you can use IE-only comments too, but I’ve been in the situation where making a modern version simultaneously IE9-compatible was just too frigging maddening.)
Detecting dumb search crawlers, that don't support major features required for my webapp, and displaying a fallback splash has been the only reasonable way I've found.
If you want to just change the styling and layout of the page depending on the user's device, then you can use css's media queries[0]. But if you want to serve two totally different pages (one for mobile and another for desktop), then I don't see how it can be done without JS or reading the user agent.
Not usually, no. CSS media queries are used to format according to display size. But as a sibling here has indicated, client hints will replace the user agent here.
I thought it used Javascript to detect screen size. At least it should react to resize events and if the dimensions are something that align with mobile, it should switch to mobile mode.
In a lot of cases you shouldn't even use Javascript for this, responsive layouts can be built using CSS media queries based on viewport size.
More advanced webapps might occasionally need to do something fancier than that if the mobile vs desktop functionality is (for some reason) substantially different instead of just rearranged.
> While removing the User-Agent completely was deemed problematic, as many sites still rely on them, Chrome will no longer update the browser version and will only include a unified version of the OS data.
So, nearly all of the information that makes User-Agent strings problematic will remain. They're just phasing out precise version information.
So basically, Google shat their own bed and is just now beginning to realize that it stinks. Attempts to invent the universal internet user toll booth are back on track.
Stupidity. user-agent spoofing is a fact of life for many projects. Whatever feature they're going to come out with to replace UA will be spoofable too soon enough.
Annoying, as I just added a user agent based workaround for another Chrome compatibility problem (the increased security on same-site cookies, which can't be handled in a compatible way with all browsers).
As usual, this will fuck up the users, and not the techy nerds making such decisions, but the average joe because things on the internet will be broken for them.
How will things be broken? Google is not removing the user agent, they're just freezing it. So all sites that currently depend on the user agent will continue to do just fine. New sites can use client hints instead, which are a much more effective replacement for user agent sniffing.
This solution very specifically places the burden on "techy nerds" and not users, so I'm not sure where you're coming from.
Right, using user agent on the client side has been unsalvageably broken for a long time. Other things, like checking the existence of window.safari or window.chrome are more reliable.
For the server side, I’m not too aware of too many cases it’s useful other than analytics, and there is too much info leakage and fingerprinting happening anyway.
So killing user agent doesn’t really seem user-hostile, save for the fact that the company doing it has near monopoly market share and doesn’t need to provide a user agent, as it’s assumed that everyone is writing code to run on Google’s browser. In that sense it’s a flex.
This last year I've been noticing things breaking on the Internet for me here and there. I'm a Firefox user. This really wasn't the case in most of the past decade.
This kinda reminded me of the late 00's. It was quite common that the odd government or enterprise website was IE6 only.
I use Safari with no plugins. Even Disney World has a broken website for buying tickets for me. The web is breaking because it's gotten way too complex and the fight against trackers is leading to random failures of things that used to work.
The web is breaking because we are reaching the point where developers are able to assume WebKit/Blink and get away with it. It is imperative that technical folk adopt Firefox to hold back the tide.
Which is a shame, but I would lay the blame squarely at the feet of the team who built a checkout that throws errors when their analytics events don't fire. QA should really include a manual run w/ an adblocker...
Thats true, but its quite likely that simply testing on a couple available browsers and avoiding browser specific checks means that safari (and other non mainstream browser) users will be fine.
There aren't really that many actual standards compliance differences between most browsers, the real problem is all the undefined garbage they are forced to run. Back when I ran a html/css/javascript validator in my browser it frankly shocked me the number mainstream sites that weren't even delivering valid html/css/javascript. In my experience developing a pretty dynamic web site (actually it was a management front-end for a rather complex application) most of our browser differences were caused by bugs that went away simply by providing correct code.
I had Build-A-Bear not work for me on Firefox at the checkout process. Had to switch to Chrome to make the purchase. But aside from that, I typically don't see any issues.
Seems they considered this issue and created a work-around:
>While removing the User-Agent completely was deemed problematic, as many sites still rely on them, Chrome will no longer update the browser version and will only include a unified version of the OS data.
Instead of relying on a user agent which doesn't tell the entire story web site developers will need to check whether or not a feature exists in a browser before using it.
