Introduction
In the last post I introduced the basics of browser and device fingerprinting and noted just how much information is available to any website or third-party tag embedded in a served page. The intention was to allow websites to optimize the user experience for the specific combination of device, operating system, browser, screen size, and more on a given viewer’s device. However, the amount of information available as a result of this open information sharing allowed for the identification of a specific individual user/user agent a majority of the time. This allowed for an alternative and very powerful form of cross-site tracking independent of cookies and other techniques.
The amount of information can be measured in terms of the concept of entropy that evolved from information theory, which you can think of as a meta-descriptor that tells you how many bits of information is needed and/or available to provide a unique identification. In this case, Eckersley in his seminal paper on fingerprinting estimated that the user agent alone contains 10 bits of information, or 210 (1,024 bits). That means that only 1 in 1,024 random browsers visiting a site are expected to share the user agent header. Add a few other features like screen resolution, timezone, and browser plugins (among others) and that number goes to 18.1 bits of information. That means only 1 in 286,777 other browsers will share its fingerprint.
That number may not seem large, but 286,777 unique visitors/day equates to several million unique visitors per month (for example if an average user visits twice a month that would equate to 4mm unique visitors). That means in one month on average 15 browsers would have the same fingerprint. Let’s take CNN, with 767.4 million viewers per month. If all those were unique viewers (which they aren’t), then that would mean on average only 2,675 browsers visiting the site would share a fingerprint with at least one other browser. That is few enough that fingerprinting becomes extremely useful in identifying individuals for marketing purposes, exactly what the Privacy Sandbox and other privacy-first technologies are trying to prevent.
In this post, we are first going to delve more deeply into the industry’s early response to limit the amount of information available for fingerprinting. Then we will explore Google’s specific responses: Client Hints Infrastructure and User Agent Client Hints. Even today, Safari or Firefox do not support Google's approach.. Edge does, as it is built on Chromium.
Early Responses to Browser Fingerprinting
It took almost 10 years for the OS and browser owners to deal with fingerprinting information in the user agent header. Mozilla was first mover in January, 2020 in Firefox 72. They then made similar changes to Mozilla on Android in Firefox 79 in July, 2020. Apple followed suit in September 2020 on both MacOs and iOS 14. The main changes fell into three categories:
- Freezing at the major browser version. Browser version no longer showed the minor version. So instead of 79.0.1 the user agent string was limited to 79.
- Hiding device-specific details. The UA string no longer provided detailed information about the specific Android version or device model.
- Hiding the minor OS version. Instead of providing the exact operating system version, the UA string in Safari 14 and later began reporting only broad version numbers or generalized information. Similarly, Firefox fixed the operating system version and did not report minor versions.
There are subtle differences between how Apple and Mozilla implemented these limitations. Those deltas are shown in Figure 1.
Figure 1 - Differences Between Mozilla’s and Apple’s Restrictions on the User Agent Header
Taking the example from the prior post here is the comparison of the before and after:
Before: Mozilla/5.0 (Linux; Android <span style={{color: '#016F01' }}>13; Pixel 7</span>) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.0.0 Mobile Safari/537.36After: Mozilla/5.0 (Linux; Android <span style={{color: '#016F01' }}>10; K</span>) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.0.0 Mobile Safari/537.36
The general structure of the user agent hasn’t changed. This avoided forcing the industry to rewrite their code to parse a user agent string (which would have been exceedingly painful). Instead the changes were subtle, as shown in Figure 2.
Figure 2: Diagram Showing Where Changes Fall in the Reduced User Agent String
The blue values in Figure 2 show those elements which will continue to be updated on all platforms (including Chrome). The green values indicate those elements which are to be unchanging on all platforms, or which in the case of Chrome will be controlled by User Agent Client Hints.
Google, obviously, chose not to implement these approaches and instead took a different tack. Beyond the privacy issues, Google also wanted to deal with the complexity of web servers had reading user agent headers in a passive mode. Many times, the server cannot reliably identify all static variables in the user agent header or infer dynamic user agent preferences. Additionally, publishers and their ad tech intermediaries have to query databases of user agent strings such as DeviceAtlas in real time to identify the device/OS/browser combination they are serving to. These database services are expensive, as is writing/maintaining the code. Google wanted to create a commonly-shared API interface that used standard metadata definitions of user agent elements to support active, standardized negotiations between browser and server to determine which elements could be shared. This would enhance privacy and lower costs even as it allowed sharing of high-entropy elements of the user agent header for optimal user experience.
Overview of the Google Client Hints Infrastructure
Google’s approach is titled Client Hints Infrastructure, with a specific user agent aspect called User Agent Client Hints. Client Hints Infrastructure provides the desired active interface while protecting privacy by requiring the high entropy descriptors to be shared only with appropriate permissions/opt-ins from the user agent. It also, by default, limits sharing of any high-entropy settings with third-parties on the page unless the top-level domain uses browser permissions (more in the next post) to specifically delegate that access to specific third-parties.
There are two ways to implement client hints infrastructure:
- An HTTP Header-Based Approach: Using HTTP request headers, which is available only for first-party contexts. There are two versions of this: request headers themselves and then through a metatag-based approach.
- A JavaScript API-based Approach: Using a JavaScript API, which can be used by an embedded script.
I will start with the header-based approach to introduce the basic concepts and then show how they translate into the Javascript API-based approach.
Universal Low Entropy Elements That Do Not Require Client Hints
Before I delve into Client Hints Infrastructure, it is important to understand the difference in how low-entropy and high-entropy client hints are handled. Low-entropy user agent elements don’t require Client Hints Infrastructure. They are sent by default to the server with all requests (see Figure 3 further down in this post). Those user agent elements are:
- Browser brand
- The browser’s significant major version
- Platform (operating system, e.g. Android, Windows, iOS, Linux)
- Indicator whether or not the client is a mobile device
These simple features do not provide enough information to be able to fingerprint a device.
The Client Hints Infrastructure Header Elements
There are only four header elements unique to the Client Hints Infrastructure API needed to implement its capabilities for accessing high-entropy hints. A fifth, the Permissions Policy header, is more broadly used and not unique to Client Hints.
- Accept-CH Header. When a server wants to access high-entropy client hints, it makes a call to the browser using the Accept-CH response header. It is a response header because the request header comes initially from the user agent calling the specific web page. The server then responds with the Accept-CH response header asking for information it needs (usually for optimizing the user experience). We’ll get into the actual mechanics shortly, but here is an example of what a simple Accept-CH header looks like:
Accept-CH: Viewport-Width, Width, Device-Memory
That is for non-user agent elements. The User Agent Client Hints specification specifically uses a slightly different nomenclature for any item that is part of the user agent, starting all requests with a Sec-CH-UA prefix:
Accept-CH: Sec-CH-UA-Model, Sec-CH-UA-Platform-Version, Sec-CH-UA-Arch
- Sec-CH-x or Sec-CH-UA-x Header. The Sec-CH-<x> and Sec-CH-UA-<x> headers are the structures by which specific high-entropy values are requested and returned from the user agent. The difference is that the former is for general client hints while the latter is for user-agent client hints specifically. The <x> is filled in with the specific property that is required. Here is an example for an item that is not part of the user agent header:
Accept-CH: Viewport-Width, Width
In a subsequent request, the client might include the following headers:
GET /image.jpg HTTP/1.1
Host: example.com
Sec-CH-Viewport-Width: 800
Sec-CH-Width: 600
Note that even though the server requested the features without the Sec-CH- prefix, the user agent returns the values using the Sec-CH-<x> header structure.
You can find the list of available client hints under the Resources section of theprivacysandbox.com here.
- Critical-CH Header. In general, the Accept-CH header only receives the allowed high-entropy hints back from the user agent on the second or any subsequent requests. If it is critical that every load, including the first, has the requested Client Hints, then the server can set a Critical-CH header to request those hints at all times. Here is an example of how the Critical-CH header is used:
HTTP/1.1 200 OK
Content-Type: text/html
Accept-CH: Device-Memory, DPR, Viewport-Width
Critical-CH: Device-Memory
- Permissions-Policy Header. As mentioned previously, client hints are only available to the top-level domain making the Accept-CH response. However, in many cases the publisher may want to share these settings with third-party vendors with JavaScript tags on the page that need access to these same settings, such as a iFrame that displays an ad and needs to know the screen resolution to correctly display the graphics. Here is an example of how the permissions policy header is used:
HTTP/2 200 OK
Content-Type: text/html
Accept-CH: Viewport-Width, Width
Permissions-Policy: ch-viewport-width=(self "https://cdn.example.com"), ch-width=(self "https://cdn.example.com")
Note that the permission is specific to one third-party site and each element has to be called out specifically. That way the top-level domain can share only those elements needed by the third-party and nothing more.
- Meta Tag Variant. Also mentioned previously is that there is a variant of the client hints infrastructure that allows developers to use a metatag to request specific client hints. That request has a form like that shown below:
<meta http-equiv="Accept-CH" content="Viewport-Width, Width" />
- Delegate-CH Header. The Delegate-CH header is used in the meta tag variant in place of the Permissions-Policy header. It appears as follows:
<meta http-equiv="Delegate-CH" content="sec-ch-ua-model; sec-ch-ua-platform; sec-ch-ua-platform-version">
Javascript-Based Approach
As mentioned earlier, only the top-level domain can use the header-based mechanic. Third-parties who have JavaScript tags embedded in a page must use the JavaScript-based version of Client Hints Infrastructure to request these values (the top-level domain can also use the JavaScript API variant). The Javascript-based mechanic uses a JavaScript navigator call - navigator.userAgentData - to access client hints. The default low-entropy elements can be accessed via the two properties: brand, mobile, and platform properties.
// Log the brand data
console.log(navigator.userAgentData.brands);
// output
[
{
brand: 'Chromium',
version: '93',
},
{
brand: 'Google Chrome',
version: '93',
},
{
brand: ' Not;A Brand',
version: '99',
},
];
// Log the mobile indicator
console.log(navigator.userAgentData.mobile);
// output
false;
// Log the platform value
console.log(navigator.userAgentData.platform);
// output
"macOS";
As always, don’t worry about what the code means. Just note the use of the navigator as well as the brand, mobile, and platform properties.
High entropy values are accessed through a getHighEntropyValues() call.
// Log the full user-agent data
navigator
.userAgentData.getHighEntropyValues(
["architecture", "model", "bitness", "platformVersion", "fullVersionList"])
.then(ua => { console.log(ua) });
Again, ignore the code per se. Just note the getHighEntropyValues() call and the way it calls five types of information. There are no Sec-UA-CH- or Sec-CH- elements. The desired information is called using the base names of the features.
The Critical-CH header is not relevant in this approach as there is no two-step round-trip as in the header-based version. However, if I am the top-level domain, it is not clear to me how the Permissions-Policy header is implemented in the JavaScript version. My guess is it isn’t and that permissions always have to be set using browser Permissions by the top-level domain in HTTP headers.
Accept-CH Cache and Accept-CH Frame
Before we delve into the mechanics and flows of Client Hints Infrastructure, there are two more critical concepts we need to introduce
The Accept-CH Cache is the location on the user’s hard drive where the permissions for what is allowed to be shared are stored. It is somewhat like an alternative cookie store in that sites can use each of the hints as a bit set on the client that will be communicated with every request. The cache allows for updates to what high-entropy hints can be shared. But because it is also like a cookie store, under the specification it is subject to similar policies as cookies. A user agent is required to clean out the Accept-CH cache whenever the user clears their cookies or the session cookies expire. There is also another header we have not covered, called the Clear-Site-Data header, which provides a mechanism to programmatically clear data stored by a website on the client-side. This can include:
- Cookies: Session and persistent cookies.
- Storage: Local storage, session storage, IndexedDB, and other client-side storage mechanisms.
- Cache: HTTP cache, including cached pages and resources. The Accept-CH cache is also subject to the policies set by this header.
The Accept-CH frame is a mechanism designed to optimize the delivery of Client Hints in HTTP/2 and HTTP/3 by leveraging the transport layer as a way to reduce the performance overhead of the multiple request-responses needed to call client hints. It is related to the Accept-CH HTTP header but operates at a different level to improve efficiency and reduce latency.
The transport layer is Layer 4 of the Open Systems Interconnection (OSI) model. OSI is a fundamental computing standard that provides for how computing devices communicate across networks that was released in 1984 (so 10 years before the Internet existed) and was fundamental for communication using old tech like dial-up modems. Explaining it is beyond the scope of this blog, but if you are interested there is a good introduction here.
We will discuss the Accept-CH frame at length in the next section, but for now just know that a "frame" refers to the smallest unit of communication in the transport layer. Frames are used to encapsulate different types of data, such as headers, data, and control information, and are transmitted over a single stream within a connection. Each frame type has a specific structure and purpose, allowing for efficient multiplexing of streams over a single connection.
The Basic Mechanics of Client Hints Infrastructure
The Basic Flows
Figure 3 shows how the client and server interface for client hints. As the diagram shows, there are five steps:
- First the TLS handshake between the browser and server occurs. (For those who really want to delve into how this works, the Chrome University videos are an excellent source to learn from).
- The Client sends a request header containing the default user agent elements that do not require Client Hints.
- The server responds with an Accept-CH header requesting the high-entropy values it needs to optimize the user experience.
- The browser now resends its original request but this time includes whatever of the requested values its permissions allow it to share.
- The server reads the specific values and then returns the page content to the user agent that is optimized for that specific set of values. At the same time it repeats the Accept-CH response header to indicate to the browser that it will want the same fields on the next request.
Figure 3- The Basic Mechanics of Client Hints Infrastructure
A Behind the Scenes Look
You can actually view this interaction for any website you visit in the Google Developer Console. Figure 4 shows what CNN uses. In this case it makes no Client Hints request and only receives back the default low-entropy settings (highlighted in yellow).
Figure 4 - CNN Client Hints Usage
Google, on the other hand, makes a large number of Clients Hints requests and gets them back (Figure 5).
Figure 5 - Google’s Client Hints Requests
While I am a big fan of the Sandbox and Google’s technology, I do find it interesting that the company that produces my browser (and thus sets the default of what can be shared) makes a call for so much information that its browser makes available to me. This could just be that Google knows the tech and thus implements it as it should be used. And perhaps this amount of information is not enough to fingerprint my browser uniquely (I’d have to do the detailed calculations and don’t have time right now). But it certainly causes me some concern as a consumer as to just how much Google is asking to know about my user agent.
Who Controls What Can Be Shared?
Which brings us to the question of who controls what gets shared from the user agent? Google sets the default, but there are settings in the browser that the user can set to prevent the sharing of certain client hints. I have not been able to find a discussion of how those work, so what I did to test it is shut off all tracking using the Data and Privacy settings in Chrome and then restarted my browser. I didn’t see anything change. And even with these settings Google was getting back a number of high-entropy signals (Figure 6):
Figure 6 - Google Client Hints Returned with All Tracking Settings in Chrome Turned Off
This is an area I wish Google would provide more documentation for so business people in the industry can understand better how much control the user has on what high-entropy client hints are shared.
Client Hint Infrastructure at the Transport Layer
As we noted in Figure 3, there is a five-step process for sharing client hints. The extra HTTP calls needed to support client hints can add significant overhead to page rendering. There is a workaround for this using something called Application Layer Protocol Negotiation. Application-Layer Protocol Negotiation (ALPN) is a Transport Layer Security (TLS) extension that allows the application layer to negotiate which protocol should be performed over a secure connection in a manner that avoids additional round trips and which is independent of the application-layer protocols. It is used to establish HTTP/2 connections without additional round trips. It can be used to implement a more efficient implementation of Client Hints Infrastructure, as shown in Figure 7. In this case, the Accept-CH response header is embedded in the TLS handshake, thus saving two steps in the process. Note that the Critical-CH header is no longer needed in this call, since there is no first step where a default set of values is sent to the server. TLS embeds the response header, not the initial request.
Figure 7 - The Transport Layer Mechanics of Client Hints
That’s a lot of information for one day, so I’ll stop here. Next up: Browser Permissions.