Google Privacy Sandbox: The Big Picture & Core Browser Elements

Welcome back! This article will cover two topics

1. An overview of the “big pieces” of the Google Privacy Sandbox.
2. A first drill down into the browser-based elements/structures that are core to the sandbox.  

We are not going to get into flows or “connective tissue” for how the various pieces talk to each other (e.g. OHTTP) for now.  We are going to build up the main static elements first, bit-by-painful-bit, exploring all underlying concepts and technologies that reside within and drive the functionality of these “big pieces”.  Then we will fit all the pieces together.  

Another aspect of my approach that differs from the official documentation is that I am attempting to provide a holistic view across all the APIs.  The existing documentation on Google’s site is organized around each API.  Makes sense.  Each has their own product manager who is responsible for the API and understands its workings.  The problem is each of these APIs runs across the different pieces of infrastructure (i.e. servers, browsers) in different ways: sometimes uniquely, sometimes in parallel but without interaction, and at other times with interactions and dependencies on one or more of the other APIs.  

The Google Privacy Sandbox is a comprehensive privacy-preserving advertising system that involves topical targeting (which is slightly different than contextual targeting), behavioral targeting, and reporting.  It depends for its success on another series of APIs – such as the Private State Tokens API – each of which has its own product owner.  Looking at it from the perspective of any single API impedes understanding of how all the parts work to form a functioning whole. Instead we will explore the Sandbox from a systems design perspective. We start with each core piece of hardware/software and identify how the APIs run and interact on that part of the system. Then we build up until the core pieces are connected, which allows us to understand the dynamics of the entire system rather than its piece parts.

An Overview of the Complete Sandbox Architecture and Timeline

So, let’s jump in.  Figure 1 shows the core browser elements, server-side applications, and “on-the-wire” improvements to web connectivity of the Google Privacy Sandbox as currently envisioned when it is fully implemented across the Web.  This diagram does not represent the technical architecture.  It is strictly intended to expose the various components that will be discussed in subsequent articles.

_{Figure 1- Overview of The Google Privacy Sandbox System}

I say “currently envisioned” because we are a long way from a stable release of any component of the architecture. It is possible there will be significant changes over the next one-two years as these technologies evolve. They are:

• The browser (client), with a number of elements, of which several are new (shown in dark grey) that we will cover in more detail in the next few articles.
• An SDK runtime for mobile apps.
• A Trusted Execution Environment (TEE).  This is a secure, attested cloud-based environment running Protected Audience API services centrally versus in-browser, usually for performance and scale reasons.
• At least two Key Management Servers run by legally and physically separate entities that provide cryptographic keys needed to support the Sandbox – especially the Key/Value service that is part of the Protected Audiences API.  There is a need for two servers because the Protected Audience API uses split private keys as part of multi-party computation to provide better security, as no single party will host any service’s private keys.  The companies that run these servers are called Coordinators.
• K-anonymity servers run by Google to provide real-time, algorithmically-derived k-anonymity thresholds to provide protection from microtargeting.
• A publisher’s server(s) where auction logic files are stored and called by the client’s browser to run an ad auction.  These server(s) will also contain code for data collection for reporting purposes
• A “buyer’s” ad server(s) where creatives are stored, bidding logic resides that is called from within the browser, and/or source event registration and data collection occur for reporting purposes.
• “On the wire” technologies to better protect browsers from being fingerprinted or data tampered with, including DNS over HTTPS, Network State Partitioning, and IP Protection.

Some of these elements are in testing today in what are called Origin Trials.  We described one of these -the FLEDGE Origin Trial - in the last article.  Others are in the process of being specified.  Others have been specified and have demo or alpha code bases available for testing.  This last category is particularly true of the server-based applications needed to complete the platform.  Figure 2 shows the history of the development of the core Sandbox APIs, the current plan for deployment to testing and general availability, and an overview of the current status of core technologies for the platform.

_{Figure 2 - Google Privacy Sandbox API Roadmap}

‍

_{Source: https://privacysandbox.com/open-web/#the-privacy-sandbox-timeline}

‍

There are similar roadmaps for the server-side technologies, which will be provided in the articles on each of those applications.  In the meantime, they can be found on Google’s Privacy Sandbox site here.

Over the next few posts, I will drill into each core element and examine them in detail.  Today we will start a series of posts about the browser and what happens there.  There are two reasons for this:

1. Replacing the functionality of cookies with a privacy-preserving variant is causing significant changes to the client side of the system.
2. The current FLEDGE Origin Trial is focused on understanding the implications of running ad auctions in the browser. Focusing on the browser elements provides a foundation to discuss the current mechanics of the sandbox in the first half of 2024 before the server-side elements come into play.  

A Quick Review of Browser Components

Figure 3 shows some core components of a typical client browser before the Google Sandbox.  There are three key groupings we need to be concerned about from this diagram:

1. The browser header
2. The main browser frame, also known as the Body Element
3. Browser storage.  

_{Figure 3 - Key Components of Browsers for Google Sandbox Discussion}

The "true" technologists will argue that I am skipping a great deal of detail necessary to understand the workings of key components of the Sandbox, such as networking.  I agree.  But these articles are intended for technically-oriented senior product leaders and AdTech executives who want to understand the Sandbox, so I will only discuss those browser components at a level of detail needed to achieve that goal.  For those readers who desire a more complete, holistic view of browsers as a way of understanding the Sandbox, this article by Deepak Puttarangaswamy provides an excellent summary.

HTTP Headers

HTML headers allow the web browser and a server to transfer specific configuration or other details needed to fulfill a request from or response to the browser.  When a user clicks to a URL, the browser sends a request that includes a header. This request HTTP header contains additional information for the web server. The web server then responds by sending specific data back to the client to be rendered on the user’s screen. A response header is attached that contains information on whether the initial request went through, type of connection, encoding etc. If the request did not go through, then HTTP response headers will contain an error code.

End-to-End Headers

End-to-end headers apply to both requests and responses, but don’t apply to the content itself. These headers must be present in any HTTP message transmitted to the final recipient.  Intermediate proxies must retransmit these headers unmodified and caches must store them.  The most common are Connection, Keep-Alive, Cache Control, and Date.

Request Headers

Request headers are important because websites tailor their layouts and design to accord with the type of machine, operating system and application making the request. Otherwise, content might be displayed incorrectly. One important header is the user agent header.  It provides critical information on the software and hardware of the source browser as shown in the example in Figure 4.  

_{Figure 4 - Example of HTTP Request Headers (with user agent header highlighted)}

‍

We will revisit the user agent header when discussing the User Agent Client Hints API.  This is because the user agent header provides a substantial amount of information that can be used to fingerprint a browser and uniquely identify it, thus violating privacy.   As part of the Privacy Sandbox, Google has proposed the Client Hints API to enables sites to request the information they need while limiting the information shared about an online user.

Response Headers

A response header is an HTTP header that can be used in an HTTP response and that doesn't relate to the content of the message. Response headers, like Age, Location or Server are used to give a more detailed context of the response.

Representation and Payload Headers

Representation headers contain information about the representation of the resource sent in an HTTP message body. Clients specify the formats that they prefer to be sent during content negotiation (using Accept-* headers), and the representation headers tell the client the format of the selected representation they actually received.

Payload headers describe payload information related to safe transport and reconstruction of the original resource representation, from one or more messages. This includes information like the length of the message payload, which part of the resource is carried in a specific payload (for a multi-part message), any encoding applied for transport, and message integrity checks, among other elements

Each entity tag is represented as a pair.  Representation headers include Content-Language, Content Encoding, and Content Location.  Payload headers include Content-Length, Content-Range, and Transfer Encoding.

‍

The Main Browser Frame

The main browser frame is where the browser’s rendering engine takes all the HTML, CSS, JavaScript and other information about the web page and displays it in the browser window.  Many of the adaptations for the Privacy Sandbox to the client-side architecture occur in the main browser frame.  We will not drill into all the sub-elements of the main browser frame here.  However, I do want to focus on a few elements/concepts that will be critical to understand which are leveraged/extended to support functionality in the Privacy Sandbox

Web Workers

As the web evolved developers, being developers, wanted to execute “bigger and better ideas” in browsers.  Often these web-based applications performed more resource intensive tasks, such as  complex calculations for image generation. So, a solution was needed to allow resource-intensive tasks to run in their own threads, in parallel, to ensure browser response times remained tolerable.

The outcome was a specification for web workers.  Workers are background threads in JavaScript that execute code independent of the main thread. Introduced in HTML5, they are designed to offload tasks that can be time-consuming or resource-intensive and to overcome the limits of single-threaded JavaScript execution.  Workers are relatively heavy-weight, and are not intended to be used in large numbers. They are intended to be long-running scripts that are not expected to respond to clicks or other interactions.  Generally, workers have a high start-up performance cost, and a high per-instance memory cost.

Worklets

Worklets are a new concept introduced in the CSSS Houdini specification and, as we will discuss in later articles, are critical to the Privacy Sandbox.  Worklets are a lightweight version of web workers that allow developers to extend the CSS rendering engine to handle custom CSS properties, functions and animations.  Worklets are similar to web workers in that they can run scripts independent of the main JavaScript execution environment.  However, there are some significant differences,  Worklets:

• Are thread-agnostic. That is, they are not designed to run on a dedicated separate thread, like each worker is. Implementations can run worklets wherever they choose (including on the main thread).
• Are able to have multiple duplicate instances of the global scope created, for the purpose of parallelism.
• Do not use an event-based API. Instead, classes are registered on the global scope, whose methods are invoked by the user agent.
• Have a reduced API surface on the global scope.
• Have a lifetime for their global object which is defined by other specifications, often in an implementation-defined manner.

These differences, especially those that allow user agents to parallelize work over multiple threads with global scope or to move work between threads as required, as well as the security features that prevent one worklet from accessing user data from another worklet, make worklets - or the Sandbox's variant called a script runner - a perfect vehicle for implementing ad auctions and bid generation in the browser.

iFrames

The term iFrame stands for “inline frame”.  iFrames are an HTML element that loads a second HMTL element within the main browser frame of another web page while allowing the user to interact with the content.  They are commonly used to embed specific content or other interactive elements into a page.  Documents rendered in an iFrame are encapsulated within their own browsing context with separate CSS, JavaScript, and HTML content.

Calling an iFrame requires a single line of HTML:

<iframe src="https://www.example.com/" width="200" height="200" sandbox="allow-downloads"></iframe>

No doubt you have heard of  iFrames as they are often used to display ads because they provide more flexibility than an inline script.  But from the perspective of the Google Sandbox they are problematic because they can provide a browser fingerprinting surface where they can identify an individual based on the ads they have been presented over a period of time.  So Google Sandbox had to create a new concept, Fenced Frames, to replace the standard iFrame.  We will be talking more about Fenced Frames in the next article.  

Navigators

The navigator object provides information about the browser, operating system, and user environment where a web page is running. The Sandbox leverages specific navigators to provide context - e.g. what browser is being used - in a privacy-preserving manner for auctions, bidding, and reporting. For example, the Protected Audiences API uses the navigator.permissions.query({name: 'geolocation'}) method to check if the user has granted geolocation permissions. This can be used as a signal during the auction to determine if the ad displayed should include location-based features. The advertiser receives a simple yes/no answer without any actual geolocation data.

But the Sandbox's use of navigators is much more extensive because Protected Audience interest groups reside in the browser and are therefore a part of its state that a navigator can access. For example, when a user’s interactions with a website indicate that the user may have a particular interest, an advertiser or someone working on behalf of the advertiser (e.g. a demand side platform) can ask the user’s browser to record this interest on-device by making a call to navigator.joinAdInterestGroup() - a new type of navigator. This indicates an intent to display to this user an advertisement relevant to this interest group in some future auction. The user agent records interest group sets, which are a list of interest groups in which owner/name pairs are unique. When an auction occurs, a second navigator for the Protected Audiences API, navigator.runAdAuction(), checks the interest group sets to determine if any of the interest groups from that particular owner qualify for the specific auction.

Navigators are extensively used in the Privacy Sandbox because so much of its functionality runs in the browser. As a result, the API has found a creative way to leverage these existing objects.

Beacons

Beacons are a type of HTTP request that operates asynchronously. This means the browser sends the request and continues executing code without waiting for a response. This is useful for sending data without interrupting the user experience.  Unlike other HTTP requests, beacons don't expect a response from the server. This simplifies implementation and reduces network overhead. Beacons are also especially good for small data transfers. The asynchronous nature of beacons and their strength in handling small data transfers make them ideally suited to send anonymized aggregated data about ad impressions, clicks, or conversions in Privacy Sandbox reporting.

Browser Storage

One of the most confusing topics for me even as I worked in AdTech for many years was browser storage.  I imagined there was actually some space in the browser itself, similar to HMTL elements in a .html document, where the data used by the browser was stored.  Ultimately, I realized that this was not the case.  That, in fact, browser storage was nothing more than a set of encrypted files in different subdirectories holding different kinds of data under my c:\users directory (on Windows).  These storage elements include:

• Local Storage
• Session Storage
• WebSQL
• Indexed DB
• (Origin Private) File System
• Application Cache
• BLOB URL Storage
• Cookies

The contents of these types of storage can be examined in the Chrome Developer Console (Figure 5).

_{Figure 5 - Display of Browser Storage Contents in the Chrome Developer Console}

‍

One very impotant thing that is unique in this display: note the item highlighted in red at the bottom right. The browser_fingerprint_id is not a standard HTML storage element. It is www.medium.com employing a unique key/value pair for some form of browser fingerprinting. You hear about this as one of the statistical methods for identifying a user when there are no third-party cookies. But you may wonder how it is or where it is done. This is a great, very visible example. I am not logged in, so to Medium I am an anonymous user. This is their fallback when I don't have a first-party cookie or haven't given them my first-party id. And it makes me wonder that if I studied many other sites if I would find an extensive use of similar, blatant tags for browser fingerprinting there.

Local Storage

Local storage holds persistent data that is needed across browser sessions or when a page refreshes for a specific domain.  Each domain has its own encrypted file within the storage folder the browser uses on the local hard drive.  One domain cannot gain access to data stored in local storage by another domain.  Stored data is not automatically sent to the server with every HTTP request. This means that the server will not have access to the data unless it is specifically requested.  

Session Storage

Session storage is similar to local storage with one key difference: the data stored in session storage is automatically deleted when the user closes the browser tab or window where the data is stored.

WebSQL 

WebSQL allowed websites, extensions or apps to store data in a structured manner on the client. It was based on an embedded database called SQLite. While WebSQL was completely deprecated from Chrome in late 2022, some elements of Chrome still use SQLite. Moreover, to understand the roots of IndexedDB, the Chrome NoSQL database, it is important to know the history of WebSQL, so we will cover it in some depth when we discuss browser storage.

IndexedDB

IndexedDB is a large-scale, NoSQL storage system that allows storage of just about anything in the user's browser. In addition to the usual search, get, and put actions, IndexDB also supports transactions. Each IndexDB database is unique to a site domain or subdomain, meaning it cannot access or be accessed by any other domain. Data storage limits are usually quite large, if they exist at all, but different browsers handle limits and data eviction differently.

Origin Private File System

There are times when applications want or need to be able to write to or read from the local hard drive on which the browser client resides. For example a user may wish to upload local files to a remote server or manipulate local files inside a rich web application. Browsers have a series of API standards that allow developers to build applications that read and write files in a sandboxed subdirectory in the Chrome directory tree (appropriately called File System) You will see the term Origina Private File System in Google documentation because an origin ((e.g. site, an extension, etc.) stores private content to disk that can be easily accessed by the user.

Application Cache

Most people nowadays have enough knowledge of browsers that they understand that the browser's cache is where websites, extensions, and web applications store data about a web page. They know this because as part of security and privacy, they are often asked or directed to "clear their cache".  What may not be clear to them is how the cache differs from other forms of web storage. Web cache is specifically designed to hold static web content (e.g. images) to:

• help reduce page load times
• allow web pages that have been previously loaded to display their static content in the event that the browser's Internet connection is temporarily interrupted.

Chrome uses the operating system's disk cache for storing web content.  Data is primarily keyed by the URL of the resource being cached. This allows Chrome to efficiently retrieve previously downloaded content for faster page loads.  The cache itself is generally not encrypted. This is because the cached data is intended for performance optimization and typically contains publicly accessible web content. The cache uses an expiration mechanism to manage storage space. Older cached files are periodically deleted to make room for new ones.   Note that browsing data in incognito mode is not stored in the cache, offering a level of privacy.

BLOB URL Storage

BLOB is an abbreviation for Binary Large Object. Basically they are very large files, can be many megabytes or even gigabytes, that can contain almost any kinds of information. Large file downloads or streaming video are examples of this kind of file. Often the application or user does not want to store these files. They are rendered by the browser and then released from memory. Instead of storing the BLOB, the browser instead stores a pointer to the URL where the content is stored. Then depending on available memory, the application slices the blob data and renders it in pieces. If the BLOB is too large to be held in memory, it can be saved to local disk until momory is available to show the next slice. BLOB storage really isn't storage in the classic sense, which is why it does not show up in the developer tools tree in Figure 5. But I mention it here as it will become important when we talk about how the Sandbox handles partitioned stroage.

Cookies

The quote above is what I first wrote when I published this post. But as I got ready to write about the Privacy Sandbox's storage elements, I realized that how cookies work is actually not obvious.  Let me give you some examples of questions that I discovered that you may or may not know the answer to:

1. Where are cookies stored in my browser/on the hard drive?
2. Are they stored in a file, a set of files, in a database like IndexedDB, or something else?
3. How is it that one site cannot "see" another site's cookies?
4. If cookies are stored in encrypted form, how are they encrypted and decrypted? What algorithm is used to encrypt/decrypt them?
5. Question #3 is important because I have plug-ins in my browser, or I can write code, to show me all the cookies  in my browser. So if I can do it or a third-party extension to Chrome can do it, why can't an evil actor web site access information, either directly or by using a browser-based attack to get permissions to my cookie status and history?

If you know the answers to these, then 10 points to Gryffindor and it just goes to show how little I've actually understood our business over the years.  But I'm guessing if I don't understand it, most likely neither do many members of my audience.

We won't cover cookies here because, after all, third-party cookies are going away, which is a major driver to create the Privacy Sandbox. But when I get into the Privacy Sandbox browser storage elements, I am going to do a special post on cookies and how they work so you have answers to those questions.

‍

_{Figure 6 - Summary of Different Types of Browser Storage (needs update)}

^{Source: https://www.geeksforgeeks.org/difference-between-local-storage-session-storage-and-cookies/}

‍

Permissioning

Permissions Policy allows the developer to control the browser features available to a page, its iframes, and subresources, by declaring a set of policies for the browser to enforce. With these tools, the  top-level site owner can define what it and its third parties intend to use, and removes the burden from an end-user of determining whether the feature access request is legitimate or not. For example, by blocking the geolocation feature for all third parties via Permissions Policy, the developer can be certain that no third party will gain access to the user's geolocation.

Many of the Privacy Sandbox APIs use the browser’s permissions policy to control access to various capabilities. With third-party cookies, the page owner has no granular control over how the cookies are used by third-party iframes.  WIth the Privacy Sandbox APIs and Permissions Policy, a page can allow or deny Privacy Sandbox APIs from being used by the page itself and third-parties on the page. For example, a page owner, such as a publisher, can use Permissions Policy to allow specified third-parties to run an ad auction, or deny all third-parties from reading the user's topics.

We will cover permissions policy and how it impacts the Privacy Sandbox in detail in a later article.

The Browser and Google Sandbox

That overview of the browser, while somewhat basic, was important because it provides a baseline of how browsers operated before the Google Privacy Sandbox was conceived.  Figure 7 is an updated architectural overview of the Chrome browser with the new Google Sandbox elements.  Clearly, a lot of changes are being made to reengineer Chrome, and online advertising as well, to adapt to the new privacy-centric view of the Web.   We will delve more deeply into those changes in the next couple of  articles.

_{Figure 7- The Browser with Updates for Google Privacy Sandbox}