Introduction
We have talked about partitioning as it relates to storage extensively two posts back. We discussed that partitioning as a concept is almost as old as browsers themselves, and that many of the current storage elements in a browser are partitioned by origin. We also noted that partitioning in the context of the Google Privacy Sandbox refers to the addition of other partition keys - most commonly the current URL, to create a partitioning tuple of <”URL”, “origin”> so that embedded elements like iFrames cannot perform cross-site reidentification of a user agent.
The problem is that partitioning exists in the larger context of a broad, on-going evolution of client-side browser storage. This evolution is being driven by the many enhancements of existing storage-related elements/APIs like IndexDB, new storage types like Shared Storage, as well as the large amounts of additional information to be stored in browsers that results from moving programmatic auctions from ad servers to the client. Moreover, the need to partition all types of browser storage for the Privacy Sandbox allows for a rearchitecting of how storage is managed that ensures a more secure and private platform overall.
The response is contained in an evolution of a browser-side storage standard called, oddly enough, the Storage Standard. We mentioned this standard and its API at the end of the first post on browser storage. It extends the basic storage concept contained in Section 12 of the official HTML Standard, which itself builds on the IETF cookie standard by adding local storage and session storage, to cover all other types of storage.
While the standard is somewhat hard to imbibe, the concepts it is built on which we need to understand are relatively straightforward. They are shown in Figure 1.
Figure 1 - The Storage Hierarchy Underlying the Storage Standard
Basically, the new storage architecture is a hierarchy of storage concepts, each of which is finer grained and is a child of the prior element in the hierarchy. You can also think of this as the storage partitioning architecture now being implemented in Chrome to support the Google Privacy Sandbox, although its design was not specifically tied to the Sandbox but rather a much wider range of use cases. The broad reasons for evolving this new standard are that it provides:
- A standardized, widely-supported way to organize key-value pairs using localStorage and sessionStorage with improved control.
- A clear separation between data that persists until cleared (localStorage) and data that persists only until the browser window/tab is closed (sessionStorage).
- A better ability to manage these storage types at a much finer-grained level.
Let’s dive into the architecture and see how it works.
Level 1: Browser Data and Session Navigation History
Let’s start right at the top and work our way down. These elements at the top (first level) of the diagram are data about the browser and the user’s session browsing history:
- User Agent. The client browser. This browser has a user agent header that describes it and data that will be held in some storage type locally in the browser. This is the “local” data that flows into the next level of the storage architecture.
- Traversable Navigable. This is a big set of words for what the average web user thinks of as a browser tab containing a web page. It’s a bit more complex than that, as it really is the open browser page and the history of any prior pages that were opened in that tab and the order in which they were opened. In fact it is a lot more complex than even that description, and to a certain extent I am at a loss as to how deep to delve as there are some basic concepts here, like navigables, which you probably should know. But at the risk of the software developers telling me “that isn’t technically accurate”, I am going to stick to a simple discussion here and then delve a bit more deeply into these concepts in the next post. For those who wish to get the complete background, jump to the next post and then navigate (Ah hah! Maybe related to a navigable? Ya think?) back here.
Think of traversable navigables as a list of entries within your browser's browsing history that represent web pages you can navigate back to using the back and forward buttons. They act like a bookmark manager specifically designed for efficient back/forward navigation. The data for each traversable navigable (i.e., a list of pages visited) is stored in a particular kind of cache called the backwards-forwards cache or bfcache and the entire browsing history for that session is deleted when the tab or browser is closed. This is the “session” data indicated in Figure 1 that flows to the next level.
As previously discussed, session data can run across multiple tabs from the same website and expires at the end of a session. So if multiple tabs from the same site are open, the session data remains in session storage until all tabs from the site are closed. That is why you will often see a second term - top-level traversable set. It indicates the multiple tabs open to a specific website session.
So what data is defined as part of this element. When a specific traversable navigable is created, the following data is used to instantiate it:
- A document. All the content for the web page (e.g. an HTML file)
- An origin. See the definition of origin if you need a refresher. Basically, this is the top-level domain of the document.
- An Initiator Origin. An initiator origin is the top-level domain of the web page that initiated the request that loaded the current page. This concept applies when an action on one webpage triggers something on another webpage. This is not a case of going to another page within the same top-level domain (origin). That is navigating within the same origin, so the origin and initiator origin are the same. An initiator origin has to be a different origin than that for the page that loads
- A Navigable Target Name. A specific section within a webpage's history. When a user navigates through a webpage using the back and forward buttons, the browser keeps track of their browsing history. The navigable target name helps identify a particular section (e.g., a heading or a specific part of the content) within that history entry. It is used for efficient back and forward navigation. It also helps avoid full page reloads, so it has performance benefits as well.
- An About Base URL. This is the URL of the pages about: schema page. Trying to explain this in detail requires its own post. Moreover, I have searched the web and cannot for the life of me figure out what function this item plays in the function of a traversable.
The data the transversable navigable holds over time grows as the user traverses through web pages. We discuss that further here. For now it is enough to know that this data will flow into the first level of the storage architecture.
Level 2: Storage Sheds
A storage shed is the highest level of the storage architecture in the browser (more exactly in a user agent, but we’ll let the term “browser” be a stand-in for now). There are two kinds of storage sheds:
- The first type of storage shed holds a set of storage keys, each of which is assigned to a particular origin. The key is the origin’s tuple. Think of this storage shed as a self-service storage facility that holds all the browser’s locally-stored data (local storage). Each unit in the shed is a secured locker for one top-level domain to store all its information. The number of the unit is its key. This number also happens to be the code to its lock.
- The second storage shed occurs at the traversable navigable (tab/session) level. It also is a storage facility, with each unit being an origin key which provides a secure storage area for an origin’s session data. This type of storage shed is specifically designed for data related to your browsing history. It might store information like the navigable target name (discussed earlier) to help you jump to specific sections within a web page when navigating back and forth.
Thus right at the top of the architecture we now see that local storage and session storage have each been partitioned and parititioned separately into their own isolated storage areas. They are partitioned by a single key - the origin.
Level 3: Storage Shelves
A storage shelf is the private storage unit for each origin/TLD within the storage facility, to continue the analogy. It ties to its key that is stored in the storage shed. It is a container for a set of strings from a specific origin, also called a map in technical parlance. These strings consist of a key (the origin) and a value (storage bucket id). When a storage shelf is created (by whom or what is discussed below), there are some other features you can set, including:
- Policies used for security checks.
- Whether or not the shelf can be used by scripts that require cross-origin isolation. This, as you can imagine, is important for the Privacy Sandbox where cross-origin isolation can be important to prevent cross-site re-identification of a user.
This is the structure that allows us to partition storage at a level, the storage bucket, that developers can use to manage their storage quotas in a finer-grained way than was previously possible.
Level 4: Storage Buckets
Within each storage shelf there can be multiple storage buckets. Storage buckets have their own proposed API (see below) and are one of the major evolutions of storage occurring beside the Privacy Sandbox. The intention is that this standard will ultimately get merged directly into the Storage Standard.
A storage bucket is a place for storage endpoints (storage types) to “store” data. I say that because, as discussed below, this is the only level of the architecture which the developer can manipulate to manage their use of storage when storage quotas become a constraint. So from a developer’s perspective, this is where and how the data is stored and managed. The “store” is in quotes because a storage bucket doesn’t contain any of that data. The data is stored in whatever storage type the developer chooses to use. All a storage bucket holds is a key value pair with a storage bottle id as the key and the storage endpoint as the value.
Every storage bucket must be associated with a storage endpoint, which is one of the browser storage types we have previously discussed. Figure 2 shows the different types of storage endpoints that are recognized by a storage bucket, their type, and their quota. As is clear from the table, storage buckets are used for all forms of storage (the serviceWorkerRegistrations endpoint is how the bucket connects to the Origin Private File System, which isn’t obvious). The quota for a storage endpoint is a number representing a recommended default storage limit (in bytes) for each storage bucket corresponding to this storage endpoint.
Figure 2 - Types of Storage Endpoints in the Storage Standard
A quota is set to null means one of two things::
- The amount of available storage on a user's device for that storage type can vary depending on factors like the operating system, device limitations, and other applications' storage usage.
- Setting a null quota allows browser vendors and storage API implementations to determine appropriate storage limitations dynamically based on the user's device and system resources.
What it does not mean is that the user can set the quota or that the storage has no restrictions.
There are two kinds of storage buckets.
- A local storage bucket is a bucket for local storage-type data. As you would expect, this data will persist beyond a single session.
- A session storage bucket is where session-type data for a particular website is stored. Session storage data persists until the browser window or tab is closed.
Level 5: Storage Bottles
So finally we are in the storage unit, we’ve have pulled out a bucket, and there are now a series of bottles in the bucket. Are they full of data? Sadly no. There is a lot of empty space in these bottles (or else they are really small). Storage bottles aren’t storage. They contain a single key-value pair that points to a specific location in an appropriate storage type where a specific piece of data is stored.
// create the data for a storage bottle
const user-data = “alvinchipmunk”;
// Store the user data in a storage bottle with the key “user-data”
localStorage.setItem("user-data", user-data);
You can access the key value (to determine if it exists) by making a call to the storage bucket of the storage bottle:
// Get the storage bottle key from your website's logic
const storageBottleKey = "user-data";
// Retrieve the storage bottle (which might involve browser-specific calls) const data = getStorageBottle(storageBottleKey);
// Check if the storage bottle exists and has a value
if (data) {
const userId = data.value;
That isn’t to say you can’t store complex data in a bottle. You can do that by creating complex data structures as a single object and then either writing them to, or retrieving them from, the appropriate storage endpoint. Here is a more complex example using a JSON structure to store multiple elements:
// construct a JSON data structure called userSettings with four elements
const userSettings = {
fontStyle: arial,
fontSize: 16,
fontColor: blue,
showGrid: true,
};
// Store the user settings as a single object as a JSON string
localStorage.setItem("user-settings", JSON.stringify(userSettings));
The Storage Bucket API Provides Access
You may have noticed that nowhere in the code examples is there any constructor like
const bottle = await storage.createStorageBottle(bottleName)
This is a made-up example because there is no createStorageBottle() capability in Chrome. In fact, the architectural elements as currently specified are not something developers can access directly. When the code above calls getStorageBottle(storageBottleKey) it can access the key name for a storage bottle that was automatically created by the system when the user saved a key called “user-data”. But the developer did not actively create the storage bottle - he didn’t have to. All of it is happening in the background, providing the benefits to the developer without forcing them to do all the work of setting up the structures. All the developer has to do is get or write data and all the mechanics happen behind the scenes.
Convenient? Yes. Problematic? Yes The problem is I need to be able to manage storage.
Remember, one of the purposes of the Storage Standard is to allow developers to have more granular control of storage management.
The problem with browser storage prior to the Storage Standard was one of a site running out of storage quota within a browser. If the user ran out of storage quota on their device, the data stored with APIs like IndexedDB or localStorage would get lost without the browser being able to intervene. The original StorageManager interface in the Storage Standard allowed developers to check for storage usage and write exception handlers when storage threatened to get too full. But in that case the browser was limited to an all-or-nothing call. It could only clear storage for that origin completely. This could be problematic because a given site might have multiple applications running in parallel and they would all have data deleted. This could end up causing a degraded or even a disrupted user experience.
So the developers of Chrome have developed an extension of the Storage API called the Storage Buckets API. This API, which has been available since the Chromium 122 release, allows developers to create storage buckets to contain data for specific applications or specific pieces of applications. A developer can create as many storage buckets as they want. When the estimated storage usage approaches the storage quota (using the updated version of the StorageManager interface), the browser may then choose to delete each bucket independently of the other buckets to free up storage space. Developers manage this by specifying an eviction priority to each bucket to ensure that the most valuable data doesn’t get deleted.
Eviction is a tad more complicated because the developer can mark a storage bucket as persistent. In this case, the contents won't be cleared by the user agent without either the data's origin or the user specifically doing so. This includes scenarios such as the user selecting a "Clear Caches" or "Clear Recent History" option. The user will be asked specifically for permission to remove persistent site storage buckets.
As noted earlier, each storage bucket is associated with a specific storage endpoint. An example of this shown below. I have highlighted the two lines of code that make the point:
// Create a storage bucket for emails that are synchronized with the
// server.
const inboxBucket = await navigator.storageBuckets.open('inbox');
const inboxDb = await new Promise(resolve => { const request = inboxBucket.indexedDB.open('messages');
request.onupgradeneeded = () => { /* migration code */ };
request.onsuccess = () => resolve(request.result);
request.onerror = () => reject(request.error); });
As you can see, you first create the bucket and then you tie it to the type of storage endpoint it needs to use.
This storage bucket is the finest-grained unit of storage that developers can control. Bottle management is happening behind the scenes. So going back to our analogy for a moment, when the developer pulls the storage bucket from the storage shelf, the bucket is a closed black box that they cannot open. However, there is a slot in the side where they can stick things into the box they want stored there or to be able to take from sometime later. Moreover, there are five different sizes of boxes on the shelves. The developer chooses the box size that best fits the amount of “stuff” they need to store. When the shelf runs out of room, the developer has to choose which box to get rid of. Sadly, removing items from the box doesn’t reduce the space taken up by the box. So if grandma's valuable jewelry is in with some unimportant papers that are not needed, the developer is SOL and grandma’s jewelry goes bye-bye along with everything else in that box. So it is important for the developer to carefully choose what data to put in which storage bucket.
A Messy Specification
Do you feel that this is a lot of complexity for something that should be relatively easy to understand? Well, you are not alone. I will say this: I have spent more time on this aspect of storage than I have on any other area of browser tech so far, I have been through more rewrites than any other feature as I have continued to try and figure it out and discovered areas that I got wrong the first time. Frankly, I’m not sure I still understand it. I thought it was me but then I found this comment from Maciej Stachowiak from Apple in June 2020 (!) that I want to quote in full because it expresses the frustration I had while writing this post. It can be found in issue #101 in the github repository for the Storage Standard
The terms "storage shed", "storage shelf", "storage bucket" and "storage bottle" are hard to understand. The terms express a size hierarchy, which is pretty clear, but other than that, they don't convey what they mean. Even the size hierarchy is based on a somewhat arbitrary ordering of keys. The ordering of the hierarchy is not motivated in the spec, and the "model" section does not directly explain what they represent.
Here's what I was able to figure out on careful reading:
storage shed: seems to exist solely to distinguish "local" vs "session". Not clear why this is the outermost container. Also, currently redundant with identifier, since any given identifier can only be one of "local" or "session", the comment that this may change does not explain why.
storage shelf: represents the storage for an origin (presumably will change for storage partitioning; will this change the key, or will it add another level of storage hierarchy?)
storage bucket: can't figure out what this represents. Currently it seems there is only one per storage shelf (keyed as "default"), but even from reading the citied issue #2, I can't figure out what a non-default bucket would represent.
storage bottle: represents the storage for a particular storage API
Perhaps something like "storage scope", "origin storage", "???", "endpoint storage"/"API storage" would be more clear? (No suggestion for the bucket because it's not clear what it is). At the very least, an overview explaining what each of the containers represents, and why they are ordered this particular way, would make it easier to understand the spec.
We will stop here for today so I can take some Advil for the headache writing this gave me.
NEXT UP: A Tech Talk on Navigables and Session Histories