General code architecture

…

Archive Storage and Indexes

This document explains how docs.rs stores rustdoc and source files in archive storage, and how single-file requests are served efficiently without downloading full archives.

Why this exists

• We store many files per crate version (HTML, CSS, JS, source files, etc.).
• Downloading full archives on every file request would be expensive.
• We need fast random access to one file at a time.

Storage model

For each crate version, docs.rs stores:

• one archive for rustdoc output;
• one archive for source files.

Both archives are ZIP files.

Archive format assumptions

ZIP compression is per-file, not whole-archive.

• Each file entry has its own compressed payload.
• This allows serving one file by reading only the corresponding byte range.

This differs from .tar.gz:

• tar concatenates files into one stream;
• gzip then compresses that full stream as a whole;
• random access to one file is much harder without scanning/decompressing much more data.

Index model

For each archive, docs.rs generates an index stored as an SQLite database.

The index is conceptually similar to a ZIP central directory, but represented in a queryable format.

Each row maps a logical file path to the location of its compressed payload in the archive:

• filename/path in archive;
• byte range start (from), inclusive;
• byte range end (to), inclusive;
• compression algorithm used for that file entry.

The range points to the compressed payload bytes of the ZIP entry (starting at ZIP data_start), not to metadata headers.

We store the index file next to the ZIP on S3.

Request flow (source or rustdoc file)

When a request asks for a single file:

1. Resolve which archive and index correspond to the crate/version.
2. Check whether the index is already present in local cache.
3. If missing, download the index file and cache it locally.
4. Query the SQLite index for the requested filename/path.
5. If no row exists, return “not found”.
6. If found, read from/to/compression information.
7. Issue an HTTP Range request to S3 for only [from, to] from the remote ZIP file.
8. Decompress the returned byte range using the per-entry compression algorithm recorded in the index.
9. Return/use the decompressed file bytes as response content.

Important details

• The local cache key includes latest_build_id (effectively ...zip.<build_id>.index) so rebuilt releases naturally use a fresh local cache entry.
• Index cache population is concurrency-safe:
  • optimistic read first (no lock),
  • then per-index lock for repair/download,
  • temp-file download + atomic rename to publish.
• If index lookup/decompression fails (for example stale offsets causing decompression errors), docs.rs purges local cached index files and retries.

Key properties

• Efficient network usage: fetches only bytes for the requested file.
• Efficient CPU usage: decompresses only one ZIP entry.
• Good cache behavior: index files can be cached locally and reused across many requests.
• better manageability or our S3 bucket, especially around rebuilds or deletions of crates or releases.

Compression layers

There are two different compression layers involved:

• ZIP entry compression (inside rustdoc/source archives): this is what the archive index stores per file and what is used to decompress range responses. At the time of writing, archive index creation supports bzip2 ZIP entries.
• Object storage compression (for regular blob uploads): this is separate and represented by blob Content-Encoding/storage metadata.

For archive file serving, the important algorithm is the ZIP entry compression from the index.

Archive downloads

• these archives can also be downloaded to be used for offline docs.
• This is also why we used a more widely supported algorithm (Bz2) instead of zstd inside zip, which would theoretically also have been possible.

Handler behavior and fallbacks

• Rustdoc/source handlers use the same archive lookup primitives.
• On missing rustdoc paths, handlers may try path fallbacks (for example appending /index.html) before returning 404.
• For target-specific misses, handlers can redirect to target fallback/search routes instead of always returning a hard 404.

Notes for maintainers

• Keep index schema and archive writer in sync.
• Any change in byte-offset computation must preserve correct range boundaries.
• If index lookup fails unexpectedly, prefer rebuilding/downloading index rather than falling back to full archive downloads.

Build Workspaces

When do we need this?

Many workspace packages do not need manual intervention and can be built simply by executing the commands listed in the main README.md file. However, some workspaces require an additional step. This is the case when values such as

version.workspace = true

are inherited from the workspaces Cargo.toml configuration file.

Fix

To build documentation, rustdoc requires a fully specified package but rustdoc does not understand workspaces which are only defined in cargo. Thus our crate needs to be packaged by cargo before running the documentation. This step will replace all of the value.workspace = true statements with their respective values.

cargo package

This will emit a packaged crate into the target/package/your_crate_name-version folder. Now the commands specified in our README.md can be executed targeting this folder.

cargo run -- build crate --local /path/to/source/target/package/your_crate_name-version/

Full MWE

To showcase when such problems can occur, take a look at the following example.

Structure

$ tree
.
├── Cargo.toml
├── my_lib
│   ├── Cargo.toml
│   └── src
│       └── lib.rs
└── README.md

3 directories, 4 files

The actual contents of my_lib do not matter, only the two configuration files.

$ cat Cargo.toml
[workspace]
members = [
        "my_lib",
]

[workspace.package]
version = "0.1.0"

and

$ cat my_lib/Cargo.toml
[package]
name = "my_lib"
version.workspace = true

[dependencies]

Building

The build command

cargo run -- build crate -l path/to/docs_rs_workspace_package/my_lib

fails with

Error: Building documentation failed

Caused by:
    Building documentation failed

Caused by:
    invalid Cargo.toml syntax

which makes sense due to

version.workspace = true

Fix

However when running the following sequence of commands

# Run this in the directory of docs_rs_workspace_package
cargo package -p my_lib

and then building again

# Run this from the docs.rs repo
cargo run -- build crate -l path/to/docs_rs_workspace_package/target/package/my_lib-0.1.0

then the build succeeds.

See list of docker images

$ docker compose logs -f db

Stop your containers and remove them with all their data

$ docker compose down -v

Check logs of the DB

$ docker compose logs -f db