General code architecture

…

Binaries

docs.rs runs several long-lived services to serve documentation, monitor crate releases, and build new documentation. We also provide an admin CLI for several admin activities.

Web Server

Our web server:

• lives in the docs_rs_web subcrate, and
• is based on the axum crate.

Besides serving some static and database-backed content, it acts as a proxy for the stored rustdoc HTML files, rewriting them on the fly to match our UI.

Because we recompress and rewrite HTML for many requests, we’re more CPU-bound than a typical web server.

Index Watcher

The index watcher is a small process that manages a clone of the crates.io-index repository.

The watcher updates the clone once a minute and uses crates-index-diff to determine the changes.

Depending on the change, we:

• add the release to the build queue,
• update the release’s yanked status, or
• delete the crate or release entirely from our storage.

The code lives in the docs_rs_watcher subcrate.

Starting position in local development

On its first run, the watcher stores the current HEAD of the registry index as its last-seen reference. It therefore watches new changes rather than queuing every release already present in the index.

To start from a particular Git reference, set it before starting the watcher:

$ just watcher queue set-last-seen-reference <GIT_REF>

Omit the reference, or pass --head, to reset it to the index’s current HEAD.

Build Server

The build servers:

• read releases from the build queue,
• use rustwide to run cargo doc, isolating each build in a Docker container for security, and
• package the documentation into a ZIP file and upload it to S3.

The code lives in the docs_rs_builder subcrate.

Build environment

Docs.rs / rustwide are internally using the crates-build-env docker images as the build environment for the crate. If you’re missing a system dependency, you can add it there.

Also see the docs.rs build info page.

The root README documents building a published release or a local package during development.

Docs.rs invokes rustdoc with its nightly toolchain. See the rustdoc book for rustdoc-specific behavior and options.

Admin CLI

The Admin CLI provides helper commands for managing a docs.rs deployment. It connects directly to the database and S3, so it can theoretically run on a separate machine without direct access to the other running web or build-servers.

For task-oriented instructions that use these commands, see the common maintenance procedures.

Features

You can get help via:

Usage: docs_rs_admin <COMMAND>

Commands:
  build     
  database  Database operations
  queue     Interactions with the build queue
  cdn       
  help      Print this message or the help of the given subcommand(s)

Options:
  -h, --help     Print help
  -V, --version  Print version

build subcommand

Usage: docs_rs_admin build <COMMAND>

Commands:
  set-toolchain  
  lock           Locks the daemon, preventing it from building new crates
  unlock         Unlocks the daemon to continue building new crates
  help           Print this message or the help of the given subcommand(s)

Options:
  -h, --help  Print help

cdn subcommand

Usage: docs_rs_admin cdn <COMMAND>

Commands:
  purge  purge pages with a surrogate key from the CDN
  help   Print this message or the help of the given subcommand(s)

Options:
  -h, --help  Print help

database subcommand

Database operations

Usage: docs_rs_admin database <COMMAND>

Commands:
  migrate                       Run database migration
  abnormality                   Manage the abnormality shown in the site header
  update-latest-version-id      temporary command to update the `crates.latest_version_id` field
  update-repository-fields      Updates GitHub/GitLab stats for crates
  backfill-repository-stats     Backfill GitHub/GitLab stats for crates
  update-crate-registry-fields  Updates info for a crate from the registry's API
  blacklist                     Blacklist operations
  limits                        Limit overrides operations
  help                          Print this message or the help of the given subcommand(s)

Options:
  -h, --help  Print help

queue subcommand

Interactions with the build queue

Usage: docs_rs_admin queue <COMMAND>

Commands:
  add                     Add a crate to the build queue
  default-priority        Interactions with build queue priorities
  repository-priority     Interactions with repository-specific build priorities
  rebuild-broken-nightly  Queue rebuilds for broken nightly versions of rustdoc, either for a single date (start) or a range (start inclusive, end exclusive)
  help                    Print this message or the help of the given subcommand(s)

Options:
  -h, --help  Print help

Legacy Daemon

We still maintain a legacy daemon binary in the cratesfyi subcrate.

It’s used only in the legacy infrastructure and combines these services into one process:

• a web server,
• an index watcher, and
• one build server.

Infrastructure

We currently run production on our legacy infrastructure while continuing to develop a new, scalable replacement.

The Fastly CDN and Fastly NgWAF will stay the same between the legacy and the new infrastructure.

Fastly CDN

Within the CDN, we run a Fastly Compute WASM module. The code lives in our simpleinfra repository.

This enables us to move performance-critical logic to the edge and write integration tests for it.

The Compute module uses Fastly NgWAF to block malicious requests at the CDN before they reach our origin servers.

The Fastly service is configured via Terraform in the same repository.

The Cache-Control headers returned by our web server determine which content is cached. There should not be any cache rules in the CDN module. For now, we also don’t want any business logic in the CDN, which makes the web server easier to test and manage.

We also use the Fastly origin shield to reduce the load on our web servers and improve cache ratios.

Changes and Deployment

We typically make changes in the simpleinfra repository, after which they are reviewed and manually applied by the infrastructure team.

Fastly NgWAF

We use the Fastly Web Application Firewall (NgWAF). It’s integrated with our Fastly Compute WASM module, so all blocking happens in the CDN and no malicious requests reach our origin servers.

When something is blocked, the user will see one of the following:

• status 406 NOT ACCEPTABLE for normal security rules, or
• status 429 Too Many Requests for rate limiting.

These status codes are only used by the NgWAF, so if a user sees one, the NgWAF is the component blocking the request.

Changes and Deployment

The integration between the Fastly CDN and NgWAF is implemented in our Compute WASM module.

In the legacy architecture, the rules are defined manually in the Signal Sciences dashboard. With the planned new infrastructure, we’ll start managing these in Terraform as well.

New or updated rules are typically distributed and active across Fastly’s CDN within one minute, though it can sometimes take two to three minutes.

Legacy Infrastructure

Here is a simplified diagram of the different moving pieces.

flowchart LR
  user[User] --> fastly[Fastly CDN]
  fastly <--> |uses| ngwaf[Fastly NgWAF]

  subgraph ec2[EC2 instance]
    nginx[nginx] --> web[web server] --> |accesses| psql[PostgreSQL database]
    watcher[index watcher] --> |enqueues builds| psql
    builder[builders × 4] --> |reads queued builds| psql
  end

  fastly --> nginx
  web -->|reads docs| s3[AWS S3]
  builder --> |uploads docs| s3

  watcher --> |pulls updates from| index[crates.io Git index]

Fastly CDN

The Fastly CDN caches responses from docs.rs and runs our Compute module at the edge. It integrates with the Fastly NgWAF to block malicious requests.

See Fastly CDN for implementation and deployment details.

Fastly NgWAF

Fastly’s Web Application Firewall filters malicious requests at the CDN before they reach our origin servers. See Fastly NgWAF for integration, rules, and deployment details.

EC2 Instance

Most of the legacy docs.rs infrastructure runs on a single large EC2 instance, including the following services:

• nginx
• web server
• index watcher
• build servers
• postgresql database

Nginx

Nginx:

• acts as a reverse proxy to our web server,
• compresses content, and
• authenticates with the CDN.

See Nginx for configuration and deployment details.

Web Server

The web server handles requests from nginx and serves docs.rs content. See Web Server for implementation details.

Index Watcher

The index watcher monitors the crates.io index and updates the build queue and stored releases. See Index Watcher for implementation details.

Build Servers

The build servers generate documentation for queued releases and upload it to S3. See Build Server for implementation details.

We currently run four parallel build servers.

PostgreSQL database

The database server runs locally on the EC2 instance. Currently we run postgres v10.

We have a psql wrapper script that directly connects to our database.

S3

We use S3 for object storage.

The service accesses S3 using the IAM role attached to the EC2 instance through its instance profile. The AWS SDK automatically obtains temporary credentials through IMDS.

Deployment

• SSH into the docs.rs server through the bastion host.
• Run docs_rs_admin build lock to lock the build queue.
• Check the build queue until no builds are in progress.
• Run the deploy-docs.rs Bash script.
• If you update content that is typically cached, run docs_rs_admin cdn purge all.
• Run docs_rs_admin build unlock to resume builds.

deploy-docs.rs will:

• tag the old HEAD as the previous release with git tag,
• pull the repository with git pull,
• run a build,
• copy the binaries,
• restart the systemd services,
• run database migrations, and
• purge the CDN of static content stored in the repository.

There is also a revert-docs.rs script that reverts to the previously tagged release. It doesn’t revert database migrations.

Configuration

To update a service’s configuration, update the corresponding dotenv file. See the systemd setup for details.

Nginx

Nginx:

• acts as a reverse proxy to our web server,
• compresses content, and
• authenticates with the CDN.

Before we had the NgWAF, nginx also handled rate limiting and IP blocking during attacks.

Changes are made manually on the server in /etc/nginx/, after which nginx must be restarted via systemd.

There is no test system.

systemd Service Setup

Each docs.rs systemd service has a configuration file in dotenv format. systemd reads these files to provide the environment in which the services run.

prune-disk-space

This scheduled daily systemd task performs cleanup to free disk space.

The cleanup commands and schedule are configured in /etc/systemd/system/prune-disk-space.{service,timer}.

# example, at the time of writing
docker container prune --force
docker image prune --force
cargo-sweep sweep /home/ubuntu/docs.rs --installed

Operations

These documents describe routine production operations and incident-response procedures for docs.rs.

See Common Maintenance Procedures for tasks involving the build queue, Rust toolchains, and crate administration.

Common Maintenance Procedures

Most procedures use the docs_rs_admin CLI, which connects directly to the database and S3. Run commands in an environment with the production configuration and credentials.

Build Queue

Rebuild a Specific Crate

Add the crate release to the queue:

$ docs_rs_admin queue add <CRATE_NAME> <VERSION>

The default priority is 5, lower than the priority 0 used for newly published releases. Override it when necessary:

$ docs_rs_admin queue add <CRATE_NAME> <VERSION> --priority <PRIORITY>

These days, crate authors can request rebuilds from the crates.io interface, so queuing rebuilds like this is more the exception.

Manage Default Build Priorities

Default priorities apply to crate names matching a PostgreSQL pattern. Lower numbers run first:

$ docs_rs_admin queue default-priority list
$ docs_rs_admin queue default-priority get <CRATE_NAME>
$ docs_rs_admin queue default-priority set 'tokio-%' -5
$ docs_rs_admin queue default-priority remove 'tokio-%'

Repository-specific overrides match repositories.name:

$ docs_rs_admin queue repository-priority list
$ docs_rs_admin queue repository-priority get <OWNER/REPOSITORY>
$ docs_rs_admin queue repository-priority set <OWNER/REPOSITORY> -10
$ docs_rs_admin queue repository-priority remove <OWNER/REPOSITORY>

Pin a Nightly Toolchain

If the latest nightly breaks documentation builds, pin a known-good nightly:

$ docs_rs_admin build set-toolchain nightly-YYYY-MM-DD

The builders read this setting from the database. No service restart is required. To resume using the latest nightly, set the toolchain back to nightly:

$ docs_rs_admin build set-toolchain nightly

After resolving an incident, use docs_rs_admin queue rebuild-broken-nightly to queue rebuilds for releases affected by one or more broken nightlies. See its --help output for the required date range.

Crate Administration

Override Build Limits for a Crate

First, inspect the crate’s current sandbox limit overrides:

$ docs_rs_admin database limits get <CRATE_NAME>

Set the overrides with docs_rs_admin database limits set. This command replaces the crate’s complete set of overrides: options you omit are cleared. Pass every existing override you want to retain. Memory is measured in bytes and timeout in seconds. For example, to allow 8 GiB of memory and a 15-minute timeout:

$ docs_rs_admin database limits set <CRATE_NAME> \
    --memory 8589934592 --timeout 900

Use the get, list, and remove subcommands to inspect or remove overrides.

Update Repository Statistics

Update GitHub and GitLab repository metadata with:

$ docs_rs_admin database update-repository-fields

Set DOCSRS_GITHUB_ACCESSTOKEN to a GitHub access token before running this command. DOCSRS_GITLAB_ACCESSTOKEN is optional; setting it raises the GitLab API rate limit, while leaving it unset uses unauthenticated requests.

Run or Revert Database Migrations

Apply all pending migrations with:

$ docs_rs_admin database migrate

Pass a migration version to move the database to that precise version, including reverting newer migrations:

$ docs_rs_admin database migrate <MIGRATION_VERSION>

Blacklist a Crate

Prevent future releases of a crate from being built:

$ docs_rs_admin database blacklist add <CRATE_NAME>

Use the list and remove subcommands to inspect or change the blacklist.

Warning: Blacklisting a crate does not remove content already published on the website.

When existing content must also be removed, use the docs_rs_watcher binary after blacklisting the crate:

$ docs_rs_watcher database delete crate <CRATE_NAME>

This deletes the crate’s existing releases from the database and storage.

Developing docs.rs

The root README contains the commands needed to set up, run, and test docs.rs.

This section contains additional development workflows and troubleshooting information that are useful after the basic setup is working.

Local Configuration

Docs.rs uses dotenv files for local configuration. Which file to edit depends on where the application process runs:

• .env configures commands and Rust binaries run directly on the host. Start by copying .env.sample.
• .docker.env configures docs.rs processes run by Docker Compose. Start by copying .docker.env.sample, or let the just recipes create an empty file.

The just command loads .env automatically. If you invoke cargo directly, source the file first or use a dotenv integration for your shell:

$ . ./.env
$ cargo run --bin docs_rs_web

CLI execution modes

The high-level CLI recipes can run their Rust binary on the host or through a one-off Docker Compose service:

The local mode uses cargo run. The docker mode builds and runs the corresponding cli, registry-watcher-cli, or builder-cli Compose service. In either mode, the recipes start the default PostgreSQL and S3 resources and apply pending database migrations when necessary.

Set a mode in .env to make it the default for the repository, or override it for one invocation:

$ DOCSRS_BUILDER_CLI_MODE=docker just builder build crate regex 1.3.1

Both mode variables accept local or docker. Use docker-run to bypass mode selection and name a Compose service explicitly:

$ just docker-run builder-cli build crate regex 1.3.1

.env controls mode selection and configures commands that run on the host. .docker.env configures the docs.rs process inside a Compose container; it does not select where a recipe runs.

The GUI end-to-end CI job sets both mode variables to docker. It prepares fixtures through the packaged admin and builder images before testing the packaged web image. The ordinary local defaults favor faster host builds where the builder supports them.

Accessing PostgreSQL

After starting the local database, open a psql session with:

$ just psql

This uses DOCSRS_DATABASE_URL from .env, falling back to the local development database configured by the Justfile.

Running PostgreSQL and S3-compatible storage outside Docker Compose is not a supported local-development configuration.

To invoke psql directly:

$ . ./.env
$ psql "$DOCSRS_DATABASE_URL"

Testing Metrics

Unit tests can inspect emitted metrics using the test metric provider from docs_rs_opentelemetry; tests using the shared context can call TestEnvironment::collected_metrics. For an end-to-end check, the Docker Compose setup provides an OpenTelemetry Collector configured to log the metrics it receives.

Start the collector with:

$ just compose-up-metrics

Configure the docs.rs process to export OTLP metrics to it:

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

Put this setting in .env when running docs.rs on the host, or in .docker.env when running it through Docker Compose. Restart the application after changing its environment.

The collector uses OTLP over gRPC on port 4317 and writes received metrics to its logs. Follow them with:

$ docker compose logs --follow opentelemetry

If OTEL_EXPORTER_OTLP_ENDPOINT is unset, docs.rs uses a no-op metric provider.

Troubleshooting Local Development

Docker reports exec user process caused "no such file or directory"

Check whether the affected entrypoint script has CRLF line endings. A CRLF line ending changes a hashbang such as #!/bin/sh into a request for the nonexistent interpreter /bin/sh\r inside the container.

Configure Git to retain LF line endings for files checked out on Windows:

$ git config core.autocrlf input

Then check out a fresh copy of the affected file.

A builder command reports Exec format error

Running builds directly on the host requires a compatible Linux environment. On platforms other than amd64 Linux, the builder recipe defaults to Docker mode automatically. If local mode was selected explicitly or the detected host is still incompatible, override it for the command:

$ DOCSRS_BUILDER_CLI_MODE=docker just builder build update-toolchain
$ DOCSRS_BUILDER_CLI_MODE=docker just builder build crate regex 1.3.1

To use Docker mode for all builder commands in this checkout, add the following to .env:

DOCSRS_BUILDER_CLI_MODE=docker

See CLI execution modes for the mode defaults and configuration behavior. To bypass mode selection entirely, run the Compose service explicitly with just docker-run builder-cli ….

Tests fail or time out unexpectedly

The test suite needs at least 4096 open file descriptors. Check the current limit and raise it for the current shell when necessary:

$ ulimit -n
$ ulimit -n 4096

Docker Compose

The local services are defined in the repository’s docker-compose.yml. The just recipes wrap common Compose operations, run database migrations when needed, and rebuild application images automatically when their inputs change.

List all available recipes with:

$ just --list

Build images directly

Docker images are defined in docker-bake.hcl. Build one image with its Bake target:

$ just docker-build build-server

Build all application images with:

$ just docker-build

After building, run the same packaged-binary smoke tests as CI with:

$ just docker-smoke-test

Or build and smoke-test the complete default group in one command:

$ just docker-test

The images are loaded into the local Docker daemon. For example, smoke-test the build-server image with:

$ docker run --rm docs-rs-build-server:ci --help

The Docker build stores cargo-chef’s compiled dependency baseline in an image layer that can be reused by remote builders. Application builds additionally share a Cargo target cache local to the Buildx builder, which enables incremental rebuilds after source changes. A new Buildx builder starts without that local incremental cache.

Run one-off CLI commands

The cli, watcher, and builder recipes select host or Compose execution through their configured CLI mode. See CLI execution modes for the defaults, environment variables, and configuration-file behavior.

Override a mode for one command when you want to keep using its high-level recipe:

$ DOCSRS_CLI_MODE=docker just cli-db-migrate
$ DOCSRS_CLI_MODE=docker just watcher queue set-last-seen-reference --head
$ DOCSRS_BUILDER_CLI_MODE=docker just builder build crate regex 1.3.1

Or bypass mode selection and run a specific Compose service directly:

$ just docker-run builder-cli build crate regex 1.3.1
$ just docker-run registry-watcher-cli queue set-last-seen-reference --head

Start services

Start the default PostgreSQL and S3 resources used by host-side development:

$ just compose-up-resources

Recipes that need these resources start them automatically. The explicit command is useful before running application binaries directly with Cargo.

Start individual application profiles with:

$ just compose-up-web
$ just compose-up-builder
$ just compose-up-watcher
$ just compose-up-metrics

Or start the complete local stack:

$ just compose-up-full

View logs

Follow logs for one or more services with:

$ just compose-logs db
$ just compose-logs web builder-a

The equivalent direct Compose command is:

$ docker compose logs --follow db

Stop or reset services

Stop services while keeping their volumes and other local data:

$ just compose-down

Remove this Compose project’s volumes, locally built images, and other local artifacts as well:

$ just compose-down-and-wipe

The second command deletes the local development database and object-storage contents.

GUI tests

Build the crate fixtures required by the GUI suite:

$ just prepare-gui-tests

This uses the builder recipe and therefore follows DOCSRS_BUILDER_CLI_MODE. The generated documentation is retained in the local PostgreSQL and S3 resources.

After preparing fixtures, start a temporary host web server and run the browser assertions with:

$ just run-gui-tests

This does not rebuild fixtures, making it the fast path for changes to templates, CSS, JavaScript, and web behavior. just compose-down preserves the fixtures; just compose-down-and-wipe removes them.

If a suitable web server is already listening on port 3000, run only the browser assertions with:

$ just run-gui-browser-tests

To reproduce the container integration path used in CI, run:

$ DOCSRS_CLI_MODE=docker DOCSRS_BUILDER_CLI_MODE=docker \
    just prepare-gui-tests run-gui-tests-e2e

This applies migrations through the packaged admin image, builds fixtures through the packaged build-server image, and serves the results from the packaged web-server image. Node and Puppeteer remain on the host so the test driver does not require a separate image.

Build Workspaces

When do we need this?

Many workspace packages do not need manual intervention and can be built simply by executing the commands listed in the main README.md file. However, some workspaces require an additional step. This is the case when values such as

version.workspace = true

are inherited from the workspaces Cargo.toml configuration file.

Fix

To build documentation, rustdoc requires a fully specified package but rustdoc does not understand workspaces which are only defined in cargo. Thus our crate needs to be packaged by cargo before running the documentation. This step will replace all of the value.workspace = true statements with their respective values.

cargo package

This will emit a packaged crate into the target/package/your_crate_name-version folder. Now the commands specified in our README.md can be executed targeting this folder.

cargo run -- build crate --local /path/to/source/target/package/your_crate_name-version/

Full MWE

To showcase when such problems can occur, take a look at the following example.

Structure

$ tree
.
├── Cargo.toml
├── my_lib
│   ├── Cargo.toml
│   └── src
│       └── lib.rs
└── README.md

3 directories, 4 files

The actual contents of my_lib do not matter, only the two configuration files.

$ cat Cargo.toml
[workspace]
members = [
        "my_lib",
]

[workspace.package]
version = "0.1.0"

and

$ cat my_lib/Cargo.toml
[package]
name = "my_lib"
version.workspace = true

[dependencies]

Building

The build command

cargo run -- build crate -l path/to/docs_rs_workspace_package/my_lib

fails with

Error: Building documentation failed

Caused by:
    Building documentation failed

Caused by:
    invalid Cargo.toml syntax

which makes sense due to

version.workspace = true

Fix

However when running the following sequence of commands

# Run this in the directory of docs_rs_workspace_package
cargo package -p my_lib

and then building again

# Run this from the docs.rs repo
cargo run -- build crate -l path/to/docs_rs_workspace_package/target/package/my_lib-0.1.0

then the build succeeds.

Updating Vendored Frontend Assets

Docs.rs keeps some frontend dependencies in the repository so builds do not depend on downloading assets at runtime.

Font Awesome

Font Awesome Free SVGs are packaged by the font-awesome-as-a-crate workspace crate. The vendored distribution, license, and generator live under crates/lib/font-awesome-as-a-crate/.

See the crate’s README for the update and release procedure.

Pure CSS

The minified Pure CSS files and their license live under crates/bin/docs_rs_web/vendor/pure-css/. The web crate’s build script reads pure-min.css and grids-responsive-min.css from that directory and combines them with the site’s generated CSS.

When updating Pure CSS, replace those minified files from an official Pure CSS release, update the vendored license when needed, and run:

$ cargo test --package docs_rs_web
$ just prepare-gui-tests run-gui-tests

For the container integration path, set both CLI modes to docker and run just prepare-gui-tests run-gui-tests-e2e instead.

Design

These documents describe parts of the docs.rs design and the decisions behind them.

Archive Storage and Indexes

This document explains how docs.rs stores rustdoc and source files in archive storage, and how single-file requests are served efficiently without downloading full archives.

Why this exists

• We store many files per crate version (HTML, CSS, JS, source files, etc.).
• Downloading full archives on every file request would be expensive.
• We need fast random access to one file at a time.

Storage model

For each crate version, docs.rs stores:

• one archive for rustdoc output;
• one archive for source files.

Both archives are ZIP files.

Archive format assumptions

ZIP compression is per-file, not whole-archive.

• Each file entry has its own compressed payload.
• This allows serving one file by reading only the corresponding byte range.

This differs from .tar.gz:

• tar concatenates files into one stream;
• gzip then compresses that full stream as a whole;
• random access to one file is much harder without scanning/decompressing much more data.

Index model

For each archive, docs.rs generates an index stored as an SQLite database.

The index is conceptually similar to a ZIP central directory, but represented in a queryable format.

Each row maps a logical file path to the location of its compressed payload in the archive:

• filename/path in archive;
• byte range start (from), inclusive;
• byte range end (to), inclusive;
• compression algorithm used for that file entry.

The range points to the compressed payload bytes of the ZIP entry (starting at ZIP data_start), not to metadata headers.

We store the index file next to the ZIP on S3.

Request flow (source or rustdoc file)

When a request asks for a single file:

1. Resolve which archive and index correspond to the crate/version.
2. Check whether the index is already present in local cache.
3. If missing, download the index file and cache it locally.
4. Query the SQLite index for the requested filename/path.
5. If no row exists, return “not found”.
6. If found, read from/to/compression information.
7. Issue an HTTP Range request to S3 for only [from, to] from the remote ZIP file.
8. Decompress the returned byte range using the per-entry compression algorithm recorded in the index.
9. Return/use the decompressed file bytes as response content.

Important details

• The local cache key includes latest_build_id (effectively ...zip.<build_id>.index) so rebuilt releases naturally use a fresh local cache entry.
• Index cache population is concurrency-safe:
  • optimistic read first (no lock),
  • then per-index lock for repair/download,
  • temp-file download + atomic rename to publish.
• If index lookup/decompression fails (for example stale offsets causing decompression errors), docs.rs purges local cached index files and retries.

Key properties

• Efficient network usage: fetches only bytes for the requested file.
• Efficient CPU usage: decompresses only one ZIP entry.
• Good cache behavior: index files can be cached locally and reused across many requests.
• better manageability or our S3 bucket, especially around rebuilds or deletions of crates or releases.

Compression layers

There are two different compression layers involved:

• ZIP entry compression (inside rustdoc/source archives): this is what the archive index stores per file and what is used to decompress range responses. At the time of writing, archive index creation supports bzip2 ZIP entries.
• Object storage compression (for regular blob uploads): this is separate and represented by blob Content-Encoding/storage metadata.

For archive file serving, the important algorithm is the ZIP entry compression from the index.

Archive downloads

• these archives can also be downloaded to be used for offline docs.
• This is also why we used a more widely supported algorithm (Bz2) instead of zstd inside zip, which would theoretically also have been possible.

Handler behavior and fallbacks

• Rustdoc/source handlers use the same archive lookup primitives.
• On missing rustdoc paths, handlers may try path fallbacks (for example appending /index.html) before returning 404.
• For target-specific misses, handlers can redirect to target fallback/search routes instead of always returning a hard 404.

Notes for maintainers

• Keep index schema and archive writer in sync.
• Any change in byte-offset computation must preserve correct range boundaries.
• If index lookup fails unexpectedly, prefer rebuilding/downloading index rather than falling back to full archive downloads.