Introduction

The cargo-bisect-rustc tool makes it super easy to find exactly when behavior has regressed in rustc. It automatically downloads rustc artifacts and tests them against a project you provide until it finds the regression.

The Installation chapter shows how to install cargo-bisect-rustc. For a quick introduction, see the Tutorial. Otherwise, start at the Basic usage chapter to learn how cargo-bisect-rustc works.

Installation

The basic method for installing cargo-bisect-rustc is:

cargo install cargo-bisect-rustc

Additional options are described below.

Requirements

Besides having a working Rust installation, you may need a few other things installed on your system:

• Unix:
  • pkg-config
  • OpenSSL (libssl-dev on Ubuntu, openssl-devel on Fedora or Alpine)
• macOS:
  • OpenSSL (homebrew is recommended to install the openssl package)
• rustup

If you’re having trouble using the system OpenSSL installation, it can be built from scratch. The following will enable the vendored OpenSSL build:

cargo install cargo-bisect-rustc --features git2/vendored-openssl

Beware that this also requires perl and make to be installed.

RUST_SRC_REPO

cargo-bisect-rustc needs to access the git log of the rust repo. You can set the default location of that when installing it:

RUST_SRC_REPO=/path/to/rust cargo install cargo-bisect-rustc

See Rust source repo for more about configuring how cargo-bisect-rustc retrieves this information.

Basic usage

Using cargo-bisect-rustc simply involves running it inside a Cargo project that reproduces the regression:

cargo bisect-rustc

For a quick introduction, see the Tutorial.

cargo-bisect-rustc works by building a Cargo project, and detecting if it succeeds or fails. It will download and use nightly Rust toolchains. It begins with two nightly boundaries, known as the start where the project successfully builds (the baseline), and the end where it is known to fail (the regression). It will then do a binary search between those dates to find the nightly where the project started to fail.

Once it finds the nightly where it started to fail, cargo-bisect-rustc will then try to find the individual PR where it regressed. The Rust project keeps the builds of every merged PR for the last 167 days. If the nightly is within that range, then it will bisect between those PRs.

And even further, if the regression is in a rollup PR, then it will bisect the individual PRs within the rollup. This final bisection is only available for x86_64-unknown-linux-gnu since it is using the builds made for the rustc performance tracker.

Rust src repo

cargo-bisect-rustc needs to read the git log of the rust-lang/rust repo in order to scan individual commits. See the Rust src repo chapter for details on how to configure how it finds the git repo.

Boundaries

Without setting any options, cargo-bisect-rustc will try to automatically find the start where the build succeeds and the end where it fails. This can take some time, depending on how far back it needs to scan. It is recommended to use the --start and --end CLI options to tell it where the boundaries are.

cargo bisect-rustc --start=2022-11-01 --end=2023-02-14

See the Bisection boundaries chapter for more details on setting these options.

Regression check

By default, cargo-bisect-rustc assumes the start boundary successfully builds, and the end boundary fails to build. You can change this using the --regress CLI option. For example, you can tell it that the start should fail, and the end should pass. There are several options you can use with the --regress flag:

See Scripting for customizing this behavior.

Custom commands

By default, cargo-bisect-rustc runs cargo build. You can change which cargo command is run by passing additional arguments after --:

cargo bisect rustc -- test --test mytest

Scripting

You can use an arbitrary script for determining what is a baseline and regression. This is an extremely flexible option that allows you to perform any action automatically. Just pass the path to the script to the --script CLI command:

cargo bisect-rustc --script ./test.sh

The script should exit 0 for the baseline, and nonzero for a regression. Since cargo-bisect-rustc sets RUSTUP_TOOLCHAIN (see Rustup toolchains), all you need to do is call cargo or rustc, and the script should automatically use the toolchain that is currently being tested.

#!/bin/sh

set -ex

# This checks that a warning is only printed once.
# See https://github.com/rust-lang/rust/issues/88256 for a regression where it
# started printing twice.

OUTPUT=`cargo check 2>&1`
COUNT=`echo "$OUTPUT" | grep -c "unnecessary parentheses"`
test $COUNT -eq 1

If you need to use the targets directly without using cargo in the script, they are available in $CARGO_TARGET_DIR/[release|debug]/..., since cargo-bisect-rustc sets $CARGO_TARGET_DIR.

Check out the examples chapters for several examples of how to use this option.

Tutorial

cargo-bisect-rustc works by building a Cargo project and checking if it succeeds or fails. This tutorial walks through an example of this process.

Finding a regression

Create a cargo project that demonstrates the regression. Let’s use issue #53157 as an example:

cargo new foo
cd foo

Edit src/main.rs with the example from the issue:

macro_rules! m {
    () => {{
        fn f(_: impl Sized) {}
        f
    }}
}

fn main() {
    fn f() -> impl Sized {};
    m!()(f());
}

Since we are testing an old regression, also edit Cargo.toml to remove the edition = "2021" field which isn’t supported in these versions.

Then run cargo bisect-rustc --end=2018-08-04.

We need to provide the end point for this particular example because that’s an old regression already fixed on the latest nightlies. We could also provide a start point if we know one; that’s going to make it faster by avoiding scanning for the start. For instance:

cargo bisect-rustc --start=2018-05-07 --end=2018-08-04

It will run cargo build in the project and check whether or not it fails. It will do a binary search between the start and end range to find exactly where the regression occurred.

Note: You can also use the flag --regress to specify other common regression criteria, e.g. --regress=ice for internal compiler errors.

In our example, in just a few steps, we can we find that it stopped working on nightly-2018-07-30.

If the regression is recent enough, then it will print out a list of PRs that were committed on that date. In this particular example, it is too old, so we’ll need to manually inspect the git log to see which PR’s were merged.

If the nightly was within the last 167 days, then cargo-bisect-rustc will then start bisecting those individual PRs.

After finding potential candidates, you can go inspect those PRs to see which one is the likely cause. In this case, since the ICE was in MIR const propagation, and #51361 is the likely candidate since it modified const evaluation.

Testing interactively

Pass/fail of cargo build may not be what you’re after. Perhaps the issue is an error message changed, so both the “good” and “bad” version will fail to compile, just with a different message. Or maybe something used to fail, and now erroneously passes. You can use the interactive feature with the --prompt flag to visually inspect a build and tell cargo-bisect-rustc what’s “good” and what’s “bad”. Let’s use issue #55036 as an example where an error message changed:

In Cargo.toml, remove the edition field (this example was before editions).

src/main.rs:

struct Foo {
    bar: i32
}

trait Baz {
    fn f(Foo { bar }: Foo) {}
}

fn main() {}

This historically emitted a bad error, was updated to emit a nice error (E0642 added in #53051), but then that nice error was lost somewhere (on the 2015 edition). Let’s find where it was lost! Grab the ranges between where it was added and where we know it fails:

cargo bisect-rustc --prompt \
    --start=2018-08-14 \
    --end=2018-10-11

At each step, cargo-bisect-rustc will show the output and ask you:

nightly-2018-08-14 finished with exit code Some(101).
please select an action to take:
> mark regressed
  mark baseline
  retry

Choose mark baseline with the nice E0642 message, and mark regressed with the less-favorable token error. Fairly quickly we find it regressed in nightly-2018-10-11. The most likely candidate is #54457 which is a rollup PR. It’s usually not too hard to look through the commits and find a likely culprit. Indeed in this example, #54415 modified function parameter parsing.

Testing with a script

Using the --script option allows you to do something more fancy than just cargo build. Maybe you need to run cargo multiple times, or just call rustc directly, or you want to automatically grep through the output. The possibilities are endless! Just write a little shell script that exits 0 for the baseline, and exits nonzero for the regression. As an example, the previous interactive session can be hands-free automated with this script:

test.sh:

#!/bin/sh

# Fail if we no longer get a `E0642` error:
cargo check 2>&1 | grep E0642

And then run:

cargo bisect-rustc --script=./test.sh \
    --start=2018-08-14 \
    --end=2018-10-11

Custom bisection messages

Available from v0.6.9

You can add custom messages when bisecting a regression. Taking inspiration from git-bisect, with term-new and term-old you can set custom messages to indicate if a regression matches the condition set by the bisection.

Example:

cargo bisect-rustc \
    --start=2018-08-14 \
    --end=2018-10-11 \
    --term-old "No, this build did not reproduce the regression, compile successful" \
    --term-new "Yes, this build reproduces the regression, compile error"

In other words, --term-old is displayed for older compilers that do not exhibit the regression. --term-new is for newer compilers which do exhibit the regression.

What counts as a “regression” is defined by the --regress CLI option. By default, a regression is a compile-error (which is equivalent to --term-new). If you flip the definition of a “regression” with --regress=success, then a regression is a successful compile (which is also equivalent to --term-new).

There are default terms based on the current --regress setting. Customizing the terms is most useful when using scripting. For example, in the Documentation changes example, the customized terms can more clearly express the results of the script of whether or not it found what it was looking for in the documentation.

Rust source repo

For cargo-bisect-rustc to work, it needs to be able to read the git log of the rust-lang/rust repo. cargo-bisect-rustc supports several methods for this described below.

GitHub API

By default, cargo-bisect-rustc uses the GitHub API to fetch the information instead of using a local checkout.

cargo bisect-rustc --access=github

Beware that GitHub has restrictive rate limits for unauthenticated requests. It allows 60 requests per hour, and cargo-bisect-rustc will use about 10 requests each time you run it (which can vary depending on the bisection). If you run into the rate limit, you can raise it to 5000 requests per hour by setting the GITHUB_TOKEN environment variable to a GitHub personal token. If you use the gh CLI tool, you can use it to get a token:

GITHUB_TOKEN=`gh auth token` cargo bisect-rustc --access=github

If you don’t use gh, you’ll just need to copy and paste the token.

Local clone

cargo-bisect-rustc can also clone the rust repo in the current directory (as rust.git). This option can be quite slow if you don’t specify the repo path at build time. You can specify this with the --access CLI argument:

cargo bisect-rustc --access=checkout

RUST_SRC_REPO environment variable

You can specify the location of the rust repo with the RUST_SRC_REPO environment variable at runtime. This is useful if you already have it checked out somewhere, but is cumbersome to use.

RUST_SRC_REPO=/path/to/rust cargo bisect-rustc

RUST_SRC_REPO environment variable (build-time)

Setting the RUST_SRC_REPO environment variable when installing cargo-bisect-rustc will set the default location for the rust repo. This is recommended if you already have the rust repo checked out somewhere.

RUST_SRC_REPO=/path/to/rust cargo install cargo-bisect-rustc

Bisection boundaries

cargo-bisect-rustc does a binary search for the regression using a start and end boundary. You can specify these boundaries with the --start and --end CLI flags. There are several ways to specify what those boundaries are. If you run the command without specifying the boundaries, it will search for them automatically:

# No --start or --end flags
cargo bisect-rustc

This will assume the latest nightly is a regression (the end boundary). It will then search backwards until it can find a nightly that passes to use as the start boundary. Bisection can usually go faster if you happen to know the start boundary, so that it doesn’t need to search for it.

--start and --end are optional. If --start is not specified, then it will try to find the start range automatically. If --end is not specified, it will assume it is the most recently available.

Date boundaries

You can pass a date in the form YYYY-MM-DD to the --start and --end flags. It will download the nightly corresponding to that date, and then begin bisecting those nightlies.

cargo bisect-rustc --start=2018-08-14 --end=2018-10-11

If the nightly with the regression was within the past 167 days, then it will automatically start bisecting the individual PRs merged on that day using Git commit boundaries.

Git commit boundaries

You can pass the particular git commit hash of a PR as a boundary. The Rust project keeps the builds of every merged PR for the last 167 days. If you happen to know the PR to use as a boundary, you can pass the SHA-1 hash of that PR.

cargo bisect-rustc \
    --start=6323d9a45bdf0ac2a9319a6a558537e0a7e6abd1 \
    --end=866a713258915e6cbb212d135f751a6a8c9e1c0a

There are several ways to determine the SHA-1 hash for a PR.

• On the PR itself, you should see a message like “bors merged commit c50c62d into rust-lang:master”. You can copy that hash to use as a boundary. If the PR was merged as part of a rollup, you will need to use the hash of the rollup instead. You’ll need to look through the PR messages to see if the PR was mentioned from a rollup PR.
• In the rust repo, run git log --first-parent upstream/master (where upstream is your origin name for rust-lang/rust). This will show all the top-level commits. You can then search for your PR.

Note: If the PR was merged after the most recent nightly, you’ll need to be sure to also specify the --end range. Otherwise it will assume the most recent nightly is the end and it won’t work if the start is after the end.

If the regression is found in a rollup PR, then cargo-bisect-rustc will bisect the individual PRs within the rollup. This final bisection is only available for x86_64-unknown-linux-gnu since it is using the builds made for the rustc performance tracker.

Note: If you specify date boundaries, then you can use the --by-commit CLI option to force it to use PR commits instead of nightlies.

Git tag boundaries

The boundary can be specified with a git release tag. This is useful if you know something works in one release and not another, but you don’t happen to know which nightly this corresponds with. When given a tag, cargo-bisect-rustc will try to find the nightly that corresponds with that release. For example:

cargo bisect-rustc --start=1.58.0 --end=1.59.0

Monotonicity

When writing your test and picking a bisection range, you should be careful to ensure that the test won’t vary between pass/fail over the bisection range. It should only transition from good to bad once in the bisection range (it must change monotonically).

In the following example, cargo-bisect-rustc will find one of the transitions, but that may not be the true root cause of the issue you are investigating.

nightly-2023-02-01 baseline **start**
nightly-2023-02-02 baseline
nightly-2023-02-03 baseline
nightly-2023-02-04 regression
nightly-2023-02-05 regression
nightly-2023-02-06 baseline
nightly-2023-02-07 regression
nightly-2023-02-08 regression **end**

Here it may either find 2023-02-04 or 2023-02-07 as the regression.

The following are some suggestions for avoiding or dealing with this problem:

• Make sure your test reliably exhibits the issue you are looking for, and does not generate any false positives or false negatives.
• Analyze the PR that was reported as the regression. Do the changes in the PR seem to be a probable cause?
• Try to keep the bisection range small to reduce the probability that you will encounter multiple regression transitions.
• Use the -vv flag (very verbose) to display the output from the compiler to make sure it is what you expect.
• Use the --prompt flag to inspect the output and verify each step.
• Beware that some issues may get fixed and then regress multiple times. Try to keep the bisection range as close to the present day as possible. Compare the output of the “regressed” commit to the latest nightly to see if they are the same.
• If the test only fails sporadically, use a script to run the compiler many times until it fails or it passes enough iterations that you feel confident that it is good.
• If the code requires relatively new language features, be careful not to pick a starting range that is too old.
• Beware of code-generation bugs that can be sensitive to code layout. Since the code to rustc changes rapidly over time, code can shift around causing different layouts and optimizations, which might cause an issue to appear and disappear several times over the bisection range.

Rustup toolchains

cargo-bisect-rustc takes advantage of rustup toolchains for installation and selecting the correct rustc to run. It will essentially run cargo +bisector-nightly-2023-03-18-x86_64-unknown-linux-gnu build using rustup toolchain override shorthand to run the toolchains that it downloads. This sets the RUSTUP_TOOLCHAIN environment variable to the toolchain name, which ensures that any call to rustc will use the correct toolchain.

By default, cargo-bisect-rustc will delete toolchains immediately after using them. You can use the --preserve option to keep the toolchains so that you can use them manually. See the Preserving toolchains example for more details.

When using the --script option, the script should just invoke cargo or rustc normally, and rely on the RUSTUP_TOOLCHAIN environment variable to pick the correct toolchain.

Git bisect a custom build

There are some rare cases where you may need to build rustc with custom options, or otherwise work around issues with pre-built compilers not being available. For this you can use git bisect to build the compiler locally.

It can be helpful to use the --first-parent option so that it only bisects the merge commits directly reachable on the master branch. Otherwise the bisecting may land on intermediate commits from within a PR which may not build or test correctly.

To start the bisection, specifying the boundaries where the bisection will start:

git bisect start --first-parent
git bisect good 96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef
git bisect bad a00f8ba7fcac1b27341679c51bf5a3271fa82df3

Then, build the compiler as needed and run your tests to check for a regression:

./x.py build std
rustc +stage1 foo.rs

You may want to consider running ./x.py clean if you are running into issues since changes to the internal structures of build artifacts aren’t always versioned, and those changes can be incompatible. Incremental caches are particularly susceptible, so you may want to turn that off if you have turned them on.

If you determine the current version is good or bad, run git bisect good or git bisect bad to mark that, and then repeat building and marking until finished.

Similar to cargo-bisect-rustc, git bisect supports scripting and lots of other goodies. Check out its documentation for more.

Alt builds

Each commit also generates what are called “alt” builds. These are builds of rustc with some different options set. As of August 2023, these include:

• rust.parallel-compiler
• llvm.assertions
• rust.verify-llvm-ir

For more information on these settings, see the config.toml docs. These alt settings are defined in ci/run.sh.

Alt builds are only available for a few targets. Look for the -alt builds in ci.yml.

This can be useful if you are bisecting an LLVM issue. With LLVM assertions enabled, alt builds have checks that can help identify broken assumptions.

Alt builds are only made for commit builds, and not nightly releases. You will need to specify --by-commit (or use a hash in the --start or --end flags) to only use commit builds.

cargo bisect-rustc --alt --by-commit

Examples

The following chapters show examples of different ways of using cargo-bisect-rustc.

Checking diagnostics

The following is an example of checking when the diagnostic output of rustc changes. For example, this can check when either the wording has changed, or a different error or warning is produced.

#109067 is an example of where this is necessary. A warning started being emitted, and it is the kind of warning that cannot be turned into an error with deny(warnings).

The following script is intended to be used with the --script option (set the executable flag on the script, chmod u+x):

#!/bin/sh

OUTPUT=`cargo check 2>&1`
# Comment out this test if your example is intended to fail.
if [ $? -ne 0 ]
then
    echo "Build unexpectedly failed: $OUTPUT"
    exit 1
fi
# Display the output for debugging purposes.
# Run `cargo-bisect-rustc` with `-vv` to view the output.
echo "$OUTPUT"
# This indicates a regression when the text "non-ASCII" is in the output.
#
# If the regression is when the text is *not* in the output, remove the `!` prefix
# (and customize the `--term-old` and `--term-new` CLI options if you want).
! echo "$OUTPUT" | grep "non-ASCII"

Then run something like:

cargo bisect-rustc --start=1.67.0 --end=1.68.0 --script ./test.sh \
    --term-old="No warning" --term-new="Found non-ASCII warning"

Scripting on Windows

Using the --script option on Windows can be cumbersome because Windows does not support #! scripts like Unix does, and the built-in scripting can also be awkward. The following sections show the different ways you can use scripting.

Batch file

You can use DOS-style .bat files:

test.bat:

(cargo check 2>&1) | find "E0642"

This can be executed directly with:

cargo-bisect-rustc --script ./test.bat

But .bat can be challenging to do more complex options, or you may not be familiar with it.

Powershell

You can’t execute .ps1 Powershell files directly, so you will need to use pwsh to launch them:

test.ps1:

( cargo check 2>&1 ) | grep E0642
if ( -Not $? ) {
    exit 1
}

This can be run with:

cargo-bisect-rustc --script pwsh -- -File ./test.ps1

Bash

If you have Git-for-Windows installed, then you can use its copy of bash to run bash scripts:

test.sh:

#!/bin/bash

cargo check 2>&1 | grep E0642

This can be run with:

cargo-bisect-rustc --script "C:\\Program Files\\Git\\usr\\bin\\bash.exe" -- ./test.sh

This also works if you have bash from something like msys2 installed.

Incremental compilation

Testing for regressions with incremental compilation may require running a command multiple times. The following illustrates an example for #87384 which only generates a warning the second time a build is run with incremental. Previously no warning was emitted.

foo.rs:

#![type_length_limit = "95595489"]

pub fn main() {
    println!("Hello, world!");
}

Create a script test.sh:

#!/bin/sh

# Exit if any command fails.
set -e

rm -rf incremental
rustc foo.rs --crate-type lib -C incremental=incremental
echo second
OUTPUT=`rustc foo.rs --crate-type lib -C incremental=incremental 2>&1`
echo $OUTPUT
! echo "$OUTPUT" | grep \
    "crate-level attribute should be in the root module"

Run this script with:

cargo-bisect-rustc --start 1.54.0 --end 1.55.0 --script ./test.sh

Slow or hung compilation

Some regressions may involve the compiler hanging or taking an unusually long time to run. The --timeout CLI option can be used to check for this. Let’s use #89524 as an example. A particular combination of factors caused the compiler to start to hang.

Change Cargo.toml to the following:

[package]
name = "slow"
version = "0.1.0"

[dependencies]
config = "=0.9.3"

[profile.release]
panic = "abort"
codegen-units = 1

Then use the timeout option:

cargo-bisect-rustc --start=2021-09-01 --end=2021-10-02 --timeout 30 -- build --release

You may need to adjust the timeout value based on the speed of your system.

Note: --timeout is currently not working on macOS. See https://github.com/rust-lang/cargo-bisect-rustc/issues/232.

Using extra components

By default, cargo-bisect-rustc only fetches rustc, cargo, rustdoc, and the standard library for the host. You may need additional Rustup Components to run your test. Some examples of when this might be needed are:

• You want to find a regression in Clippy (see Bisecting Clippy), or miri.
• Scanning for when some documentation changed (see Documentation changes).
• The platform needs additional things. For example, bisecting x86_64-pc-windows-gnu host may need the rust-mingw component.

If you are testing cross-compilation, use the --target option to download the standard library for the target you are using.

The following example shows how to use components to do a bisection with Cargo’s build-std feature.

cargo-bisect-rustc --start=2022-11-01 --end=2022-11-20 -c rust-src -- build -Zbuild-std

Note: The --with-src option is an alias for -c rust-src.
The --with-dev option is an alias for -c rustc-dev -c llvm-tools.

Running without cargo

Some bisections don’t require Cargo. You can use the --without-cargo option to skip installing cargo which can speed up the bisection since it doesn’t need to download cargo, and doesn’t have the overhead of running cargo. You will need to pair this with --script since cargo-bisect-rustc assumes projects use Cargo.

For example, using a simple rustc command:

cargo-bisect-rustc --start=2022-11-01 --end=2022-11-20 --without-cargo --script=rustc -- foo.rs

Note: You can use --without-cargo while still using a Cargo project. Rustup will fall back to using cargo from your installed nightly, beta, or stable toolchain. However, this isn’t recommended since cargo is only intended to work with the version it is released with, and can sometimes be incompatible with different versions. But if you are bisecting a very recent change, then you can probably get away with it.

Preserving toolchains

You may want to reuse the toolchains downloaded by cargo-bisect-rustc for doing further analysis or debugging. Or, while setting up your regression test, you may need to adjust your test and script several times, and downloading the same toolchains multiple times can be quite slow.

You can do this with the --preserve option.

cargo bisect-rustc --start=2023-01-01 --end=2023-02-01 --preserve

The toolchains will be kept in your Rustup home directory (typically ~/.rustup/toolchains).

Toolchains for nightlies will have the form of bisector-nightly-YYYY-MM-DD-<target>. Toolchains for PR artifacts will have the form of bisector-ci-<hash>-<target>.

You can run these toolchains using a Rustup override, like this:

cargo +bisector-nightly-2023-03-18-x86_64-unknown-linux-gnu build
# or...
cargo +bisector-ci-e187f8871e3d553181c9d2d4ac111197a139ca0d-x86_64-unknown-linux-gnu build

When you are done, you’ll probably want to clean up these directories since they use a lot of space. The easiest method is to just delete the directories:

rm -rf ~/.rustup/toolchains/bisector-*

Manually installing

The --install option can be used to only install a toolchain. This won’t do a bisection, it is just for fetching a toolchain for testing.

cargo bisect-rustc --install e187f8871e3d553181c9d2d4ac111197a139ca0d

Note: See also rustup-toolchain-install-master which is specialized for installing CI artifacts.

Bisecting Rustdoc

cargo-bisect-rustc can be used to check for Rustdoc regressions, too. All you need to do is instruct it to use the correct command.

The following example will check to find a regression when cargo doc suddenly starts to fail.

cargo bisect-rustc --start=2022-08-05 --end=2022-09-09 -- doc

Some rustdoc regressions might be in the generated HTML output. To scan the output, you can use a script like the following:

test.sh:

#!/bin/sh

# Exit if any command fails.
set -e

cargo doc

grep "some example text" $CARGO_TARGET_DIR/doc/mycrate/fn.foo.html

This can be used with the --script option:

cargo-bisect-rustc --start=2023-01-22 --end=2023-03-18 --script=./test.sh \
    --term-old="Found example text" --term-new="Failed, or did not find text"

Bisecting clippy

cargo-bisect-rustc can be used to check for Clippy regressions, too. You’ll need to instruct it to download clippy, and run the command correctly:

cargo bisect-rustc --start=1.67.0 --end=1.68.0 -c clippy -- clippy

Note that depending on what you are looking for, this may just find a PR that syncs the rust-clippy repo to rust-lang/rust. You may be able to scan the list of changes in that PR to discover what you are looking for. If the list of changes is too big or nothing is jumping out as a possible culprit, then consider using git bisect on the clippy repo itself (which will require building clippy).

Documentation changes

cargo-bisect-rustc can be used to scan for changes in the documentation shipped with each release. This includes all the books and standard library documentation. To do this, instruct it to download the component, and use a script that scans for whatever you are looking for. You can use rustup doc --path or rustc --print=sysroot to find the proper location. For example:

test.sh:

#!/bin/sh

# Exit if any command fails.
set -e

STD=`dirname $(rustup doc --std --path)`

# Checks if a particular file exists.
# This could also be `grep` or any other kinds of tests you need.
if [ -e $STD/io/error/type.RawOsError.html ]
then
    echo "found"
    exit 1
fi

And run with:

cargo bisect-rustc --start 1.68.0 --end 1.69.0 -c rust-docs --script ./test.sh \
    --term-old="Did not find" --term-new="Found"

Note: This may not work on all targets since cargo-bisect-rustc doesn’t properly handle rustup manifests, which alias some targets to other targets. Use --host x86_64-unknown-linux-gnu in that situation.

Flaky errors

Some tests may fail randomly. The following script is an example that will run rustc repeatedly to check for a failure. This example is from #108216 (which requires macOS).

test.sh:

#!/bin/sh

rm -rf *.o incremental foo

echo "fn main() { let a: i64 = 1 << 64; }" > foo1.rs
echo "fn main() { let a: i64 = 1 << 63; }" > foo2.rs

ARGS="--crate-name foo -C split-debuginfo=unpacked -C debuginfo=2 -C incremental=incremental"

for i in {1..20}
do
    echo run $i
    rustc foo1.rs $ARGS && { echo "ERROR: first build should have failed"; exit 1; }
    rustc foo2.rs $ARGS || { echo "ERROR: second build should have passed"; exit 1; }
    ./foo || { echo "ERROR: executing should have passed"; exit 1; }
done

This test can be run with:

cargo bisect-rustc --start=1.57.0 --end=1.58.0 --script=./test.sh

In general, configure the script to perform whichever actions you need in a for loop that runs enough times that you have a high confidence it has found the regression.