Introduction
The cargo-bisect-rustc
tool makes it super easy to find exactly when behavior has regressed in rustc.
It automatically downloads rustc artifacts and tests them against a project you provide until it finds the regression.
The Installation chapter shows how to install cargo-bisect-rustc
.
For a quick introduction, see the Tutorial.
Otherwise, start at the Basic usage chapter to learn how cargo-bisect-rustc
works.
Installation
The basic method for installing cargo-bisect-rustc
is:
cargo install cargo-bisect-rustc
Additional options are described below.
Requirements
Besides having a working Rust installation, you may need a few other things installed on your system:
- Unix:
- pkg-config
- OpenSSL (
libssl-dev
on Ubuntu,openssl-devel
on Fedora or Alpine)
- macOS:
- OpenSSL (homebrew is recommended to install the
openssl
package)
- OpenSSL (homebrew is recommended to install the
- rustup
If you’re having trouble using the system OpenSSL installation, it can be built from scratch. The following will enable the vendored OpenSSL build:
cargo install cargo-bisect-rustc --features git2/vendored-openssl
Beware that this also requires perl
and make
to be installed.
RUST_SRC_REPO
cargo-bisect-rustc
needs to access the git log of the rust repo.
You can set the default location of that when installing it:
RUST_SRC_REPO=/path/to/rust cargo install cargo-bisect-rustc
See Rust source repo for more about configuring how cargo-bisect-rustc
retrieves this information.
Basic usage
Using cargo-bisect-rustc
simply involves running it inside a Cargo project that reproduces the regression:
cargo bisect-rustc
For a quick introduction, see the Tutorial.
Note: On Windows, due to an issue with rustup, you will need to execute
cargo-bisect-rustc
with a-
betweencargo
andbisect
.
cargo-bisect-rustc
works by building a Cargo project, and detecting if it succeeds or fails.
It will download and use nightly Rust toolchains.
It begins with two nightly boundaries, known as the start where the project successfully builds (the baseline), and the end where it is known to fail (the regression).
It will then do a binary search between those dates to find the nightly where the project started to fail.
Once it finds the nightly where it started to fail, cargo-bisect-rustc
will then try to find the individual PR where it regressed.
The Rust project keeps the builds of every merged PR for the last 167 days.
If the nightly is within that range, then it will bisect between those PRs.
And even further, if the regression is in a rollup PR, then it will bisect the individual PRs within the rollup.
This final bisection is only available for x86_64-unknown-linux-gnu
since it is using the builds made for the rustc performance tracker.
Rust src repo
cargo-bisect-rustc
needs to read the git log of the rust-lang/rust
repo in order to scan individual commits.
See the Rust src repo chapter for details on how to configure how it finds the git repo.
Boundaries
Without setting any options, cargo-bisect-rustc
will try to automatically find the start where the build succeeds and the end where it fails.
This can take some time, depending on how far back it needs to scan.
It is recommended to use the --start
and --end
CLI options to tell it where the boundaries are.
cargo bisect-rustc --start=2022-11-01 --end=2023-02-14
See the Bisection boundaries chapter for more details on setting these options.
Regression check
By default, cargo-bisect-rustc
assumes the start boundary successfully builds, and the end boundary fails to build.
You can change this using the --regress
CLI option.
For example, you can tell it that the start should fail, and the end should pass.
There are several options you can use with the --regress
flag:
Option | Start | End | Description |
---|---|---|---|
error | Succeed | Fail | The default setting checks for a failure as the regression. |
success | Fail | Succeed | Reverses the check to find when something is fixed. |
ice | No ICE | ICE | Scans when an Internal Compiler Error (ICE) was introduced. |
non-ice | ICE | No ICE | Scans when an ICE was fixed. |
non-error | Non-ICE Failure | Succeed or ICE | Scans when an ill-formed program stops being properly rejected, or the compiler starts generating an ICE. |
See Scripting for customizing this behavior.
Custom commands
By default, cargo-bisect-rustc
runs cargo build
.
You can change which cargo
command is run by passing additional arguments after --
:
cargo bisect rustc -- test --test mytest
Scripting
You can use an arbitrary script for determining what is a baseline and regression.
This is an extremely flexible option that allows you to perform any action automatically.
Just pass the path to the script to the --script
CLI command:
cargo bisect-rustc --script ./test.sh
The script should exit 0 for the baseline, and nonzero for a regression.
Since cargo-bisect-rustc
sets RUSTUP_TOOLCHAIN
(see Rustup toolchains), all you need to do is call cargo
or rustc
, and the script should automatically use the toolchain that is currently being tested.
#!/bin/sh
set -ex
# This checks that a warning is only printed once.
# See https://github.com/rust-lang/rust/issues/88256 for a regression where it
# started printing twice.
OUTPUT=`cargo check 2>&1`
COUNT=`echo "$OUTPUT" | grep -c "unnecessary parentheses"`
test $COUNT -eq 1
If you need to use the targets directly without using cargo
in the script, they are available in $CARGO_TARGET_DIR/[release|debug]/...
, since cargo-bisect-rustc
sets $CARGO_TARGET_DIR
.
Check out the examples chapters for several examples of how to use this option.
Tutorial
cargo-bisect-rustc
works by building a Cargo project and checking if it succeeds or fails.
This tutorial walks through an example of this process.
Finding a regression
Create a cargo project that demonstrates the regression. Let’s use issue #53157 as an example:
cargo new foo
cd foo
Edit src/main.rs
with the example from the issue:
macro_rules! m { () => {{ fn f(_: impl Sized) {} f }} } fn main() { fn f() -> impl Sized {}; m!()(f()); }
Since we are testing an old regression, also edit Cargo.toml
to remove the edition = "2021"
field which isn’t supported in these versions.
Then run cargo bisect-rustc --end=2018-08-04
.
We need to provide the end point for this particular example because that’s an old regression already fixed on the latest nightlies. We could also provide a start point if we know one; that’s going to make it faster by avoiding scanning for the start. For instance:
cargo bisect-rustc --start=2018-05-07 --end=2018-08-04
It will run cargo build
in the project and check whether or not it fails.
It will do a binary search between the start and end range to find exactly where the regression occurred.
Note: You can also use the flag
--regress
to specify other common regression criteria, e.g.--regress=ice
for internal compiler errors.
In our example, in just a few steps, we can we find that it stopped working on nightly-2018-07-30
.
If the regression is recent enough, then it will print out a list of PRs that were committed on that date. In this particular example, it is too old, so we’ll need to manually inspect the git log to see which PR’s were merged.
If the nightly was within the last 167 days, then cargo-bisect-rustc
will then start bisecting those individual PRs.
After finding potential candidates, you can go inspect those PRs to see which one is the likely cause. In this case, since the ICE was in MIR const propagation, and #51361 is the likely candidate since it modified const evaluation.
Testing interactively
Pass/fail of cargo build
may not be what you’re after.
Perhaps the issue is an error message changed, so both the “good” and “bad” version will fail to
compile, just with a different message.
Or maybe something used to fail, and now erroneously passes.
You can use the interactive feature with the --prompt
flag to visually inspect a build and tell cargo-bisect-rustc
what’s “good” and what’s “bad”.
Let’s use issue #55036 as an example where an error message changed:
In Cargo.toml
, remove the edition
field (this example was before editions).
src/main.rs
:
struct Foo { bar: i32 } trait Baz { fn f(Foo { bar }: Foo) {} } fn main() {}
This historically emitted a bad error, was updated to emit a nice error (E0642 added in #53051), but then that nice error was lost somewhere (on the 2015 edition). Let’s find where it was lost! Grab the ranges between where it was added and where we know it fails:
cargo bisect-rustc --prompt \
--start=2018-08-14 \
--end=2018-10-11
At each step, cargo-bisect-rustc
will show the output and ask you:
nightly-2018-08-14 finished with exit code Some(101).
please select an action to take:
> mark regressed
mark baseline
retry
Choose mark baseline
with the nice E0642 message, and mark regressed
with the less-favorable token error.
Fairly quickly we find it regressed in nightly-2018-10-11.
The most likely candidate is #54457 which is a rollup PR.
It’s usually not too hard to look through the commits and find a likely culprit.
Indeed in this example, #54415 modified function parameter parsing.
Testing with a script
Using the --script
option allows you to do something more fancy than just cargo build
.
Maybe you need to run cargo multiple times, or just call rustc
directly, or you want to automatically grep through the output.
The possibilities are endless!
Just write a little shell script that exits 0 for the baseline, and exits nonzero for the regression.
As an example, the previous interactive session can be hands-free automated with this script:
test.sh
:
#!/bin/sh
# Fail if we no longer get a `E0642` error:
cargo check 2>&1 | grep E0642
And then run:
cargo bisect-rustc --script=./test.sh \
--start=2018-08-14 \
--end=2018-10-11
Custom bisection messages
Available from v0.6.9
You can add custom messages when bisecting a regression. Taking inspiration from git-bisect, with term-new
and term-old
you can set custom messages to indicate if a regression matches the condition set by the bisection.
Example:
cargo bisect-rustc \
--start=2018-08-14 \
--end=2018-10-11 \
--term-old "No, this build did not reproduce the regression, compile successful" \
--term-new "Yes, this build reproduces the regression, compile error"
In other words, --term-old
is displayed for older compilers that do not exhibit the regression. --term-new
is for newer compilers which do exhibit the regression.
What counts as a “regression” is defined by the --regress
CLI option. By default, a regression is a compile-error (which is equivalent to --term-new
). If you flip the definition of a “regression” with --regress=success
, then a regression is a successful compile (which is also equivalent to --term-new
).
There are default terms based on the current --regress
setting. Customizing the terms is most useful when using scripting. For example, in the Documentation changes example, the customized terms can more clearly express the results of the script of whether or not it found what it was looking for in the documentation.
Rust source repo
For cargo-bisect-rustc
to work, it needs to be able to read the git log of the rust-lang/rust
repo.
cargo-bisect-rustc
supports several methods for this described below.
GitHub API
By default, cargo-bisect-rustc
uses the GitHub API to fetch the information instead of using a local checkout.
cargo bisect-rustc --access=github
Beware that GitHub has restrictive rate limits for unauthenticated requests.
It allows 60 requests per hour, and cargo-bisect-rustc
will use about 10 requests each time you run it (which can vary depending on the bisection).
If you run into the rate limit, you can raise it to 5000 requests per hour by setting the GITHUB_TOKEN
environment variable to a GitHub personal token.
If you use the gh
CLI tool, you can use it to get a token:
GITHUB_TOKEN=`gh auth token` cargo bisect-rustc --access=github
If you don’t use gh
, you’ll just need to copy and paste the token.
Local clone
cargo-bisect-rustc
can also clone the rust repo in the current directory (as rust.git
).
This option can be quite slow if you don’t specify the repo path at build time.
You can specify this with the --access
CLI argument:
cargo bisect-rustc --access=checkout
RUST_SRC_REPO
environment variable
You can specify the location of the rust repo with the RUST_SRC_REPO
environment variable at runtime.
This is useful if you already have it checked out somewhere, but is cumbersome to use.
RUST_SRC_REPO=/path/to/rust cargo bisect-rustc
RUST_SRC_REPO
environment variable (build-time)
Setting the RUST_SRC_REPO
environment variable when installing cargo-bisect-rustc
will set the default location for the rust repo.
This is recommended if you already have the rust repo checked out somewhere.
RUST_SRC_REPO=/path/to/rust cargo install cargo-bisect-rustc
Bisection boundaries
cargo-bisect-rustc
does a binary search for the regression using a start and end boundary.
You can specify these boundaries with the --start
and --end
CLI flags.
There are several ways to specify what those boundaries are.
If you run the command without specifying the boundaries, it will search for them automatically:
# No --start or --end flags
cargo bisect-rustc
This will assume the latest nightly is a regression (the end boundary). It will then search backwards until it can find a nightly that passes to use as the start boundary. Bisection can usually go faster if you happen to know the start boundary, so that it doesn’t need to search for it.
--start
and --end
are optional.
If --start
is not specified, then it will try to find the start range automatically.
If --end
is not specified, it will assume it is the most recently available.
Date boundaries
You can pass a date in the form YYYY-MM-DD to the --start
and --end
flags.
It will download the nightly corresponding to that date, and then begin bisecting those nightlies.
cargo bisect-rustc --start=2018-08-14 --end=2018-10-11
If the nightly with the regression was within the past 167 days, then it will automatically start bisecting the individual PRs merged on that day using Git commit boundaries.
Git commit boundaries
You can pass the particular git commit hash of a PR as a boundary. The Rust project keeps the builds of every merged PR for the last 167 days. If you happen to know the PR to use as a boundary, you can pass the SHA-1 hash of that PR.
cargo bisect-rustc \
--start=6323d9a45bdf0ac2a9319a6a558537e0a7e6abd1 \
--end=866a713258915e6cbb212d135f751a6a8c9e1c0a
There are several ways to determine the SHA-1 hash for a PR.
- On the PR itself, you should see a message like “bors merged commit c50c62d into
rust-lang:master
”. You can copy that hash to use as a boundary. If the PR was merged as part of a rollup, you will need to use the hash of the rollup instead. You’ll need to look through the PR messages to see if the PR was mentioned from a rollup PR. - In the rust repo, run
git log --first-parent upstream/master
(whereupstream
is your origin name forrust-lang/rust
). This will show all the top-level commits. You can then search for your PR.
Note: If the PR was merged after the most recent nightly, you’ll need to be sure to also specify the
--end
range. Otherwise it will assume the most recent nightly is the end and it won’t work if the start is after the end.
If the regression is found in a rollup PR, then cargo-bisect-rustc
will bisect the individual PRs within the rollup.
This final bisection is only available for x86_64-unknown-linux-gnu
since it is using the builds made for the rustc performance tracker.
Note: If you specify date boundaries, then you can use the
--by-commit
CLI option to force it to use PR commits instead of nightlies.
Git tag boundaries
The boundary can be specified with a git release tag.
This is useful if you know something works in one release and not another, but you don’t happen to know which nightly this corresponds with.
When given a tag, cargo-bisect-rustc
will try to find the nightly that corresponds with that release.
For example:
cargo bisect-rustc --start=1.58.0 --end=1.59.0
Monotonicity
When writing your test and picking a bisection range, you should be careful to ensure that the test won’t vary between pass/fail over the bisection range. It should only transition from good to bad once in the bisection range (it must change monotonically).
In the following example, cargo-bisect-rustc
will find one of the transitions, but that may not be the true root cause of the issue you are investigating.
nightly-2023-02-01 baseline **start**
nightly-2023-02-02 baseline
nightly-2023-02-03 baseline
nightly-2023-02-04 regression
nightly-2023-02-05 regression
nightly-2023-02-06 baseline
nightly-2023-02-07 regression
nightly-2023-02-08 regression **end**
Here it may either find 2023-02-04 or 2023-02-07 as the regression.
The following are some suggestions for avoiding or dealing with this problem:
- Make sure your test reliably exhibits the issue you are looking for, and does not generate any false positives or false negatives.
- Analyze the PR that was reported as the regression. Do the changes in the PR seem to be a probable cause?
- Try to keep the bisection range small to reduce the probability that you will encounter multiple regression transitions.
- Use the
-vv
flag (very verbose) to display the output from the compiler to make sure it is what you expect. - Use the
--prompt
flag to inspect the output and verify each step. - Beware that some issues may get fixed and then regress multiple times. Try to keep the bisection range as close to the present day as possible. Compare the output of the “regressed” commit to the latest nightly to see if they are the same.
- If the test only fails sporadically, use a script to run the compiler many times until it fails or it passes enough iterations that you feel confident that it is good.
- If the code requires relatively new language features, be careful not to pick a starting range that is too old.
- Beware of code-generation bugs that can be sensitive to code layout. Since the code to rustc changes rapidly over time, code can shift around causing different layouts and optimizations, which might cause an issue to appear and disappear several times over the bisection range.
Rustup toolchains
cargo-bisect-rustc
takes advantage of rustup toolchains for installation and selecting the correct rustc
to run.
It will essentially run cargo +bisector-nightly-2023-03-18-x86_64-unknown-linux-gnu build
using rustup toolchain override shorthand to run the toolchains that it downloads.
This sets the RUSTUP_TOOLCHAIN
environment variable to the toolchain name, which ensures that any call to rustc
will use the correct toolchain.
By default, cargo-bisect-rustc
will delete toolchains immediately after using them.
You can use the --preserve
option to keep the toolchains so that you can use them manually.
See the Preserving toolchains example for more details.
When using the --script
option, the script should just invoke cargo
or rustc
normally, and rely on the RUSTUP_TOOLCHAIN
environment variable to pick the correct toolchain.
Git bisect a custom build
There are some rare cases where you may need to build rustc
with custom options, or otherwise work around issues with pre-built compilers not being available.
For this you can use git bisect
to build the compiler locally.
It can be helpful to use the --first-parent
option so that it only bisects the merge commits directly reachable on the master branch.
Otherwise the bisecting may land on intermediate commits from within a PR which may not build or test correctly.
To start the bisection, specifying the boundaries where the bisection will start:
git bisect start --first-parent
git bisect good 96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef
git bisect bad a00f8ba7fcac1b27341679c51bf5a3271fa82df3
Then, build the compiler as needed and run your tests to check for a regression:
./x.py build std
rustc +stage1 foo.rs
You may want to consider running ./x.py clean
if you are running into issues since changes to the internal structures of build artifacts aren’t always versioned, and those changes can be incompatible.
Incremental caches are particularly susceptible, so you may want to turn that off if you have turned them on.
If you determine the current version is good or bad, run git bisect good
or git bisect bad
to mark that, and then repeat building and marking until finished.
Similar to cargo-bisect-rustc
, git bisect
supports scripting and lots of other goodies.
Check out its documentation for more.
Alt builds
Each commit also generates what are called “alt” builds. These are builds of rustc with some different options set. As of August 2023, these include:
rust.parallel-compiler
llvm.assertions
rust.verify-llvm-ir
For more information on these settings, see the config.toml
docs.
These alt settings are defined in ci/run.sh
.
Alt builds are only available for a few targets.
Look for the -alt
builds in ci.yml
.
This can be useful if you are bisecting an LLVM issue. With LLVM assertions enabled, alt builds have checks that can help identify broken assumptions.
Alt builds are only made for commit builds, and not nightly releases.
You will need to specify --by-commit
(or use a hash in the --start
or --end
flags) to only use commit builds.
cargo bisect-rustc --alt --by-commit
Examples
The following chapters show examples of different ways of using cargo-bisect-rustc
.
Checking diagnostics
The following is an example of checking when the diagnostic output of rustc
changes.
For example, this can check when either the wording has changed, or a different error or warning is produced.
#109067 is an example of where this is necessary.
A warning started being emitted, and it is the kind of warning that cannot be turned into an error with deny(warnings)
.
The following script is intended to be used with the --script
option (set the executable flag on the script, chmod u+x
):
#!/bin/sh
OUTPUT=`cargo check 2>&1`
# Comment out this test if your example is intended to fail.
if [ $? -ne 0 ]
then
echo "Build unexpectedly failed: $OUTPUT"
exit 1
fi
# Display the output for debugging purposes.
# Run `cargo-bisect-rustc` with `-vv` to view the output.
echo "$OUTPUT"
# This indicates a regression when the text "non-ASCII" is in the output.
#
# If the regression is when the text is *not* in the output, remove the `!` prefix
# (and customize the `--term-old` and `--term-new` CLI options if you want).
! echo "$OUTPUT" | grep "non-ASCII"
Then run something like:
cargo bisect-rustc --start=1.67.0 --end=1.68.0 --script ./test.sh \
--term-old="No warning" --term-new="Found non-ASCII warning"
Scripting on Windows
Using the --script
option on Windows can be cumbersome because Windows does not support #!
scripts like Unix does, and the built-in scripting can also be awkward.
The following sections show the different ways you can use scripting.
Batch file
You can use DOS-style .bat
files:
test.bat
:
(cargo check 2>&1) | find "E0642"
This can be executed directly with:
cargo-bisect-rustc --script ./test.bat
But .bat
can be challenging to do more complex options, or you may not be familiar with it.
Powershell
You can’t execute .ps1
Powershell files directly, so you will need to use pwsh
to launch them:
test.ps1
:
( cargo check 2>&1 ) | grep E0642
if ( -Not $? ) {
exit 1
}
This can be run with:
cargo-bisect-rustc --script pwsh -- -File ./test.ps1
Bash
If you have Git-for-Windows installed, then you can use its copy of bash to run bash scripts:
test.sh
:
#!/bin/bash
cargo check 2>&1 | grep E0642
This can be run with:
cargo-bisect-rustc --script "C:\\Program Files\\Git\\usr\\bin\\bash.exe" -- ./test.sh
This also works if you have bash from something like msys2 installed.
Incremental compilation
Testing for regressions with incremental compilation may require running a command multiple times. The following illustrates an example for #87384 which only generates a warning the second time a build is run with incremental. Previously no warning was emitted.
foo.rs
:
#![type_length_limit = "95595489"] pub fn main() { println!("Hello, world!"); }
Create a script test.sh
:
#!/bin/sh
# Exit if any command fails.
set -e
rm -rf incremental
rustc foo.rs --crate-type lib -C incremental=incremental
echo second
OUTPUT=`rustc foo.rs --crate-type lib -C incremental=incremental 2>&1`
echo $OUTPUT
! echo "$OUTPUT" | grep \
"crate-level attribute should be in the root module"
Run this script with:
cargo-bisect-rustc --start 1.54.0 --end 1.55.0 --script ./test.sh
Slow or hung compilation
Some regressions may involve the compiler hanging or taking an unusually long time to run.
The --timeout
CLI option can be used to check for this.
Let’s use #89524 as an example.
A particular combination of factors caused the compiler to start to hang.
Change Cargo.toml
to the following:
[package]
name = "slow"
version = "0.1.0"
[dependencies]
config = "=0.9.3"
[profile.release]
panic = "abort"
codegen-units = 1
Then use the timeout option:
cargo-bisect-rustc --start=2021-09-01 --end=2021-10-02 --timeout 30 -- build --release
You may need to adjust the timeout value based on the speed of your system.
Note:
--timeout
is currently not working on macOS. See https://github.com/rust-lang/cargo-bisect-rustc/issues/232.
Using extra components
By default, cargo-bisect-rustc
only fetches rustc
, cargo
, rustdoc
, and the standard library for the host.
You may need additional Rustup Components to run your test.
Some examples of when this might be needed are:
- You want to find a regression in Clippy (see Bisecting Clippy), or miri.
- Scanning for when some documentation changed (see Documentation changes).
- The platform needs additional things.
For example, bisecting
x86_64-pc-windows-gnu
host may need therust-mingw
component.
If you are testing cross-compilation, use the --target
option to download the standard library for the target you are using.
The following example shows how to use components to do a bisection with Cargo’s build-std feature.
cargo-bisect-rustc --start=2022-11-01 --end=2022-11-20 -c rust-src -- build -Zbuild-std
Note: The
--with-src
option is an alias for-c rust-src
.
The--with-dev
option is an alias for-c rustc-dev -c llvm-tools
.
Running without cargo
Some bisections don’t require Cargo.
You can use the --without-cargo
option to skip installing cargo which can speed up the bisection since it doesn’t need to download cargo, and doesn’t have the overhead of running cargo.
You will need to pair this with --script
since cargo-bisect-rustc
assumes projects use Cargo.
For example, using a simple rustc
command:
cargo-bisect-rustc --start=2022-11-01 --end=2022-11-20 --without-cargo --script=rustc -- foo.rs
Note: You can use
--without-cargo
while still using a Cargo project. Rustup will fall back to usingcargo
from your installed nightly, beta, or stable toolchain. However, this isn’t recommended sincecargo
is only intended to work with the version it is released with, and can sometimes be incompatible with different versions. But if you are bisecting a very recent change, then you can probably get away with it.
Preserving toolchains
You may want to reuse the toolchains downloaded by cargo-bisect-rustc
for doing further analysis or debugging.
Or, while setting up your regression test, you may need to adjust your test and script several times, and downloading the same toolchains multiple times can be quite slow.
You can do this with the --preserve
option.
cargo bisect-rustc --start=2023-01-01 --end=2023-02-01 --preserve
The toolchains will be kept in your Rustup home directory (typically ~/.rustup/toolchains
).
Toolchains for nightlies will have the form of bisector-nightly-YYYY-MM-DD-<target>
.
Toolchains for PR artifacts will have the form of bisector-ci-<hash>-<target>
.
You can run these toolchains using a Rustup override, like this:
cargo +bisector-nightly-2023-03-18-x86_64-unknown-linux-gnu build
# or...
cargo +bisector-ci-e187f8871e3d553181c9d2d4ac111197a139ca0d-x86_64-unknown-linux-gnu build
When you are done, you’ll probably want to clean up these directories since they use a lot of space. The easiest method is to just delete the directories:
rm -rf ~/.rustup/toolchains/bisector-*
Manually installing
The --install
option can be used to only install a toolchain.
This won’t do a bisection, it is just for fetching a toolchain for testing.
cargo bisect-rustc --install e187f8871e3d553181c9d2d4ac111197a139ca0d
Note: See also
rustup-toolchain-install-master
which is specialized for installing CI artifacts.
Bisecting Rustdoc
cargo-bisect-rustc
can be used to check for Rustdoc regressions, too.
All you need to do is instruct it to use the correct command.
The following example will check to find a regression when cargo doc
suddenly starts to fail.
cargo bisect-rustc --start=2022-08-05 --end=2022-09-09 -- doc
Some rustdoc regressions might be in the generated HTML output. To scan the output, you can use a script like the following:
test.sh
:
#!/bin/sh
# Exit if any command fails.
set -e
cargo doc
grep "some example text" $CARGO_TARGET_DIR/doc/mycrate/fn.foo.html
This can be used with the --script
option:
cargo-bisect-rustc --start=2023-01-22 --end=2023-03-18 --script=./test.sh \
--term-old="Found example text" --term-new="Failed, or did not find text"
Bisecting clippy
cargo-bisect-rustc
can be used to check for Clippy regressions, too.
You’ll need to instruct it to download clippy, and run the command correctly:
cargo bisect-rustc --start=1.67.0 --end=1.68.0 -c clippy -- clippy
Note that depending on what you are looking for, this may just find a PR that syncs the rust-clippy
repo to rust-lang/rust
.
You may be able to scan the list of changes in that PR to discover what you are looking for.
If the list of changes is too big or nothing is jumping out as a possible culprit, then consider using git bisect
on the clippy repo itself (which will require building clippy).
Documentation changes
cargo-bisect-rustc
can be used to scan for changes in the documentation shipped with each release.
This includes all the books and standard library documentation.
To do this, instruct it to download the component, and use a script that scans for whatever you are looking for.
You can use rustup doc --path
or rustc --print=sysroot
to find the proper location.
For example:
test.sh
:
#!/bin/sh
# Exit if any command fails.
set -e
STD=`dirname $(rustup doc --std --path)`
# Checks if a particular file exists.
# This could also be `grep` or any other kinds of tests you need.
if [ -e $STD/io/error/type.RawOsError.html ]
then
echo "found"
exit 1
fi
And run with:
cargo bisect-rustc --start 1.68.0 --end 1.69.0 -c rust-docs --script ./test.sh \
--term-old="Did not find" --term-new="Found"
Note: This may not work on all targets since
cargo-bisect-rustc
doesn’t properly handle rustup manifests, which alias some targets to other targets. Use--host x86_64-unknown-linux-gnu
in that situation.
Flaky errors
Some tests may fail randomly.
The following script is an example that will run rustc
repeatedly to check for a failure.
This example is from #108216 (which requires macOS).
test.sh
:
#!/bin/sh
rm -rf *.o incremental foo
echo "fn main() { let a: i64 = 1 << 64; }" > foo1.rs
echo "fn main() { let a: i64 = 1 << 63; }" > foo2.rs
ARGS="--crate-name foo -C split-debuginfo=unpacked -C debuginfo=2 -C incremental=incremental"
for i in {1..20}
do
echo run $i
rustc foo1.rs $ARGS && { echo "ERROR: first build should have failed"; exit 1; }
rustc foo2.rs $ARGS || { echo "ERROR: second build should have passed"; exit 1; }
./foo || { echo "ERROR: executing should have passed"; exit 1; }
done
This test can be run with:
cargo bisect-rustc --start=1.57.0 --end=1.58.0 --script=./test.sh
In general, configure the script to perform whichever actions you need in a for
loop that runs enough times that you have a high confidence it has found the regression.