Finish the libtest json output experiment

Summary

Finish the libtest json experiment.

Motivation

libtest is the test harness used by default for tests in cargo projects. It provides the CLI that cargo calls into and enumerates and runs the tests discovered in that binary. It ships with rustup and has the same compatibility guarantees as the standard library.

Before 1.70, anyone could pass --format json despite it being unstable. When this was fixed to require nightly, this helped show how much people have come to rely on programmatic output.

Cargo could also benefit from programmatic test output to improve user interactions, including

• Wanting to run test binaries in parallel, like cargo nextest
• Lack of summary across all binaries
• Noisy test output (see also #5089(https://github.com/rust-lang/cargo/issues/5089))
• Confusing command-line interactions (see also #8903(https://github.com/rust-lang/cargo/issues/8903), #10392(https://github.com/rust-lang/cargo/issues/10392))
• Poor messaging when a filter doesn’t match
• Smarter test execution order (see also #8685(https://github.com/rust-lang/cargo/issues/8685), #10673(https://github.com/rust-lang/cargo/issues/10673))
• JUnit output is incorrect when running multiple test binaries
• Lack of failure when test binaries exit unexpectedly

Most of that involves shifting responsibilities from the test harness to the test runner which has the side effects of:

• Allowing more powerful experiments with custom test runners (e.g. cargo nextest) as they’ll have more information to operate on
• Lowering the barrier for custom test harnesses (like libtest-mimic) as UI responsibilities are shifted to the test runner (cargo test)

The status quo

The next 6 months

The “shiny future” we are working towards

• Reporting shifts from test harnesses to Cargo
• We run test harnesses in parallel

Team asks

Frequently asked questions