Add tooling for generating coverage

[maxammann]

It would be great to have tooling like "cargo fuzz coverage ", that creates a binary with coverage instrumentation, then runs it on the corpus and generates an LLVM .profdata file.

This can then be used to generate an HTML report using:

cargo cov -- show fuzz/target/<target triple>/release/my_compiler \
    --format=html \
    -instr-profile=fuzz/coverage/my_compiler/coverage.profdata \
    > index.html

From my perspective this is maybe one of the last key things that are missing from test-fuzz that are available in cargo-fuzz.

[smoelius]

I think it should be not-to-hard to implement something like this.

cargo-test-fuzz already has --replay corpus, replay-queue, etc. Moreover, the way those options are implemented is by running a test binary. I think we could repurpose that code to run the test binary under cargo-llvm-cov. So we could then have --coverage corpus, --coverage queue, etc.

We would have to insist that the user has installed cargo-llvm-cov, but we already do that for cargo-afl, so it's probably okay.

One thing I am uncertain about is: what is the best interface for specifying the toolchain? My understanding is that cargo-llvm-cov requires a nightly compiler.

One approach could be to check whether the current rustc is nightly (using a trick like this). If it's not, then print a warning like "the currently installed nightly toolchain will be used," and go with that. But I am open to other ideas.

Please let me know if what I've written is significantly different from what you were imagining.

[maxammann]

I think we could repurpose that code to run the test binary under cargo-llvm-cov. So we could then have --coverage corpus, --coverage queue, etc.

That sounds good!

We would have to insist that the user has installed cargo-llvm-cov, but we already do that for cargo-afl, so it's probably okay.

I also think that is fine. We just have to pass the essential options along. cargo-fuzz currently just generates a profdata file and then you are supposed to work this that. I think this may be preferable over actually generating HTML reports, so users still have control.
Supporting every feature of llvm-cov is probably hard to maintain in the long run.

One thing I am uncertain about is: what is the best interface for specifying the toolchain? My understanding is that cargo-llvm-cov requires a nightly compiler.

I will test that. Something was stabilized around ~1.60 with regards to coverage.

If it's not, then print a warning like "the currently installed nightly toolchain will be used," and go with that.

cargo-fuzz also requires nightly. So maybe we can check how they do it.

---

I belive a major blocker with the approach you mentinoed is that LLVM will overwrite the profile data. So if you rerun a binary, all the previous data is lost. So either you collect many files and then combine them or run all tests in one single binary. I feel like the first approach works better and only relies on some temporary storage.

[smoelius]

So either you collect many files and then combine them...

I don't know how cargo-llvm-cov does it, but I think it has something like this "built in": https://github.com/taiki-e/cargo-llvm-cov#merge-coverages-generated-under-different-test-conditions

But your overall point that this idea requires further investigation is certainly on the mark.

[maxammann]

I just experimented a little bit with cargo-llvm-cov and it seems to require a quite default layout of the target/ directory. At least that is the case for binary target like they are used by cargo-fuzz.

With cargo fuzz the target directory is in fuzz/target. Binaries in there are named after the target, e.g. target_1.

I got it working using the following environment:

export RUSTFLAGS='-C instrument-coverage --cfg=coverage --cfg=trybuild_no_target'
export CARGO_LLVM_COV="1"
export CARGO_LLVM_COV_SHOW_ENV="1"
export CARGO_LLVM_COV_TARGET_DIR="/project/fuzz/target/"
export LLVM_PROFILE_FILE="$CARGO_LLVM_COV_TARGET_DIR/fuzz-%p-%8m.profraw"

Then I did from the fuzz/ directory:

cargo +nightly fuzz run fuzz_target_1 -- -runs=0 corpus/fuzz_target_1
cargo llvm-cov report --profile release --target x86_64-unknown-linux-gnu

I'm continuing now to see how to integrate it into test-fuzz.

[smoelius]

🙌

[maxammann]

Also got it working 🎉

1. cargo test-fuzz
2. Export:

export CARGO_LLVM_COV="1"
export CARGO_LLVM_COV_SHOW_ENV="1"
export CARGO_LLVM_COV_TARGET_DIR="/project/target"
export LLVM_PROFILE_FILE="/project/target/$project_dir-%p-%8m.profraw"
export RUSTFLAGS="-C instrument-coverage --cfg=coverage --cfg=trybuild_no_target"

4. cargo test-fuzz --no-instrumentation --replay corpus
5. cargo llvm-cov report --profile debug --html --hide-instantiations
6. rm target/*.profraw

The report looks slightly weird though: