Improve support for doctests

[taiki-e]

STATUS:

#40 fixed all of the errors and many of the warnings

The remaining warnings are probably the column offset errors mentioned in rust-lang/rust#79417 (comment), and it's not clear whether we can fix them on our side at this time.

---

This is currently available via --doctests flag as an unstable feature.

cargo llvm-cov --doctests

Refs:

• https://doc.rust-lang.org/rustc/instrument-coverage.html#including-doc-tests
• Rustdoc doctests should have their file/lines remapped when using -Zinstrument-coverage rust-lang/rust#79417
• Tracking issue: --persist-doctests flag for rustdoc rust-lang/rust#56925

[taiki-e]

STATUS: I think it's better than before, but there are still a few problems.

warning: 4 functions have mismatched data

[cloudhead]

It would be great to have this working better. This tool is super handy.

Currently, running with --doctests I get some error output at the end, and it seems like the files that contain doctests get included twice in the coverage output:

error: src/block.rs: No such file or directory
warning: The file 'src/block.rs' isn't covered.
error: src/block/filter.rs: No such file or directory
warning: The file 'src/block/filter.rs' isn't covered.
error: src/block/iter.rs: No such file or directory
warning: The file 'src/block/iter.rs' isn't covered.
error: src/network.rs: No such file or directory
warning: The file 'src/network.rs' isn't covered.
error: src/protocol/addrmgr.rs: No such file or directory
warning: The file 'src/protocol/addrmgr.rs' isn't covered.
error: src/time.rs: No such file or directory
warning: The file 'src/time.rs' isn't covered.

[taiki-e]

@cloudhead Is there an example that can reproduce the error you encountered?

[cloudhead]

Yeah, though after reproducing this I realize some of these errors come from the --html flag, and on its own, --doctests doesn't output these errors, only in combination with --html.

The repo is here:
https://github.com/cloudhead/nakamoto

I ran: cargo llvm-cov --all --doctests --html

The issue with coverage output for files included twice appears with only --doctests though. Basically the paths are not matched, eg.:

net/poll/src/time.rs                     45                22    51.11%          14                 5    64.29%          78                30    61.54%           0                 0         -
src/time.rs                              22                 2    90.91%           4                 0   100.00%          89                 0   100.00%           0                 0         -

These are actually the same file, but I'm guessing the second one is from the doctest run, while the first is from the unit test run, I'm not sure.

Note that there is no src/time.rs, that path doesn't exist.

[taiki-e]

Thanks. I'll take a look.

[taiki-e]

#40 fixed all of the errors and many of the warnings that happen in cloudhead/nakamoto.

Before #40:

warning: 5 functions have mismatched data
error: src/block.rs: No such file or directory
warning: The file 'src/block.rs' isn't covered.
error: src/block/filter.rs: No such file or directory
warning: The file 'src/block/filter.rs' isn't covered.
error: src/block/iter.rs: No such file or directory
warning: The file 'src/block/iter.rs' isn't covered.
error: src/network.rs: No such file or directory
warning: The file 'src/network.rs' isn't covered.
error: src/protocol/addrmgr.rs: No such file or directory
warning: The file 'src/protocol/addrmgr.rs' isn't covered.
error: src/time.rs: No such file or directory
warning: The file 'src/time.rs' isn't covered.
warning: 5 functions have mismatched data

After #40:

warning: 5 functions have mismatched data
warning: 5 functions have mismatched data

The remaining warnings are probably the column offset errors mentioned in rust-lang/rust#79417 (comment), and it's not clear whether we can fix them on our side at this time.

[cloudhead]

Awesome, thanks a bunch!

[adamreichold]

I am not sure whether I have missed any of the prerequisites, but I have tried to use the --doctests option in the rust-numpy crate, c.f. PyO3/rust-numpy#286, but it appears that while the doctests are executed, they are not included in the coverage computation (whereas both unit tests and integration tests work). There are no error messages and no warnings besides the initial warning that the feature is unstable. I can also reproduce this locally, e.g. if I run cargo llvm-cov --doc --html I get an empty report.

Can anybody advise if am missing some dependency or step? Thank you!

[taiki-e]

My understanding is that the coverage is broken due to a bug of rustc's --remap-path-prefix (#140, #122).
Could you try #141?

[adamreichold]

Could you try #141?

I installed cargo-llvm-cov using

> cargo install --force --git https://github.com/taiki-e/cargo-llvm-cov.git --branch disable-remap-path-prefix cargo-llvm-cov

but did not observe any changes in the output. I guess I need to wait for #122 and therefore rust-lang/rust#92648 to land?

[dominikwilkowski]

I don't know if this is related to this issue here but adding the below lines to a test will give me warning: 4 functions have mismatched data.

let candy_color = [
	String::from("#ea3223"),
	String::from("#377d22"),
	String::from("#fffd54"),
	String::from("#ea3df7"),
	String::from("#74fbfd"),
	String::from("#ee776d"),
	String::from("#8cf57b"),
	String::from("#fffb7f"),
	String::from("#6974f6"),
	String::from("#ee82f8"),
	String::from("#8dfafd"),
];
assert!(candy_color.contains(&color2hex(&Colors::Candy, &options)));

These lines are within a *_test.rs file inside the test folder and inside a #[cfg(test)] and #[test] block.
They test the output of a function that randomizes it's output.

And I'm running the test without --doctests via cargo llvm-cov --html on stable

[taiki-e]

@dominikwilkowski I believe that's also a rustc bug. (maybe related to code generated by #[test] attr)

[jonhoo]

fwiw, now that instrument-coverage is stabilized, https://doc.rust-lang.org/rustc/instrument-coverage.html#including-doc-tests is the link to the relevant paragraph in the upstream docs

[taiki-e]

@jonhoo Thanks. I updated the link.

[taiki-e]

@dominikwilkowski If an issue you have encountered has not yet been fixed, would you mind submitting an issue about it at https://github.com/rust-lang/rust?

[dominikwilkowski]

@taiki-e what's a good way to re-produce this bug without llvm-cov? Once I know how to re-produce this I'll happily submit a ticket to rust

[taiki-e]

I think we can get the commands that cargo-llvm-cov internally calls and adjust them. For example, the output of the following command should be able to be run as-is:

cargo llvm-cov --html -vv 2>&1 | grep CARGO_INCREMENTAL | sed -E -e 's/^ *Running `//g; s/`$//g; s|-sparse -f .*-o|-sparse target/llvm-cov-target/*.profraw -o|; s|.*/bin/llvm-|"$(rustc --print target-libdir)"/../bin/llvm-|g' -e "s|$(pwd)/||g"

[jcape]

Here's the output from mobilecoinfoundation/from-random

CARGO_INCREMENTAL=0 LLVM_PROFILE_FILE='target/llvm-cov-target/from-random-%p-%m.profraw' RUSTFLAGS='-C instrument-coverage --cfg=coverage --cfg=trybuild_no_target' RUST_TEST_THREADS=1 /home/user/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/cargo test --tests --manifest-path Cargo.toml --target-dir target/llvm-cov-target -v

"$(rustc --print target-libdir)"/../bin/llvm-profdata merge -sparse target/llvm-cov-target/*.profraw -o target/llvm-cov-target/from-random.profdata

"$(rustc --print target-libdir)"/../bin/llvm-cov show -format=html -instr-profile=target/llvm-cov-target/from-random.profdata -object target/llvm-cov-target/debug/deps/mc_from_random-0b047475ff4e3b36 -ignore-filename-regex '/rustc/[0-9a-f]+/|^/home/user/src/mobilecoinfoundation/from\-random(/.*)?/(tests|examples|benches)/|^/home/user/src/mobilecoinfoundation/from\-random/target/llvm\-cov\-target($|/)|^/home/user/\.cargo/(registry|git)/|^/home/user/\.rustup/toolchains($|/)' -show-instantiations=true -show-line-counts-or-regions -show-expansions -Xdemangler=/home/user/.cargo/bin/cargo-llvm-cov -Xdemangler=llvm-cov -Xdemangler=demangle -output-dir=target/llvm-cov/html