Change configs for source-base code coverage

[wszdexdrf]

Description

This also changes the code coverage front end to coveralls instead of codecov, which had some issues with other changes in the PR. This will provide better and more accurate code coverage reports.

Notes to the reviewers

The tests run before generating the report are not exhaustive (not exhaustive earlier too, but I added as many as I could), and hence the report won't be 100% accurate.

Checklists

All Submissions:

• I've signed all my commits
• I followed the contribution guidelines
• I ran cargo fmt and cargo clippy before committing

[rajarshimaitra]

Concept ACK.. I am wondering should we have separate code coverage job and redo all the tests or should we just create coverage report in the cont_integration jobs itself to get coverage for the full test suit..

Wondering what others thinks about this..

[wszdexdrf]

create coverage report in the cont_integration jobs itself to get coverage for the full test suit..

I actually tried that, but I couldn't get the reports from the separate (parallel) tests to merge, even though it should be possible according to the documentation for coveralls. I will try it some more and see if it works.

[afilini]

Maybe you could also upload the coverage report as an artifact so that it can be manually downloaded and inspected with a different tool if coveralls isn't accurate.

We have some code to upload artifacts in the "build nightly docs" job, you can take it from there.

[rajarshimaitra]

In the last dev call we converged on approach of

• Update the existing coverage tool from codecov to better stuffs.
• Make the coverage test as exhaustive as possible..

[afilini]

Just one last small comment, sorry

[afilini]

You can probably add test-esplora and test-md-docs as well

[wszdexdrf]

I can't add test-esplora in the same command as esplora fails to compile alongside other features. On the other hand, codecov doesn't seem to like it if I try to upload more than one lcov file from two different test runs. (To be specific, codecov reports less total lines in project, indicative of the fact that features are disabled in the tests).

[afilini]

Sorry, I thought it was gonna be easier to generate an html report from the coverage file, but it actually depends on many files in your current environment. I would suggest also generating a report and uploading that as well for convenience. You can do that with the genhtml command

[wszdexdrf]

In the last dev call we converged on approach of

* Update the existing coverage tool from codecov to better stuffs.

* Make the coverage test as exhaustive as possible..

The problem with codecov was that it reported many extra lines (which are not present in the source file) which results in lower % coverage. I found coveralls to be more accurate.

[notmandatory]

The problem with codecov was that it reported many extra lines (which are not present in the source file) which results in lower % coverage. I found coveralls to be more accurate.

Yes I think this is what @rajarshimaitra meant, that's it's fine to switch to a new tool like grcov as you're doing.

[notmandatory]

this fixes #698

[notmandatory]

With this PR looks like codecov report shows 90.87% coverage 🥇!

https://app.codecov.io/gh/wszdexdrf/bdk

Old config only showing 77.21%:

https://app.codecov.io/gh/bitcoindevkit/bdk

[wszdexdrf]

With this PR looks like codecov report shows 90.87% coverage 🥇!

https://app.codecov.io/gh/wszdexdrf/bdk

Old config only showing 77.21%:

https://app.codecov.io/gh/bitcoindevkit/bdk

I hate to say this, but that's not entirely correct. That was an older report I had generated while testing. That report has lesser number of total lines and hence it has higher percentage of coverage. The current version is
https://coveralls.io/github/wszdexdrf/bdk
Which is 79.18 % (81% if I can get it to remove examples from the report) :(

One good thing to note will be that even though the percentage is relatively unchanged, the number of tracked lines is almost doubled, so I think this should provide better tracking. And also, directly fixes the issue that some modules weren't being tracked...

[notmandatory]

ACK 0143624

[rajarshimaitra]

ACK 0143624

Whatever it is, its better than before.. So LFG!!

[danielabrozzoni]

utACK 0143624

[danielabrozzoni]

Oh, I just noticed... You should change the badge in the README not to point to codecov anymore, but to point to coveralls

[rajarshimaitra]

Sorry wrong review..

The problem seems to be there still..

[rajarshimaitra]

ReACK 28dab48

[notmandatory]

I just noticed on coveralls that it doesn't show the source code so there's no way to see what lines are covered or not, is there something you need to enable to get coveralls to show this info?

https://coveralls.io/builds/52019084/source?filename=src%2Fpsbt%2Fmod.rs

[wszdexdrf]

I just noticed on coveralls that it doesn't show the source code so there's no way to see what lines are covered or not, is there something you need to enable to get coveralls to show this info?

https://coveralls.io/builds/52019084/source?filename=src%2Fpsbt%2Fmod.rs

That's weird. It does show the code for me...

Edit: I found this. I guess we could use the HTML report for line by line coverage. I am waiting for any idea about how to proceed now.

[notmandatory]

Edit: I found this. I guess we could use the HTML report for line by line coverage. I am waiting for any idea about how to proceed now.

Ahh! OK I can confirm that once I login to coveralls with my github account I'm able to see the code and coverage details. As mentioned in the issue you found this is annoying, but I wouldn't say it's a show stopper.

[notmandatory]

@wszdexdrf looks like this needs another re-base to pickup new CI tests.

[danielabrozzoni]

ACK 0010ecd