Reporting accurate code coverage

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Reporting accurate code coverage

Boost - Dev mailing list
 General Code Coverage Tip (for everyone):

Measure your coverage using branches, not lines.  Branch coverage tells you
how well error cases are tested and therefore a far better measure of
quality than line-based.  I have seen  over 90% line coverage but the
branch coverage is more like 60% to 70% on average over the boost
CMT-managed repositories.

Here's the shell script I use on travis to generate the data and feed it to
codecov.io:

https://github.com/jeking3/boost-ci/blob/master/ci/travis/codecov.sh

The trick is to feed --rc lcov_branch_coverage=1 to the correct commands to
generate and preserve the branch information.

Thanks,

- Jim

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Reporting accurate code coverage

Boost - Dev mailing list
On Wed, Jul 11, 2018 at 7:34 AM James E. King III <[hidden email]> wrote:

> General Code Coverage Tip (for everyone):
>
> Measure your coverage using branches, not lines.  Branch coverage tells
> you how well error cases are tested and therefore a far better measure of
> quality than line-based.  I have seen  over 90% line coverage but the
> branch coverage is more like 60% to 70% on average over the boost
> CMT-managed repositories.
>
>
Below is some proof of what I mentioned about line coverage being (quite)
deceptive - Boost.Rational has 98.8% "line" coverage but only 59.7% branch
coverage.  Line coverage reported by lcov in this case means the line was
hit in some way.  It doesn't mean every branch on that line was exercised
for true and false cases to ensure proper behavior.  Good unit tests will
exercise all the branch cases and this typically proves that error handling
code works as intended.

In short, don't rely on line coverage for your quality measurements.
Always rely on branch coverage.

Extracting /home/travis/build/boostorg/boost-root/boost/rational.hpp
Extracted 1 files
Writing data to coverage.info
Summary coverage rate:
  lines......: 98.9% (281 of 284 lines)
  functions..: 97.5% (424 of 435 functions)
  branches...: 59.7% (964 of 1615 branches)
+lcov --gcov-tool=gcov-7 --rc lcov_branch_coverage=1 --list coverage.info
Reading tracefile coverage.info
                  |Lines       |Functions  |Branches
Filename          |Rate     Num|Rate    Num|Rate     Num
========================================================
[/home/travis/build/boostorg/boost-root/boost/]
rational.hpp      |98.9%    284|97.5%   435|59.7%   1615
========================================================
            Total:|98.9%    284|97.5%   435|59.7%   1615

- Jim

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Reporting accurate code coverage

Boost - Dev mailing list
Dear James,

> On 13. Jul 2018, at 16:45, James E. King III via Boost <[hidden email]> wrote:
>
> In short, don't rely on line coverage for your quality measurements.
> Always rely on branch coverage.

I generally see the wisdom in your argument.

I use coveralls.io to display and track coverage. On Travis, I execute:

pip install --user cpp-coveralls
coveralls -r .. -b . --verbose --exclude ${TRAVIS_BUILD_DIR}/deps  --gcov=`which ${GCOV}` --gcov-options '\-lp';

Seems like coveralls may have added tracking of branch coverage in 2017 for some languages,
https://github.com/lemurheavy/coveralls-public/issues/31 <https://github.com/lemurheavy/coveralls-public/issues/31>
but I can't see it for my project. The situation is not clear.

Do I have to migrate to codecov.io <http://codecov.io/> to follow your advice?
Has anyone been able to get branch coverage working with coveralls?

Best regards,
Hans

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Reporting accurate code coverage

Boost - Dev mailing list
Yes: See the PR linked in the issue:
https://github.com/openssl/openssl/pull/4447/files

Simply replace '\-lp'; by '\-lpbc';


Side note on branch coverage: I think it was coveralls who first
supported "partial" coverage. That is a line, where not all branches are
covered. You can configure, which types of such partials are counted as
"hit" or "miss". Doing that I quickly found the branch coverage in C++
is incredibly hard. Even for non-optimized debug builds the compiler
will add branches where you won't expect them and you have no clue in
how to get them covered. This is much easier for scripting languages.

@James What is the current state on that? Did you test this extensively?
Did you also encounter such spurious branch misses? If not and the state
is the same as like 2 years ago, then branch coverage as a metric is not
really useful in C++, as you have many false negatives.
Note: I'm not talking about branch coverage in general, just that it is
(was?) to unreliable in C++. You can't achieve 100% (not that you really
should) due to the compiler.

Alex Grund


Am 13.07.2018 um 17:12 schrieb Hans Dembinski via Boost:

> Dear James,
>
>> On 13. Jul 2018, at 16:45, James E. King III via Boost <[hidden email]> wrote:
>>
>> In short, don't rely on line coverage for your quality measurements.
>> Always rely on branch coverage.
> I generally see the wisdom in your argument.
>
> I use coveralls.io to display and track coverage. On Travis, I execute:
>
> pip install --user cpp-coveralls
> coveralls -r .. -b . --verbose --exclude ${TRAVIS_BUILD_DIR}/deps  --gcov=`which ${GCOV}` --gcov-options '\-lp';
>
> Seems like coveralls may have added tracking of branch coverage in 2017 for some languages,
> https://github.com/lemurheavy/coveralls-public/issues/31 <https://github.com/lemurheavy/coveralls-public/issues/31>
> but I can't see it for my project. The situation is not clear.
>
> Do I have to migrate to codecov.io <http://codecov.io/> to follow your advice?
> Has anyone been able to get branch coverage working with coveralls?
>
> Best regards,
> Hans

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost