Code coverage for .Net multi-targeting library

Background

When we develop an open-source project in GitHub, it is a good practice to show how perfect we are doing things through badges. One of the measures is code coverage. Code coverage means how deep our code is tested. 

This is a post to discuss how to set up code coverage for a multi-targeting .Net library by analyzing how it is done for DotNet.Helpers nuget library. Below is the live badge of DotNet.Helpers.

Test frameworks

There are multiple choices here. The best way is to go with the native MSTest.

Coverage file formats

There are different formats to store the code coverage. OpenCover is a stable format that uses XML to store the results.

Tools

OpenCover

This tool support .Net Core natively. It has 1.2k GitHub stars as of writing this post. The problem with the tool is that we cannot generate the coverage report using dotnet cli. Also, it supports only Windows. Is that a problem in the .Net world. Maybe not now, if we are using CI/CD provider with Windows. But it will be a consideration when Microsoft stops Windows and asks all of us to use Linux. Their current direction is 'Develop anywhere, use any tool but host in Azure'.

//This doesn't seem working
dotnet test with --collect: "Code Coverage"

Coverlet

This is another great tool to get the results. It can be integrated with the .Net build system. Coverlet doesn't merge results automatically when we are using the multi-targeting library. Though it defeats the purpose, we can work around using different methods.

The main reason for using Coverlet is that it works natively with dotnet test command. We will see the syntax later in this post.

Coverage visualizers

Generating a file is not enough. We need somebody who can track coverage and display beautiful reports. Often hosted services with freemium pricing model.

CodeCov v/s CoverAlls

Both CodeCov and CoverAlls good to use. In case anyone needs to do research, below are the links.
https://stackshare.io/stackups/codecov-vs-coveralls
https://www.appveyor.com/blog/2017/03/17/codecov/

DotNet.Helpers library uses CodeCov for tracking coverage.

How DotNet.Helpers implemented code coverage

It has 3 steps

Setting up the coverage generator tool

This step is a one-time activity. It installs coverlet.msbuild nuget package

Install-Package coverlet.msbuild

Please make sure the package is installed on the test project having the test classes.

Generate coverage report

This step has to happen with each build. The step is mentioned in the appveyor.yml. Similar to AppVeyor, it can be configured with any CI/CD provider. Below is the simple command to generate the report.

dotnet test "DotNet.Helpers.Tests\DotNet.Helpers.Tests.csproj" /p:CollectCoverage=true /p:CoverletOutputFormat=opencover
/p:Threshold=100 /p:ThresholdType=line --configuration Debug --no-build

The above command should be executed as a single unit. Multiple lines are used in this post for easier understanding.

The switch /p:Threshold=100 means it requires 100% code coverage and /p:ThresholdType=line tells the coverage is at the line level. Meaning all the lines in the library should be executed at least once during the unit testing.

The command will execute tests against all the framework versions mentioned in the test project (.csproj) file. But overwrites instead of merging as we didn't mention the target version in the output file name.

Upload coverage report

Uploading to service is optional but it helps to show the badge on how good our library is tested also reports. DotNet.Helpers library uses CodeCov service. Uploading the report to CodeCov can be done using the below command.

choco install codecov --no-progress
codecov.exe -f "DotNet.Helpers.Tests\coverage.opencover.xml" -t $env:CODECOV_TOKEN

The choco is the command to install software using Chocolatey. The $env:CODECOV_TOKEN is optional. CodeCov can check whether it is running as part of the CI/CD process inside AppVeyor and if the criteria satisfy, it can magically upload without the token.

Adding Badge

In GitHub readme.md we can embed the badge code as below.

[![codecov](https://codecov.io/gh/joymon/dotnet-helpers/branch/master/graph/badge.svg)](https://codecov.io/gh/joymon/dotnet-helpers)

Below is the badge code in HTML to embed in any web site

<a href="https://codecov.io/gh/joymon/dotnet-helpers">
  <img src="https://codecov.io/gh/joymon/dotnet-helpers/branch/master/graph/badge.svg" />
</a>

Source of DotNet.Helpers can be found in GitHub. Happy testing...