Code coverage for .Net multi-targeting library

Background

When we develop an open-source project in GitHub, it is a good practice to show how perfect we are doing things through badges. One of the measures is code coverage. Code coverage means how deep our code is tested. 

This is a post to discuss how to set up code coverage for a multi-targeting .Net library by analyzing how it is done for DotNet.Helpers nuget library. Below is the live badge of DotNet.Helpers.

Test frameworks

There are multiple choices here. The best way is to go with the native MSTest.

Coverage file formats

There are different formats to store the code coverage. OpenCover is a stable format that uses XML to store the results.

Tools

OpenCover

This tool support .Net Core natively. It has 1.2k GitHub stars as of writing this post. The problem with the tool is that we cannot generate the coverage report using dotnet cli. Also, it supports only Windows. Is that a problem in the .Net world. Maybe not now, if we are using CI/CD provider with Windows. But it will be a consideration when Microsoft stops Windows and asks all of us to use Linux. Their current direction is 'Develop anywhere, use any tool but host in Azure'.

//This doesn't seem working
dotnet test with --collect: "Code Coverage"

Coverlet

This is another great tool to get the results. It can be integrated with the .Net build system. Coverlet doesn't merge results automatically when we are using the multi-targeting library. Though it defeats the purpose, we can work around using different methods.

The main reason for using Coverlet is that it works natively with dotnet test command. We will see the syntax later in this post.

Coverage visualizers

Generating a file is not enough. We need somebody who can track coverage and display beautiful reports. Often hosted services with freemium pricing model.

CodeCov v/s CoverAlls

Both CodeCov and CoverAlls good to use. In case anyone needs to do research, below are the links.
https://stackshare.io/stackups/codecov-vs-coveralls
https://www.appveyor.com/blog/2017/03/17/codecov/

DotNet.Helpers library uses CodeCov for tracking coverage.

How DotNet.Helpers implemented code coverage

It has 3 steps

Setting up the coverage generator tool

This step is a one-time activity. It installs coverlet.msbuild nuget package

Install-Package coverlet.msbuild

Please make sure the package is installed on the test project having the test classes.

Generate coverage report

This step has to happen with each build. The step is mentioned in the appveyor.yml. Similar to AppVeyor, it can be configured with any CI/CD provider. Below is the simple command to generate the report.

dotnet test "DotNet.Helpers.Tests\DotNet.Helpers.Tests.csproj" /p:CollectCoverage=true /p:CoverletOutputFormat=opencover
/p:Threshold=100 /p:ThresholdType=line --configuration Debug --no-build

The above command should be executed as a single unit. Multiple lines are used in this post for easier understanding.

The switch /p:Threshold=100 means it requires 100% code coverage and /p:ThresholdType=line tells the coverage is at the line level. Meaning all the lines in the library should be executed at least once during the unit testing.

The command will execute tests against all the framework versions mentioned in the test project (.csproj) file. But overwrites instead of merging as we didn't mention the target version in the output file name.

Upload coverage report

Uploading to service is optional but it helps to show the badge on how good our library is tested also reports. DotNet.Helpers library uses CodeCov service. Uploading the report to CodeCov can be done using the below command.

choco install codecov --no-progress
codecov.exe -f "DotNet.Helpers.Tests\coverage.opencover.xml" -t $env:CODECOV_TOKEN

The choco is the command to install software using Chocolatey. The $env:CODECOV_TOKEN is optional. CodeCov can check whether it is running as part of the CI/CD process inside AppVeyor and if the criteria satisfy, it can magically upload without the token.

Adding Badge

In GitHub readme.md we can embed the badge code as below.

[![codecov](https://codecov.io/gh/joymon/dotnet-helpers/branch/master/graph/badge.svg)](https://codecov.io/gh/joymon/dotnet-helpers)

Below is the badge code in HTML to embed in any web site

<a href="https://codecov.io/gh/joymon/dotnet-helpers">
  <img src="https://codecov.io/gh/joymon/dotnet-helpers/branch/master/graph/badge.svg" />
</a>

Source of DotNet.Helpers can be found in GitHub. Happy testing...

Developing .Net nuget package that targets multiple frameworks

Background

Every serious developer might have accumulated a set of reusable code snippets over the course of their career. Mostly those snippets help them to speed up work. With the advent of packaging techniques such as nuget, NPM it is very easy to share between projects. 'DotNet.Helpers' is such a library for sharing reusable .Net code which makes the development faster.

.Net uses the nuget package repository. There are different versions in .Net and maintaining a separate project and nuget package for each version is not practical.

.Net has a solution for the same and its called multi-targeting.

This post is to discuss some of the use cases, issues faced during the development of 'DotNet.Helpers' library. Below is the list.

What are the use cases in multi-targeting?

• Conditional references
• Conditional compilation
• Conditional inclusion of files
• Unit testing against each framework
• Documentation against each framework

But before discussing in details lets see how the internet says about it.

Microsoft way

The first thing Microsoft ensure that the nuget package structure is independent of how we compile and build package. Basically, the nuget package is a zip file. If it has the right folder structure, and metadata it just works.

https://docs.microsoft.com/en-us/nuget/create-packages/supporting-multiple-target-frameworks

The tooling support for multi-targeting started from VS 2015 onwards. SDK-Style projects help us mention all the frameworks in a single .csproj file itself.

How other libraries did it?

Though Microsft has tooling for multi-targeting, it is difficult to exact guidance to start our own multi-targeting nuget package. Better to look at how other libraries did the same.

NewtonSoft.JSON - This is a famous library for JSON related operations. It uses the SDK-Style project and CI/CD done using AppVeyor and Azure Pipelines. But it is a little difficult to start a new project by just following it.

An easier way is documented at west-wind.com. It explains how single repo can be used to hold source code which can be used to build a nuget package having different target frameworks.

Use cases

Most of the solutions presented below are tested against VS 2019. None of the features require a paid version of VS. Just community will be enough.

Conditional references

When we develop our library, we may be referring to other assemblies but not required in certain frameworks. For example, mscorlib.dll is only needed when we target the full framework. Similarly, there are more. Those references can be made only to the full framework using the below method.

<ItemGroup Condition="Condition"> 

A sample can be found in the repo

The same which works for assemblies can be extended to nuget library references too. Here is the link from the same repo.

Conditional compilation

This is the same as what is there from the beginning. We can write code using the conditional compilation constants and the value of constants can be controlled from the SDK-Style .csproj file.

Here is the link to defining the constant based on the framework version and here is the link to using it in the source code.

Conditional inclusion of files

Another scenario is including the files based on the framework. This is useful mainly if the entire file is framework dependent than some sections inside. 

It can be done by using the same technique which can be done in the .csproj file.

<Compile Remove="<file pattern>"

A sample link is here.

Unit testing against each framework

The approach is the same as how we are developing the library. Create a unit test project with the SDK-Style template. Use conditional inclusion and compilation techniques described above. Based on the below tag in the test project file, tests will be run against all the mentioned frameworks.

<TargetFrameworks>net472;netcoreapp2.1;netcoreapp3.1</TargetFrameworks>

Here is the link to DotNet.Helpers.Tests.csproj

Documentation against each framework

This is the tricky part. To be frank there is no solution implemented for this in DotNet.Helpers library. It uses DocFX to generate documentation and generate using a full framework which is kind of superset of others.

Generated DoxFX contents, doesn't seem to be providing any way to filter APIs based on frameworks.

Support for AppVeyor for CI & CD

This is a bonus section. If we get the project set up correctly, we can easily integrate with CI/CD systems such as AppVeyor. It is free for open-source projects.

Links

http://blogs.microsoft.co.il/iblogger/2017/04/05/easily-supporting-multiple-target-frameworks-tfms-with-vs2017-and-nuget/
https://www.cyotek.com/blog/targeting-multiple-versions-of-the-net-framework-from-the-same-project

Failing AppVeyor build on low test coverage in .Net application using Coverlet

Introduction

Normally this scenario may seem straight forward. In fact, t is very simple. We need to fail a .Net build in CI/CD process if the test coverage is less than a defined threshold. We are using Coverlet for generating the test coverage reports and AppVeyor for CI/CD.

The command to generate reports and error out the process is very simple as below.

---

dotnet test "<target>" /p:CollectCoverage=true /p:CoverletOutputFormat=opencover 

/p:Threshold=100 /p:ThresholdType=line 

---

<target> is the path to solution or project file.

Problem

The command is self explainable. There are parameters to mention the Threshold and the ThresholdType. It errors out when we run locally. But when the same run from AppVeyor, it just shows the error message and continues to succeed.

One such sample can be found in the below link.
https://ci.appveyor.com/project/joymon/dotnet-helpers/builds/30446410#L79

It just says there is an error but continues.

---

C:\Users\appveyor\.nuget\packages\coverlet.msbuild\2.2.1\build\netstandard2.0\coverlet.msbuild.targets(23,5): error : 'DotNet.Helpers' has a line coverage '92.9%' below specified threshold '100%' [C:\projects\dotnet-helpers\DotNet.Helpers.Tests\DotNet.Helpers.Tests.csproj]

---

Troubleshooting

There were no direct google results found to solve the issue. So created the below question in the AppVeyor forum.

https://help.appveyor.com/discussions/problems/26109-net-build-is-not-failing-when-the-test-coverage-is-low-using-coverlet

But later when searched for similar issues in other CI/CD platforms, one question got the attention.

https://stackoverflow.com/questions/56631066/azure-devops-build-dont-fail-when-coverage-is-below-target-with-dotnet-test

As per the answer in this post, the action has to be performed differently to fail the build. It feels something custom. So decided to fail the AppVeyor build by code based on the result from dotnet test command. 

Solution

Not sure its actual solution or a hack to fail the build as the command is not failing the build. Below goes the solution.

---

dotnet test "<target>" /p:CollectCoverage=true /p:CoverletOutputFormat=opencover 

/p:Threshold=100 /p:ThresholdType=line

if($LastExitCode -ne 0) { $host.SetShouldExit($LastExitCode ) }

---

It is simple as checking the $LastExitCode and setting it to the $host to indicate the runtime that it failed. Normally any non zero result code is considered as a failure.

Working sample

Full working AppVeyor.yml can be viewed in the below GitHub repo

https://github.com/joymon/dotnet-helpers

A build log showing it failed due to code coverage can be found here.

Happy debugging...

Another .Net helper library via nuget package system

Background

Over the past 13 years, to be precisely from Nov 2005 till today, I had written lot of .Net code for day job as well as to personal projects. When I started .Net, I though yes I will master it and enjoy rest of my career. But I soon realized that is not going to work with the collapse of Silverlight. Microsoft was telling or people was arguing that Silverlight will not die as Microsoft is using it for their Azure portal. All of a sudden MSFT replaced Azure Silverlight site with HTML and that was kind of last nail on Silverlight. More details on it can be found in my last post in SilverlightedWeb blog which is a readonly blog now.  Then I thought Silverlight technology's end is inevitable as it is replaced by HTML5 but .Net will live long.

That thought got shaken when MSFT released their so called code editor now becoming full fledged IDE named VS Code. That didn't use WPF which was the star of desktop programming from MSFT at that time. Instead it used Electron from Github which depends on Chrome. Yes, the browser from Google powering web. Essentially we develop browser application and show as standalone executable. That was the time I said good bye to WPF technology. More details here. Then whats left, only ASP.Net which was and still struggling to compete with NodeJS. Don't bring Windows Phone here as that is the one of very hand countable things MSFT properly shut down. No idea how long something called UWP will live.

.Net Core

Finally something came named .Net Core. Its like Angular 1.x and 2. Only name is same, internally all most new. That is what now MSFT fans betting on, as return of .Net. .It is advertised as another true cross platform which will run on Linux! Yes its the second cross platform .Net. The original one also advertised as cross platform with the intermediate language and JIT similar to JVM echo system.

The another factor is performance. .Net Core is expected to beat NodeJS for serving http responses. There are some case studies people are claiming it is faster such as on Bing.

Another areas where .Net was weak is AI, Machine Learning, distributed computing etc... Now there is ML.Net SDK also announced.

Yes it may be faster and may become powerful than Python in AI programming.  But will this technology enough to feed my family in future?

So what is next?

Personally I don't see a bright future for .Net unless .Net Core becomes a big hit. So better to reduce focus on .Net and consider other technologies as well seriously. Electron for desktop development, Angular + NodeJS for web front end, Scala for distributed programming etc...

But what I should do with all my .Net knowledge acquired in past 13 years as I still have hope on huge return of .Net core?

Offload from brain and move on. The better place to offload code level techniques is a nuget package at this point. I could have added the helper classes to my first nuget package, but unfortunately that was towards a specific problem of Orchestration. Hence I had to start another nuget library for my helper classes and coding techniques. Link to Github repo below.

https://github.com/joymon/dotnet-helpers

This library uses multi targeting feature so that one code base can be compiled to multiple targets. This is useful especially to provide libraries for .Net Core.

Thanks to AppVeyor for giving free CI&CD support to publish to nuget repo.

Why I didn't join with other helper nuget libraries is described in the readme of the repo.

Protractor - CI with AppVeyor issue on webdriver-manager update

Protractor is a nice tool to do testing of Angular applications. It really shines when used in continuous delivery scenarios because of its end to end testing capabilities. We can simply host the site and run our tests with Protractor. But when we use it with hosted CI solutions there is a chance for getting the below exception in the build process.

(node:1184) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: Failed to make Github request, rate limit reached.

This happens when we issue the below command which we cannot avoid to get it working.

webdriver-manager update

One such failure can be seen here. What is really happening is that during the update, the webdriver-manager is trying to download something from GitHub for Gecko driver. Since most of the hosted CI solutions use same single IP to access outside internet, the GitHub API limits comes into play.

This is already known issue and the only way is to disable Gecko driver for now. Command to disable Gecko driver is as follows.

webdriver-manager update --gecko false

Thanks to the AppVeyor support team.

My JavaScript Module experiments - CI & CD

This is second in the series "My JavaScript Module experiments". This post talks about setting up a simple CI & CD pipeline in modularied JavaScript environment using AppVeyor as CI & CD runner.

Other posts in this series below.

1 My JavaScript Module experiments
2 My JavaScript Module experiments - CI & CD

Introduction

This post directly jumps to setting up CI & CD for modularized JavaScript. Here also Webpack is used as module loader as well as to do other operations such as TypeScript transpilation, minification etc...

AppVeyor & GitHub Pages

AppVeyor & GitHub pages are 2 external services used here. AppVeyor provides free CI & CD service for open source projects and GitHub Pages provides free hosting from Github repositories. Basic knowledge about these services are considered as prerequisites.

AppVeyor is mainly targeted to Microsoft .Net stack and it automatically takes care, if there is a Visual Studio solution file. But we can make it work for Node as well by disabling the default build process. That can be done by simple changes inside appveyor.yml file. AppVeyor supports configuring CI & CD activities either in a appveyor.yml file found in the GitHub repo or through their UI. Here we are using the appveyor.yml  file as its easy to understand. Though is a file publicly accessible file in repo, we can include sensitive information inside it with encryption.

Separation of Concerns - What does what

When we look at the technologies used, we can see those can so same tasks. AppVeyor can run normal commands as well as PowerShell and even start NodeJS which in turn starts Webpack. Using normal commands, PowerShell & Node we can do file copy operations. Webpack which can be started as command can also execute code which does file copy. What is the problem if all the technologies can do same task? The problem is lack of clarity on which component does what.

For small projects, it is fine. We can have AppVeyor scripts and Webpack do file copy. But when we think of larger projects we need to have clear distinction on that should do what. Else it would be difficult to maintain in future.

The simple rule can be to assign the dev related tasks such as compilation and bundling  to Webpack and deployment tasks to the AppVeyor scripts such as pushing to GitHub. Ultimately AppVeyor is ruling the environment. So there should be one AppVeyor command which starts Webpack. Summarized duties as follows.

• AppVeyor
• Downloads the source code
• Install proper Node version and setup environment by installing packages.
• This include the Webpack NPM package too.
• Starts Webpack
• Run tests.
• Collect the artifact and deploy  
• Webpack
• Compilation
• Do minification / Uglify
• Create bundle
• Emit the output to \dist folder. (\dist is just a convention. It can be any folder)

The CI & CD workflow

The CI & CD workflow can be simple. After the transpilation &  bundling the output HTML app which includes html, js, images etc...needs to be pushed to the \docs folder of repository. The GitHub repository can be configured to serve web pages from \docs folder.

Sample

It is easy to explain the but there are high possibility for missing steps. So to make it more understandable, please refer sample repo located at below location

https://github.com/joymon/webpack-starters

• It does CI & CD on 'TypeScript-Angular1-Adv folder'.Other folders are showing different Webpack samples.
• Push the output to the \docs folder in same repo. \Docs folder is configured to GitHub pages to serve from.
• It is little advanced Angular 1 sample done using TypeScript.
• Node & NPM used as dev technology.
• AppVeyor.yml has all the CI & CD steps. The GitHub token is encrypted and kept inside it.
• Since the aim of that folder is to minimally show the webpack features, at this point there are no tests included.