Failing AppVeyor build on low test coverage in .Net application using Coverlet

Introduction

Normally this scenario may seem straight forward. In fact, t is very simple. We need to fail a .Net build in CI/CD process if the test coverage is less than a defined threshold. We are using Coverlet for generating the test coverage reports and AppVeyor for CI/CD.

The command to generate reports and error out the process is very simple as below.

---

dotnet test "<target>" /p:CollectCoverage=true /p:CoverletOutputFormat=opencover 

/p:Threshold=100 /p:ThresholdType=line 

---

<target> is the path to solution or project file.

Problem

The command is self explainable. There are parameters to mention the Threshold and the ThresholdType. It errors out when we run locally. But when the same run from AppVeyor, it just shows the error message and continues to succeed.

One such sample can be found in the below link.
https://ci.appveyor.com/project/joymon/dotnet-helpers/builds/30446410#L79

It just says there is an error but continues.

---

C:\Users\appveyor\.nuget\packages\coverlet.msbuild\2.2.1\build\netstandard2.0\coverlet.msbuild.targets(23,5): error : 'DotNet.Helpers' has a line coverage '92.9%' below specified threshold '100%' [C:\projects\dotnet-helpers\DotNet.Helpers.Tests\DotNet.Helpers.Tests.csproj]

---

Troubleshooting

There were no direct google results found to solve the issue. So created the below question in the AppVeyor forum.

https://help.appveyor.com/discussions/problems/26109-net-build-is-not-failing-when-the-test-coverage-is-low-using-coverlet

But later when searched for similar issues in other CI/CD platforms, one question got the attention.

https://stackoverflow.com/questions/56631066/azure-devops-build-dont-fail-when-coverage-is-below-target-with-dotnet-test

As per the answer in this post, the action has to be performed differently to fail the build. It feels something custom. So decided to fail the AppVeyor build by code based on the result from dotnet test command. 

Solution

Not sure its actual solution or a hack to fail the build as the command is not failing the build. Below goes the solution.

---

dotnet test "<target>" /p:CollectCoverage=true /p:CoverletOutputFormat=opencover 

/p:Threshold=100 /p:ThresholdType=line

if($LastExitCode -ne 0) { $host.SetShouldExit($LastExitCode ) }

---

It is simple as checking the $LastExitCode and setting it to the $host to indicate the runtime that it failed. Normally any non zero result code is considered as a failure.

Working sample

Full working AppVeyor.yml can be viewed in the below GitHub repo

https://github.com/joymon/dotnet-helpers

A build log showing it failed due to code coverage can be found here.

Happy debugging...