Add support for merging coverage files from multiple test projects using dotnet test --collect:"Code Coverage"

[lomholdt]

Description

When collecting code coverage using dotnet test --collect:"Code Coverage" I would like to be able to merge multiple coverage files into one if I have multiple test projects in a solution.

See #981 for reference

Expected behavior

A single coverage file that contains coverage for all test projects in a solution.

Actual behavior

Multiple coverage files are created in each test project

AB#1272814

[kennethkryger]

+1 🤞

[KLuuKer]

Any decent size application (heck even small ones) will have many projects each with there own tests, so this is a must!

[mayankbansal018]

@cltshivash , @PBoraMSFT can you check where it can be fit in our backlog

[michael-colclough]

This is very important to our team. Without it, the code coverage results are meaningless. Please could this be implemented as soon as possible

[cheenamalhotra]

We are also waiting for this support to be added, it's a basic requirement in order to gather cumulative code coverage. Is there an ETA to this request?

[ManishJayaswal]

bumping the priority and also assigning to @jakubch1 as he is working on code coverage. Let's see if we can get to this in 16.6 release.

[nohwnd]

For what it's worth, the types that can merge the code coverage files already ship with the test platform. You can parse the coverage files out from the output and merge them yourself. The only hiccup here is that the interop library is built for specific architecture, so you need the correct bitness. Both 32-bit and 64-bit versions are shipped in TP.

Here is an example that works for solution that mixes netcoreapp2.1 and 3.1 and produces two coverage files as a result.

dotnet build
dotnet test --collect:"Code Coverage" | Tee-Object -Variable output

$coverage = if ($null -ne $output) { 
    $output | 
    Select-String \.coverage | 
    ForEach-Object { $_.Line.Trim() }
}

if ($null -ne $coverage){ 
    if (1 -eq $coverage.Count) {
        $coverage
    }
    else {
        $architecture = if (64 -eq [System.IntPtr]::Size * 8) { "Amd64" } else { "X86" }

        $libs = Get-ChildItem  "~\.nuget\packages\microsoft.testplatform\*\tools\net451\Common7\IDE\Extensions\TestPlatform\Microsoft.VisualStudio.Coverage.Interop.dll" -Recurse | 
            Sort-Object -Property CreationTime -Descending

        $lib = foreach ($l in $libs) { 
            $an = [Reflection.Assemblyname]::GetAssemblyName($l)
            if ($architecture -eq $an.ProcessorArchitecture) {
                $l
                break
            }
        }

        if (-not $lib) { 
            throw "Could not find Code Coverage interop."
        }

        Add-Type -Path $lib
        $cc = [Microsoft.VisualStudio.Coverage.Interop.CoverageData]

        $firstFile = $coverage[0]
        $mergedFile = [IO.Path]::Combine([IO.Path]::GetDirectoryName($firstFile), [IO.Path]::GetFileNameWithoutExtension($firstFile) + "_merged" + [IO.Path]::GetExtension($firstFile))
        $mergeWith = $firstFile
        foreach ($file in $coverage | Select-Object -Skip 1)
        {
            $cc::MergeCoverageFiles($file, $mergeWith, $mergedFile)
            $mergeWith = $mergedFile
        }
        
        $mergeWith
    }

    if ($mergeWith -and $dte) { 
        # try opening open the file if we are in VS 
        # package manager console
        $dte.itemoperations.OpenFile($mergeWith)
    }
}

[nohwnd]

@jakubch1 let's syncup on this tomorrow, maybe this will be easy to fix in test console. ^^^

[nohwnd]

I've been looking into this and the main issue here is that we are just invoking msbuild which is in turn running VSTest task which invokes vsconsole. This only outputs into console but does not report back to dotnet test which invoked the build and could possibly merge the results before returning. This requires quite a big change in how the projects are invoked. Possibly simplifying the vstest task to just collect the data produced by msbuild (most importantly the target path which is the dll that we will invoke against), and then communicate that back to the test task. The test task would then spawn processes and register callbacks.

Or do it similarly but instead of starting the vstest console that would write into console we would invoke it in the client mode like in VS and collect the results.

I will try the second approach because that has smaller impact, hopefully that won't be too slow.

[nohwnd]

@jakubch1 let's sync on this this week, we need to start pushing to get it to 16.6, and I have a lot of other prio1 issues :)

[MarcoRossignoli]

@nohwnd this could resolve also coverlet users complaints on report merge.

[pavelhorak]

AB#1018529

[jakubch1]

We will soon add proper logic to support merging and aggregating test results.

In the meantime you can use 2 workarounds using our new tool https://docs.microsoft.com/en-us/dotnet/core/additional-tools/dotnet-coverage:

1. After running dotnet test --collect "Code Coverage" you can merge all coverage reports using command dotnet-coverage merge --remove-input-files -r *.coverage - this will merge all *.coverage files from all subdirectories and remove it at the end. You can also use format option to convert result to cobertura.
2. You can use command dotnet-coverage collect "dotnet test" in solution directory to collect code coverage for all tests. Format option can be used to generate cobertura file and publish it using https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/test/publish-code-coverage-results?view=azure-devops. This has 1 drawback that coverage is collected by the tool not by tests data collector which means that report is disconnected from trx and diff code coverage in azure pipelines will not work. Collect command is supported on Windows and Linux x64.

[ygoe]

Is there a resolution to this? Can I use the feature now? I'm left with this issue closed but no information on how to use it.

[MarcoRossignoli]

Is there a resolution to this? Can I use the feature now? I'm left with this issue closed but no information on how to use it.

There's no specific setup the it; automatically at the end of the dotnet test command all the coverage files will be merged in one in the test result folder.

cc: @jakubch1

[silkfire]

@MarcoRossignoli Is there a way to disable this default behavior when running the command on a solution?

[MarcoRossignoli]

@MarcoRossignoli Is there a way to disable this default behavior when running the command on a solution?

At the moment there's no way to disable the solution merging when dotnet test is used on a solution.

cc: @jakubch1