Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unexpected code coverage results with runsettings inclusions / exclusions #3462

Closed
tonyhallett opened this issue Mar 13, 2022 · 1 comment
Closed
Assignees

Comments

@tonyhallett
Copy link

Unexpected code coverage results - please can you explain the algorithm in more detail.

I have read the documentation https://docs.microsoft.com/en-us/visualstudio/test/customizing-code-coverage-analysis?view=vs-2022#include-or-exclude-assemblies-and-members

What I expect is that included code is the union of all includes. This is not the case.

What I discovered.

PublicKeyTokens includes have no effect - alone acts if was not present - all loaded assemblies with pdbs. With ModulePaths includes the included assemblies are the ones that match the ModulePaths.
Should the docs state that only Exclude applies here ?

Attributes includes have no effect.
Assembly level excludes have no effect such as https://docs.microsoft.com/en-us/dotnet/api/system.diagnostics.codeanalysis.excludefromcodecoverageattribute?view=net-6.0
Is this what "elements" refers to "Matches elements that have the specified attribute" ? If so it does seem odd that an attribute with a use case of excluding from code coverage is unable to.
Should the docs state that only Exclude applies here ?

CompanyNames includes behaves strangely.
Alone they work.
When combined with ModulePaths includes they have no effect. Assemblies with the company name are not included and the ModulePaths includes are present despite not having the company name. #
Please explain further.

Functions includes.
Alone they work.
When combined with CompanyNames includes - included assemblies are assemblies that match the Functions includes and the CompanyNames includes.
When combined with ModulePaths includes - included assemblies are assemblies that match the Functions includes and the ModulePaths includes.
When combined with both CompanyNames and ModulePaths - see the strange CompanyNames behaviour again.

Sources includes
Alone they work.
When combined with CompanyNames includes - included assemblies are assemblies that match the Sources includes and the CompanyNames includes.
When combined with ModulePaths includes - included assemblies are assemblies that match the Sources includes and the ModulePaths includes.
When combined with both CompanyNames and ModulePaths - see the strange CompanyNames behaviour again.

ModulePaths include
Alone they work.
When combined with others exaplained above.

dotnet test "C:\Users\tonyh\source\repos\MsFCCCoverage\TestProject1\TestProject1.csproj"

With project <RunSettingsFilePath/>
With run settings with
`
PathTo\msCodeCoverage\17.1.0\build\netstandard1.0
<DataCollector friendlyName="Code Coverage"...

`

@fhnaseer
Copy link
Member

@tonyhallett Code coverage inclusion or exclusion for an assembly is done in two parts.

  1. Assembly level (, , , etc.)
  2. Code Level (, , , etc.)

First we check for assembly level flags and then we go for code level flags if required.

For all exclude and include attributes in runsettings it works in the following way

  1. If is empty, then everything is included except that is in
  2. If is not empty, then only those items will be included.

For there is an exception. If a module is in then it will always be added for code coverage. It will ignore all other assembly level flags. Code level flags will work.

Please review all of your cases with this. If something is not working as expected then please do let me know. And if possible then please share a minimal solution so that I can verify cases.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants