Skip to content

Latest commit

 

History

History
147 lines (113 loc) · 6.21 KB

filter.md

File metadata and controls

147 lines (113 loc) · 6.21 KB

TestCase filter

This document will help you to selectively execute tests based on filtering conditions through --filter for dotnet test and --testcasefilter for vstest.console.exe.

Syntax

dotnet test --filter <Expression> or vstest.console.exe --testcasefilter:<Expression>

Expression is in the format <property><operator><value>[|&<Expression>]. Expressions can be enclosed in paranthesis. e.g. (Name~MyClass) | (Name~MyClass2).

vstest 15.1+: An expression without any operator is automatically considered as a contains on FullyQualifiedName property. E.g. dotnet test --filter xyz is same as dotnet test --filter FullyQualifiedName~xyz.

Property is an attribute of the Test Case. For example, the following are the properties supported by popular unit test frameworks.

Test Framework Supported properties
MSTest
  • FullyQualifiedName
  • Name
  • ClassName
  • Priority
  • TestCategory
Xunit
  • FullyQualifiedName
  • DisplayName
  • Traits

Allowed operators:

  • = implies an exact match
  • != implies an exact not match
  • ~ implies a contains lookup
  • !~ implies a not contains lookup

Value is a string. All the lookups are case insensitive.

Escape Sequences must be used to represent characters in the value that have special meanings in the filter, i.e. filter operators.

Escape Sequence Represents
\\ \
\( (
\) )
\& &
\| |
\= =
\! !
\~ ~

A helper method Microsoft.VisualStudio.TestPlatform.ObjectModel.Utilities.FilterHelper.Escape is also available by referencing the Microsoft.VisualStudio.TestPlatform.ObjectModel NuGet package, which can be used to escape strings programatically.

Expressions can be joined with boolean operators. The following boolean operators are supported:

  • | implies a boolean OR
  • & implies a boolean AND

Examples

The following examples use dotnet test, if you're using vstest.console.exe replace --filter with --testcasefilter:.

MSTest

namespace MSTestNamespace
{
    using Microsoft.VisualStudio.TestTools.UnitTesting;

    [TestClass]
    public class UnitTestClass1
    {
        [Priority(2)]
        [TestMethod]
        public void TestMethod1()
        {
        }

        [TestCategory("CategoryA")]
        [Priority(3)]
        [TestMethod]
        public void TestMethod2()
        {
        }
    }
}
Expression What it does?
dotnet test --filter Method Runs tests whose FullyQualifiedName contains Method. Available in vstest 15.1+.
dotnet test --filter Name~TestMethod1 Runs tests whose name contains TestMethod1.
dotnet test --filter ClassName=MSTestNamespace.UnitTestClass1 Runs tests which are in the class MSTestNamespace.UnitTestClass1.
Note: The ClassName value should have a namespace, ClassName=UnitTestClass1 won't work.
dotnet test --filter FullyQualifiedName!=MSTestNamespace.UnitTestClass1.TestMethod1 Runs all tests except MSTestNamespace.UnitTestClass1.TestMethod1.
dotnet test --filter TestCategory=CategoryA Runs tests which are annotated with [TestCategory("CategoryA")]
dotnet test --filter Priority=3 Runs tests which are annotated with [Priority(3)].Note: Priority~3 is invalid as Priority is an int not a string.

Using Logical operators | and &

Expression What it does?
dotnet test --filter "FullyQualifiedName~UnitTestClass1|TestCategory=CategoryA" Runs tests which have UnitTestClass1 in FullyQualifiedName or TestCategory is CategoryA.
dotnet test --filter "FullyQualifiedName~UnitTestClass1&TestCategory=CategoryA" Runs tests which have UnitTestClass1 in FullyQualifiedName and TestCategory is CategoryA.
dotnet test --filter "(FullyQualifiedName~UnitTestClass1&TestCategory=CategoryA)|Priority=1" Runs tests which have either FullyQualifiedName contains UnitTestClass1 and TestCategory is CategoryA or Priority is 1.

xUnit

Expression What it does?
dotnet test --filter DisplayName=XUnitNamespace.TestClass1.Test1 Runs only one test XUnitNamespace.TestClass1.Test1.
dotnet test --filter FullyQualifiedName!=XUnitNamespace.TestClass1.Test1 Runs all tests except XUnitNamespace.TestClass1.Test1
dotnet test --filter DisplayName~TestClass1 Runs tests whose display name contains TestClass1.

Using traits for filter

namespace XUnitNamespace
{
    public class TestClass1
    {
        [Trait("Category", "bvt")]
        [Trait("Priority", "1")]
        [Fact]
        public void foo()
        {
        }

        [Trait("Category", "Nightly")]
        [Trait("Priority", "2")]
        [Fact]
        public void bar()
        {
        }
    }
}

In above code we defined traits with keys Category and Priority which can be used for filtering.

Expression What it does?
dotnet test --filter XUnit Runs tests whose FullyQualifiedName contains XUnit. Available in vstest 15.1+.
dotnet test --filter Category=bvt Runs tests which have [Trait("Category", "bvt")].

Using Logical operators | and &

Expression What it does?
dotnet test --filter "FullyQualifiedName~TestClass1|Category=Nightly" Runs tests which have TestClass1 in FullyQualifiedName or Category is Nightly.
dotnet test --filter "FullyQualifiedName~TestClass1&Category=Nightly" Runs tests which have TestClass1 in FullyQualifiedName and Category is Nightly.
dotnet test --filter "(FullyQualifiedName~TestClass1&Category=Nightly)|Priority=1" Runs tests which have either FullyQualifiedName contains TestClass1 and Category is CategoryA or Priority is 1.