Incremental dotnet test

[eatdrinksleepcode]

Description

Incremental build is a critical feature for decreasing build times for large solutions. Currently we have incremental behavior for dotnet build and dotnet publish, but not dotnet test: tests are always run, even if nothing has changed since they last completed successfully.

In the early days of .NET core, before dotnet test existed, I worked with the dotnet build tools team to add this behavior (see dotnet/buildtools#116). It has since been lost in the various tooling changes that have happened.

Steps to reproduce

Consider the following project structure, where capital letters are projects with production code and lower-case letters are projects with test code:

    c
    |
a   C    b
 \ / \  /
  A   B

I run dotnet test at the root: all six projects build, and all three test projects run tests, which pass.

Now I change code in B, and run dotnet test from the root again.

Expected behavior

• B/b and C/c are built, but A/a are not built because they are up-to-date.
• b and c run tests, but a does not run tests because none of its inputs have changed since the tests last passed.

Actual behavior

• B/b and C/c are built, but A/a are not built because they are up-to-date.
• a, b and c run tests, irrespective of whether they have changed since they last passed.

[nohwnd]

Note: This is different from incremental build (which we also fail at, here microsoft/vstest#3266). Incremental build ensures that we don't re-build the assemblies, but we will run tests from all of them.

This asks to only run tests from the assemblies that changed or whose dependencies changed. This seems like a special case of a bigger "run only impacted tests" functionality, that could use other sources of change than just build. @kendrahavens should we consider adding this functionality?

[eatdrinksleepcode]

@nohwnd while "run only impacted tests" would also be a very useful feature, I don't see this as particularly related to that. This feature doesn't have to understand what tests are impacted, or even really that tests are involved at all. I am in fact simply asking to apply the concept of "incremental build" to running tests; in other words, if dotnet test were implemented via a MSBuild target, with the correct inputs and outputs specified, then MSBuild "incremental build" would mean that my suggested behavior would happen automatically. That's how we implemented it in the early versions of the dotnet build tools.

I'm not familiar with the relationship between dotnet test and MSBuild, so I can't say whether this could actually be implemented that way or not. But "run only impacted tests" sounds like a complicated and expensive feature that might be years in development, if it is ever delivered at all. This functionality is simple and straightforward and could be delivered much sooner.

[eatdrinksleepcode]

@nohwnd I continue to see interest in making dotnet test incremental, e.g. to help with monorepos. I can probably contribute the implementation if we can agree on an approach.

Looking at how dotnet test is implemented, It looks like it delegates to a "VSTest" target in the SDK, which executes a task named VSTestTask2, which uses dotnet exec to execute the test runner. Unfortunately this makes it difficult to take advantage of the incremental functionality in MSBuild, since most of the logic of running the tests doesn't happen until after the target has already started executing. One possible solution would be to create a VSTestPrep task class, which uses logic extracted from VSTestTask2 (and TestTaskUtils) to write out the inputs and outputs of the test into dedicated item groups. Those item groups could then be used to specify the inputs and outputs of VSTestTask2.

Do you think that could work? Or do you have any other ideas?

[nohwnd]

@eatdrinksleepcode I don't know enough about this to suggest a better way. Maybe you could tell me what the ideal situation would be and we can work from there? :)

There also was a recent change in net9 to run tfms in parallel that collects tfms to build and runs inner msbuild: #38569

I don't know if that will makes things significantly more difficult, but thought it might be interesting to you.

[nohwnd]

Also you should know that we are also quite unlikely to invest significant effort into this path, we would rather have a more complete and more granular solution.

[eatdrinksleepcode]

we would rather have a more complete and more granular solution.

Is this referring to your previous comment:

This seems like a special case of a bigger "run only impacted tests" functionality, that could use other sources of change than just build.

I'm still all for it. I'm just concerned that it won't happen before I retire 😄 Is there an issue for that I can watch?