Test disappears from GTA after exception

[vko-insiders]

Hey, I'm using VS2017 v15.9.16 with GTA 0.16.1.1369 (installed via Tools -> Extensions and Updates).

When there is an unhandled exception during a test execution, the test is disappearing from the GTA-list.
In comparison: When reproducing the same behavior with MS-Test, the test remains with a red error message: Message: Failed to set up the execution context to run the test.

So I think this is a improvement-request to keep tests that failed hard in the GTA-list.

[csoltenborn]

This is surprising, I haven't seen such behavior before. Can you provide a minimal sample solution which demonstrates the issue?

[vko-insiders]

Unfortunately I cannot reproduce my issue at the moment.
I will close this ticket and reopen a new ticket if I can reproduce it with a minimal example.

[c-stubbe]

I work in the same company as vko-insiders and frequently face the same problem.
I've built a sample solution which demonstrates the issue, see Tests.zip. The solutions consists out of three projects: Dll, GoogleTest and MsTest.

Steps to reproduce:

1. Open the solution and click "Run All" in the Test Explorer.
2. All three projects will be built, both tests will be executed and should be green.
3. Delete the file "Dll.dll" in the "Debug" subfolder and rerun both tests (do not click "Run All" as it will rebuild the DLL.dll).
4. MsTest will get red with the message "Failed to set up the execution context to run the test". GoogleTest will disappear from the Test Explorer.

You can now build the Dll project or even the whole solution or click "Run All". GoogleTest will not appear in the Test Explorer.
The only way to make it appear again is to rebuild the GoogleTest project or to close and reopen the solution. Both take considerable time in our real solution.

[csoltenborn]

Ok, I have watched the behavior, but this is a rather weird scenario, isn't it? Why would you delete that DLL from outside VS?

Anyways. here's what's happening: GTA gets the order from VS to execute the test. However, execution fails due to the missing DLL. GTA recognizes this and prints an according warning: [GTA 20:54:55.666 Warning] 1 test cases seem to not have been run - are you repeating a test run, but tests have changed in the meantime? Test cases: TestCaseName.TestName. And since GTA does not report a test result, VS apparently removes the test from test explorer.

IMHO, this is reasonable behavior, because the test is not available - test discovery would also result in the test not existing (try running GoogleTest.exe on the console after removing that DLL).

Btw, I would argue that your solution has a minor issue: The GoogleTest project lacks a reference to the Dll project. In fact, if that reference exists, the described behavior appears to not be possible - might it be a good idea to add that reference?

[c-stubbe]

This was just a sample solution to reproduce the problem. We faced the problem with external DLLs (mainly from Qt) which were not built in the same solution.
E. g. a test suite is green, I add another test which uses a new Qt module (and thus a new DLL), I rerun the test suite and the test suite disappears. I notice my mistake and add the missing DLL, but cannot rerun the test suite as it is missing.
IMHO, the test should be red, so I can fix it (be adding a DLL) and run it again.
The test executables are not in the same folder than the main executables, which makes the question which DLLs are needed in which folder even more complicated -- I can't be sure that the test will find all necessary DLLs even when the main executable finds all DLLs.

If I understood correctly, it would be possible for GTA to report "test not found" as test result which would solve the problem.
Can this be implemented?

[csoltenborn]

Yes, it can! Feel free to give this build a try...

[c-stubbe]

Thanks!
With "ReportAsFailed" it behaves like I thought it should. I can add the missing DLLs and rerun the one test I am interested in, without having to rebuild anything.
The only thing I noticed is the test result message ("Test case has not been run due to unkown reasons"). This could be made more clearly (maybe "The test case could not be run. There may be a dependency missing since test discovery.").

Will this feature be included in the next release?

Again, thanks for implementing it so quickly.

[csoltenborn]

Yes, it will go intothe next release, and this won't take too long (maybe next weekend). I haven't yet decided which setting will be the default one, though - I might go for Do not report to not change existing behavior...

The test message is not as obvious as it appears to you: For instance, perform discovery, comment out a certain test, and run that same test via right-clicking and selecting Run selected tests. Same behavior, although the test executable runs just fine. That's the use case for printing the warning in debug mode as GTA already does...

[c-stubbe]

Good. I'm looking forward to the next release.

Both is OK with me.
The default setting is not that important. It won't be the only setting which we change after new Visual Studio installs -- and could even be included in that gta.runsettings file.
I agree that there are other reasons why the test can't be run, so "unknown reasons" may be the best message.

[csoltenborn]

Enhancement has been released - enjoy!

Closing...

[c-stubbe]

Thanks.
The new release, installed from the VS marketplace, works as it should.
I like the detailed test message.