TimeoutTestNodeStateProperty not causing test session failure and summary count is wrong

[thomhurst]

This seemed to have worked in 1.3.2, but is failing on 1.4.0-preview.24422.11.

Reporting just tests that have timed out causes:

• The test session to pass, when it should fail
• The count at the end to be wrong

Test run summary: Passed! - TestingPlatformTimeoutStateRepro.dll (net8.0|x64)
  total: 1
  failed: 0
  succeeded: 0
  skipped: 0
  duration: 66ms

As you can see, there's a total of 1, but 0 for all the actual states it could be.

Repro:

// See https://aka.ms/new-console-template for more information

using Microsoft.Testing.Platform.Builder;
using Microsoft.Testing.Platform.Capabilities;
using Microsoft.Testing.Platform.Capabilities.TestFramework;
using Microsoft.Testing.Platform.Extensions.Messages;
using Microsoft.Testing.Platform.Extensions.TestFramework;
using Microsoft.Testing.Platform.TestHost;

var builder = await TestApplication.CreateBuilderAsync(args);
builder.RegisterTestFramework(_ => new TestFrameworkCapabilities(), (_, _) => new DummyAdapter());
var app = await builder.BuildAsync();
return await app.RunAsync();

internal class DummyAdapter() : ITestFramework, IDataProducer
{
    public string Uid => nameof(DummyAdapter);

    public string Version => string.Empty;

    public string DisplayName => string.Empty;

    public string Description => string.Empty;

    public Type[] DataTypesProduced => new[] { typeof(TestNodeUpdateMessage) };

    public Task<CloseTestSessionResult> CloseTestSessionAsync(CloseTestSessionContext context) => Task.FromResult(new CloseTestSessionResult { IsSuccess = true });

    public Task<CreateTestSessionResult> CreateTestSessionAsync(CreateTestSessionContext context) => Task.FromResult(new CreateTestSessionResult { IsSuccess = true });

    public Task ExecuteRequestAsync(ExecuteRequestContext context)
    {
        context.MessageBus.PublishAsync(this, new TestNodeUpdateMessage(new SessionUid("1"), new TestNode()
        {
            Uid = "2",
            DisplayName = "Blah",
            Properties = new PropertyBag(new TimeoutTestNodeStateProperty(new Exception()))
        }));
     
        context.Complete();
        
        return Task.CompletedTask;
    }

    public Task<bool> IsEnabledAsync() => Task.FromResult(true);
}

internal class Capabilities : ITestFrameworkCapabilities
{
    IReadOnlyCollection<ITestFrameworkCapability> ICapabilities<ITestFrameworkCapability>.Capabilities
        => Array.Empty<ITestFrameworkCapability>();
}

[MarcoRossignoli]

@thomhurst can you confirm that the exit code is not 0?

@nohwnd I suppose some issue with the new UX?

[thomhurst]

@MarcoRossignoli ExitCode looks to be 2, so that's good. Just a misleading UI then.

Test run summary: Passed! - bin\Debug\net8.0\TUnit.TestProject.dll (net8.0|x64)
  total: 1
  failed: 0
  succeeded: 0
  skipped: 0
  duration: 30s 608ms

PS C:\git\TUnit\TUnit.TestProject> echo $LASTEXITCODE
2

[nohwnd]

Will have a look. :)

We need that intermediate service that summarizes the data for consumers, and the ux will just render them so all different possible renditions of the result (like exit code, terminal and trx) are reporting the same results.