Tests pass despite GD.PrintErr(...) or GD.PushError(...)

[bjkarasek]

In documentation (https://mikeschulze.github.io/gdUnit4/testing/assert-error/) it seems that assert-error is not supported in C#. Does that mean there is no way to make tests fail when you print GD.PrintErr or GD.PushErr? Is there a workaround where I can "log" something that will make the tests fail?

[MikeSchulze]

Correct the c# API do not have a monitor to scan for Godot error logs.
Using the logging of an error to force a task to fail is very uncommon in C#.
You should use exception or return error codes to enforce error handling in your code.

[bjkarasek]

Exceptions thrown in the Godot main thread are caught by Godot itself regarding the Godot exception handling.

But you can still track exceptions when you test your components that throw an exception.

Can you please describe your use case in more detail, what you want to test.

I want to be able to implement an integration test where I create a gameplay instance, run some logic on it and then maybe do some asserts. This works fine.

But additionally if in gameplay there is some unexpected issue (in my case there was a desync between host and client as I test a multiplayer game with deterministic simulation) I want to be able to log an error or raise en exception that will fail the tests.

I tried catching the exception and somehow failing the tests this way but failed to do so, though maybe I didn't do it correctly.

[MikeSchulze]

Ok, if I understand you correctly, you have a running game server-client interaction and you have an error handling you want to test.

The best way is to test the components as unit and not as an integration test.
So I would recommend testing the error handling directly.
For example, if there is an error in the client that fails due to a server communication, the test should be structured as follows.
You should use a mocked server so that it corresponds to the error image, call the client in the test and check whether the error is handled.

[bjkarasek]

Ok, if I understand you correctly, you have a running game server-client interaction and you have an error handling you want to test.

The best way is to test the components as unit and not as an integration test. So I would recommend testing the error handling directly. For example, if there is an error in the client that fails due to a server communication, the test should be structured as follows. You should use a mocked server so that it corresponds to the error image, call the client in the test and check whether the error is handled.

Instead of testing error handling I simply want the integration test to fail if something unexpected happens. So another example would be let's say a test where I spawn a building on a map (a strategy game) and the building is actually spawned properly and it exists so asserts pass, but there is some null reference exception thrown because some internal building logic failed after it was properly spawned.

I guess it's reasonable that a test should verify a single unit, but I hoped to do maybe an unusual approach where instead of testing a single thing, I can run a complicated integration test and other than asserting some conditions, simply verify that there were no errors and no exceptions were thrown.

[MikeSchulze]

I understand your problem, but unfortunately, Godot's internal exception logic is thwarting my plans.

Therefore, I would need an exception handler in Godot with which I can forward such internally intercepted exceptions in order to be able to react to them.

For GDScript I had achieved this with a lot of effort with a hack via the Godot.log. I didn't want to install such a hack again ;)

[bjkarasek]

I understand your problem, but unfortunately, Godot's internal exception logic is thwarting my plans.

Therefore, I would need an exception handler in Godot with which I can forward such internally intercepted exceptions in order to be able to react to them.

For GDScript I had achieved this with a lot of effort with a hack via the Godot.log. I didn't want to install such a hack again ;)

That's unfortunate. I guess I'll try to somehow hack it for a partial solution so that at least the exceptions that I throw will fail the tests.

Still thank you for your time and for a great project :)

[MikeSchulze]

Update

I was able to hook into the Godot Exception handling by prototyping an exception hook handler.

    [TestCase]
    public void Example()
    {
        var sceneTree = (SceneTree)Engine.GetMainLoop();
        sceneTree.Root.AddChild(new ExceptionTestNode());
    }


    // Test class to verify the interceptor
    public partial class ExceptionTestNode : Node
    {
        public override void _Ready() =>
            // Trigger a test exception
            throw new InvalidOperationException("This is a internal exception.");
    }

it aborts the test now and the report contains the exception source

[MikeSchulze]

Update

You can now test with release candidate 6 the exception hook.
Update your project

    <PackageReference Include="gdUnit4.api" Version="4.4.0-rc*"/>
    <PackageReference Include="gdUnit4.test.adapter" Version="2.1.0-rc*"/>

The hock will now report exceptions naturally hidden by Godot main thread as test failure.
Godot PushError and PrintErr are not affected, you should use exceptions to propagate unexpected behaviors.

Please give me feedback, Thanks

[bjkarasek]

Update

You can now test with release candidate 6 the exception hook. Update your project

    <PackageReference Include="gdUnit4.api" Version="4.4.0-rc*"/>
    <PackageReference Include="gdUnit4.test.adapter" Version="2.1.0-rc*"/>

The hock will now report exceptions naturally hidden by Godot main thread as test failure. Godot PushError and PrintErr are not affected, you should use exceptions to propagate unexpected behaviors.

Please give me feedback, Thanks

Amazing! Great to hear that it can be done! I tested it a bit and it seems to be working well, however it only works when I run tests from JetBrains Rider. If I run them from the Godot Editor, then they won't fail.

[MikeSchulze]

I have reopened the issue because correctly caught exceptions are still reported as errors, but this should not be the case.

 public override void _Process(double delta)
    {
        // we throw an example exception
        if (frameCount == 10)
            throw new InvalidProgramException("Exception during scene processing");
        try
        {
            if (frameCount == 5)
                // is reported too, but should not
                throw new InvalidProgramException("Exception during scene processing inside a catch block");
        }
        catch (InvalidProgramException)
        {
            // ignore
        }

        frameCount++;
    }

[MikeSchulze]

For running inside Godot the Exception hook is not enabled.

[MikeSchulze]

I'm actually struggling, I opened a discussion here