Skip to content
This repository has been archived by the owner on Nov 3, 2023. It is now read-only.

Commit

Permalink
[Harness] Fix exit code of 0 being reported in case of exceptions in …
Browse files Browse the repository at this point in the history
…the harness

This caused a weird issue where we had the following:
Most of the MD assemblies built against GuiUnit.
MonoDevelop.MonoDroid.Tests built against GuiUnit-Ng.

Since we switched to test deployment, GuiUnit seemed to be the last one
being copied.

Thus, the Android tests would fail to load:
```
2018-08-02T08:55:18.2470490Z Can't find custom attr constructor image: /Users/vsts/agent/2.138.4/work/r1/a/VS for Mac/drop/tests/MonoDevelop.MonoDroid.Tests.dll mtoken: 0x0a000008 due to: Could not resolve type with token 01000013 (from typeref, class/assembly NUnit.Framework.OneTimeTearDownAttribute, GuiUnit, Version=1.0.6397.19727, Culture=neutral, PublicKeyToken=8ebbaf189675e490) assembly:GuiUnit, Version=1.0.6397.19727, Culture=neutral, PublicKeyToken=8ebbaf189675e490 type:NUnit.Framework.OneTimeTearDownAttribute member:(null) signature:<none>
2018-08-02T08:55:18.2513870Z System.TypeLoadException: Could not resolve type with token 01000013 (from typeref, class/assembly NUnit.Framework.OneTimeTearDownAttribute, GuiUnit, Version=1.0.6397.19727, Culture=neutral, PublicKeyToken=8ebbaf189675e490)
2018-08-02T08:55:18.2534360Z   at (wrapper managed-to-native) System.MonoCustomAttrs.IsDefinedInternal(System.Reflection.ICustomAttributeProvider,System.Type)
2018-08-02T08:55:18.2557740Z   at System.MonoCustomAttrs.IsDefined (System.Reflection.ICustomAttributeProvider obj, System.Type attributeType, System.Boolean inherit) [0x00027] in /Users/builder/jenkins/workspace/build-package-osx-mono/2018-02/external/bockbuild/builds/mono-x64/mcs/class/corlib/System/MonoCustomAttrs.cs:317
2018-08-02T08:55:18.2581720Z   at System.Reflection.MonoMethod.IsDefined (System.Type attributeType, System.Boolean inherit) [0x00000] in /Users/builder/jenkins/workspace/build-package-osx-mono/2018-02/external/bockbuild/builds/mono-x64/mcs/class/corlib/System.Reflection/MonoMethod.cs:386
2018-08-02T08:55:18.2605460Z   at NUnit.Framework.Internal.Reflect.GetMethodsWithAttribute (System.Type fixtureType, System.Type attributeType, System.Boolean inherit) [0x0001a] in /Users/vsts/agent/2.138.4/work/1/s/md-addins/external/guiunit/src/framework/Internal/Reflect.cs:71
2018-08-02T08:55:18.2628860Z   at NUnit.Framework.Internal.TestFixture..ctor (System.Type fixtureType, System.Object[] arguments) [0x00008] in /Users/vsts/agent/2.138.4/work/1/s/md-addins/external/guiunit/src/framework/Internal/Tests/TestFixture.cs:53
2018-08-02T08:55:18.2653880Z   at NUnit.Framework.Builders.NUnitTestFixtureBuilder.BuildSingleFixture (System.Type type, NUnit.Framework.TestFixtureAttribute attr) [0x00033] in /Users/vsts/agent/2.138.4/work/1/s/md-addins/external/guiunit/src/framework/Internal/Builders/NUnitTestFixtureBuilder.cs:160
2018-08-02T08:55:18.2677470Z   at NUnit.Framework.Builders.NUnitTestFixtureBuilder.BuildFrom (System.Type type) [0x00038] in /Users/vsts/agent/2.138.4/work/1/s/md-addins/external/guiunit/src/framework/Internal/Builders/NUnitTestFixtureBuilder.cs:110
2018-08-02T08:55:18.2700470Z   at NUnit.Framework.Internal.TestFixtureBuilder.BuildFrom (System.Type type) [0x00000] in /Users/vsts/agent/2.138.4/work/1/s/md-addins/external/guiunit/src/framework/Internal/TestFixtureBuilder.cs:65
2018-08-02T08:55:18.2724530Z   at NUnit.Framework.Internal.NUnitLiteTestAssemblyBuilder.GetFixtures (System.Reflection.Assembly assembly, System.Collections.IList names) [0x0002a] in /Users/vsts/agent/2.138.4/work/1/s/md-addins/external/guiunit/src/framework/Internal/NUnitLiteTestAssemblyBuilder.cs:110
2018-08-02T08:55:18.2748530Z   at NUnit.Framework.Internal.NUnitLiteTestAssemblyBuilder.Build (System.Reflection.Assembly assembly, System.Collections.IDictionary options) [0x00018] in /Users/vsts/agent/2.138.4/work/1/s/md-addins/external/guiunit/src/framework/Internal/NUnitLiteTestAssemblyBuilder.cs:52
2018-08-02T08:55:18.2772390Z   at NUnit.Framework.Internal.NUnitLiteTestAssemblyRunner.Load (System.Reflection.Assembly assembly, System.Collections.IDictionary settings) [0x00007] in /Users/vsts/agent/2.138.4/work/1/s/md-addins/external/guiunit/src/framework/Internal/NUnitLiteTestAssemblyRunner.cs:96
2018-08-02T08:55:18.2795930Z   at GuiUnit.TestRunner.ExecuteWithListener (System.String[] args, NUnitLite.Runner.TcpWriter tcpWriter) [0x0016f] in /Users/vsts/agent/2.138.4/work/1/s/md-addins/external/guiunit/src/framework/GuiUnit/TestRunner.cs:188
```

Since GuiUnit did not report exit code failure for any exception in the harness,
most of the shell scripts which would verify the exit code would just continue
and consider the test run successful.

In case of any exception other than FileNotFoundException, we return 1 as exit code.

Fixes VSTS 657366 - CI does not fail if the test runner fails to run
  • Loading branch information
Therzok authored and alanmcgovern committed Aug 2, 2018
1 parent 9b7497c commit 0c3159a
Showing 1 changed file with 1 addition and 0 deletions.
1 change: 1 addition & 0 deletions src/framework/GuiUnit/TestRunner.cs
Original file line number Diff line number Diff line change
Expand Up @@ -242,6 +242,7 @@ void ExecuteWithListener (string[] args, TcpWriter tcpWriter)
catch (Exception ex)
{
writer.WriteLine(ex.ToString());
ExitCode = 1;
}
finally
{
Expand Down

0 comments on commit 0c3159a

Please sign in to comment.