[wasm] Wasm.Build.Tests: add support for shared builds #49398

radical · 2021-03-09T23:50:44Z

Essentially, we want to share builds wherever possible. Example cases:
- Same build, but run with different hosts like v8/chrome/safari, as
  separate test runs
- Same build but run with different command line arguments
Sharing builds especially helps when we are AOT'ing, which is slow!
This is done by caching the builds with the key:

public record BuildArgs(string ProjectName, string Config, bool AOT, string ProjectFileContents, string? ExtraBuildArgs);
Also, SharedBuildClassFixture is added, so that the builds can be
cleaned up after all the tests in a particular class have finished
running.
Each test run gets a randomly generated test id. This is used for
creating:
1. build paths, like artifacts/bin/Wasm.Build.Tests/net6.0-Release/browser-wasm/xharness-output/logs/n1xwbqxi.ict
2. and the log for running with xharness, eg. for Chrome, are in
  artifacts/bin/Wasm.Build.Tests/net6.0-Release/browser-wasm/xharness-output/logs/n1xwbqxi.ict/Chrome/
split WasmBuildAppTest.cs into : BuildTestBase.cs, and
MainWithArgsTests.cs.

Note: In interest of lowering test run times, this builds the tests in Release config only, on CI.
But when running the tests locally, both Debug, and Release configs are used.

ghost · 2021-03-09T23:50:52Z

Tagging subscribers to this area: @directhex
See info in area-owners.md if you want to be subscribed.

Issue Details

Essentially, we want to share builds wherever possible. Example cases:
- Same build, but run with different hosts like v8/chrome/safari, as
  separate test runs
- Same build but run with different command line arguments
Sharing builds especially helps when we are AOT'ing, which is slow!
This is done by caching the builds with the key:

public record BuildArgs(string ProjectName, string Config, bool AOT, string ProjectFileContents, string? ExtraBuildArgs);
Also, SharedBuildClassFixture is added, so that the builds can be
cleaned up after all the tests in a particular class have finished
running.
Each test run gets a randomly generated test id. This is used for
creating:
1. build paths, like artifacts/bin/Wasm.Build.Tests/net6.0-Release/browser-wasm/xharness-output/logs/n1xwbqxi.ict
2. and the log for running with xharness, eg. for Chrome, are in
  artifacts/bin/Wasm.Build.Tests/net6.0-Release/browser-wasm/xharness-output/logs/n1xwbqxi.ict/Chrome/
split WasmBuildAppTest.cs into : BuildTestBase.cs, and
MainWithArgsTests.cs.

Author:	radical
Assignees:	-
Labels:	`area-Infrastructure-mono`
Milestone:	-

ghost · 2021-03-09T23:51:24Z

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

Essentially, we want to share builds wherever possible. Example cases:
- Same build, but run with different hosts like v8/chrome/safari, as
  separate test runs
- Same build but run with different command line arguments
Sharing builds especially helps when we are AOT'ing, which is slow!
This is done by caching the builds with the key:

public record BuildArgs(string ProjectName, string Config, bool AOT, string ProjectFileContents, string? ExtraBuildArgs);
Also, SharedBuildClassFixture is added, so that the builds can be
cleaned up after all the tests in a particular class have finished
running.
Each test run gets a randomly generated test id. This is used for
creating:
1. build paths, like artifacts/bin/Wasm.Build.Tests/net6.0-Release/browser-wasm/xharness-output/logs/n1xwbqxi.ict
2. and the log for running with xharness, eg. for Chrome, are in
  artifacts/bin/Wasm.Build.Tests/net6.0-Release/browser-wasm/xharness-output/logs/n1xwbqxi.ict/Chrome/
split WasmBuildAppTest.cs into : BuildTestBase.cs, and
MainWithArgsTests.cs.

Author:	radical
Assignees:	-
Labels:	`arch-wasm`, `area-Infrastructure-mono`
Milestone:	-

radical · 2021-03-11T02:40:56Z

This will be rebased once #49446 is merged.

- Essentially, we want to share builds wherever possible. Example cases: - Same build, but run with different hosts like v8/chrome/safari, as separate test runs - Same build but run with different command line arguments - Sharing builds especially helps when we are AOT'ing, which is slow! - This is done by caching the builds with the key: `public record BuildArgs(string ProjectName, string Config, bool AOT, string ProjectFileContents, string? ExtraBuildArgs);` - Also, ` SharedBuildClassFixture` is added, so that the builds can be cleaned up after all the tests in a particular class have finished running. - Each test run gets a randomly generated test id. This is used for creating: 1. build paths, like `artifacts/bin/Wasm.Build.Tests/net6.0-Release/browser-wasm/xharness-output/logs/n1xwbqxi.ict` 2. and the log for running with xharness, eg. for Chrome, are in `artifacts/bin/Wasm.Build.Tests/net6.0-Release/browser-wasm/xharness-output/logs/n1xwbqxi.ict/Chrome/` - split `WasmBuildAppTest.cs` into : `BuildTestBase.cs`, and `MainWithArgsTests.cs`.

…zation

.. tests. Code stolen from @maximlipin's dotnet#49204

For AOT we generate `pinvoke-table.h` in the obj directory. But there is one present in the runtime pack too. In my earlier changes the order in which these were passed as include search paths was changed from: `"-I/runtime/pack/microsoft.netcore.app.runtime.browser-wasm/Release/runtimes/browser-wasm/native/include/wasm" "-Iartifacts/obj/mono/Wasm.Console.Sample/wasm/Release/browser-wasm/wasm/"` .. which meant that the one from the runtime pack took precedence, and got used. So, fix the order! And change the property names to indicate where they are sourced from.

The environment variable is set on helix. During local testing it can be useful when using a locally built xharness.

radical · 2021-03-12T17:31:50Z

Unrelated test failures.

steveisok

LGTM. I'm curious the average helix run time w/ these changes.

radical · 2021-03-16T16:32:01Z

AFAICS, the test time stays at about 45mins. With this one we actually are building only Release, but #49559 will enable testing for Debug too. And I will add support to enable more parallelization.

Though when I look at the run times for the individual runs for wasm.build tests, they seem to be around ~25mins.

radical · 2021-03-16T17:09:58Z

AFAICS, the test time stays at about 45mins.

The helix step runs in total 45mins when I looked at one of the earlier main builds. The wasm.build test run itself takes ~25mins, it runs in parallel to the library tests.

radical · 2021-03-16T19:52:54Z

Repeatedly getting test run failures:

.packages/microsoft.dotnet.helix.sdk/6.0.0-beta.21155.1/tools/Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(47,5): error : (NETCORE_ENGINEERING_TELEMETRY=Helix) RestApiException`1: The response contained an invalid status code 500 Internal Server Error

Body: {"Message":"An error occured.","ActivityId":"0ade13a01812c1468a790d153e2fc3a0"}
   at Microsoft.DotNet.Helix.Client.Storage.OnNewFailed(Request req, Response res) in /_/src/Microsoft.DotNet.Helix/Client/CSharp/generated-code/Storage.cs:line 202
   at Microsoft.DotNet.Helix.Client.Storage.NewAsync(ContainerCreationRequest body, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.Helix/Client/CSharp/generated-code/Storage.cs:line 164
   at Microsoft.DotNet.Helix.Client.ApiBlobHelper.GetContainerAsync(String requestedName, String targetQueue, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.Helix/JobSender/StorageHelpers/ApiBlobHelper.cs:line 22
   at Microsoft.DotNet.Helix.Client.JobDefinition.SendAsync(Action`1 log, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.Helix/JobSender/JobDefinition.cs:line 157
   at Microsoft.DotNet.Helix.Sdk.SendHelixJob.ExecuteCore(CancellationToken cancellationToken) in /_/src/Microsoft.Dot

.. tracked in https://github.com/dotnet/core-eng/issues/12499 .

radical · 2021-03-17T16:33:59Z

Added a few more tests, and some cleanup.

radical · 2021-03-26T02:22:35Z

investigating the test run time

TLDR; - this might help with the job getting scheduled first, and thus having a chance at completing at the same time as others. Reasoning: The problem this is trying to fix is: 1. The helix step submits 3 jobs: a. library tests to be run with v8 b. library tests to be run with browser (scenario=wasmtestonbrowser) c. Wasm.Build.Tests (scenario=buildwasmapps) 2. The 3 jobs, individually take roughly 30mins each 3. And they get submitted at roughly the same time 4. But .. the first two seem to complete earlier, and the 3rd one completes 25-30mins later. The hypothesis is that all the machines might be busy processing the 200+ work items from each of the first two steps, and so Wasm.Build.Tests get scheduled pretty late. So, here we move that to be submitted first, in the hope that it would be able to run in parallel with the other jobs.

radical · 2021-03-26T19:48:56Z

@akoeplinger with your suggested change (7caed5a) it ran in 43mins, instead of 59! 👍

radical · 2021-03-26T22:21:48Z

@lewing @steveisok this is ready for another look! And hopefully merge :D

dotnet-issue-labeler bot added the area-Infrastructure-mono label Mar 9, 2021

radical marked this pull request as draft March 9, 2021 23:51

radical added the arch-wasm WebAssembly architecture label Mar 9, 2021

radical force-pushed the shared-wasm-build-tests branch 3 times, most recently from 3f95a20 to 78840c3 Compare March 10, 2021 21:52

radical marked this pull request as ready for review March 10, 2021 22:43

radical requested a review from marek-safar as a code owner March 10, 2021 22:43

radical requested review from lewing, steveisok and akoeplinger March 10, 2021 22:44

radical force-pushed the shared-wasm-build-tests branch from 5a03409 to cedeeca Compare March 11, 2021 02:35

radical force-pushed the shared-wasm-build-tests branch from cedeeca to 2dbfc6a Compare March 11, 2021 05:36

radical added 9 commits March 11, 2021 17:35

[wasm] Wasm.Build.Tests: Add test for relinking with InvariantGlobali…

af44fea

…zation

[wasm] Wasm.Build.Tests - Check CultureInfo for invariant culture

1b102b3

.. tests. Code stolen from @maximlipin's dotnet#49204

fix invariant+aot test

33d87f2

[wasm] Only test with Release config on CI

250653e

[wasm] Fallback to dotnet xharness if XHARNESS_CLI_PATH is not set.

3cd29e0

The environment variable is set on helix. During local testing it can be useful when using a locally built xharness.

[wasm] fix invariant test - 'en-ES' -> 'es-ES'

739881f

[wasm] RunWithEmSdkEnv: log the working directory also

fcf0da6

radical force-pushed the shared-wasm-build-tests branch from 808e761 to fcf0da6 Compare March 11, 2021 23:52

radical added 2 commits March 11, 2021 19:18

Merge remote-tracking branch 'origin/main' into wasm-build-tests

cb7aff7

[wasm] Re-enable wasm build tests

1ce676f

runfoapp bot mentioned this pull request Mar 12, 2021

Test failure: System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CertificateRequestChainTests/CreateChain_Hybrid #25979

Closed

radical added 2 commits March 16, 2021 10:45

[wasm] Add regression test for issue dotnet#49588

522424a

Merge remote-tracking branch 'origin/main' into wasm-build-tests

86b4b36

steveisok approved these changes Mar 16, 2021

View reviewed changes

fix test

aae95e2

radical added 2 commits March 17, 2021 12:30

[wasm] Cleanup, and add more tests

3540462

Merge remote-tracking branch 'origin/main' into wasm-build-tests

5e90d6f

radical added 4 commits March 17, 2021 13:16

Merge remote-tracking branch 'origin/main' into wasm-build-tests

02045a3

Update tests to track wasm relinking being default in some cases

c21da91

Fix InvariantGlobalization to track change in wasm relinking defaults

d6e9732

Merge remote-tracking branch 'origin/main' into wasm-build-tests

da64faf

runfoapp bot mentioned this pull request Mar 22, 2021

System.Security.Cryptography.X509Certificates.Tests.ChainTests.BuildInvalidSignatureTwice #48851

Closed

radical added 3 commits March 25, 2021 10:46

Merge remote-tracking branch 'origin/main' into wasm-build-tests

eaf1615

[wasm] Update emsdk check message to track changes

97b83b8

[wasm] Update tests to track changes

f463714

steveisok approved these changes Mar 26, 2021

View reviewed changes

radical mentioned this pull request Mar 26, 2021

[wasm] Fix order of include paths, to have the obj dir first #50303

Merged

radical merged commit 617a18d into dotnet:main Mar 29, 2021

radical deleted the shared-wasm-build-tests branch March 29, 2021 19:33

ghost locked as resolved and limited conversation to collaborators Apr 28, 2021

karelz added this to the 6.0.0 milestone May 20, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[wasm] Wasm.Build.Tests: add support for shared builds #49398

[wasm] Wasm.Build.Tests: add support for shared builds #49398

radical commented Mar 9, 2021 •

edited

Loading

ghost commented Mar 9, 2021

ghost commented Mar 9, 2021

radical commented Mar 11, 2021

radical commented Mar 12, 2021

steveisok left a comment

radical commented Mar 16, 2021

radical commented Mar 16, 2021 •

edited

Loading

radical commented Mar 16, 2021

radical commented Mar 17, 2021

radical commented Mar 26, 2021

radical commented Mar 26, 2021

radical commented Mar 26, 2021

[wasm] Wasm.Build.Tests: add support for shared builds #49398

[wasm] Wasm.Build.Tests: add support for shared builds #49398

Conversation

radical commented Mar 9, 2021 • edited Loading

ghost commented Mar 9, 2021

ghost commented Mar 9, 2021

radical commented Mar 11, 2021

radical commented Mar 12, 2021

steveisok left a comment

Choose a reason for hiding this comment

radical commented Mar 16, 2021

radical commented Mar 16, 2021 • edited Loading

radical commented Mar 16, 2021

radical commented Mar 17, 2021

radical commented Mar 26, 2021

radical commented Mar 26, 2021

radical commented Mar 26, 2021

radical commented Mar 9, 2021 •

edited

Loading

radical commented Mar 16, 2021 •

edited

Loading