-
Notifications
You must be signed in to change notification settings - Fork 18.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Windows: Build and use gotestsum for running all tests #39998
Windows: Build and use gotestsum for running all tests #39998
Conversation
Jenkinsfile
Outdated
@@ -828,6 +828,7 @@ pipeline { | |||
} | |||
post { | |||
always { | |||
junit testResults: 'bundles/junit-report.xml', allowEmptyResults: true |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can disable the allowEmptyResults
for now, so that it's clear if it failed
Perhaps we should do a dir
(e.g.) too see what files are present, and if the location in which it's stored is correct
one failure, which is known to be flaky; #32219 https://ci.docker.com/public/job/moby/job/PR-39998/1/execution/node/209/log/?consoleFull
It did run the archive junit results step; https://ci.docker.com/public/job/moby/job/PR-39998/1/execution/node/355/log/ But I don't see the test-results for windows; |
To help with development iteration, you could throw in a temporary git commit to disable tests except for Lines 11 to 17 in 5b57f41
|
3ebfb8f
to
af4e179
Compare
Looks like it still doesn't write results in the right location; https://ci.docker.com/public/job/moby/job/PR-39998/5/execution/node/356/log/
|
0595e1d
to
1818270
Compare
1818270
to
3cb4def
Compare
3cb4def
to
8fab97f
Compare
@vikramhh I rebased your branch again; I think your local git was not up-to-date with the upstreams. |
3b8b468
to
413a18e
Compare
59c3482
to
7408acd
Compare
Dockerfile.windows
Outdated
` | ||
Write-Host INFO: Completed | ||
Write-Host INFO: Building gotestsum...; ` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we move this inside the Build-GoTestSum
function?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I do not know a compelling argument either way. Why would you choose one over the other?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
All other messages (including "Build done ..") are printed as part of the script, so it would make sense to include this as well. If it's move to a script, we could just call Build-GoTestSum
and be done
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
} ` | ||
} ` | ||
} ` | ||
Function Build-GoTestSum() { ` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The scripts that we're writing in the Dockerfile now are a bit lengthy, and more complex to maintain (having to keep all the line-continuations (```) into account.
Perhaps we should put these function in a .ps
file (easier to write/maintain), and;
Somewhere at the top of the Dockerfile (assuming the functions themself don't change often);
COPY hack/dockerfile/install/gotestsum.installer.ps1 hack/dockerfile/install/gotestsum.installer.ps1
And later on, call those functions
RUN . "hack/dockerfile/install/gotestsum.installer.ps1" Build-GoTestSum
(or however that should be done in PowerShell)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good suggestion and something I have put on my plate. It will be done in a separate PR
hack/make.ps1
Outdated
ForEach($dir in $integration_api_dirs) { | ||
#$integration_api_dirs += $dir | ||
#Write-Host "Building test suite binary $dir" | ||
#go test -c -o "$dir\test.exe" $dir |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same here.
Wondering though why we're no longer building the binaries here? (perhaps you can explain why this changed)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Instead of building the test binary and then running it, we are just running the test. Why we need to do it in two different stages?
hack/make.ps1
Outdated
$p = New-Object System.Diagnostics.Process | ||
$p.StartInfo = $pinfo | ||
$p.Start() | Out-Null | ||
$p.WaitForExit() | ||
$err = $p.StandardError.ReadToEnd() | ||
Write-Host "$err" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this is to print stderr
output if there's no error that happened, correct?
In case of an error ($LASTEXITCODE -ne 0
), this will now be printed twice. Think it should be changed to either;
if (($LASTEXITCODE -ne 0) -and ($err -notlike "*warning: no tests to run*")) {
Throw "Integration tests failed: $err"
} else {
Write-Host "$err"
}
Or (if Throw
exits the script (not sure if if does so?)); perhaps add a comment as well to explain why we're printing $err
if (($LASTEXITCODE -ne 0) -and ($err -notlike "*warning: no tests to run*")) {
Throw "Integration tests failed: $err"
}
# Write stderr output
Write-Host "$err"
7408acd
to
eb5e871
Compare
9c500f5
to
79ebfeb
Compare
This PR is valuable as is with RS5 junit test results in jenkins. It is important to expose the failures since the return code is always success. I'd recommend getting this PR in as soon as possible and having a separate PR for RS1. |
The issue we noticed when discussing, is that the naming of the tests for Windows is different than the ones for Linux, therefore the tests show in a different place; for reference, this is how the naming is set for the Linux ones for the junit.xml; moby/hack/make/.integration-test-helpers Lines 69 to 90 in e780565
|
The above is true only for integration tests and could be handled by transforming the output of all tests [i.e. the xml files] by applying the same rules to each test phase [unit/integration and integration-cli] once we figure out where each result type should show up. This is not a blocker. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
@vikramhh could you rebase this PR, because I think the failing test is fixed / skipped on master, so after rebasing, it should be green |
5b8d715
to
3c3b9bf
Compare
fef5b43
to
10786ba
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
(rebased, to resolve merge-conflict due to Golang bump)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No idea about the powershell, but LGTM! 🐑
remaining failures were tests that don't work on windows, and are now skipped through #40199 |
10786ba
to
5fc0f6b
Compare
rebased to run CI once more, but moving to "merge" |
Remaining "failure" is
Which will be addressed by #40193 |
1. Dockerfile.Windows modified to build gotestsum.exe 2. Use gotestsum.exe in invoking the execution of: (a) Unit tests (run in containers), (b) Integration tests (run outside containers) (c) Integration-cli (run outside containers) No changes made to other categories of tests (e.g. LCOW). 3. Copy .xml files produced by gotestsum in appropriate paths where Jenkins can ingest them 4. Modify Jenkinsfile to mark results output as being jUnit "type" as well as to archive the .xml test result files as artifacts. Signed-off-by: Vikram bir Singh <vikrambir.singh@docker.com> Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
5fc0f6b
to
8d2e1ee
Compare
#40193 was merged, so rebased |
All green now. As a follow-up we need to rewrite the test names to match what's used on Linux, so that the Windows tests will appear separate (instead of under Windows currently produces: <testsuite tests="22" failures="0" time="28.840000" name="github.com/docker/docker/integration/build">
<properties>
<property name="go.version" value="go1.13.4 windows/amd64"></property>
</properties>
<testcase classname="github.com/docker/docker/integration/build" name="TestBuildWithSession" time="0.000000"> The same test on Linux (amd64) produces: <testsuite tests="32" failures="0" time="171.970000" name="amd64.integration.build">
<properties>
<property name="go.version" value="go1.13.4 linux/amd64"></property>
</properties>
<testcase classname="amd64.integration.build" name="TestBuildWithSession" time="0.000000">
<skipped message="=== RUN TestBuildWithSession
--- SKIP: TestBuildWithSession (0.00s)
 build_session_test.go:25: TODO: BuildKit
"></skipped> |
carry of #39971 to make it pick-up the Jenkinsfile changes