Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restore Seperate test builds, but fix broken pipe #35378

Merged
merged 6 commits into from
Apr 27, 2020

Conversation

sdmaclea
Copy link
Contributor

No description provided.

* Build CoreClr tests separately

* Build coreclr targetGeneric tests separately for CI
* Build coreclr targetGeneric tests separately in runtime.yml
* Distinguish generic build based on Libraries config
* Build OSX release libraries when CoreCLR changed
* Always use tar.gz compression for generic tests

* Mark all CoreCLR Interop/COM projects OsSpecific
@ghost
Copy link

ghost commented Apr 23, 2020

Tagging subscribers to this area: @ViktorHofer
Notify danmosemsft if you want to be subscribed.

@@ -57,6 +58,11 @@ jobs:
- '${{ parameters.runtimeFlavor }}_common_test_build_p0_${{ parameters.managedTestBuildOsGroup }}${{ parameters.managedTestBuildOsSubgroup }}_${{ parameters.archType }}_${{parameters.buildConfig }}'
- ${{ if ne(parameters.testGroup, 'innerloop') }}:
- '${{ parameters.runtimeFlavor }}_common_test_build_p1_${{ parameters.managedTestBuildOsGroup }}${{ parameters.managedTestBuildOsSubgroup }}_${{ parameters.archType }}_${{parameters.buildConfig }}'
- ${{ if eq(parameters.testBuildPhased, true) }}:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we have to still depend on the non-phased build if this is to true?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

testBuildPhased here means that the tests were built in two phases. TargetGeneric and TargetSpecific tests. We join them here by unzipping into the same directory.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So if testBuildPhased==true we still depend on:
- '${{ parameters.runtimeFlavor }}_common_test_build_p0_${{ parameters.managedTestBuildOsGroup }}${{ parameters.managedTestBuildOsSubgroup }}_${{ parameters.archType }}_${{parameters.buildConfig }}'

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@trylek And I discussed this briefly during the original review.

We may eventually refactor and build the TargetSpecific tests as part of run-test-job. He was suggesting collecting stats to determine whether the extra build legs added value or not.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That sounds like a reasonable, interesting idea.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In fact, I was mostly worried (based on earlier discussions with @jashook) that adding too many legs can be counterproductive by increasing the load on machine allocation and provisioning. I however think that having the specific test build legs initially separate is probably a good idea to get an initial reading on the perf characteristics.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, but if we refactor and just have build-test legs for the generic tests and then the target specific builds are done at the run-test-job, before sending to helix, we would be reducing the number of legs we will have by the number of target specific tests we currently have, so I think it would improve perf and reliability.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suspect you are right.

  • We have reduce the target specific build by at least 90%.
  • The extra leg appears expensive (maybe 50% overhead). I was seeing target specific 10 minute builds in the pipelines. I don't think it was that long on my local machine
  • We should only have to rerun a given leg < 1% of the time (if we achieve 95% success).
  • Fewer legs should reduce infra burden, so we could se benefits elsewhere.
  • Seems like the savings could easily amortize the cost.

Filtering to build the TargetSpecific tests is currently slow. But if we make msbuild improvements this may go away.

eng/pipelines/runtime.yml Outdated Show resolved Hide resolved
@sdmaclea sdmaclea force-pushed the debugOnPrReleaseOnRolling branch from 9bced7d to ca27831 Compare April 23, 2020 23:47
eng/pipelines/runtime.yml Outdated Show resolved Hide resolved
@sdmaclea
Copy link
Contributor Author

I just forced pushed...

eng/pipelines/runtime.yml Outdated Show resolved Hide resolved
Use isFullMatrix rather than debugOnPrReleaseOnRolling
Simplify logic
Fix another duplicate job case
Copy link
Member

@trylek trylek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks even better to me than the last time, thank you!

@@ -57,6 +58,11 @@ jobs:
- '${{ parameters.runtimeFlavor }}_common_test_build_p0_${{ parameters.managedTestBuildOsGroup }}${{ parameters.managedTestBuildOsSubgroup }}_${{ parameters.archType }}_${{parameters.buildConfig }}'
- ${{ if ne(parameters.testGroup, 'innerloop') }}:
- '${{ parameters.runtimeFlavor }}_common_test_build_p1_${{ parameters.managedTestBuildOsGroup }}${{ parameters.managedTestBuildOsSubgroup }}_${{ parameters.archType }}_${{parameters.buildConfig }}'
- ${{ if eq(parameters.testBuildPhased, true) }}:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In fact, I was mostly worried (based on earlier discussions with @jashook) that adding too many legs can be counterproductive by increasing the load on machine allocation and provisioning. I however think that having the specific test build legs initially separate is probably a good idea to get an initial reading on the perf characteristics.

unpackFolder: '$(managedTestArtifactRootFolderPath)'
artifactFileName: '$(managedGenericTestArtifactName).tar.gz'
artifactName: '$(managedGenericTestArtifactName)'
displayName: 'generic managed test artifacts'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might be worth 1-2 more comments explaining the generic vs. specific test management. It would be probably too nitpicky to alternate the displayName of the pre-existing download between "managed test artifacts" and "generic managed test artifacts" but we should try to keep these scripts easy to understand and possibly refactor.

@ghost
Copy link

ghost commented Apr 24, 2020

Hello @sdmaclea!

Because this pull request has the auto-merge label, I will be glad to assist with helping to merge this pull request once all check-in policies pass.

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (@msftbot) and give me an instruction to get started! Learn more here.

@sdmaclea sdmaclea merged commit 6d64040 into dotnet:master Apr 27, 2020
@sdmaclea sdmaclea deleted the debugOnPrReleaseOnRolling branch April 27, 2020 18:24
sdmaclea added a commit to sdmaclea/runtime that referenced this pull request May 5, 2020
sdmaclea added a commit that referenced this pull request May 6, 2020
…35868)

* Revert "Restore Seperate test builds, but fix broken pipe (#35378)"

This reverts commit 6d64040.

* Add issues related to CI coverage outage
@ghost ghost locked as resolved and limited conversation to collaborators Dec 9, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants