Skip to content
This repository has been archived by the owner on Jan 23, 2023. It is now read-only.

Switch to Arcade yaml stage-based publishing with custom steps #7725

Merged
merged 7 commits into from
Aug 15, 2019

Conversation

dagood
Copy link
Member

@dagood dagood commented Aug 14, 2019

For https://github.com/dotnet/core-setup/issues/7118.

This PR splits the current publish step into three parts:

  1. Prepare artifacts for Arcade publishing (sign them, filter them)
  2. Run Arcade publishing steps (nupkgs to blob feed, symbol packages to symbol servers)
  3. Run custom steps that Arcade doesn't handle
    • Publish binaries to blob storage.
    • Generate and publish checksums to blob storage.
    • Publish to dotnet/versions.
    • Copy bits to dotnetcli and dotnetclichecksums Latest channel ("finalize").

The custom publishing is configured based on channel, so it hooks in nicely with the Arcade way of specifying when a certain type of publishing should happen. By default, the custom steps aren't run. When a new channel is introduced, the configuration must be updated if custom publish should happen for it. (I was concerned earlier about how this would attach, I think this is actually pretty ideal.)

Workarounds:

  1. -warnAsError:$false is a workaround for [post-build] Allow passing additional parameters to sdk-task.ps1 invocation in the post-build yaml template arcade#3666. It's a bit of a time bomb (modifications to eng/common are overwritten by the next Maestro++ update), so I don't currently intend to merge this PR with it still there. Edit: I've replaced this workaround with a manual integration of Make post-build.yml accept parameters to sdk-task.ps1 arcade#3676, so this workaround is fine to merge.
  2. Add --dry-run to darc gather-drop test to (try to) avoid flakiness observed in test builds. See Change gather-drop post build script to do a dry-run arcade#3675.

Example build: https://dev.azure.com/dnceng/internal/_build/results?buildId=308903&view=results. (Enabling the stages view preview feature recommended to see the flow. I find myself turning it off and on very frequently to deal with its issues.)

@riarenas @JohnTortugo @mmitche @wtgodbe

@@ -38,7 +37,7 @@
<PropertyGroup>
<!-- arcade -->
<MicrosoftDotNetBuildTasksPackagingPackageVersion>1.0.0-beta.19308.1</MicrosoftDotNetBuildTasksPackagingPackageVersion>
<MicrosoftDotNetBuildTasksFeedPackageVersion>2.2.0-beta.19308.1</MicrosoftDotNetBuildTasksFeedPackageVersion>
<MicrosoftDotNetBuildTasksFeedPackageVersion>2.2.0-beta.19410.2</MicrosoftDotNetBuildTasksFeedPackageVersion>
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I filed https://github.com/dotnet/core-setup/issues/7693 to get these up to date more generally and get them auto-updating.

@mmitche
Copy link
Member

mmitche commented Aug 14, 2019

@dagood, @riarenas Is working on the publishing steps for dotnetclichecksums and dotnetcli blob storage publishing, though not copy to latest or checksum generation.

@dagood
Copy link
Member Author

dagood commented Aug 14, 2019

I'm aware, it's tracked at dotnet/arcade#3607. I'm not waiting on it, or planning on using that as part of this effort unless migrating to it is trivial.

@dagood dagood added the * NO MERGE * The PR is not ready for merge yet (see discussion for detailed reasons) label Aug 14, 2019
@dagood dagood marked this pull request as ready for review August 14, 2019 20:21
This is possible now that signing is in a different job.
eng/jobs/prepare-signed-artifacts.yml Outdated Show resolved Hide resolved
eng/stages/custom-publish.yml Outdated Show resolved Hide resolved
eng/stages/custom-publish.yml Outdated Show resolved Hide resolved
@dagood dagood removed the * NO MERGE * The PR is not ready for merge yet (see discussion for detailed reasons) label Aug 15, 2019
@dagood
Copy link
Member Author

dagood commented Aug 15, 2019

Final validation build succeeded: https://dev.azure.com/dnceng/internal/_build/results?buildId=311523&view=results. Going ahead with the merge. I'll be keeping a close eye on the next build.

@dagood dagood merged commit 002bc9e into dotnet:master Aug 15, 2019
@dagood dagood deleted the stages-publishing branch August 15, 2019 18:18
@dagood
Copy link
Member Author

dagood commented Aug 15, 2019

Publish didn't work cleanly in the official build, the build is blocked right now. Check out the .NET Core Eng Services Partners Teams team channel First Responders for ongoing status.

@JohnTortugo
Copy link

Publish didn't work cleanly in the official build, the build is blocked right now. Check out the .NET Core Eng Services Partners Teams team channel First Responders for ongoing status.

Tracking a workaround for that here: https://github.com/dotnet/core-eng/issues/7448

picenka21 pushed a commit to picenka21/runtime that referenced this pull request Feb 18, 2022
…et/core-setup#7725)

* Switch to stage-based publishing with custom steps

* Workaround: allow known symbol publisher warnings

Use Arcade-supported symbol publish workaround

* Workaround: run gather drop dry to avoid flakiness

* Use hosted pool for custom publish


Commit migrated from dotnet/core-setup@002bc9e
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants