Skip to content

Commit

Permalink
Updates to patch release for the release portion (#30279)
Browse files Browse the repository at this point in the history
* Updates to patch release for the release portion

* move save package properties to build

* Update patch-release for multiple service directory testing of Build/Analyze/Verify. Release will not be performed.

* couple of minor fixes

* Update timeout for DocsMS publish and remove TestPipeline variable from patch-release.yml

* Add README file for patch-release

* Update Patch-Release-Readme

* Update Patch-Release-README
  • Loading branch information
JimSuplizio authored Aug 5, 2022
1 parent caddd5a commit f0d1693
Show file tree
Hide file tree
Showing 5 changed files with 74 additions and 18 deletions.
49 changes: 49 additions & 0 deletions eng/pipelines/Patch-Release-README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# Path Release Pipeline

The patch release pipeline is a pipeline designed to release a patch of updated azure-sdk-for-java Artifacts in preparation for release of the Azure Java SDK BOM (Bill of Materials).

## Prerequisites

The patch release pipeline differs from service directory pipelines in that the Artifacts list should change, every time, prior to this pipeline being run. The list of what Artifacts and external dependencies that needs updated, built, and released, to enable the BOM release, is computed by other Java scripts. The last part of this computation feeds in the computed artifacts list to the [Update-Artifacts-List-For-Patch-Release.ps1][update_for_release_script] script which, in turn, updates the [patch-release.yml][patch_release_yml] file's Artifacts and AdditionalModules lists. The updated patch-release.yml should be checked in along with the version_client.txt, pom, CHANGELOG and README updates when prepping for the patch release.

### [Update-Artifacts-List-For-Patch-Release.ps1][update_for_release_script]

The script uses the powershell-yml module to load up all the ci.yml files in the repository to create a dictionary. It then uses the input project list to output the Artifacts and AdditionalModules into the YmlToUpdate with the correct metadata. For example; If a library had skipPublishDocMs: true it would be preserved here. There's one additional thing that's being done by the script which makes the Artifacts list it generates different from other ci.yml files and that's the addition of the ServiceDirectory on each Artifact. The reason for this is because there's common tooling that needs the ServiceDirectory as part of it's input. Things like README and Changelog verification are good examples.

**arguments:**
**SourcesDirectory** The root of the repository.
**YmlToUpdate** The yml file to update. *This should be the [patch-release.yml][patch_release_yml] file for the patch release pipeline*.
**ProjectList** Comma separated list of artifacts.

**Example**:
./eng/scripts/Update-Artifacts-List-For-Patch-Release.ps1 -SourcesDirectory \$\(Build.SourcesDirectory\) -YmlToUpdate $(Build.SourcesDirectory)/eng/pipelines/patch-release.yml -ProjectList com.azure:azure-sdk-template,com.azure:azure-sdk-template-two,com.azure:azure-sdk-template-three

### [patch-release.yml][patch_release_yml]

This is the yml file that belongs to the [java - patch-release][java_patch_release] pipeline. It's updated by running [Update-Artifacts-List-For-Patch-Release.ps1][update_for_release_script] and it's expected that this will be checked into the branch where the patch release is being prepped prior to running.

### [java - patch-release][java_patch_release] pipeline

The [java - patch-release][java_patch_release] pipeline is manual only. There is no purpose to running this pipeline other than than to release a patch build. The pipeline will build, analyze, verify and release the set of Artifacts that are in the patch-release.yml.

### How build works in the patch build pipeline

Normal service directory pipelines are only concerned with the set of Artifacts that live within that service directory. Any interdependencies within that service directory use current versions of each other and any dependencies outside of that service directory use the dependency versions. When a artifacts from a service directory are released, the released version becomes the new dependency version and everything that depends on it gets the updated version. This is why monthly releases have a release order. The From Source runs, which are part of every test matrix, is how we know this process works.

The patch-release pipeline utilizes the same tools that the From Source runs use. The repository is prepped in such a way that everything is built from source, regardless of the service directory. This is what allows us to release a set of Artifacts, regardless of their original service directories, to be released as part of the same release job instead of having to order the releases.

## What the patch release pipeline does and doesn't do

**Release verifications are done in the AnalyzeAndVerify Job.** In the patch-release pipeline, verifications normally done as part of the release task in service directories pipelines are now being done as part of the AnalyzeAndVerify job. This can be done since there's no ambiguity as to whether or not the artifacts are going to be released.

**Testing is not done in the patch-release pipeline.** Unlike service directory pipelines, there are no tests being run. We have a highly customizable test matrix which many service directories have made changes to. Because of this, it's nearly impossible to rectify a test matrix for a set of artifacts that spans the entire repository. There is a mitigation for this; as part of the preparation for patch release versions, poms, CHANGELOG and README files all need to be updated. These updates will cause each individual service directory pipeline to get run as part of the process for the update PR. In between the update PR and kicking off the patch-release pipeline, there should be no outside changes that aren't related to the aforementioned update. The matrix of FromSource and non-FromSource runs should provide enough verification. We already make the same assumptions when we release service directory, update the dependency version and then release a service directory that depends on what was previously released.

## Testing changes/updates to the patch-release pipeline

The easiest way to test the patch release pipeline is to use the example above which sets the Artifacts list to the
list of template libraries. After running, add the following parameter **TestPipeline**: **true**. This will allow the release of the template libraries without any version preparation, just run and release. The parameter will cause the test pipeline version code, which sets template libraries to a new beta version, based upon the buildId, for release and then auto closes the version increment PR.

<!-- LINKS -->
[java_patch_release]: https://dev.azure.com/azure-sdk/internal/_build?definitionId=5015&_a=summary
[update_for_release_script]: https://github.com/Azure/azure-sdk-for-java/blob/main/eng/scripts/Update-Artifacts-List-For-Patch-Release.ps1
[patch_release_yml]: https://github.com/Azure/azure-sdk-for-java/blob/main/eng/pipelines/patch-release.yml
14 changes: 8 additions & 6 deletions eng/pipelines/templates/stages/archetype-java-release-patch.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@ parameters:
- name: Artifacts
type: object
default: []
# JRS this is here for testing purposes
- name: TestPipeline
type: boolean
default: false
Expand Down Expand Up @@ -34,7 +33,7 @@ stages:
jobs:
- deployment: SignPackage
environment: esrp
timeoutInMinutes: 20
timeoutInMinutes: 240
variables:
- template: /eng/pipelines/templates/variables/globals.yml
pool:
Expand Down Expand Up @@ -99,9 +98,10 @@ stages:
- deployment: PublishPackage
displayName: "Publish to Maven Central"
condition: and(succeeded(), ne(variables['Skip.PublishPackage'], 'true'))
# This timeout shouldn't be necessary once we're able to parallelize better. Right now,
# this is here to ensure larger areas (30+) libraries don't time out.
timeoutInMinutes: 120
# There are over 130 com.azure libraries. At roughly 3m 20s per ESRP release
# this timeout should be enough to allow release of everything. Things should
# change once ESRP has batch release enabled.
timeoutInMinutes: 540
environment: ${{ parameters.Environment }}
dependsOn: TagRepository
variables:
Expand Down Expand Up @@ -223,11 +223,12 @@ stages:
CommitMsg: "Increment package versions for patch releases"
PRTitle: "Increment versions for patch releases"
PRLabels: "auto-merge"
CloseAfterOpenForTesting: 'false'
CloseAfterOpenForTesting: '${{ parameters.TestPipeline }}'

- deployment: PublishDocsMs
displayName: Docs.MS Release
condition: and(succeeded(), ne(variables['Skip.PublishDocs'], 'true'))
timeoutInMinutes: 180
environment: ${{ parameters.Environment }}
dependsOn: PublishPackage

Expand Down Expand Up @@ -285,6 +286,7 @@ stages:
- deployment: PublishDocs
displayName: Publish Docs to GitHubIO Blob Storage
condition: and(succeeded(), ne(variables['Skip.PublishDocs'], 'true'))
timeoutInMinutes: 120
environment: ${{ parameters.Environment }}
dependsOn: PublishPackage
variables:
Expand Down
21 changes: 12 additions & 9 deletions eng/pipelines/templates/stages/archetype-sdk-client-patch.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,10 +33,9 @@ parameters:
- name: JavaBuildVersion
type: string
default: $(JavaBuildVersion)
# JRS this is here for testing purposes only since I'm using template to test
- name: TestPipeline
type: boolean
default: true
default: false


stages:
Expand Down Expand Up @@ -109,6 +108,17 @@ stages:
pip install markdown2 BeautifulSoup4
displayName: 'pip install markdown2 and BeautifulSoup4'
# Save the Package Properties
# ServiceDirectories variable is a plain, comma separated list of ServiceDirectories that is
# output by the generate_from_source_pom script above.
- pwsh: |
$ServiceDirs = '$(ServiceDirectories)'
foreach($ServiceDir in $ServiceDirs.Split(',')) {
$(Build.SourcesDirectory)/eng/common/scripts/Save-Package-Properties.ps1 -ServiceDirectory $ServiceDir -OutDirectory $(Build.ArtifactStagingDirectory)/PackageInfo
}
workingDirectory: $(Pipeline.Workspace)
displayName: "Save Package Properties"
# Why is this necessary? Normal build and release jobs don't install reporting tools prior
# to building however they're still able to find the the build tools through the root aggregate
# pom file. The ClientFromSourcePom only contains what is necessary for the list of libraries
Expand Down Expand Up @@ -262,7 +272,6 @@ stages:
- pwsh: |
$ArtifactsList = $('${{ convertToJson(parameters.Artifacts) }}' | ConvertFrom-Json | Select-Object name, groupId, ServiceDirectory, uberJar | Where-Object -Not "uberJar")
$ArtifactsList | Format-Table -Property GroupId, Name, ServiceDirectory | Out-String | Write-Host
python --version
foreach($artifact in $ArtifactsList) {
$(Build.SourcesDirectory)/eng/versioning/scan_for_unreleased_dependencies.ps1 $($artifact.groupId) $($artifact.name) $(Build.SourcesDirectory)/sdk/$($artifact.ServiceDirectory)
}
Expand Down Expand Up @@ -299,12 +308,6 @@ stages:
# changelog since the only time this pipeline should run is for a patch release.
ForRelease: true

# We `install` the code quality reports tooling into our local m2 cache separately from building the Maven project
# reports. This means it is available as part of that, but also so that this is not documented in the project report.
- template: ../steps/install-reporting-tools.yml
parameters:
JdkVersion: ${{ parameters.JavaBuildVersion }}

- template: /eng/common/pipelines/templates/steps/eng-common-workflow-enforcer.yml

# This pipeline should never get kicked off outside of an internal, manual run. Put the checks here
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ foreach ($project in $ProjectList) {
}
}

# This is lame. Because resourcemanager and resourcemanagerhybrid contain the some of the same
# This is sad. Because resourcemanager and resourcemanagerhybrid contain the some of the same
# artifacts and we don't know which one is actually needed, if resourcemanager is in the list
# then add resourcemanagerhybrid.
Write-Host "sparseCheckoutDirectories=$sparseCheckoutDirectories"
Expand Down
6 changes: 4 additions & 2 deletions eng/scripts/generate_from_source_pom.py
Original file line number Diff line number Diff line change
Expand Up @@ -120,8 +120,10 @@ def create_from_source_pom(project_list: str, set_skip_linting_projects: str, ma

# output the ServiceDirectories environment variable
service_dirs = list(sorted(service_directories))
# JRS need to create service_directory_full_path which will add root_path
# and service_directory_sdk_relative_path which doesn't remove the SDK string
# Create service_directory_full_path which will add root_path and service_directory_sdk_relative_path
# which doesn't remove the SDK string. These are necessary because inline powershell within the yml
# won't work correctly if the results, in the yml, are in single quotes that necessary in case the
# path contains spaces.
service_dirs_sdk_rel_path = ""
service_dirs_full_path = ""
service_dirs_plain = ""
Expand Down

0 comments on commit f0d1693

Please sign in to comment.