Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Eng Common Sync Improvements. #1081

Merged
5 commits merged into from
Oct 13, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 10 additions & 9 deletions eng/common/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,14 @@ languages repos as they will be overwritten the next time an update is taken fro

### Workflow

The 'Sync eng/common directory' PRs will be created in the language repositories once a pull request that touches the eng/common directory is submitted against the master branch. This will make it easier for changes to be tested in each individual language repo before merging the changes in the azure-sdk-tools repo. The workflow is explained below:
The 'Sync eng/common directory' PRs will be created in the language repositories when a pull request that touches the eng/common directory is submitted against the master branch. This will make it easier for changes to be tested in each individual language repo before merging the changes in the azure-sdk-tools repo. The workflow is explained below:

1. Create a PR against Azure/azure-sdk-tools:master. This is the **Tools PR**.
2. `azure-sdk-tools - sync - eng-common` is run automatically. It creates **Sync PRs** in each of the connected language repositories using the format `Sync eng/common directory with azure-sdk-tools for PR {Tools PR Number}`. Each **Sync PR** will contain a link back to the **Tools PR** that triggered it.
3. More changes pushed to the **Tools PR**, will automatically triggered new pipeline runs in the respective **Sync PRs**. The **Sync PRs** are used to make sure the changes would not break any of the connected pipelines.
4. Once satisfied with the changes;
- First make sure all checks in the **Sync PRs** are green and approved. The **Tools PR** contains links to all the **Sync PRs**. If for some reason the PRs is blocked by a CI gate get someone with permission to override and manually merge the PR.
- To test the state of all the **Sync PRs**, you can download the `PRsCreated.txt` artifact from your `azure-sdk-tools - sync - eng-common` pipeline, then run `./eng/scripts/Verify-And-Merge-PRs.ps1 <path to PRsCreated.txt>` which will output the status of each associated PR.
- Next approve the `VerifyAndMerge` job for the `azure-sdk-tools - sync - eng-common` pipeline triggered by your **Tools PR** which will automatically merge all the **Sync PRs**. You need `azure-sdk` devops contributor permissions to reach the `azure-sdk-tools - sync - eng-common` pipeline.
- Finally merge the **Tools PR**.
1. Create a PR (**Tools PR**) in the `azure-sdk-tools` repo with changes to eng/common directory.
2. `azure-sdk-tools - sync - eng-common` pipeline is triggered for the **Tools PR**
3. The `azure-sdk-tools - sync - eng-common` pipeline queues test runs for template pipelines in various languages. These help you test your changes in the **Tools PR**.
4. If there are changes in the **Tools PR** that will affect the release stage you should approve the release test pipelines by clicking the approval gate. The test (template) pipeline will automatically release the next eligible version without needing manual intervention for the versioning. Please approve your test releases as quickly as possible. A race condition may occur due to someone else queueing the pipeline and going all the way to release using your version while yours is still waiting. If this occurs manually rerun the pipeline that failed.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this does happen, what is the recovery step, delete the sync PRs and trigger again?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no need to delete the sync pipeline. You'll just need to rerun the particular template pipeline that experienced the race condition. Again you only have to do this if your changes will potentially affect release.

5. If you make additional changes to your **Tools PR** repeat steps 1 - 4 until you have completed the necessary testing of your changes. This includes full releases of the template package, if necessary.
6. Sign off on CreateSyncPRs stage of the sync pipeline using the approval gate. This stage will create the **Sync PRs** in the various language repos with the `auto-merge` label applied. A link to each of the **Sync PRs** will show up in the **Tools PR** for you to click and review.
7. Go review and approve each of your **Sync PRs**. The merging will happen automatically.
8. Sign off on VerifyAndMerge stage of the sync pipeline using the approval gate. This stage will merge any remaining open **Sync PRs** and also append `auto-merge` to the **Tools PR** so it will automatically merge once the pipeline finishes.
7. Sign Off on the VerifyAndMerge stage. This will merge any remaining open **Sync PR** and also append `auto-merge` to the **Tools PR**.
chidozieononiwu marked this conversation as resolved.
Show resolved Hide resolved
53 changes: 0 additions & 53 deletions eng/common/scripts/Add-Issue-Comment.ps1

This file was deleted.

28 changes: 28 additions & 0 deletions eng/common/scripts/Add-IssueComment.ps1
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
[CmdletBinding(SupportsShouldProcess = $true)]
param(
[Parameter(Mandatory = $true)]
[string]$RepoOwner,

[Parameter(Mandatory = $true)]
[string]$RepoName,

[Parameter(Mandatory = $true)]
[string]$IssueNumber,

[Parameter(Mandatory = $true)]
[string]$Comment,

[Parameter(Mandatory = $true)]
[string]$AuthToken
)

. "${PSScriptRoot}\common.ps1"

try {
Add-IssueComment -RepoOwner $RepoOwner -RepoName $RepoName `
-IssueNumber $IssueNumber -Comment $Comment -AuthToken $AuthToken
}
catch {
LogError "Add-IssueComment failed with exception:`n$_"
exit 1
}
28 changes: 28 additions & 0 deletions eng/common/scripts/Add-IssueLabels.ps1
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
[CmdletBinding(SupportsShouldProcess = $true)]
param(
[Parameter(Mandatory = $true)]
[string]$RepoOwner,

[Parameter(Mandatory = $true)]
[string]$RepoName,

[Parameter(Mandatory = $true)]
[string]$IssueNumber,

[Parameter(Mandatory = $true)]
[string]$Labels,

[Parameter(Mandatory = $true)]
[string]$AuthToken
)

. "${PSScriptRoot}\common.ps1"

try {
Add-IssueLabels -RepoOwner $RepoOwner -RepoName $RepoName `
-IssueNumber $IssueNumber -Labels $Labels -AuthToken $AuthToken
}
catch {
LogError "Add-IssueLabels failed with exception:`n$_"
exit 1
}
232 changes: 232 additions & 0 deletions eng/common/scripts/Invoke-GitHubAPI.ps1
Original file line number Diff line number Diff line change
@@ -0,0 +1,232 @@
$GithubAPIBaseURI = "https://api.github.com/repos"

function Get-GitHubHeaders ($token) {
$headers = @{
Authorization = "bearer $token"
}
return $headers
}

function Invoke-GitHubAPIPost {
param (
[Parameter(Mandatory = $true)]
$apiURI,
[Parameter(Mandatory = $true)]
$body,
[Parameter(Mandatory = $true)]
$token
)

$resp = Invoke-RestMethod `
-Method POST `
-Body ($body | ConvertTo-Json) `
-Uri $apiURI `
-Headers (Get-GitHubHeaders -token $token) `
-MaximumRetryCount 3

return $resp
}

function Invoke-GitHubAPIPatch {
param (
[Parameter(Mandatory = $true)]
$apiURI,
[Parameter(Mandatory = $true)]
$body,
[Parameter(Mandatory = $true)]
$token
)

$resp = Invoke-RestMethod `
-Method PATCH `
-Body ($body | ConvertTo-Json) `
-Uri $apiURI `
-Headers (Get-GitHubHeaders -token $token) `
-MaximumRetryCount 3

return $resp
}

function Invoke-GitHubAPIGet {
param (
[Parameter(Mandatory = $true)]
$apiURI,
$token
)

if ($token)
{
$resp = Invoke-RestMethod `
-Method GET `
-Uri $apiURI `
-Headers (Get-GitHubHeaders -token $token) `
-MaximumRetryCount 3
}
else {
$resp = Invoke-RestMethod `
-Method GET `
-Uri $apiURI `
-MaximumRetryCount 3
}

return $resp
}

function SplitMembers ($membersString)
{
if (!$membersString) { return $null }
return @($membersString.Split(",") | % { $_.Trim() } | ? { return $_ })
}

function List-PullRequests {
param (
[Parameter(Mandatory = $true)]
$RepoOwner,
[Parameter(Mandatory = $true)]
$RepoName,
[ValidateSet("open","closed","all")]
$State = "open",
$Head,
$Base,
[ValidateSet("created","updated","popularity","long-running")]
$Sort,
[ValidateSet("asc","desc")]
$Direction
)

$uri = "$GithubAPIBaseURI/$RepoOwner/$RepoName/pulls"
if ($State -or $Head -or $Base -or $Sort -or $Direction) { $uri += '?'}
if ($State) { $uri += "state=$State&" }
if ($Head) { $uri += "head=$Head&" }
if ($Base) { $uri += "base=$Base&" }
if ($Sort) { $uri += "sort=$Sort&" }
if ($Direction){ $uri += "direction=$Direction&" }

return Invoke-GitHubAPIGet -apiURI $uri
}

function Add-IssueComment {
param (
[Parameter(Mandatory = $true)]
$RepoOwner,
[Parameter(Mandatory = $true)]
$RepoName,
[Parameter(Mandatory = $true)]
$IssueNumber,
[Parameter(Mandatory = $true)]
$Comment,
[Parameter(Mandatory = $true)]
$AuthToken

)
$uri = "$GithubAPIBaseURI/$RepoOwner/$RepoName/issues/$IssueNumber/comments"

$parameters = @{
body = $Comment
}

return Invoke-GitHubAPIPost -apiURI $uri -body $parameters -token $AuthToken
}

# Will add labels to existing labels on the issue
function Add-IssueLabels {
param (
[Parameter(Mandatory = $true)]
$RepoOwner,
[Parameter(Mandatory = $true)]
$RepoName,
[Parameter(Mandatory = $true)]
$IssueNumber,
[ValidateNotNullOrEmpty()]
[Parameter(Mandatory = $true)]
$Labels,
[Parameter(Mandatory = $true)]
$AuthToken
)

if ($Labels.Trim().Length -eq 0)
{
throw "The 'Labels' parameter should not not be whitespace..`
You can use the 'Update-Issue' function if you plan to reset the labels"
}

$uri = "$GithubAPIBaseURI/$RepoOwner/$RepoName/issues/$IssueNumber/labels"
$labelAdditions = SplitMembers -membersString $Labels
$parameters = @{
labels = @($labelAdditions)
}

return Invoke-GitHubAPIPost -apiURI $uri -body $parameters -token $AuthToken
}

# Will add assignees to existing assignees on the issue
function Add-IssueAssignees {
param (
[Parameter(Mandatory = $true)]
$RepoOwner,
[Parameter(Mandatory = $true)]
$RepoName,
[Parameter(Mandatory = $true)]
$IssueNumber,
[ValidateNotNullOrEmpty()]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't it valid to pass in empty to clear the assignees?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not for this function. That can be done with the Update-Issue function.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not that I think it is important but it would appear as though the Update-Issue function also blocks empty.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I see what you mean. I should check for whitespace in this.
For Update-Issue you can pass white space " " to clear labels or assignees.

[Parameter(Mandatory = $true)]
$Assignees,
[Parameter(Mandatory = $true)]
$AuthToken
)

if ($Assignees.Trim().Length -eq 0)
{
throw "The 'Assignees' parameter should not be whitespace.`
You can use the 'Update-Issue' function if you plan to reset the Assignees"
}

$uri = "$GithubAPIBaseURI/$RepoOwner/$RepoName/issues/$IssueNumber/assignees"
$assigneesAdditions = SplitMembers -membersString $Assignees
$parameters = @{
assignees = @($assigneesAdditions)
}

return Invoke-GitHubAPIPost -apiURI $uri -body $parameters -token $AuthToken
}

# For labels and assignee pass comma delimited string, to replace existing labels or assignees.
# Or pass white space " " to remove all labels or assignees
function Update-Issue {
param (
[Parameter(Mandatory = $true)]
$RepoOwner,
[Parameter(Mandatory = $true)]
$RepoName,
[Parameter(Mandatory = $true)]
$IssueNumber,
[string]$Title,
[string]$Body,
[ValidateSet("open","closed")]
[string]$State,
[int]$Milestome,
[ValidateNotNullOrEmpty()]
[string]$Labels,
[ValidateNotNullOrEmpty()]
[string]$Assignees,
[Parameter(Mandatory = $true)]
$AuthToken
)

$uri = "$GithubAPIBaseURI/$RepoOwner/$RepoName/issues/$IssueNumber"
$parameters = @{}
if ($Title) { $parameters["title"] = $Title }
if ($Body) { $parameters["body"] = $Body }
if ($State) { $parameters["state"] = $State }
if ($Milestone) { $parameters["milestone"] = $Milestone }
if ($Labels) {
$labelAdditions = SplitMembers -membersString $Labels
$parameters["labels"] = @($labelAdditions)
}
if ($Assignees) {
$assigneesAdditions = SplitMembers -membersString $Assignees
$parameters["assignees"] = @($assigneesAdditions)
}

return Invoke-GitHubAPIPatch -apiURI $uri -body $parameters -token $AuthToken
}
1 change: 1 addition & 0 deletions eng/common/scripts/common.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ $EngScriptsDir = Join-Path $EngDir "scripts"
. (Join-Path $EngCommonScriptsDir ChangeLog-Operations.ps1)
. (Join-Path $EngCommonScriptsDir Package-Properties.ps1)
. (Join-Path $EngCommonScriptsDir logging.ps1)
. (Join-Path $EngCommonScriptsDir Invoke-GitHubAPI.ps1)

# Setting expected from common languages settings
$Language = "Unknown"
Expand Down
Loading