Skip to content

Commit

Permalink
ongoing
Browse files Browse the repository at this point in the history
  • Loading branch information
Konrad Jamrozik committed Dec 23, 2022
1 parent 83309a9 commit fe603e2
Showing 1 changed file with 53 additions and 38 deletions.
91 changes: 53 additions & 38 deletions eng/common/scripts/job-matrix/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
# Azure Pipelines Matrix Generator

* [Azure Pipelines Matrix Generator](#azure-pipelines-matrix-generator)
* [How does the matrix generator work](#how-does-the-matrix-generator-work)
* [How to use matrix generator from your pipeline](#how-to-use-matrix-generator-from-your-pipeline)
* [Matrix generator usage example](#matrix-generator-usage-example)
* [Matrix generator pipeline usage example](#matrix-generator-pipeline-usage-example)
* [Runtime matrix generation customization](#runtime-matrix-generation-customization)
* [Matrix config file syntax](#matrix-config-file-syntax)
* [Fields](#fields)
Expand Down Expand Up @@ -34,9 +35,18 @@ This implementation aims to address that, by replicating the cross-product matri
and [excludes](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstrategymatrixexclude)
filters, but also adds some additional features like sparse matrix generation and programmable matrix filters.

## How does the matrix generator work

This matrix generator implementation works by generating a json value for [`jobs.job.strategy.matrix`](https://learn.microsoft.com/en-us/azure/devops/pipelines/yaml-schema/jobs-job-strategy?view=azure-pipelines#strategy-matrix-maxparallel) and passing it
to the definition, which is possible because [matrix can accept a runtime expression containing a stringified json object](https://docs.microsoft.com/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml#multi-job-configuration) (see the code sample at the bottom of the linked section).

You can use the matrix generator in two ways: [from the pipeline](#how-to-use-matrix-generator-from-your-pipeline),
or by calling [`Create-JobMatrix.ps1`](https://github.com/Azure/azure-sdk-tools/blob/main/eng/common/scripts/job-matrix/Create-JobMatrix.ps1) script directly and then passing the generated matrix json as argument to `jobs.job.strategy.matrix`.

The pipeline usage is recommended.

If you call the generator from the pipeline, you will rely on the [archetype](https://github.com/Azure/azure-sdk-tools/blob/main/eng/common/pipelines/templates/jobs/archetype-sdk-tests-generate.yml), which ends up calling `Create-JobMatrix.ps1` behind the scenes.

## How to use matrix generator from your pipeline

Assume you have a job defined in azure pipelines yaml file. You want to run
Expand All @@ -48,7 +58,7 @@ based on one or more matrix json configs referenced in its `MatrixConfigs` param
That job will use as template the definition
[`archetype-sdk-tests-generate.yml`](https://github.com/Azure/azure-sdk-tools/blob/main/eng/common/pipelines/templates/jobs/archetype-sdk-tests-generate.yml).

### Matrix generator usage example
### Matrix generator pipeline usage example

Here is an example. Let's assume your job definition has following path:

Expand Down Expand Up @@ -163,13 +173,13 @@ Example:
"matrix": {
"operatingSystem": [
"windows-2022",
"ubuntu-18.04",
"ubuntu-22.04",
"macos-11"
],
"framework": [
"net461",
"netcoreapp2.1",
"net50"
"net6.0"
],
"additionalTestArguments": [
"",
Expand Down Expand Up @@ -458,9 +468,9 @@ In the matrix job output that azure pipelines consumes, the format is a map of m
"framework": "net461",
"operatingSystem": "macos-11"
},
"net50_ubuntu1804": {
"framework": "net50",
"operatingSystem": "ubuntu-18.04"
"net60_ubuntu2204": {
"framework": "net6.0",
"operatingSystem": "ubuntu-22.04"
},
"netcoreapp21_windows2022": {
"framework": "netcoreapp2.1",
Expand Down Expand Up @@ -507,16 +517,19 @@ named `ExcludedKey`, a `framework` parameter with value either `461` or `5.0`, a
-Filters @("ExcludedKey=^$", "framework=(461|5\.0)", "SupportedClouds=^$|.*Public.*")
```

Note that `Create-JobMatrix.ps1` is [called internally by the archetype](https://github.com/Azure/azure-sdk-tools/blob/main/eng/common/pipelines/templates/jobs/archetype-sdk-tests-generate.yml#L72) and you are never expected to call it directly.
Instead, use use the generator from the pipelines, as explained in [Matrix generator pipeline usage example](#matrix-generator-pipeline-usage-example).

### Replace/Modify/Append Values

Replacements for values can be passed to the matrix as an array of strings, in `MatrixReplace` parameter, each matching the format of `<keyRegex>=<valueRegex>/<replacementValue>`.
The replace argument will find any combinations where the key fully matches the key regex and the value fully matches the value regex, and replace the value with
the replacement specified.
The replace argument will find any combinations where the key fully matches the key regex and the value fully matches
the value regex, and replace the value with the replacement specified.

NOTE:

* The replacement value supports regex capture groups, enabling substring transformations, e.g. `Foo=(.*)-replaceMe/$1-replaced`. See the below examples for usage.
* For each key/value, the first replacement provided that matches will be the only one applied.
* For each key/value pair, the first replacement provided that matches will be the only one applied.
* If `=` or `/` characters need to be part of the regex or replacement, escape them with `\`.

For example, given a matrix config like below:
Expand All @@ -525,7 +538,7 @@ For example, given a matrix config like below:
{
"matrix": {
"Agent": {
"ubuntu-1804": { "OSVmImage": "MMSUbuntu18.04", "Pool": "azsdk-pool-mms-ubuntu-1804-general" }
"ubuntu-2204": { "OSVmImage": "MMSUbuntu22.04", "Pool": "azsdk-pool-mms-ubuntu-2204-general" }
},
"JavaTestVersion": [ "1.8", "1.11" ]
}
Expand All @@ -535,48 +548,49 @@ For example, given a matrix config like below:

The normal matrix output (without replacements), looks like:

``` powershell
``` json
$ ./Create-JobMatrix.ps1 -ConfigPath <test> -Selection all
{
"ubuntu1804_18": {
"OSVmImage": "MMSUbuntu18.04",
"Pool": "azsdk-pool-mms-ubuntu-1804-general",
"ubuntu2204_18": {
"OSVmImage": "MMSUbuntu22.04",
"Pool": "azsdk-pool-mms-ubuntu-2204-general",
"JavaTestVersion": "1.8"
},
"ubuntu1804_111": {
"ubuntu2204_111": {
"OSVmImage": "MMSUbuntu18.04",
"Pool": "azsdk-pool-mms-ubuntu-1804-general",
"Pool": "azsdk-pool-mms-ubuntu-2204-general",
"JavaTestVersion": "1.11"
}
}
```

Passing in multiple replacements, the output will look like below. Note that replacing key/values that appear nested within a grouping
will not affect that segment of the job name, since the job takes the grouping name (in this case "ubuntu1804").
Passing in multiple replacements, the output will look like below. Note that replacing key/value pairs that appear
nested within a grouping will not affect that segment of the job name, since the job takes the grouping name (in this case `ubuntu2204`).

The below example includes samples of regex grouping references, and wildcard key/value regexes:

``` powershell
$ $replacements = @('.*Version=1.11/2.0', 'Pool=(.*ubuntu.*)-general/$1-custom')
$ ../Create-JobMatrix.ps1 -ConfigPath ./test.Json -Selection all -Replace $replacements
{
"ubuntu1804_18": {
"OSVmImage": "MMSUbuntu18.04",
"Pool": "azsdk-pool-mms-ubuntu-1804-custom",
"ubuntu2204_18": {
"OSVmImage": "MMSUbuntu22.04",
"Pool": "azsdk-pool-mms-ubuntu-2204-custom",
"JavaTestVersion": "1.8"
},
"ubuntu1804_20": {
"OSVmImage": "MMSUbuntu18.04",
"Pool": "azsdk-pool-mms-ubuntu-1804-custom",
"OSVmImage": "MMSUbuntu22.04",
"Pool": "azsdk-pool-mms-ubuntu-2204-custom",
"JavaTestVersion": "2.0"
}
}
```

#### NonSparseParameters
### NonSparseParameters

Sometimes it may be necessary to generate a sparse matrix, but keep the full combination of a few parameters. The
NonSparseParameters argument allows for more fine-grained control of matrix generation. For example:
Sometimes it may be necessary to generate a sparse matrix, but keep the full combination of a few parameters.
The `MatrixConfigs` `NonSparseParameters` parameter allows for more fine-grained control of matrix generation.
For example:

``` powershell
./Create-JobMatrix.ps1 `
Expand All @@ -592,7 +606,7 @@ Given a matrix like below with `JavaTestVersion` marked as a non-sparse paramete
"matrix": {
"Agent": {
"windows-2022": { "OSVmImage": "MMS2022", "Pool": "azsdk-pool-mms-win-2022-general" },
"ubuntu-1804": { "OSVmImage": "MMSUbuntu18.04", "Pool": "azsdk-pool-mms-ubuntu-1804-general" },
"ubuntu-2204": { "OSVmImage": "MMSUbuntu22.04", "Pool": "azsdk-pool-mms-ubuntu-2204-general" },
"macos-11": { "OSVmImage": "macos-11", "Pool": "Azure Pipelines" }
},
"JavaTestVersion": [ "1.8", "1.11" ],
Expand All @@ -602,27 +616,28 @@ Given a matrix like below with `JavaTestVersion` marked as a non-sparse paramete
}
```

A matrix with 6 combinations will be generated: A sparse matrix of Agent, AZURE_TEST_HTTP_CLIENTS and ArmTemplateParameters
(3 total combinations) will be multiplied by the two `JavaTestVersion` parameters `1.8` and `1.11`.
A matrix with 6 combinations will be generated: A sparse matrix of `Agent`, `AZURE_TEST_HTTP_CLIENTS` and `ArmTemplateParameters`
(3 total combinations) will be multiplied by the two `JavaTestVersion` parameter values of `1.8` and `1.11`.

NOTE: NonSparseParameters are also applied when generating an imported matrix.
NOTE: `NonSparseParameters` are also applied when generating an imported matrix.

#### Under the hood
## Under the hood

The script generates an N-dimensional matrix with dimensions equal to the parameter array lengths. For example,
the below config would generate a 2x2x1x1x1 matrix (five-dimensional):
The [`Create-JobMatrix.ps1`](https://github.com/Azure/azure-sdk-tools/blob/main/eng/common/scripts/job-matrix/Create-JobMatrix.ps1) script generates an N-dimensional matrix with dimensions equal to the parameter array lengths.
For example, the below config would generate a 2x2x1x1x1 matrix (five-dimensional):

``` json
"matrix": {
"framework": [ "net461", "netcoreapp2.1" ],
"framework": [ "net461", "net6.0" ],
"additionalTestArguments": [ "", "/p:SuperTest=true" ]
"pool": [ "ubuntu-18.04" ],
"container": [ "ubuntu-18.04" ],
"pool": [ "ubuntu-22.04" ],
"container": [ "ubuntu-22.04" ],
"testMode": [ "Record" ]
}
```

The matrix is stored as a one-dimensional array, with a row-major indexing scheme (e.g. `(2, 1, 0, 1, 0)`).
The matrix is stored as a one-dimensional array, with a [row-major](https://en.wikipedia.org/wiki/Row-_and_column-major_order)
indexing scheme (e.g. `(2, 1, 0, 1, 0)`).

## Testing

Expand Down

0 comments on commit fe603e2

Please sign in to comment.