Skip to content

Latest commit

 

History

History
385 lines (299 loc) · 12.4 KB

PhaseToJobSchemaChange.md

File metadata and controls

385 lines (299 loc) · 12.4 KB
Raw

Phase to Job Schema Change

Overview

The Azure DevOps schema originally defined a schema which included phase and queue objects. Azure DevOps modified their schema and replaced the phase object with the job object, and the queue with a pool. This change is not a direct one-to-one change as "queue" was a member of "phase" but now "pool" is not a member of "job". Additionally, a "matrix" is no longer a part of the "queue/pool" object, it is now a member of the "job" (defined as a strategy).

Definitions

Phase schema - The Azure pipeline schema originally included phase and queue objects. That schema which refers to a "phase" object is called the "phase schema".

Job schema - Azure DevOps replaced the "phase" object with a "job" object and changed the members of that object. Authoring yaml which refer to a job object instead of a phase object is referred to as the "job schema". "Job" and "phase" are not interchangeable objects, they are mutually exclusive.

Why

- Why use Arcade's templates

  • Using Arcade's templates allows you to write "cleaner" pipeline files by conditionally adding tasks depending on the context your build is running in. You can use the same yaml file for both PR and official builds. ie, the templates will add the "install Microbuild plugin" step if you are running in an official build context.

  • Arcade's templates enable sending telemetry so that we can gather information about build reliability and we can use that information to target what areas should be improved to help our customers be impacted by infrastructure less often.

  • Arcade's templates simplify using Helix

  • Arcade's templates simplify enabling dependency flow

- Why change from phase to job schema

  • Azure DevOps no longer supports the phase schema

  • BYOC (scalable agent pools) is not supported by the phase schema

What

- What is the eng\common\templates\job\job.yml template

job.yml is the base.yml replacement which uses the "job" schema instead of the "phase" schema.

See the job template.

Example:

In the main pipeline you would reference the template like this...

jobs:
- template: /eng/common/templates/job/job.yml
  parameters:
    enableMicrobuild: true
    enablePublishBuildArtifacts: true
    name: Windows_NT_Job
    pool: dotnet-external-temp
    steps:
    - script: echo Hello World!
- template: /eng/common/templates/job/job.yml
  parameters:
    enableMicrobuild: true
    enablePublishBuildArtifacts: true
    name: Linux_Job
    pool: dnceng-linux-external-temp
    steps:
    - script: echo Hello World!

Additional example of the phase to job schema transition

- What is the eng\common\templates\jobs\jobs.yml template

The "jobs" template wraps the "job" template, providing all of the functionality of the "job" template, but additionally adding a dependent job; the publish-build-assets job. Previously, repos participating in Arcade's dependency flow model had to explicitly add this job to their yml file and define its dependencies. If you're referencing the jobs.yml template, you can just specify enablePublishBuildAssets as "true" and that job will be included in your build.

If you are participating in dependency flow (publishing assets), it is recommended that you use the "jobs" template.

See the jobs template.

Example:

In the main pipeline you would reference the template like this...

jobs:
- template: /eng/common/templates/jobs/jobs.yml
  parameters:
    enableMicrobuild: true
    enablePublishBuildArtifacts: true
    jobs:
    - job: Windows_NT_Job
      pool: dotnet-external-temp
      steps:
      - script: echo Hello World!
    - job: Linux_Job
      pool: dnceng-linux-external-temp
      steps:
      - script: echo Hello World!

Additional example of the phase to jobs schema transition

How

Phase to job schema transition

  • The "job" template makes use of variable array syntax. The array syntax is the only way to reference "variable groups" in yaml and so, that is the supported way to list variables in yaml.

    Sample array syntax:

    variables:
# Create a "foo" variable
- name: foo
  value: bar
# Reference a variable group
- group: myvargroup

    If your variables are defined using the object syntax, they will not work with job.yml/jobs.yml.

    Sample object syntax (not valid when referencing job.yml or jobs.yml):

    # will not evaluate properly if you're using the job.yml or jobs.yml templates
variables:
  foo: bar

  • There are a couple of parameters which job.yml supports that base.yml did not.

Transition example:

Original yaml referencing eng\common\templates\phases\base.yml.

azure-pipelines.yml

variables:
  Build.Repository.Clean: true
  _HelixType: build/product/
  _HelixSource: pr/dotnet/arcade-minimalci-sample/$(Build.SourceBranch)

trigger:
- master
pr:
- master

phases:
- template: /eng/common/templates/phases/base.yml
  parameters:
    name: Windows_NT
    enableTelemetry: true
    queue:
      name: dotnet-external-temp
      parallel: 99
      matrix:
        debug_configuration:
          _BuildConfig: Debug
        release_configuration:
          _BuildConfig: Release
    steps:
    - script: eng\common\cibuild.cmd
        -configuration $(_BuildConfig)
        -prepareMachine
      name: Build
      displayName: Build
      condition: succeeded()
    - task: PublishBuildArtifacts@1
      displayName: Publish Logs to VSTS
      inputs:
        PathtoPublish: '$(Build.SourcesDirectory)/artifacts/log/$(_BuildConfig)'
        PublishLocation: Container
        ArtifactName: $(Agent.Os)_$(Agent.JobName)
      continueOnError: true
      condition: always()
    variables:
      _HelixBuildConfig: $(_BuildConfig)

Updated yaml referencing eng\common\templates\job\job.yml.

azure-pipelines.yml

trigger:
- master
pr:
- master

jobs:
- template: /eng/common/templates/job/job.yml
  parameters:
    name: Windows_NT
    enableTelemetry: true
    # Allow job.yml to add the "PublishBuildArtifacts" step - https://github.com/dotnet/arcade/blob/93b39c3209a5929662190c7e85b43b4f2a32bab1/eng/common/templates/job/job.yml#L38
    enablePublishBuildArtifacts: true
    helixRepo: dotnet/arcade-minimalci-sample
    pool:
      name: dotnet-external-temp
    strategy:
      matrix:
        debug_configuration:
          _BuildConfig: Debug
        release_configuration:
          _BuildConfig: Release
    steps:
    - checkout: self
      clean: true
    - script: eng\common\cibuild.cmd
        -configuration $(_BuildConfig)
        -prepareMachine
      name: Build
      displayName: Build
      condition: succeeded()

Phase to jobs schema transition

Transition example:

Original yaml referencing eng\common\templates\phases\base.yml.

azure-pipelines.yml

variables:
  Build.Repository.Clean: true
  _HelixType: build/product/
  _HelixSource: pr/dotnet/arcade-minimalci-sample/$(Build.SourceBranch)

trigger:
- master
pr:
- master

phases:
- template: /eng/common/templates/phases/base.yml
  parameters:
    name: Windows_NT
    enableTelemetry: true
    queue:
      name: dotnet-external-temp
      parallel: 99
      matrix:
        debug_configuration:
          _BuildConfig: Debug
        release_configuration:
          _BuildConfig: Release
    steps:
    - script: eng\common\cibuild.cmd
        -configuration $(_BuildConfig)
        -prepareMachine
      name: Build
      displayName: Build
      condition: succeeded()
    - task: PublishBuildArtifacts@1
      displayName: Publish Logs to VSTS
      inputs:
        PathtoPublish: '$(Build.SourcesDirectory)/artifacts/log/$(_BuildConfig)'
        PublishLocation: Container
        ArtifactName: $(Agent.Os)_$(Agent.JobName)
      continueOnError: true
      condition: always()
    variables:
      _HelixBuildConfig: $(_BuildConfig)

Updated yaml referencing eng\common\templates\jobs\jobs.yml.

azure-pipelines.yml

trigger:
- master
pr:
- master

jobs:
- template: /eng/common/templates/jobs/jobs.yml
  parameters:
    enableTelemetry: true
    enablePublishBuildArtifacts: true
    # Add a dependent job which publishes build assets for dependency flow
    enablePublishBuildAssets: true
    helixRepo: dotnet/arcade-minimalci-sample
    # define jobs in the jobs object
    jobs:
    - job: Windows_NT
      pool:
        name: dotnet-external-temp
      strategy:
        matrix:
          debug_configuration:
            _BuildConfig: Debug
          release_configuration:
            _BuildConfig: Release
      steps:
      - checkout: self
        clean: true
      - script: eng\common\cibuild.cmd
          -configuration $(_BuildConfig)
          -prepareMachine
        name: Build
        displayName: Build
        condition: succeeded()

Notes:

  • The jobs.yml template, unlike job.yml, does not have support for handling explicit templates.

    Example:

    jobs:
- template: /eng/common/templates/jobs/jobs.yml
  parameters:
    jobs:
    - job: Windows_NT
      steps:
      - script: echo Hello world!
    # This won't work, you can't pass a template as a job to jobs.yml
    - template: sayhi.yml

    If you need to reference a template, then you should use job.yml to wrap each individual job rather than using jobs.yml to wrap all jobs.

  • See https://github.com/dotnet/arcade for an example of a repo that uses the "jobs" template.

Azure DevOps schemas

Phase schema

phase

The phase schema is no longer supported or documented. Including the schema format here for reference.

- phase: string # name
  displayName: string
  dependsOn: string | [ string ]
  condition: string
  continueOnError: true | false
  queue: string | queue
  server: true | server
  variables: { string: string } | [ variable ]
  steps: [ script | bash | powershell | checkout | task | templateReference ]

queue

name: string
demands: string | [ string ] # Supported by private pools
timeoutInMinutes: number
cancelTimeoutInMinutes: number
parallel: number
matrix: { string: { string: string } }

Job schema

job

The job schema has replaced the phase schema and is publicly documented