Support Azure Devops self-hosted agents

[onybo]

Description

I would really like to see support for Self-hosted agents in addition to the current Microsoft hosted agents.

I am aware that a similar issue was raised some time ago, but in the previous issue it was more about on-prem agents while I think it is really about supporting the more general concept of self-hosted agents, which may be on-prem or anywhere else.

It seems that the current implementation of AzurePipelines in nuke is only for Microsoft hosted agents.
As far as I know, the generated yaml only works when using Microsoft hosted agents and the AzurePipelines attribute can't be configured so that the yaml file required for a Self-hosted agent is generated.

Furthermore the generated yaml is assuming that all jobs/targets should go into all stages, and each stage is linked to a specific vmImage.
E.g. adding an AzurePipelines attribute like this will produce two almost identical stages including all targes as jobs (only difference would be the vm image):

[AzurePipelines( 
    AzurePipelinesImage.UbuntuLatest,
    AzurePipelinesImage.Ubuntu1804,
    InvokedTargets = new[] { nameof(Compile) })]

Producing a yaml wich would look like this:

Running all targets (jobs) on all images may make sense when all agents have the same set of installed tools like the Microsoft hosted agents, but when using self hosted agents this may not make as much sense.

When using self hosted agents we should not be specifying the vmImage property of the pool, but rather the name of the pool that we want to use.
When self hosted agents they may also have different capabilities and we need to control what pool is used for a specific target/job.
E.g. the test targets/jobs may require different tools than what is needed for the build job.

I suggest to introduce a new AzureStageAttribute which could accept either a stage name or the image.
As we can have any combination of Stages running on either Microsoft hosted and some on self-hosted I think we should allow for multiple AzureStageAttributes on the Build object.
The reason for this is that attempting to support both in a single AzurePipelinesAttribute as was done in the previous attempt to solve issue 1019, may become a bit "not a great implementation".
This would require a change in BuildServerConfigurationGenerationAttributeBase as well.

The attribute would be used when generating the pool property in the yaml.
We would set either vmImage or name depending on what was passed to the AzureStageAttribute.
vmImage would be used for Microhost hosted agents as today, name would be used for self-hosted agents.

That is we would could have two constructors like this:

   // For Microsoft hosted agents
    public AzureStageAttribute(
        [CanBeNull] string suffix,
        AzurePipelinesImage image
    {
       // todo: implement this
    }
    
    public AzureStageAttribute(
        [CanBeNull] string suffix,
        string stageName
    {
       // todo: implement this
    }

There are of course some details that need to be ironed out surrounding this, but before investing too much into this I would like to know what you think about the idea?

Is this something you would consider including if I make a PR for this?

This would still not fix the need to control the pool used for which target/job, but I am thinking that would be a possible future step (possibly adding a AzureJobAttribute to set the pool on a job) , if you are at all agreeing to allow for some improvements in this area?

Usage Example

[AzureStage(
    name: "stage_build_stuff",
    displayName: "Build my stuff",
    poolName: "my-self-hosted-agents",
    InvokedTargets = new[] { nameof(Compile) })]

Generated yaml:

stages:
  - stage: stage_build_stuff
    displayName: 'Build my stuff'
    dependsOn: [  ]
    pool:
      name: my-self-hosted-agents
    jobs:
      - job: Restore
      ....

Alternative

No response

Could you help with a pull-request?

Yes

[FrankM-Bosch]

Hi,
I am also looking for the possibility to make use of Azure self-hosted-agent pool with an on-premise AZD server.
Glad, that the topic seems to be moving on.
What I also noticed is, that the yaml makes use of the "Cache@2" build task:

- task: Cache@2
        displayName: 'Cache: nuke-temp'
        inputs:
          key: $(Agent.OS) | nuke-temp | **/global.json, **/*.csproj, **/Directory.Packages.props
          restoreKeys: $(Agent.OS) | nuke-temp
          path: .nuke/temp

However, this is (currently) not supported for Azure Dev Ops on-premise Server (2022)!

##[error]TF400734: This service is only available in hosted Azure DevOps.

[shlomiassaf]

Same here!

This is a big problem!
the vmImage property is not a close list of enum values.

Since it's already here, it's a legacy constraint.

Can I suggest using source generators to generate the enum so the current list will not change
however, one might be able to "extend" it by some declerative way and from here it will be available within that project.

This makes sense as those images are private.

@matkoch is this possible?

[shlomiassaf]

Honestly, it seems like the current implementation is somewhat narrow and limiting.

Binding an image to a stage is not the right approach here (IMHO)...
It belongs to the AzureDevops approach where we have build pipelines and release pipelines and seperate entities.
I see it as old and outdated, most probably just use pipelines and the release/deploy are now just additional stages.

Nuke can only use pipelines, it's not exposed to the release entity.
Its a must to be able to support the different CIs.

Anyway, with that in mind...
An image (AzurePipelinesImage) should actually resolve to a matrix in a strategy...

- stage: TestWithCrossPlatform
  dependsOn: Build
  jobs:  
  - job:
    strategy:
      matrix:
        Linux:
          imageName: 'ubuntu-latest'
        Mac:
          imageName: 'macOS-latest'
        Windows:
          imageName: 'windows-latest'
    pool:
      vmImage: $(imageName)
    steps:
    - powershell: |
        "OS = $($env:AGENT_OS)" | Out-Host
      displayName: 'Test with Agent'

And with that, we should actually be able to express multiple stages within a pipeline.

So, declare which target belongs to which stage, and for each stage we should be able to define the AzurePipelinesImage varients which will automatically, if more then 1, render the matrix...

I understand it's a bit against the general approach.
And there is an alternative here, by doing it all within one stage that will run all targets (build -> deploy)
but it's not complete.

We loose alot of the pipeline capabilities such as gating, approvals, gradual staging etc...

For example:

If i could express this in one NukeBuild class and a single [AzurePipeline] declaration but internally it will have multipple stages, I'll be able to do the above.

Rougly, something similar to this:

In AzurePipelineAttribute we replace the target definitions with a collection of target definitions...
A stage now can not auto-generate it's name, as it is a logical unit created from composition so intent is not clear.

public class StageGroupDefinition
{
    public string Name { get; set; }
    public string DisplayName { get; set; }
    public AzurePipelinesImage[]? Images { get; set; }
    
    public string[] InvokedTargets { get; set; }
    public string[] NonEntryTargets { get; set; }
    public string[] ExcludedTargets { get; set; }
}

Now AzurePipelinesStage & AzurePipelinesJob

(i've combined them to make it fast, see the comments:

public class AzurePipelinesStage : ConfigurationEntity
{
    public string Name { get; set; }
    public string DisplayName { get; set; }
    public AzurePipelinesImage[]? Images { get; set; }
    public AzurePipelinesStage[] Dependencies { get; set; }
    public AzurePipelinesJob[] Jobs { get; set; }

    public override void Write(CustomFileWriter writer)
    {
        using (writer.WriteBlock($"- stage: {Name}"))
        {
            writer.WriteLine($"displayName: {DisplayName.SingleQuote()}");
            writer.WriteLine($"dependsOn: [ {Dependencies.Select(x => x.Name).JoinCommaSpace()} ]");
            
            if (Images?.Length == 1)
            {
                using (writer.WriteBlock("pool:"))
                {
                    writer.WriteLine($"vmImage: {Images[0].GetValue().SingleQuote()}");
                }
            }

            using (writer.WriteBlock("jobs:"))
            {
                // Jobs.ForEach(x => x.Write(writer));
                // This code should be in AzurePipelinesJob.Write

                Jobs.ForEach(job =>
                {
                    using (writer.WriteBlock($"- job: {Name}"))
                    {
                        // Job definitions here...
                        
                        if (Images?.Length > 1) // job.Images
                        {
                            using (writer.WriteBlock("strategy:"))
                            {
                                using (writer.WriteBlock("matrix:"))
                                {
                                    foreach (var img in Images)
                                    {
                                        using (writer.WriteBlock($"{img}:")) // TODO: Use some name generation/validation logic
                                        {
                                            writer.WriteLine($"imageName: {img.GetValue().SingleQuote()}");
                                        }
                                    }
                                }
                            }
                        
                            using (writer.WriteBlock("pool:"))
                            {
                                writer.WriteLine("vmImage: $(imageName)");
                            }
                        }
                    
                        using (writer.WriteBlock("steps:"))
                        {
                            job.Steps.ForEach(x => x.Write(writer));
                        }
                    }
                });
            }
        }
    }
}