EPIC: Decouple Azure type information from a particular version of Bicep

[anthony-c-martin]

Currently Bicep types are embedded with a Bicep release - meaning there is often a lag between types being available, and usable in Bicep, and upgrading requires the user to upgrade their VSCode extension or Bicep CLI.

Now that Bicep has the ability to talk to OCI for the Bicep module registry, it would make sense to re-use this infrastructure to be able to deliver Bicep types on a shorter cadence, and avoid having to tie the types to a version of Bicep - meaning that upgrading/downgrading the Bicep version will not change the types that are used for validation.

We may want to use the import statement to lock Bicep to a particular set of types - e.g.:

import 'az@0.1'

Triage

Preview Give feedback

1. Publish provider artifacts (using minor version to start off) #11752
   +0
2. Implement versioning detection strategy for az
3. Determine course of action regarding Fix dynamic loading feature to load providers from custom bicepconfig.json aliases #12267 (comment)

Todo

Preview Give feedback

1. The Bicep Language Server should provide completions for import 'az@ by inspection of the OCI registry #11610 Completions for available versions

In Progress

Preview Give feedback

1. Propagate az ArmTemplateProviderVersion to backend, support reading / logging of version in backend
2. Design and implement a solution to cache the AzResourceProvider for dynamic type loading #11307 Design caching strategy for providers

Deprioritized

Preview Give feedback

1. {@majastrz} - We can have service connection to an OCI registry in an official OneBranch pipeline to publish directly from ADO
2. Optimize decompressing of provider package - to file system or in memory
3. Move function definitions to the provider package

Done

Preview Give feedback

1. Implement provider declaration aliasing #12127
   story: dynamic type loading +1
   
2. Discussion Proposal: Versioning of provider artifacts and provider resources #11894
3. Get approval for ADR: Azure provider registry aliasing #12110
4. Prototype: Publish az types to Bicep Registry for consumption by dynamic type loading Bicep feature bicep-types-az#1553
   +0
   
5. Proposal: Azure provider registry aliasing #11598 Design for aliasing of registry module paths
6. Refactoring of registry classes to decouple module & provider
7. We want to create sufficient tests to assert that BEFORE we publish a new AZ package, this package will load properly in Bicep. And we have a clear strategy for compatibility testing (which bicep versions work with which packages)
8. Proposal: Distribute Bicep providers using OCI registry #10662 Initial design
9. Prototype: Support for dynamic loading of provider definitions #10624 Support import syntax with loading versioned provider types from disk
10. Provider restore for resource type providers #11458 Support restore for resource type providers
11. ADR: Process for publishing the AZ provider types (alpha) #11757
    story: dynamic type loading +1
    
12. Proposal: Versioning of provider artifacts and provider resources  #11593 Versioning of provider packages
13. Document strategy for publishing to MCR (including naming / path conventions)
14. Document the OCI manifest format for providers

Open questions/tasks

• Implement publishing pipeline for providers
• Removal of feature flag
• Close on serialization format for types (types.json)
• Build in ability to version the contract (to support back/format compatibility wile being able to make changes)
  • Old versions of Bicep need to be able to understand they don't support the new format (and show an error if the user is using an incompatible version of Bicep)
  • New versions of Bicep need to be able to support N numbers of older versioned format
• Generalize to support other providers
• Ensure we have a plan for airgapped clouds (and have verified it)

[slavizh]

I am not sure if this is the issue I should comment upon. In latest community call there was a demo for Az provider and the presenter said that they will paste link to the GitHub issue but I do not have that link as I was watching the recording. There was some debate around having define always the latest provider version something like:

provider 'br:biceptestdf.azurecr.io/bicep/proivders/az@latest'

I am not sure why it was said that every time you run your deployment the result will change based on what is pulled as latest. After all we define the API version for every RP and that version does not change. Even if there is a change in specific version, let's say version 2023-01-01 for storage account was updated and new property was added. It should not matter what version of the az provider I use as the actual update for 2023-01-01 for storage account happens in ARM (in the cloud on that resource provider). The only breaking change I can see happening is if you are using let's say 2020-01-01 for storage account and in the latest provider that version is no longer available. But even if t is no longer available in the published az provider that storage account version can still be available in Azure (ARM) as usually deprecation has some notice in advance. If that storage account version is deprecated fully (not available in ARM) than your template will not work at all no matter which az provider version you use.

[slavizh]

Also I would prefer one time import via the main.bicep file and option to define the az provider version via bicepconfig and az CLI command.

[brwilkinson]

If the available type versions were discoverable in the config (intellisense), similar to resource type versions. Then it would be easy to manually update the settings. Without just using a 'latest' tag.

A linter rule to show that you are pinned to an old types version would also be useful.

[slavizh]

No, discoverability instead of latest does not work for us. We have over 160 separate templates so updating each one of them is quite job. Currently we just update the bicep CLI and everything works so if this can be defined outside of bicep CLI we need latest.

[anthony-c-martin]

No, discoverability instead of latest does not work for us. We have over 160 separate templates so updating each one of them is quite job. Currently we just update the bicep CLI and everything works so if this can be defined outside of bicep CLI we need latest.

@slavizh would #12202 support what you're looking for? The goal is for the default behavior to match the current behavior (each version of Bicep is released with a particular version of the Azure types, so upgrading your CLI will pick up the 'latest' types), while providing the flexibility to override it if desired.

[brwilkinson]

I guess I don't understand.

The types version is for a collection of types + versions.

We are statically setting the "type collection" version. Which includes all current types and all current type versions.

If a new version of a resource provider definition is released in Azure, then I don't have access to this.

However if a new "type collection" becomes available... I can bump this in bicep config, without updating the Bicep version.

Now I can write a new bicep file with the new resource api version. All other type versions in all other bicep templates are still available. No code changes required; I just now have access to the new api version from the new "type collection"?

Am I missing something?

[slavizh]

@anthony-c-martin I still think latest or at least something like 1.x.x, 2.x.x, etc. is needed when you have some constrain to currently update bicep cli but you need the latest types for resource providers.