Move Azure provider from Service Management Provider API to ARM

[phinze]

Sounds like the Service Management Provider API is on its way out, and the Azure Resource Manager APIs are the replacement.

Opening this to acknowledge our the need to switch the provider to the supported APIs.

Not sure how much work its going to be - pinging @aznashwan - any ideas on that front?

[aznashwan]

@phinze: I've actually been looking over the new APIs and started planning Terraform's transition to them already.

Unfortunately, the switch will require an almost complete re-write of what we already have (tests included), and I can't provide a timeline on it too confidently just now. 😞

Will follow-up with a game plan early next week. 😄

[spuder]

If I might purpose a radical idea. Azure is unique in that it has the ability to do some of the same things as terraform through its declarative templates

Instead of calling the new API Directly, Terraform could simply convert the .tf files to an azure compatible template and resource group (which are just json).

https://azure.microsoft.com/en-us/documentation/articles/resource-group-overview/
https://azure.microsoft.com/en-us/documentation/articles/resource-group-define-dependencies/

Update
Most of the azure quickstart templates are stored on github gists
Here is an example of what one looks like:
https://azure.microsoft.com/en-us/documentation/templates/elasticsearch/

and the raw json

https://raw.githubusercontent.com/azure/azure-quickstart-templates/master/elasticsearch/azuredeploy.json

All templates refer to a schema provided by microsoft

https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json

[stack72]

@aznashwan have you managed to start the work on this cut over to Azure RM?

[chevalpartners]

I am a hashicorp partner and lack of ARM support is a big problem for me.
Most of my customers use Azure.
Service management(SMA) is on its way out. To make matters worse there is no compatibility between the resources created with Service management api and ARM api.
Knowing that SMA is on its way out makes it difficult for us to use Terraform.

I know that you will have to do a complete rewrite but it will be worth it.

Most of the Azure shops are not doing infrastructure as code yet. They typically don't use CHEF, Puppet, Ansible etc.
Making it easier for them to provision resources with the help of terraform will increase adoption of your products.

[panazzo]

Hi,

I'm here to join my friends to say that I'm really looking forward to the ARM support for Terraform.

I'm building a new project on top of Terraform and there's no way I'll use ASM for Azure.

Thank you =)

[clausasbjorn]

Chiming in to show my interest in this. I would like to help out with the transition, @aznashwan let me know if you are interested.

[chevalpartners]

I can help as well. If I have to learn go so be it.

[luisrudge]

I like @spuder's idea of just converting stuff to ARM templates.

[jmspring]

Chiming in, I work at Microsoft and was perusing the Azure Provider this week. I have a potential interest in helping as well @aznashwan.

I haven't scoped the idea of template translation, but have started diving a bit into the Azure for Go SDK.

Has work started on this?

@chevalpartners, I'd be interested in hearing a bit about your customer needs as well.

[aznashwan]

First and foremost; sorry for my random disappearing this last couple of weeks.
I've been meaning to have this open discussion over e-mail/IRC for a while now, but I guess here would be a much more public and/or persistent place for it, so here goes:

I'm pleased to say work on the ARM provider has started (I already have a basic outline of the credentials bit of the provider set up and running so as to get a feel for the new sdk), but I'd like to spend the rest of this week submitting a patch to fix the last couple of bugs still out on the ASM-based provider.

Luckily, this gives us some time to debate how we might go about with the actual 'architecture' of the thing.

On simply converting to ARM template format:

@spuder, @luisrudge: the idea of having Terraform simply create an ARM template in the backend is a very good one; unfortunately, as I've been spending my last couple of months or so working on a Heat/CFN to ARM template converter, I have the following objections towards such an approach:

• the whole idea of Terraform is to provide a single tool for managing all your infrastructure providers; converting to ARM template format in the backed would thus make it (from Azure's point of view) nothing more than a glorified version of the already existing PowerShell commandlets which let you deploy ARM templates.
• the ARM templating language is very rigid; being completely Azure-only and thus negating all of Terraform's cross-provider goodness (i.e. there'd be no way to link ARM template resources to the resources of the other providers (outside-in it'll work anyhow; of course....)).
• Terraform's way of deploying resources requires that each one's deployment be handled individually, making the translation process much much more tedious (i.e. we'll have to use a template for each resource).
• ARM template translation would only really be useful for the Creation of the deployment, so we'd still end up making API calls for Reading, Updating and Deleting the resources (which cannot be done through the templates).

All in all; I'd argue that this approach is the much more complex and prone to over-engineering one, and I'd much prefer we stick to the ARM API bindings themselves (which the azure-sdk-for-go folks did an amazing job with).

On the 'architectural' side of things:

Considering that the old ASM and new ARM API's do things differently with different and incompatible results; it would not make sense to nuke and pave over the old provider as it'd be a complete loss on everyone's side. ARM's novelty, however, guarantees that the ARM implementation will have to be written pretty much completely from scratch (even down the provider auth details).

I've spent quite some time thinking of the most elegant way we could add ARM support (other than the obvious write a fresh new provider way). Here are my ideas together with their associated pluses and minuses:

Separate provider:

• + exactly as straightforward as it sounds.
• - naming/branding issues (we must call one of them something other than 'provider/azure', or have both be 'provider/asm' and 'provider/arm', respectively [which might get a lot of people really confused]).
• - renaming the existing ASM-based provider will be tedious as hell...

Add new resources on existing provider:

Simply add the auth details to the existing provider and define the other resource on top of it (i.e. have an 'azure_asm_instance' AND an 'azure_arm_instance'...
Pros/Cons:

• - auth details for both required up front; even when using one of the two
• - major confusion with offered resource types
• - old resources will have to be renamed => no backwards compatibility...
• - simply the worst idea in this list...

Single provider; switch individual resource implementation at runtime:

We could add the required ARM authentication parameters to the existing provider and have each resource have a ''use_asm_api" flag which is set to false by default.
We'd still have a single provider whose resources' CRUD functions would switch to the corresponding ASM or ARM resource-CRUD-ing function. Code speaks better:

func resourceAzureInstanceCreate(d *schema.ResourceData, meta interface{}) error {
        if d.Get("use_asm_api").(bool) {
                return resourceAzureASMInstanceCreate(d, meta)
        }

       return resourceAzureARMInstanceCreate(d, meta)
}

Pros/Cons:

• + it's still a single provider, a simple flag being (in my opinion at least), much easier to grasp than two separate providers.
• + considering people will want to use ARM 99% of the time anyhow, usability seems good.
• + lines of code-wise, ~same as the 'just write a new provider' base case
• + complexity ~same as base case
• - old .tf files will have to be updated, but just a single line/resource, which is not that bad...
• - we might have to check per resource if the authentication details were provided for the API that resource will be defined with (an if per resource function; not too bad...)

Single provider; choose underlying resource offering:

I.e. we would still have a single provider with the auth fields for both ASM and ARM API's and set that "use_asm_api" flag in the provider itself; depending on that, we'd then switch the offered resource map. Envisioning something along the lines of:

func Provider() *terraform.ResourceProvider {
    return &schema.Provider {
        // ...
        ResourceMap: hypotheticalChooseAzureResourceMapFunc(),
    }
}

Pros/cons:

• + in the case people would like to define both ASM and ARM resources, they'd simply go ahead and instantiate the provider twice.
• + code mass and complexity ~same as separate provider
• ---- no current way to access provider data within the Provider func. Have not researched Terraform internals and cannot estimate how major the required changes to terraform-core are...

TL;DR:

ARM support is on it's way ASAP.

Considering Terraform is by far the most community-focused project I've had the pleasure to work on; I'd love to hear any and all opinions you guys might have regarding how you'd like the provider(s) layed out.

CC: @svanharmelen @phinze @catsby @mitchellh @everyone

[luisrudge]

I like the idea of creating a new provider. Something like provider/azure and provider/azure-arm. The azure-cli already does this kind of separation (you have to set arm/asm mode).

[phinze]

Welcome back @aznashwan! And thank you for the thoughtful post. 😄

I could not have stated better why the ARM Templates-based implementation is probably not the right route for Terraform. Well put, @aznashwan.

As for how to model the ARM implementation, I believe that Single provider; switch individual resource implementation at runtime provides the best user experience if the implementation is feasible. One question there: would the ARM/ASM resources have the same attributes from Terraform's perspective? If not, it seems like it could be a validation / documentation nightmare for both implementors and users to figure out which attributes behave differently depending on the API.

In OpenStack we went with Add new resources to existing provider for the _v1 and _v2 split. I don't think it's necessarily the worst option - it is probably the easiest from an implementation perspective, and it does guarantee that no existing configurations are broken. We could probably figure out how to make the authentication details conditionally required.

In summary, if we can expect a certain amount of consistency between schemas, I think it's worth pursuing further the improved UX of an API-switch flag. If there are significant semantic differences between the APIs, I think we probably have to implement separately either at the resource or the provider level.

Further opinions are welcome!

[aznashwan]

@phinze: the probable differences between resource parameters is something I did not take into too much consideration; good catch.

They should be very marginal (the couple I can think of off the top of my head would be easily solved with a little pre-processing on one side or another).
What might be the real issue is regarding resources which are not definable with the old API at all; but we could just error people trying to declare those on ASM.

Considering that the Single provider; switch individual resource implementation at runtime approach allows for great decoupling between old and new resource implementations; I say I get to writing the extra bit required in the provider itself and sending it in early next week after which we can start getting individual resources up and running. Should we reach a point where we decide the hassle is too big; it shouldn't take more than a day to split the codebase into a separate provider and just go with that from there...

@clausasbjorn @chevalpartners @jmspring: I forgot to thank you for your help offer last post; would very much love to have you guys on board!

Sound like a fair plan?

[clausasbjorn]

Thanks @aznashwan for a thorough write-up!

At first glance I find the "single provider with a flag" approach most appealing, I do however wonder if it will be a cause of confusion later on as the provider evolves.

It is my understanding that part of the benefit of using ARM over ASM is that ARM provides access to a broader set of resources, and will continue to grow. Are we likely to end up with a provider where resources appear to be available using either API, but are in fact only supported by ARM?

[clausasbjorn]

I didn't see your response before I posted. You addressed my question, explanatory error messages and documentation is probably sufficiently clear. Your plan of action sounds fine :)

I've got my environment up and running, and have started reading up on the code!

[jmspring]

ARM is the proper approach and where new APIs will be showing up. That said, even in "arm mode" the xplat cli will use the ASM APIs for things that haven't been brought over to ARM yet.

I like the plan of action and am looking forward to helping out.

One thing I haven't looked into is the APIs exposed by the Azure Go SDK as compared to other language bindings (node, .net, java).

[rajinders]

There is a new release for Azure PowerShell in preview.

https://azure.microsoft.com/en-us/blog/azps-1-0-pre/

It combines the service management and arm providers.

It may be worth a looking at to see how they are doing it and to see if Terraform can use similar approach.

[joseph-holland]

@rajinders I was just looking into this and it appears that Microsoft have packaged up both Service Manager and Resource Manager cmdlets into the same Azure PowerShell download, but have now split the commands out:

• Service Manager cmdlets are named/prefixed: Azure
• The newer Resource Manager cmdlets are named/prefixed: AzureRM
  So they connect separately to the different APIs anyway.

Given this and my knowledge of Azure I am of the opinion that a new provider is the best way to add this new functionality - provider/azure and provider/azurerm

Going forward there will be more resources added into Resource Manager that will never exist in the older Service Manager API and the old API will get phased out eventually. If these are two providers the separation is very clear and easier to manage IMHO.

[aznashwan]

@everyone: thank you all for your support.

I'm pleased to say that all the base work for ARM support in the provider itself has been done.
I have taken the liberty of creating a PR so as to have it in an utmost visible place: #3808

Now; work should be ready to start on the fun part: writing the resource definitions themselves.
I suggest that we keep syncing over IRC to ensure that no two people are working on the same resource type.

[clausasbjorn]

@aznashwan Awesome :) Sorry for the late response. I will start catching up this week!

[stack72]

Hi all

I am going to close this issue - ARM is now a thing in Terraform and has active development

P.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.