azurerm_linux_virtual_machine_scale_set with publicIPPrefix lost during scale event

[datadot]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Terraform v0.13.4
+ provider registry.terraform.io/hashicorp/azurerm v2.30.0

Affected Resource(s)

• azurerm_linux_virtual_machine_scale_set

Terraform Configuration Files

terraform {
  required_version = ">= 0.13"

  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 2.30.0"
    }
  }

  backend "local" {}

}

provider "azurerm" {
    features {
        virtual_machine_scale_set {
            roll_instances_when_required = false
        }
    }
}

locals {
    admin_username = "adminuser"
    public_key = "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEA1RzHexmc3tLReJxU180Po4DORbiF1p73cPUrlLDgHEnBRtaWasHET3OXX7bOhlNtBJ2TYrWgAJM4/4L/LRx9i7qFK40LsOfU0/f4T38WsZew5gZgVc0Stdoo6ateKT0hFz0EhaddkbfJilp91JlPDEsgqnNN6NE3y7xHyRNi33ozqgRM1XFO7LL2E1mTrCGgGDC/NZtxgxyDhednTE+oin+kneBbLdNfdPCzhM7MbP9GXPiujjpnZqgOe/L8IbXXeVd/+t8TlHktrKWdGXfec3oozcDqhfFg2MBgK2y+VEzx/FN0EmX8vojt8IKjvOj9BDi9qp/xo1GgkGJ8FtGINQ== example-key"
    location = "uksouth"
}

variable "instances" {
  description   = "VMSS Instance Count"
  type          = number
  default       = 2
}

resource "azurerm_resource_group" "rg_vnets" {
    name     = "rgp-example-vnets"
    location = local.location
}

resource "azurerm_resource_group" "rg_vmss" {
    name     = "rgp-example-vmss"
    location = local.location
}

resource "azurerm_resource_group" "rg_pubips" {
    name     = "rgp-example-pubips"
    location = local.location
}

resource "azurerm_virtual_network" "vnet" {
    name                = "vnet-vmss"
    resource_group_name = azurerm_resource_group.rg_vnets.name
    location            = local.location
    address_space       = ["10.10.0.0/16"]
}

resource "azurerm_subnet" "subnet" {
    name                 = "vnet-subnet"
    resource_group_name  = azurerm_resource_group.rg_vnets.name
    virtual_network_name = azurerm_virtual_network.vnet.name
    address_prefixes     = ["10.10.1.0/24"]
}

resource "azurerm_public_ip_prefix" "pubip_prefix" {
    name                = "ip-prefix-vmss"
    location            = local.location
    resource_group_name = azurerm_resource_group.rg_pubips.name
    prefix_length = 29
}

resource "azurerm_network_security_group" "nsg_vmss" {
  name                = "nsg-external"
  location            = local.location
  resource_group_name = azurerm_resource_group.rg_vmss.name

  security_rule {
    name                       = "ingress-ssh"
    priority                   = 100
    direction                  = "Inbound"
    access                     = "Allow"
    protocol                   = "Tcp"
    source_port_range          = "*"
    destination_port_range     = "22"
    source_address_prefix      = "*"
    destination_address_prefix = "*"
  }

}

resource "azurerm_linux_virtual_machine_scale_set" "vmss" {
    name                            = "example-vmss"
    computer_name_prefix            = "vmss"
    resource_group_name             = azurerm_resource_group.rg_vmss.name
    location                        = local.location
    sku                             = "Standard_D2s_v3"
    instances                       = var.instances
    admin_username                  = local.admin_username
    disable_password_authentication = true
    overprovision                   = false
    zones                           = []

    source_image_reference {
        publisher = "Canonical"
        offer     = "UbuntuServer"
        sku       = "18_04-lts-gen2"
        version   = "latest"
    }

    admin_ssh_key {
        username   = local.admin_username
        public_key = local.public_key
    }

    os_disk {
        storage_account_type = "StandardSSD_LRS"
        caching              = "ReadWrite"
    }

    data_disk {
        storage_account_type = "StandardSSD_LRS"
        caching              = "ReadWrite"
        lun                  = 10
        disk_size_gb         = 256
    }

    network_interface {
        name    = "external"
        primary = true
        dns_servers = []

        ip_configuration {
            application_gateway_backend_address_pool_ids  = []
            application_security_group_ids                = []
            load_balancer_backend_address_pool_ids        = []
            load_balancer_inbound_nat_rules_ids           = []
            name      = "external"
            primary   = true
            subnet_id = azurerm_subnet.subnet.id

            public_ip_address {
                name                = "public-ip-address"
                public_ip_prefix_id = azurerm_public_ip_prefix.pubip_prefix.id
            }

        }

        network_security_group_id = azurerm_network_security_group.nsg_vmss.id

    }

    lifecycle {
        ignore_changes = [
        tags,
        ]
    }

}

Debug Output

I believe the below is the cause of problem:

2020/10/06 11:57:40 [WARN] Provider "registry.terraform.io/hashicorp/azurerm" produced an unexpected new value for azurerm_linux_virtual_machine_scale_set.vmss, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .network_interface[0].ip_configuration[0].public_ip_address[0].public_ip_prefix_id: was cty.StringVal("/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rgp-example-pubips/providers/Microsoft.Network/publicIPPrefixes/ip-prefix-vmss"), but now cty.StringVal("")

Panic Output

No panic output, Terraform exists with code 0.

Expected Behavior

Apply the above Terraform to create example VMSS scale set.
Initial instances have IPs from Public IP Prefix
Increase scale set through Terraform from 2 to 3 instances.
Apply the updated Terraform manifest increasing instance count.
New instances should have IPs within the defined Public IP Prefix.

Actual Behavior

Initial created instances when applying first time have IPs from Public IP Prefix.
VMSS has configuration for Public IP Prefix - confirmed within Azure Resource Explorer (https://resources.azure.com/)
After applying for second time increasing instance count, new instances contain generally available public IPs not from Public IP Prefix.
VMSS configuration for Public IP Prefix has been removed by Terraform - confirmed within Azure Resource Explorer.
Next attempt to apply this Terraform plan will cause Terraform to attempt a destroy and re-create.

Steps to Reproduce

1. terraform init
2. terraform apply -var "instances=2"
3. terraform apply -var "instances=3"

Important Factoids

Not region or tenant specific.

[ArcturusZhang]

Hi @datadot thanks for this issue!

This issue is caused by some issues in the REST API of VMSS and have been reported here: Azure/azure-rest-api-specs#10190. We will update this resource to make this attribute updatable as soon as the service team fixes their swaggers.