Fail to destroy subnet in use by gateway ip configuration of application gateway

[genevieve]

Terraform Version

Azure Provider v0.3.3
Terraform v0.11.0

Affected Resource(s)

• azurerm_subnet
• azurerm_application_gateway

Terraform Configuration Files

resource "azurerm_subnet" "sub1" {
  name                 = "${var.env_id}-cf-subnet1"
  address_prefix       = "${cidrsubnet(var.network_cidr, 8, 1)}"
  resource_group_name  = "${azurerm_resource_group.bosh.name}"
  virtual_network_name = "${azurerm_virtual_network.bosh.name}"
}

...

resource "azurerm_application_gateway" "network" {
  name                = "${var.env_id}-app-gateway"
  resource_group_name = "${azurerm_resource_group.bosh.name}"
  location            = "West US"

  sku {
    name     = "Standard_Small"
    tier     = "Standard"
    capacity = 2
  }

  probe {
    name                = "Probe01"
    protocol            = "Http"
    path                = "/login"
    host                = "login.${var.system_domain}"
    interval            = 60
    timeout             = 60
    unhealthy_threshold = 3
  }

  gateway_ip_configuration {
    name      = "${var.env_id}-cf-gateway-ip-configuration"
    subnet_id = "${azurerm_virtual_network.bosh.id}/subnets/${azurerm_subnet.sub1.name}"
  }

  frontend_port {
    name = "frontendporthttps"
    port = 443
  }

  frontend_port {
    name = "frontendportlogs"
    port = 4443
  }

  frontend_ip_configuration {
    name                 = "${var.env_id}-cf-frontend-ip-configuration"
    public_ip_address_id = "${azurerm_public_ip.lb.id}"
  }

  backend_address_pool {
    name = "${var.env_id}-cf-backend-address-pool"
  }

  backend_http_settings {
    name                  = "${azurerm_virtual_network.bosh.name}-be-htst"
    cookie_based_affinity = "Disabled"
    port                  = 80
    protocol              = "Http"
    request_timeout       = 1
    probe_name            = "Probe01"
  }

  ssl_certificate {
    name     = "ssl-cert"
    data     = "${var.pfx_cert_base64}"
    password = "${var.pfx_password}"
  }

  http_listener {
    name                           = "${azurerm_virtual_network.bosh.name}-httplstn"
    frontend_ip_configuration_name = "${var.env_id}-cf-frontend-ip-configuration"
    frontend_port_name             = "frontendporthttps"
    protocol                       = "Https"
    ssl_certificate_name           = "ssl-cert"
  }

  request_routing_rule {
    name                       = "${azurerm_virtual_network.bosh.name}-rqrt"
    rule_type                  = "Basic"
    http_listener_name         = "${azurerm_virtual_network.bosh.name}-httplstn"
    backend_address_pool_name  = "${var.env_id}-cf-backend-address-pool"
    backend_http_settings_name = "${azurerm_virtual_network.bosh.name}-be-htst"
  }
}

Debug Output

Error: Error applying plan:

1 error(s) occurred:

* azurerm_subnet.cf-lb (destroy): 1 error(s) occurred:

* azurerm_subnet.cf-lb: network.SubnetsClient#Delete: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="InUseSubnetCannotBeDeleted" Message="Subnet bbl-env-cf-sn is in use by /subscriptions/XXXX/resourceGroups/XXXX/providers/Microsoft.Network/applicationGateways/bbl-env-app-gateway/gatewayIPConfigurations/bbl-env-cf-gateway-ip-configuration and cannot be deleted." Details=[]

Expected Behavior

terraform destroy should have successfully destroyed.

Actual Behavior

It failed to destroy the subnet as it is in use by the gateway ip configuration that references the subnet in the configuration of the application gateway.

After a second terraform destroy, it successfully destroyed the subnet.

Steps to Reproduce

1. terraform apply
2. terraform destroy

[genevieve]

I see there are related issues, specifically: Azure/azure-rest-api-specs#1233

If this is an issue with the azure api becoming consistent, could we check for the InUseSubnetCannotBeDeleted error message && whether that resource causing the delete conflict actually exists? We might be able to make this a retryableError type, and loop over in an effort to delete the subnet again as we wait for the api to become consistent?

[tombuildsstuff]

hey @genevievelesperance

Thanks for opening this issue - apologies for the delayed response here!

I've been looking into Application Gateways over the past few days and have been trying to find a way around this issue. As it stands - I'd agree that it's definitely related to Azure/azure-rest-api-specs#1233 - however I've opened a new issue specifically for the bugs relating to Application Gateways here: Azure/azure-rest-api-specs#2187

Thanks!

[genevieve]

Thanks Tom! Would you prefer I close this issue now or when your issue gets closed? (However that happens..)

[tombuildsstuff]

@genevieve sorry, missed this notification - let's leave this issue open until the other issues resolved, since others may find it when searching. FWIW I'm chasing the API issue through our internal channels too.

[genevieve]

Thanks Tom!

[tombuildsstuff]

👋

To give an update here: we're still waiting for the Application Gateway API to be fixed before proceeding with any enhancements or bug fixes for this resource; once the API is fixed (I've just requested an update). As such I've de-prioritised this issue for the moment, but we'll circle around and take another look once the API's in a usable state - sorry for the inconvenience here!

Thanks!

[tombuildsstuff]

hi @genevieve

Given this issue is blocked on an upstream issue in the Azure API rather than keeping multiple issues open and trying to ensure they all remain up to date - I'm going to close this in favour of #1576 which is the Meta-Issue tracking the Bugs and Enhancements for the Application Gateway Resource. Once the bug in the API is fixed we should be able to take a look into this, but we'll keep track of the status of this issue there for the moment.

Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!