Spot pools can't set max surge

[spatel-15]

Is there an existing issue for this?

• I have searched the existing issues

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

1.3.2

AzureRM Provider Version

3.12.0

Affected Resource(s)/Data Source(s)

azurerm_kubernetes_cluster_node_pool

Terraform Configuration Files

resource "azurerm_kubernetes_cluster_node_pool" "node_pools" {
  for_each              = var.node_pools
  kubernetes_cluster_id = azurerm_kubernetes_cluster.aks_cluster.id
  enable_node_public_ip = false
  zones                 = lookup(each.value, "availability_zones", [])
  tags                  = {}
  orchestrator_version  = var.kubernetes_version
  name                  = each.value["name"]
  vm_size               =  "Standard_D2_v2"
  max_pods              = 110
  os_type               = "Linux"
  os_disk_size_gb       = 256
  os_disk_type          = "Ephemeral"
  enable_auto_scaling   = true
  node_count            = 3
  min_count             = 1
  max_count             = 5
  vnet_subnet_id        = "/subscriptions/${data.azurerm_subscription.current.subscription_id}/resourceGroups/${local.vnet_resource_group}/providers/Microsoft.Network/virtualNetworks/${local.vnet_name}/subnets/${lookup(each.value, "subnet_name", local.subnet_name)}"
  priority              = "Spot"
  eviction_policy       = "Deallocate"
  spot_max_price      = -1

  upgrade_settings {
    max_surge = local.is_dev ? null : lookup(each.value, "max_surge", "1")
  }
}

Debug Output/Panic Output

Error: Missing required argument
│ 
│   with module.kubernetesCluster.azurerm_kubernetes_cluster_node_pool.node_pools["axxsv2"],
│   on ../modules/aks/cluster-aks.tf line 149, in resource "azurerm_kubernetes_cluster_node_pool" "node_pools":
│  149:     max_surge = local.is_dev ? null : lookup(each.value, "max_surge", "1")
│ 
│ The argument "upgrade_settings.0.max_surge" is required, but no definition
│ was found.

Expected Behaviour

max_surge attribute is not required for SPOT nodepools

upgrade_settings {
     max_surge = local.is_dev ? null : lookup(each.value, "max_surge", "1")
} 

Actual Behaviour

max_surge is a required field in Terraform azurerm_kubernetes_cluster_node_pool resource but Azure API does not support value for max_surge .

So when i defined the below settings -

upgrade_settings {
    max_surge = lookup(each.value, "max_surge", "1")
} 

i get this error
│ Error: creating Node Pool: (Agent Pool Name "x2xx3" / Managed Cluster Name "wowdk8suiaae" / Resource Group "xxx-dev-k8s-xxxxxxxx-xxx"): containerservice.AgentPoolsClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidParameter" Message="The value of parameter agentPoolProfile.upgrade.maxSurge is invalid. Error details: Spot pools can't set max surge. Please see https://aka.ms/aks-naming-rules for more details." Target="agentPoolProfile.upgrade.maxSurge"

Steps to Reproduce

terraform {
  required_version = ">= 1.3.2"
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "3.12.0"
    }

    kubernetes = {
      source = "hashicorp/kubernetes"
      version = "2.12.0"
    }
  }
}

provider "azurerm" {
  skip_provider_registration = true
  features {}
}

resource "azurerm_resource_group" "example" {
  name     = "example-resources"
  location = "West Europe"
}

resource "azurerm_kubernetes_cluster" "example" {
  name                = "example-aks1"
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name
  dns_prefix          = "exampleaks1"

  default_node_pool {
    name       = "default"
    node_count = 1
    vm_size    = "Standard_D2_v2"
  }

  service_principal {
    client_id     = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
    client_secret = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
  }
}

resource "azurerm_kubernetes_cluster_node_pool" "example" {
  name                  = "spotpool1"
  kubernetes_cluster_id = azurerm_kubernetes_cluster.example.id
  vm_size               = "Standard_DS2_v2"
  node_count            = 1
  priority              = "Spot"
  eviction_policy       = "Deallocate"
  spot_max_price        = -1
  
  tags = {
    Environment = "Dev"
  }
  
  upgrade_settings {
    max_surge = 1
  }
}

Important Factoids

No response

References

No response

[alexismosquera]

👍

[julianninotovar]

👍

[robin-wayve]

You need a dynamic block for your upgrade_settings.

[rcskosir]

@spatel-15 Thank you for opening this issue. Were you able to resolve the issue by using dynamic block for your upgrade_setting like @robin-wayve suggested?

[mtoumi990]

@rcskosir here is an example that might require some updates depending on your azurerm_kubernetes_cluster_node_pool settings.

dynamic "upgrade_settings" {
    for_each = lookup(each.value, "node_pool_type", "Regular") != "Spot" ? [1] : []
    content {
      max_surge = lookup(each.value, "max_surge", "33%")
    }
  }

[stephybun]

Thanks for opening this issue @spatel-15.

As the error suggests, max_surge shouldn't be set for Spot node pools and so the upgrade_settings block which is optional should be removed from the configuration. Alternatively as the comments above suggest you could use a dynamic block for upgrade_settings.

Since this is working as intended and the suggestions above should resolve your issue I'm going to close this issue, do let us know if that is not the case.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.