Invalid address to set: []string{"integrations_server", "0", "symbols_https_endpoint"} or Invalid address to set: []string{"integrations_server", "0", "profiling_https_endpoint"}

[davidg-datascene]

• [ x] I am running the latest version
• [ x] I checked the documentation and found no answer
• [ x] I checked to make sure that this issue has not already been filed
• [ x] I am reporting the issue to the correct repository (for multi-repository projects)

Expected Behavior

The terraform plan runs through without error

## Terraform definition

# Create an Elastic Cloud deployment
resource "ec_deployment" "elasticcloud" {
  # Optional name.
  name = "my_example_deployment"

  region                 = "us-east-1"
  version                = data.ec_stack.latest.version
  deployment_template_id = "aws-io-optimized-v2"

  elasticsearch = {
    hot = {
      autoscaling = {}
    }
  }

  kibana = {}

  enterprise_search = {}

  integrations_server = {}
}

Steps to Reproduce

ElasticCloud have introduced some new endpoints (Profiling and Symbols) as per screen shot.

EC version: 8.11.3

which is causing the plan to fail on subsequent runs (symbols_https_endpoint or profiling_https_endpoint):

Error: Invalid address to set: []string{"integrations_server", "0", "symbols_https_endpoint"}

  with module.main.module.elastic_cluster[0].ec_deployment.elasticcloud,
  on .terraform/modules/main.elastic_cluster/elastic.tf line 7, in resource "ec_deployment" "elasticcloud":
   7: resource "ec_deployment" "elasticcloud" {
   

[davidg-datascene]

Might need to be added to the Nested Schema for integrations_server.endpoints

Thanks.

[davidg-datascene]

Apologies to pester but this is impacting use hard i.e. we can't apply elastic changes through our terraform code beacuse of the issue. Thanks.

[tobio]

@davidg-datascene I'm struggling to reproduce this with just the resource definition included here. Are you able to produce a minimal set of steps to reproduce this error? Specifically covering:

1. Terraform versions terraform version from within your TF module dir would be great, it would include both the CLI and provider versions.
2. The steps you're taking to enable the symbols/profiling URLS, IIUC these would only be available after enabling universal profiling.
3. Anything else required to go from a clean TF state to a failing state.

[davidg-datascene]

We have introduced APM (integrations_server) into Kibana via the terraform resource "ec_deployment" as shown by the snippet below. We have a nonprod and prod cluster using APM (integrations_server) built using the same terraform module. On the nonprod cluster the issue doesn't exist, probably because the Symbols and Profiling endpoints haven't been activated. We haven't introduced Universal Profiling on either cluster but suspect someone may have clicked on the Universal Profiling link in the Observability part of the PROD Kibana to activate the endpoints.

Elastic cloud version: 8.11.3

Terrafrom provider version (we currently can't upgrade to the latest EC provider but I suspect the issue is with the latest provider as well)

terraform {
  required_providers {
    ec = {
      source  = "elastic/ec"
      version = "< 0.6"
    }
  }
  required_version = ">= 0.14"
}

Include APM (integrations_server) in Cluster deployment

resource "ec_deployment" "elasticcloud" {
        name                   = "test-with-apm"
       
### Include integrations_server (APM)
      integrations_server {
         topology {
                 size = "4g"
                zone_count   = 3
            }
        }
}

1 - Go into Kibana and click on Universal Profiling in Kibana?
2 - Check in Elastic deployment has the 2 new endpoints 'Symbols' and 'Profiling' as shown by image above.
3 - Re-run terraform plan and should fail as demonstrated originally (if new endpoints exist else plan is good).

Let me know if you need more info. Thanks

[3kt]

@tobio the below config can be copy-pasted to reproduce the issue:

Run the below main.tf:

terraform {
  required_providers {
    ec = {
      source  = "elastic/ec"
      version = "< 0.6"
    }
  }
  required_version = ">= 0.14"
}

provider "ec" {
    apikey = "YOUR_API_KEY"
}

resource "ec_deployment" "elasticcloud" {
  name = "test-with-apm"
  version = "8.11.3"
  deployment_template_id = "azure-storage-optimized"
  region                 = "azure-northeurope"

  elasticsearch {}

  kibana {}

  integrations_server {
     topology {
            size = "4g"
            zone_count   = 1
        }
    }
}

Once deployment is finished, you can verify that a terraform refresh completes without issue:

❯ terraform refresh
ec_deployment.elasticcloud: Refreshing state... [id=ID]

If you then navigate to https://YOUR_DEP.kb.northeurope.azure.elastic-cloud.com:9243/app/profiling and click on "Set up universal profiling", any subsequent refresh will fail because of the added endpoint, per the initial description:

❯ terraform refresh
ec_deployment.elasticcloud: Refreshing state... [id=ID]
╷
│ Error: Invalid address to set: []string{"integrations_server", "0", "symbols_https_endpoint"}
│
│   with ec_deployment.elasticcloud,
│   on main.tf line 15, in resource "ec_deployment" "elasticcloud":
│   15: resource "ec_deployment" "elasticcloud" {
│
╵

[3kt]

Might be as simple as adding symbols and profiling there

From the API call response:

          "metadata" : {
            "version" : 8,
            "last_modified" : "2024-02-19T16:12:50.124Z",
            "endpoint" : "ID.apm.northeurope.azure.elastic-cloud.com",
            "service_url" : "https://ID.apm.northeurope.azure.elastic-cloud.com",
            "aliased_endpoint" : "NAME.apm.northeurope.azure.elastic-cloud.com",
            "aliased_url" : "https://NAME.apm.northeurope.azure.elastic-cloud.com",
            "sso_deep_linking_supported" : false,
            "ccr" : false,
            "ports" : {
              "http" : 9200,
              "https" : 443,
              "transport_passthrough" : 9400
            },
            "services_urls" : [
              {
                "service" : "profiling",
                "url" : "https://NAME.profiling.northeurope.azure.elastic-cloud.com"
              },
              {
                "service" : "symbols",
                "url" : "https://NAME.symbols.northeurope.azure.elastic-cloud.com"
              },
              {
                "service" : "fleet",
                "url" : "https://NAME.fleet.northeurope.azure.elastic-cloud.com"
              },
              {
                "service" : "apm",
                "url" : "https://NAME.apm.northeurope.azure.elastic-cloud.com"
              }
            ]
          },

[tobio]

@3kt @davidg-datascene thanks for the repro steps. I can repro this with 0.5, but this issue has been fixed in 0.6+.

Is there another issue blocking an upgrade?

[davidg-datascene]

@3kt @davidg-datascene thanks for the repro steps. I can repro this with 0.5, but this issue has been fixed in 0.6+.

Is there another issue blocking an upgrade?

@tobio - Hi, thanks for reproducing it. We are looking at the options to upgrade to v0.6 as that is a breaking change and then from there to the latest.