Infinite ec_deployment.elasticsearch.config drift when using ec 0.6.0 and terraform 1.4.0

[lra]

Readiness Checklist

• I am running the latest version
• I checked the documentation and found no answer
• I checked to make sure that this issue has not already been filed
• I am reporting the issue to the correct repository (for multi-repository projects)

Expected Behavior

$ terraform plan
No changes. Your infrastructure matches the configuration.

It should not try to recreate the cluster if we don't have any ec_deployment.elasticsearch.config set.

Current Behavior

With:

resource "ec_deployment" "REDACTED" {
  name                   = "REDACTED"
  region                 = "azure-westus2"
  version                = "7.16.3"
  deployment_template_id = "azure-io-optimized-v3"

  elasticsearch = {
    cold = {
      autoscaling = {}
    }

    config = {}

    coordinating = {
      autoscaling = {}
    }

    frozen = {
      autoscaling = {}
    }

    hot = {
      autoscaling = {}
    }

    master = {
      autoscaling = {}
    }

    ml = {
      autoscaling = {}
    }

    warm = {
      autoscaling = {}
    }
  }

  kibana = {}
}

I get this infinite diff:

$ terraform plan
Terraform will perform the following actions:

  # ec_deployment.REDACTED will be updated in-place
  ~ resource "ec_deployment" "REDACTED" {
      + apm_secret_token       = (sensitive value)
      ~ elasticsearch          = {
          ~ cloud_id       = "REDACTED" -> (known after apply)
          ~ config         = {
              + plugins = (known after apply)
            }
          ~ http_endpoint  = "REDACTED" -> (known after apply)
          ~ https_endpoint = "REDACTED" -> (known after apply)
          ~ resource_id    = "REDACTED" -> (known after apply)
            # (12 unchanged attributes hidden)
        }
        id                     = "REDACTED"
        name                   = "REDACTED"
        # (5 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

And with

    config = {
      plugins = []
    }

I get:

Terraform will perform the following actions:

  # ec_deployment.REDACTED will be updated in-place
  ~ resource "ec_deployment" "REDACTED" {
      + apm_secret_token       = (sensitive value)
      ~ elasticsearch          = {
          ~ cloud_id       = "REDACTED" -> (known after apply)
          ~ config         = {
              + plugins = []
            }
          ~ http_endpoint  = "REDACTED" -> (known after apply)
          ~ https_endpoint = "REDACTED" -> (known after apply)
          ~ resource_id    = "REDACTED" -> (known after apply)
            # (12 unchanged attributes hidden)
        }
        id                     = "REDACTED"
        name                   = "REDACTED"
        # (5 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

and finally without any config block I get:

Terraform will perform the following actions:

  # ec_deployment.REDACTED will be updated in-place
  ~ resource "ec_deployment" "REDACTED" {
      + apm_secret_token       = (sensitive value)
      ~ elasticsearch          = {
          ~ cloud_id       = "REDACTED" -> (known after apply)
          - config         = {} -> null
          ~ http_endpoint  = "REDACTED" -> (known after apply)
          ~ https_endpoint = "REDACTED" -> (known after apply)
          ~ resource_id    = "REDACTED" -> (known after apply)
            # (12 unchanged attributes hidden)
        }
        id                     = "REDACTED"
        name                   = "REDACTED"
        # (5 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

We don't have any way out with tf 1.4 and ec 0.6.

Steps to Reproduce

1. terraform plan

Possible Solution

• This works with terraform v1.3.9 and ec v0.6.0
• This works with terraform v1.4.0 and ec v0.5.1
• This works with v1.3.9 and ec v0.5.1
• This does not work with v1.4.0 and ec v0.6.0

So we have workarounds for now, but we had to rollback from ec v0.6.0 back to ec v0.5.1 to be able to upgrade terraform to v1.4

Your Environment

Terraform v1.4.0
on darwin_arm64
+ provider registry.terraform.io/elastic/ec v0.6.0

[fludo]

Same problematic behavior with ec 0.7.0, terraform 1.4.6.

[Zawadidone]

@tobio will this issue be part of the next milestone?

[tobio]

It doesn't look like we've committed to a timeframe on this issue as yet sorry @Zawadidone.

[gaima8]

We downgraded terraform from v1.4.x to v1.3.9 for this about 4 weeks ago, and it worked for a while.
Recently however we're starting to have a similar, but very slightly different, problem.

Our terraform sets up two separate deployments. One started showing this problem in a few days, the other just started doing it this week.

  ~ resource "ec_deployment" "esd" {
      ~ apm_secret_token       = (sensitive value)
      ~ elasticsearch          = {
          ~ cloud_id       = "....hidden...." -> (known after apply)
          ~ config         = {
              + plugins            = (known after apply)
                # (1 unchanged attribute hidden)
            }
          ~ http_endpoint  = "http://....hidden....:9200" -> (known after apply)
          ~ https_endpoint = "https://....hidden....:443" -> (known after apply)
          ~ resource_id    = "....hidden...." -> (known after apply)

This is terraform v1.3.9 and elastic/ec 0.7.0.

[Zawadidone]

I have the same issue

[ghost]

I have a similar issue on Terraform v1.5.2 and ec v0.7.0 but for us, whenever I run terraform apply the diff boils down to:

resource "ec_deployment" "anyline_datalake_deployment" {
      + apm_secret_token       = (sensitive value)
      ~ elasticsearch          = {
          ~ cloud_id       = "cloud_id" -> (known after apply)
          ~ http_endpoint  = "some.cloud.es.io:9200" -> (known after apply)
          ~ https_endpoint = "some.cloud.es.io:443" -> (known after apply)
          ~ resource_id    = "ID" -> (known after apply)
            # (8 unchanged attributes hidden)
        }
        id                     = "id"
        name                   = "name"
        # (7 unchanged attributes hidden)
    }

The ids stay the same between the deployments, so I am not sure what triggers the change in the first place.

[rpatal]

We have the same issue. Despite there are no changes in the Terraform configuration, every time we run a plan o apply, we have the following result:

~ resource "ec_deployment" "deployment" {
      + apm_secret_token       = (sensitive value)
      ~ elasticsearch          = {
          ~ cloud_id       = "<cloud_id_value>" -> (known after apply)
          ~ http_endpoint  = "<http_endpoint_value>" -> (known after apply)
          ~ https_endpoint = "<https://endpoint:443>" -> (known after apply)
          ~ resource_id    = "<resource_id_value" -> (known after apply)
            # (11 unchanged attributes hidden)
        }
        id                     = "<id_value>"
        name                   = "<name>"
        # (8 unchanged attributes hidden)
    }

Terraform Version: 1.5.2
Provider Version: 0.7.0

The id values remain the same.

[nirsv]

We have the same issue:
Every terraform apply command attempts to make changes. After confirming with yes, it reports that modifications were successful, but when I check the activity center, it doesn't specify what changes were made.

      ~ elasticsearch          = {
          ~ cloud_id       = "%%%%HIDDEN%%%%" -> (known after apply)
          ~ config         = {
              + plugins            = (known after apply)
                # (1 unchanged attribute hidden)
            }
          ~ http_endpoint  = "http://%%%%HIDDEN.us-east-1.aws.found.io:9200" -> (known after apply)
          ~ https_endpoint = "https://%%%%HIDDEN.us-east-1.aws.found.io:443" -> (known after apply)
          ~ resource_id    = "%%%%HIDDEN%%%%%" -> (known after apply)
            # (8 unchanged attributes hidden)
        }
        id                     = "%%%%HIDDEN%%%%"
        name                   = "%%%%HIDDEN%%%%"
        # (11 unchanged attributes hidden)
    }```

Terraform v1.4.6
+ provider registry.terraform.io/elastic/ec v0.7.0
+ provider registry.terraform.io/elastic/elasticstack v0.6.2

[vl-kp]

same issue

[tobio]

A fix for this was released with 0.8.0 yesterday

[nirsv]

Thanks @tobio it solved the error for me.
Terraform v1.4.6
provider registry.terraform.io/elastic/ec v0.8.0

[Zawadidone]

The same for me. Thank you for fixing this issue!