Persistent diff with data sources when referencing 0-count instances in 0.13

[lorengordon]

Terraform Version

$ terraform -version
Terraform v0.13.1
+ provider registry.terraform.io/hashicorp/external v1.2.0
+ provider registry.terraform.io/hashicorp/random v2.3.0
+ provider registry.terraform.io/hashicorp/template v2.1.2

Terraform Configuration Files

locals {
  template = "$${random}"
  vars = {
    random = join("", random_pet.this.*.id)
    empty  = join("", random_pet.empty.*.id)
  }
}

resource random_pet this {
  count = 1
}

resource random_pet empty {
  count = 0
}

data template_file this {
  template = local.template
  vars     = local.vars
}

data external this {
  program = ["jq", "-r", "."]

  query = {
    template = local.template
    vars     = jsonencode(local.vars)
  }
}

output external_result {
  value = data.external.this.result
}

output template_rendered {
  value = data.template_file.this.rendered
}

Expected Behavior

Following a successful apply, terraform plan should result in no changes.

Actual Behavior

terraform plan outputs that data sources will be read during apply, despite there being no other changes.

Steps to Reproduce

1. terraform init
2. terraform apply
3. terraform plan

$ terraform apply
...
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Outputs:

external_result = {
  "template" = "${random}"
  "vars" = "{\"empty\":\"\",\"random\":\"star-tuna\"}"
}
template_rendered = star-tuna

$ terraform apply
random_pet.this[0]: Refreshing state... [id=star-tuna]

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
 <= read (data resources)

Terraform will perform the following actions:

  # data.external.this will be read during apply
  # (config refers to values not yet known)
 <= data "external" "this"  {
      + id      = "-"
      + program = [
          + "jq",
          + "-r",
          + ".",
        ]
      + query   = {
          + "template" = "${random}"
          + "vars"     = jsonencode(
                {
                  + empty  = ""
                  + random = "star-tuna"
                }
            )
        }
      + result  = {
          + "template" = "${random}"
          + "vars"     = jsonencode(
                {
                  + empty  = ""
                  + random = "star-tuna"
                }
            )
        }
    }

  # data.template_file.this will be read during apply
  # (config refers to values not yet known)
 <= data "template_file" "this"  {
      + id       = "228051b9820bc0592514d4f122899016cdb24beccdba197c4a4d9f495e9e5d36"
      + rendered = "star-tuna"
      + template = "${random}"
      + vars     = {
          + "empty"  = ""
          + "random" = "star-tuna"
        }
    }

Plan: 0 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value:

Additional Information

The persistent diff occurs when the data source references a "conditional" resource (e.g. count = 0), such that the value is empty. In the repro above, this is the empty key/value in the vars map. Remove that value and it works fine.

References

There are a handful of similar reported issues, but the ones I found were closed as duplicates or otherwise resolved in terraform 0.13.1.

• Handle semantically equivalent changes from the legacy SDK when planning data sources #25812

[jbardin]

Thanks for the complete example @lorengordon!

The difference here is that 0.12 stored random_pet.empty with 0 instances in the state, while 0.13 does not. This causes the evaluation during Refresh to not be able to determine if random_pet.empty.*.id is supposed to be empty, or has yet to be created. Since this unknown value is being used in the data sources, they are deferred to plan when there is more information available. The removal of random_pet.empty, or changing it to be a positive count removes the planned data reads; as does running plan -refresh=false.

There may be some way to re-introduce the empty resource state, however I have a feeling that the cleanup was done to prevent other side effects of having no resources in the state and it might be more complicated than it appears.

In the long run, this is one of the cases that merging of refresh into the plan walk will take care of, which we can look forward to in an upcoming major release.

[lorengordon]

Thanks @jbardin, that explanation makes total sense. Of course if a resource with 0 instances is not stored in the state, then the whole thing will need to be read every time and create this persistent diff.

I would be very interested in learning which specific edge cases led to this change in behavior.

In the meantime, I did find one workaround, using the same condition on the resource and on any reference to its attributes. That way terraform can evaluate the expression properly. Here's an example:

locals {
  create_this  = true
  create_empty = false

  template = "$${random}"
  vars = {
    random = local.create_this ? random_pet.this[0].id : ""
    empty  = local.create_empty ? random_pet.empty[0].id : ""
  }
}

resource random_pet this {
  count = local.create_this ? 1 : 0
}

resource random_pet empty {
  count = local.create_empty ? 1 : 0
}

data template_file this {
  template = local.template
  vars     = local.vars
}

output template_rendered {
  value = data.template_file.this.rendered
}

[quentin9696]

Hello,

This issue will only be fix im TF 0.14 ?

Quentin.

[jbardin]

This is fixed in master by the new data lifecycle changes, and will be included in the 0.14 release.

• Move resource refreshing into plan #26270
• save read data in the refresh state during plan #26285
• simplify data lifecycle #26321

Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.