Unhelpful "changes made outside of terraform" due to AWS vs tf representation in 1.0

[rvandegrift]

I'm filing this as a bug report, though perhaps it's a feature request.

terraform 1.0 warns the user when a resource has been changed outside of terraform. In some cases, I'm seeing differences that are semantically equivalent. This looks to be due to the AWS provider's representation differing from the AWS API's representation. As a result, terraform seems to always think someone has changed the resources. This significantly reduces the utility of the outside changes feature.

The reference issues contains a terraform core issue about the lexicographic iteration behavior of jsonecode. In the case of the policy document, this is probably the cause.

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

$ terraform -v
Terraform v1.0.2
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v3.48.0

Affected Resource(s)

• aws_iam_policy_document
• aws_s3_bucket (probably any resource which support optional tags?)

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

Sample for IAM:

resource "aws_iam_role" "role" {
  name                = var.role_name
  assume_role_policy  = data.aws_iam_policy_document.arpd.json
  managed_policy_arns = [var.managed_policy]
}

data "aws_iam_policy_document" "arpd" {
  statement {
    actions = ["sts:AssumeRole"]
    principals {
      type = "AWS"
      identifiers = [
        "arn:aws:iam::012345678901:root",
        "arn:aws:iam::987654321098:root",
      ]
    }
  }
}

Sample for S3:

resource "aws_s3_bucket" "bucket" {
  bucket = "my-bucket"
}

Expected Behavior

An empty tags map or a re-ordered list in a policy shouldn't count as an outside change, since there is no semantic difference.

Actual Behavior

Terraform produces copious details on changes which aren't really changes:

$ terraform plan
...
Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform apply":

  # aws_s3_bucket.bucket has been changed
  ~ resource "aws_s3_bucket" "bucket" {
        id                          = "my-bucket"
      + tags                        = {}
        # (10 unchanged attributes hidden)


        # (2 unchanged blocks hidden)
    }
  # aws_iam_role.role has been changed
  ~ resource "aws_iam_role" "role" {
      ~ assume_role_policy    = jsonencode(
          ~ {
              ~ Statement = [
                  ~ {
                      ~ Principal = {
                          ~ AWS = [
                              - "arn:aws:iam::012345678901:root",
                                "arn:aws:iam::987654321098:root",
                              + "arn:aws:iam::012345678901:root",
                            ]
                        }
                        # (3 unchanged elements hidden)
                    },
                ]
                # (1 unchanged element hidden)
            }
        )
        id                    = "role"
        name                  = "role"
        tags                  = {}
        # (8 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

Steps to Reproduce

1. terraform apply
2. terraform plan

Important Factoids

References

• Preserving object key order in jsonencode terraform#27880

[ewbankkit]

Potentially related: jen20/awspolicyequivalence#10.

[robomon1]

For json objects you should be doing a jsondiff and not a text diff. That would take care of this. In json there is no difference for something like this:

                          ~ Service = [
                              - "vpc-flow-logs.amazonaws.com",
                                "ec2.amazonaws.com",
                                "ssm.amazonaws.com",
                              + "vpc-flow-logs.amazonaws.com",
                            ]

The list is the same but it comes out in different order. But json doesn't care about the order so neither should the aws terraform provider.

[rvandegrift]

this core bug report is also related: hashicorp/terraform#28803

[danieladams456]

I'm seeing what might be a related issue with inline_policy of aws_iam_role and aws_iam_role_policy. I generally use the separate resource to manage inline policies, but now the role object shows a "changes made outside of Terraform" difference after every change of the role_policy object since its state has not "caught up" with the desired change from the other resource.

[KyleKotowick]

@robomon1

For json objects you should be doing a jsondiff and not a text diff. That would take care of this. In json there is no difference for something like this:

                          ~ Service = [
                              - "vpc-flow-logs.amazonaws.com",
                                "ec2.amazonaws.com",
                                "ssm.amazonaws.com",
                              + "vpc-flow-logs.amazonaws.com",
                            ]

The list is the same but it comes out in different order. But json doesn't care about the order so neither should the aws terraform provider.

This is not true, unfortunately. The Service field in your example is an array (a.k.a. "list"), and per RFC 7159:

An array is an ordered sequence of zero or more values.

So a JSON diff should show a difference if the order of elements in a list/array changes. The problem isn't that Terraform is detecting a difference, the problem is that AWS isn't respecting the order of elements that are provided in a policy and IAM is re-ordering them as it pleases. i.e. AWS is modifying the policy document on its own and Terraform is just detecting the change.

[julian-alarcon]

Similar error with AWS WAF aws_wafv2_web_acl.

Terraform v1.0.4
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v3.53.0

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform apply":
  # aws_wafv2_web_acl.search_web_acl_prod has been changed
  ~ resource "aws_wafv2_web_acl" "my_acl" {
        id          = "35435f-3455-6533-1234-88ad904517b"
        name        = "my_acl"
        tags        = {
            "deployment-tool" = "terraform"
        }
        # (6 unchanged attributes hidden)



        # (8 unchanged blocks hidden)
    }

Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan may include actions to undo or respond to these changes.

───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

No changes. Your infrastructure matches the configuration.

Your configuration already matches the changes detected above. If you'd like to update the Terraform state to match, create and apply a refresh-only plan:
  terraform apply -refresh-only
Releasing state lock. This may take a few moments...

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

[thiagofanfoni]

Hey all,
I'm facing the same thing (terraform v1.0.4 , aws v3.54.0) ...
The state is fixed if I run apply twice every time I do a change.

module.iam.aws_iam_role.it["test2"]: Refreshing state... [id=test2]
module.iam.aws_iam_role.it["test1"]: Refreshing state... [id=test1]
module.iam.aws_iam_role_policy_attachment.it["test1_AWSDirectConnectReadOnlyAccess_policy"]: Refreshing state... [id=test1-20210818015852396300000001]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # module.iam.aws_iam_role_policy.it["test1_test_policy"] will be created
  + resource "aws_iam_role_policy" "it" {
      + id     = (known after apply)
      + name   = "test"
      + policy = jsonencode(
            {
              + Statement = [
                  + {
                      + Action   = [
                          + "iam:List*",
                        ]
                      + Effect   = "Allow"
                      + Resource = "*"
                    },
                ]
              + Version   = "2012-10-17"
            }
        )
      + role   = "test1"
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

module.iam.aws_iam_role_policy.it["test1_test_policy"]: Creating...
module.iam.aws_iam_role_policy.it["test1_test_policy"]: Still creating... [10s elapsed]
module.iam.aws_iam_role_policy.it["test1_test_policy"]: Creation complete after 11s [id=test1:test]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

❯ terraform plan
module.iam.aws_iam_role.it["test1"]: Refreshing state... [id=test1]
module.iam.aws_iam_role.it["test2"]: Refreshing state... [id=test2]
module.iam.aws_iam_role_policy_attachment.it["test1_policy_AWSDirectConnectReadOnlyAccess"]: Refreshing state... [id=test1-20210818015852396300000001]
module.iam.aws_iam_role_policy.it["test1_test_policy"]: Refreshing state... [id=test1:test]

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform apply":

  # module.iam.aws_iam_role.it["test1"] has been changed
  ~ resource "aws_iam_role" "it" {
        id                    = "test1"
        name                  = "test1"
        tags                  = {}
        # (10 unchanged attributes hidden)

      - inline_policy {}
      + inline_policy {
          + name   = "test"
          + policy = jsonencode(
                {
                  + Statement = [
                      + {
                          + Action   = [
                              + "iam:List*",
                            ]
                          + Effect   = "Allow"
                          + Resource = "*"
                        },
                    ]
                  + Version   = "2012-10-17"
                }
            )
        }
    }

Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan may include
actions to undo or respond to these changes.

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

No changes. Your infrastructure matches the configuration.

Your configuration already matches the changes detected above. If you'd like to update the Terraform state to match, create and apply a refresh-only plan:
  terraform apply -refresh-only

[rvandegrift]

Here's a new, particularly useless example from aws provider 3.64.0:

Terraform detected the following changes made outside of Terraform since the last "terraform apply":

  # aws_iam_role.runner has been changed
  ~ resource "aws_iam_role" "runner" {
        id                    = "runner-node"
        name                  = "runner-node"
        tags                  = {
            "node-role" = "runner"
        }
        # (10 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

That is: there were no changes!

[YakDriver]

Thank you for reporting this problem! The op issue should be resolved in #21997. The role issue may be releated/resolved by #22004. As such, I'm going to close this. If after trying out those fixes you are still seeing problems, please reopen this or submit a new issue.

[sblask]

This has not been fixed. It's better now, but I still see plenty of:

  ~ resource "aws_iam_role" "instance" {
      ~ assume_role_policy    = jsonencode(
          ~ {
              ~ Statement = [
                  ~ {
                      ~ Principal = {
                          ~ Service = [
                                "sagemaker.amazonaws.com",
                              - "ec2.amazonaws.com",
                                "glue.amazonaws.com",
                              + "ec2.amazonaws.com",
                            ]
                        }
                        # (3 unchanged elements hidden)
                    },
                ]
                # (1 unchanged element hidden)
            }
        )
        id                    = "foo"
        name                  = "foo"
        tags                  = {}
        # (9 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

[forty]

We still have this issue with Terraform 1.1.7 Provider AWS 4.8.0

The issue is on Principal / Federated attribute of the policy though, rather than on Principal / AWS or Principal / Service

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.