Can't use heredoc with AWS SQS policy

[b-ryan]

When I use a heredoc, the policy always appears to have changed. Example:

resource "aws_sqs_queue" "test" {
  name = "test"

  policy = <<EOF
{
    "Version": "2012-10-17",
    "Id": "test",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {"AWS": "*"},
            "Action": "SQS: SendMessage",
            "Resource": "arn:aws:sqs:us-east-1:blahblah:test",
            "Condition": {
                "ArnEquals": {
                    "aws: SourceArn": "sns_arn"
                }
            }
        }
    ]
}
EOF
}

After creating this, terraform tells me the policy needs to change in the next plan. This appears to be due to this policy containing newline characters and spaces. I think AWS strips all whitespace characters from the policy. Doing aws sqs get-queue-attributes --attribute-names Policy --queue-url https://... gives me a policy without any whitespace characters.

I assume the fix here would be to strip whitespace from the resource.

[jen20]

Hi @b-ryan! Thanks for reporting this - I have a feeling we are missing normalisation of the JSON as is applied in several other resources with policies - for example

terraform/builtin/providers/aws/resource_aws_vpc_endpoint.go

Line 25 in cf98cbc

StateFunc: normalizeJson,

- should be a straightforward fix!

[b-ryan]

Hi @jen20 -- thanks. I thought a function like that might exist. I'll try to fix this if I have some time.

[mmlb]

Chiming in that this doesn't just apply to heredoc type vars. I have a queue policy that looks like

policy = "${replace(file("some-policy-file.json"), "@ENV@", "${var.env}")}"

that also hits this.

[r39132]

Hi Folks!
This bug has been a constant irritant for our company. We use terraform (and ansible) to setup/deploy all of our infrastructure. A large part of that is setting up data pipelines based on SNS/SQS. Every time we do a terraform apply, all of our SQS pipelines break. This is a major bug!
-s

[aflury]

+1 (Disclosure: I work with @r39132 ).

It seems like the fix is simple. I came up with the exact fix that @chancefeick did before noticing that he had already done so.

[rsutton1]

👍 on this (disclosure: I work with @b-ryan )

I'm currently seeing this bug when trying to get my policy in Terraform to match the policy currently in AWS. The policies are the same, however Terraform shows a policy change just because the whitespace is off. It looks like there's no way to get the whitespace to match without changing Terraform.

I set the policy in Terraform to be the same as the AWS policy, yet Terraform is giving a diff in the plan because it thinks they're different. I copied the Terraform output for the current policy and the new policy into a text file to see the difference and I see this:

$ diff current.txt new.txt
25a26
> 

As thediff shows, the only difference is that Terraform policies include an extra line break at the end. I tried to add that linebreak to the policy through the AWS console UI, but when I try to save the policy AWS automatically removes the linebreak at the end.

If Terraform could normalize JSON in the same way as AWS, this problem would be fixed.

[scheffield]

Hey there, this issue seems to be fixed since 0.6.15 (#5888). It does not occur for me anymore.

[dmangot]

We're on 0.6.16 and it still tells us it's going to change the policy every single time. Maybe there is something different about what we're doing. It happens on our SQS queues.

[sepulworld]

This still appears to be an issue. On 0.6.16 and it still looks to modify SNS resources with EOF of JSON

[djosephsen]

We're still having this issue in 0.7.11. I spent some time poking at it this week and I think there are two reasons we're experiencing it.

First, terraform doesn't seem to be normalizing the json it creates from aws_iam_policy_document data sources. This is causing newline characters and indentation to creep into the schema.Resource's json field.

Second, terraform doesn't seem to be respecting the schema options on policy documents during diffing. Ie, adding DiffSuppressFunc: suppressEquivalentAwsPolicyDiffs and other options to the json schema field in the schema.Resource returned by dataSourceAwsIamPolicyDocument() doesn't affect the diff output, so terraform is showing false positives with respect to things like fields being out of order and etc when it should be using suppressEquivalentAwsPolicyDiffs to compare those fields.

I have a working fix for the first problem, which is to wrap the output of dataSourceAwsIamPolicyDocumentRead in normalizeJsonString(), and I'll PR that tonight. I'm not really sure what to do about the second problem though and I'd super-appreciate some guidance from someone who understands the diffing code-paths and might know off the top of their head what happens internally when an iam policy generated from a policy_document in the local config is compared to json gleaned from the aws api.

Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.