[Bug]: TF_AWS_DEFAULT_TAGS works in plan and fails in apply

[nitrocode]

Terraform Core Version

1.8.1

AWS Provider Version

5.62.0

Affected Resource(s)

• aws provider

Expected Behavior

Apply works as expected without an error

Actual Behavior

The plan shows this which is affected

  # aws_iam_role.default will be updated in-place
! resource "aws_iam_role" "default" {
!       tags_all              = {
+           "repository"             = "github.com/org/repo"
+           "repository_dir"         = "file-directory-of-terraform/"
            # (9 unchanged elements hidden)
        }
        # (11 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

Relevant Error/Panic Output Snippet

The apply shows this

Error: Provider produced inconsistent final plan

When expanding the plan for aws_iam_role.default to include new values
learned so far during apply, provider "registry.terraform.io/hashicorp/aws"
produced an invalid new value for .tags_all: element "repository" has
vanished.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Error: Provider produced inconsistent final plan

When expanding the plan for aws_iam_role.default to include new values
learned so far during apply, provider "registry.terraform.io/hashicorp/aws"
produced an invalid new value for .tags_all: element "repository_dir" has
vanished.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Terraform Configuration Files

Create a simple role without setting tags or tags_all or default_tags on the aws provider

Steps to Reproduce

1. Set the env vars

   export TF_AWS_DEFAULT_TAGS_repository="github.com/org/repo"
   export TF_AWS_DEFAULT_TAGS_repository_dir="file-directory-of-terraform/"

2. Run plan and save to a plan file
3. Verify the tag changes from the env var
4. Run apply on the plan file and see the failure

Debug Output

No response

Panic Output

No response

Important Factoids

I saw this in an atlantis deployment with the following configuration

workflows:
  terraform:
    plan:
      steps:
        # These env vars TF_AWS_DEFAULT_TAGS_ will work for aws provider 5.62.0+
        # https://github.com/hashicorp/terraform-provider-aws/releases/tag/v5.62.0
        - &env_default_tags_repository
          env:
            name: TF_AWS_DEFAULT_TAGS_repository
            command: 'echo "github.com/${HEAD_REPO_OWNER}/${HEAD_REPO_NAME}"'
        - &env_default_tags_repository_dir
          env:
            name: TF_AWS_DEFAULT_TAGS_repository_dir
            command: 'echo "${REPO_REL_DIR}"'

The workaround was to add the same env vars for both plan and apply stages which is odd because this is not needed for TF_VAR_ inputs.

    apply:
      steps:
        # These env vars TF_AWS_DEFAULT_TAGS_ will work for aws provider 5.62.0+
        # https://github.com/hashicorp/terraform-provider-aws/releases/tag/v5.62.0
        - *env_default_tags_repository
        - *env_default_tags_repository_dir

References

• https://github.com/hashicorp/terraform-provider-aws/releases/tag/v5.62.0
• Allow default_tags to be set by environment variables #33339

Would you like to implement a fix?

No

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[nitrocode]

Looks like the same input vars for the plan stage needed to be added to the apply stage in the same atlantis workflow.

This is different from the plan output when using TF_VAR_ inputs which persist in plan files whereas TF_AWS_DEFAULT_TAGS_ do not which leads to the inconsistent plan vs apply when using the TF_AWS_DEFAULT_TAGS_ env vars.

[justinretzolk]

Hey @nitrocode 👋 Thank you for taking the time to raise this, and for following up with your findings! Given that it wound up being an issue of how Atlantis is preserving the environment variables between stages, I'm going to close this. If you run into any further issues with the provider, please do let us know and we'd be happy to look into it!

[github-actions]

Warning

This issue has been closed, meaning that any additional comments are hard for our team to see. Please assume that the maintainers will not see them.

Ongoing conversations amongst community members are welcome, however, the issue will be locked after 30 days. Moving conversations to another venue, such as the AWS Provider forum, is recommended. If you have additional concerns, please open a new issue, referencing this one where needed.

[nitrocode]

Hi @justinretzolk , even though I was able to workaround the issue, the main issue is the difference between TF_VAR vs TF_AWS_DEFAULT_TAGS.

The former persists in the plan file. The latter does not persist in the plan file so the apply shows inconsistent results.

I believe these different types of env vars should be consistent, no?

The former is the behavior in terraform/opentodu. The latter is the behavior in aws provider.

[nitrocode]

cc @justinretzolk

[nitrocode]

cc: @justinretzolk

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.