Failing to upgrade Aurora Cluster for Postgres database from version 11.9 to 11.12 using Terraform

[rawashdeha]

Terraform Version = v0.14.6
Provider = AWS
Using TFE
There error is during Apply:

Plan Result

Terraform will perform the following actions:

module.aurora_01_cluster.aws_rds_cluster.this will be updated in-place
~ resource "aws_rds_cluster" "this" {
~ engine_version = "11.9" -> "11.12"
id = "snaurorapsgcluster"
tags = {

        "Core"            = "true"
        "Description"     = "Aurora Cluster"

    }
    # (34 unchanged attributes hidden)

    # (1 unchanged block hidden)
}
module.aurora_01_cluster_instance_reader.aws_rds_cluster_instance.default will be updated in-place
~ resource "aws_rds_cluster_instance" "default" {
~ engine_version = "11.9" -> "11.12"
id = "snaurorapsginstance"
~ tags = {
+ "Environment" = "Test"
# (10 unchanged elements hidden)
}
~ tags_all = {
+ "Environment" = "Test"
# (10 unchanged elements hidden)
}
# (28 unchanged attributes hidden)
}

module.aurora_01_cluster_instance_writer.aws_rds_cluster_instance.default will be updated in-place
~ resource "aws_rds_cluster_instance" "default" {
~ engine_version = "11.9" -> "11.12"
id = "snaurorapsginstance"
~ tags = {
+ "Environment" = "Test"
# (10 unchanged elements hidden)
}
~ tags_all = {
+ "Environment" = "Test"
# (10 unchanged elements hidden)
}
# (28 unchanged attributes hidden)
}

Plan: 0 to add, 3 to change, 0 to destroy.

Apply result:

�Terraform v0.14.6
Initializing plugins and modules...
�[0m�[1mmodule.aurora_01_cluster.aws_rds_cluster.this: Modifying... [id=snaurorapsgcluster]�[0m�[0m
�[0m�[1mmodule.aurora_01_cluster.aws_rds_cluster.this: Still modifying... [id=snaurorapsgcluster, 10s elapsed]�[0m�[0m
�[0m�[1mmodule.aurora_01_cluster.aws_rds_cluster.this: Still modifying... [id=snaurorapsgcluster, 20s elapsed]�[0m�[0m
�[0m�[1mmodule.aurora_01_cluster.aws_rds_cluster.this: Still modifying... [id=snaurorapsgcluster, 30s elapsed]�[0m�[0m
�[0m�[1mmodule.aurora_01_cluster.aws_rds_cluster.this: Modifications complete after 31s [id=snaurorapsgcluster]�[0m�[0m
�[31m
�[1m�[31mError: �[0m�[0m�[1mProvider produced inconsistent final plan�[0m

�[0mWhen expanding the plan for
module.aurora_01_cluster_instance_writer.aws_rds_cluster_instance.default to
include new values learned so far during apply, provider
"registry.terraform.io/hashicorp/aws" produced an invalid new value for
.engine_version: was cty.StringVal("11.12"), but now cty.StringVal("11.9").

This is a bug in the provider, which should be reported in the provider's own
issue tracker.
�[0m�[0m
�[31m
�[1m�[31mError: �[0m�[0m�[1mProvider produced inconsistent final plan�[0m

�[0mWhen expanding the plan for
module.aurora_01_cluster_instance_reader.aws_rds_cluster_instance.default to
include new values learned so far during apply, provider
"registry.terraform.io/hashicorp/aws" produced an invalid new value for
.engine_version: was cty.StringVal("11.12"), but now cty.StringVal("11.9").

This is a bug in the provider, which should be reported in the provider's own
issue tracker.
�[0m�[0m

[justinretzolk]

Hey @rawashdeha 👋 Thank you for taking the time to submit this issue. So that we have all of the necessary information in order to investigate, can you update the issue description to include all of the information found in the bug template?

[paulo-space-walker]

I was banging my head last week and this week when trying to upgrade the cluster/instance to a newer major version and found the same issue.

I'm not sure this is the case here, since the terraform template was not provided above, but when I tried to upgrade postgresql cluster from version 10.14 to version 11.15, the same error showed up to me, I change the instance engine version to a string instead of a reference to the cluster, workaround the error but the instance didn't get updated.

Terraform CLI and Terraform AWS Provider Version
Terraform v1.0.7
on darwin_amd64

• provider registry.terraform.io/hashicorp/aws v4.20.1
• provider registry.terraform.io/hashicorp/external v2.2.2
• provider registry.terraform.io/hashicorp/local v2.2.3
• provider registry.terraform.io/hashicorp/null v3.1.1
• provider registry.terraform.io/hashicorp/random v3.1.0

Affected Resource(s)

• aws_rds_cluster
• aws_rds_cluster_instance

Terraform Configuration Files
[initial state]
main.tf

resource "aws_rds_cluster" "default" {
  cluster_identifier = "aurora-cluster-demo"
  engine             = "aurora-postgresql"
  engine_version     = "10.14"
  database_name      = "mydb"
  deletion_protection = true

  availability_zones = ["us-west-2a", "us-west-2b", "us-west-2c"]
  backup_retention_period = 1
  preferred_maintenance_window    = "mon:00:00-mon:03:00"
  enabled_cloudwatch_logs_exports = ["postgresql"]

  iam_database_authentication_enabled = true

  master_username    = "foo"
  master_password    = "barbut8chars"

  allow_major_version_upgrade = true
}

resource "aws_rds_cluster_instance" "cluster_instances" {
  count              = 1
  identifier         = "aurora-cluster-demo-${count.index}"
  cluster_identifier = aws_rds_cluster.default.id
  instance_class     = "db.t3.medium"
  engine             = aws_rds_cluster.default.engine
  engine_version     = aws_rds_cluster.default.engine_version
  apply_immediately          = false
  auto_minor_version_upgrade = false

  lifecycle {
    prevent_destroy = true
  }
}

Debug Output

Terraform used the selected providers to generate the following execution plan. Resource
actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # module.database.aws_rds_cluster.default will be updated in-place
  ~ resource "aws_rds_cluster" "default" {
      ~ engine_version                      = "10.14" -> "11.15"
        id                                  = "aurora-cluster-demo"
        tags                                = {}
        # (35 unchanged attributes hidden)
    }

  # module.database.aws_rds_cluster_instance.cluster_instances[0] will be updated in-place
  ~ resource "aws_rds_cluster_instance" "cluster_instances" {
      ~ apply_immediately                     = false -> true
      ~ engine_version                        = "10.14" -> "11.15"
        id                                    = "aurora-cluster-demo-0"
        tags                                  = {}
        # (25 unchanged attributes hidden)
    }

Plan: 0 to add, 2 to change, 0 to destroy.

Do you want to perform these actions in workspace "my_workspace_name"?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

module.database.aws_rds_cluster.default: Modifying... [id=aurora-cluster-demo]
module.database.aws_rds_cluster.default: Still modifying... [id=aurora-cluster-demo, 10s elapsed]
module.database.aws_rds_cluster.default: Still modifying... [id=aurora-cluster-demo, 20s elapsed]
module.database.aws_rds_cluster.default: Still modifying... [id=aurora-cluster-demo, 30s elapsed]
module.database.aws_rds_cluster.default: Modifications complete after 31s [id=aurora-cluster-demo]
╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for module.database.aws_rds_cluster_instance.cluster_instances[0]
│ to include new values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/aws" produced an invalid new value for .engine_version:
│ was cty.StringVal("11.15"), but now cty.StringVal("10.14").
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue
│ tracker.

Expected Behavior
Upgrade Postgresql engine version from version 10.14 to 11.15 and apply changes immediately

Actual Behavior
Got Error: Provider produced inconsistent final plan

Steps to Reproduce
1 - Write the initial state of main.tf
2 - run terraform apply
3 - change main.tf, changing the engine version to 11.15
4 - run terraform apply
5 - got the error above
6 - changed aws_rds_cluster_instance (below)
7 - run terraform apply

WORKAROUND THE ERROR
Instead of setting engine_version = aws_rds_cluster.default.engine_version I put a string instead engine_version = "11.15" in the resource "aws_rds_cluster_instance" "cluster_instances" and the output is the following

Terraform will perform the following actions:

  # module.database.aws_rds_cluster.default will be updated in-place
  ~ resource "aws_rds_cluster" "default" {
      ~ engine_version                      = "10.14" -> "11.15"
        id                                  = "aurora-cluster-demo"
        tags                                = {}
        # (35 unchanged attributes hidden)
    }

  # module.database.aws_rds_cluster_instance.cluster_instances[0] will be updated in-place
  ~ resource "aws_rds_cluster_instance" "cluster_instances" {
      ~ apply_immediately                     = false -> true
      ~ engine_version                        = "10.14" -> "11.15"
        id                                    = "aurora-cluster-demo-0"
        tags                                  = {}
        # (25 unchanged attributes hidden)
    }

Plan: 0 to add, 2 to change, 0 to destroy.

Do you want to perform these actions in workspace "my_workspace_name"?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

module.database.aws_rds_cluster.default: Modifying... [id=aurora-cluster-demo]
module.database.aws_rds_cluster.default: Still modifying... [id=aurora-cluster-demo, 10s elapsed]
module.database.aws_rds_cluster.default: Still modifying... [id=aurora-cluster-demo, 20s elapsed]
module.database.aws_rds_cluster.default: Still modifying... [id=aurora-cluster-demo, 30s elapsed]
module.database.aws_rds_cluster.default: Modifications complete after 31s [id=aurora-cluster-demo]
module.database.aws_rds_cluster_instance.cluster_instances[0]: Modifying... [id=aurora-cluster-demo-0]
module.database.aws_rds_cluster_instance.cluster_instances[0]: Modifications complete after 0s [id=aurora-cluster-demo-0]
Releasing state lock. This may take a few moments...

Apply complete! Resources: 0 added, 2 changed, 0 destroyed.

Finally, it looks like it worked, but the engine version never gets updated in the aws console and no matter how many times I try to run this template it will try to update but never happens.

[paulo-space-walker]

I was banging my head last week and this week when trying to upgrade the cluster/instance to a newer major version and found the same issue.
...
Finally, it looks like it worked, but the engine version never gets updated in the aws console and no matter how many times I try to run this template it will try to update but never happens.

After writing the comment above, I realized there was some stuff I could try
Here is what I did that finally worked to upgrade the version immediatelly

I set apply_immediately=true flag in the resource aws_rds_cluster and removed prevent_destroy=true from the resource aws_rds_cluster_instance although I am not sure the latter was responsible for getting a positive outcome out of it.

Here is the final template that worked for me:

resource "aws_rds_cluster" "default" {
  cluster_identifier = "aurora-cluster-demo"
  engine             = "aurora-postgresql"
  engine_version     = "11.15"
  database_name      = "mydb"
  deletion_protection = true

  availability_zones = ["us-west-2a", "us-west-2b", "us-west-2c"]
  backup_retention_period = 1
  preferred_maintenance_window    = "mon:00:00-mon:03:00"
  enabled_cloudwatch_logs_exports = ["postgresql"]

  iam_database_authentication_enabled = true

  master_username    = "foo"
  master_password    = "barbut8chars"

  allow_major_version_upgrade = true
  apply_immediately          = true
}

resource "aws_rds_cluster_instance" "cluster_instances" {
  count              = 1
  identifier         = "aurora-cluster-demo-${count.index}"
  cluster_identifier = aws_rds_cluster.default.id
  instance_class     = "db.t3.medium"
  engine             = aws_rds_cluster.default.engine
  engine_version     = "11.15"
  apply_immediately          = true
  auto_minor_version_upgrade = false
}

[github-actions]

This functionality has been released in v4.62.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.