Handle pending RDS engine version updates

[jar-b]

Description

This change fixes "inconsistent final plan" errors caused by deferred engine version updates (apply_immediately = false) in the RDS cluster, cluster instance, and instance resources. Specifically, read operations will now check the PendingModifiedValues for a non-nil EngineVersion. If present, and equal to the existing configured value, this will be stored in state to ensure a consistent final plan and avoid persistent diffs until the update takes place during the next maintenance window.

Additionally, this change improves the cluster update workflow when there are pending updates by:

• Setting the EngineVersion field if the configured and "actual" engine versions differ.
  • This can occur when multiple applies happen prior to a maintenance window. In this situation the desired engine version is sent again, rather than omitting the field entirely (the current behavior).
• Setting the DBInstanceParameterGroupName field any time the value is set.
  • This attribute is not returned from the DescribeDBClusters API, so once set there is no way to detect drift and determine when state does not match the remote value. Sending the value whenever configured avoids errors where the desired value should be sent, but isn't because no drift could be detected. Sending a value that matches the existing remote should have no negative impact.

Relations

Closes #29861
Closes #22022
Closes #28219
Relates #30245

References

• ModifyDBCluster
• CreateDBCluster
• DescribeDBClusters

Output from Acceptance Testing

$ make testacc PKG=rds TESTS=TestAccRDSCluster_allowMajorVersionUpgrade
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/rds/... -v -count 1 -parallel 20 -run='TestAccRDSCluster_allowMajorVersionUpgrade'  -timeout 180m
=== RUN   TestAccRDSCluster_allowMajorVersionUpgrade
=== PAUSE TestAccRDSCluster_allowMajorVersionUpgrade
=== RUN   TestAccRDSCluster_allowMajorVersionUpgradeNoApplyImmediately
=== PAUSE TestAccRDSCluster_allowMajorVersionUpgradeNoApplyImmediately
=== RUN   TestAccRDSCluster_allowMajorVersionUpgradeWithCustomParametersApplyImm
=== PAUSE TestAccRDSCluster_allowMajorVersionUpgradeWithCustomParametersApplyImm
=== RUN   TestAccRDSCluster_allowMajorVersionUpgradeWithCustomParameters
=== PAUSE TestAccRDSCluster_allowMajorVersionUpgradeWithCustomParameters
=== CONT  TestAccRDSCluster_allowMajorVersionUpgrade
=== CONT  TestAccRDSCluster_allowMajorVersionUpgradeWithCustomParameters
=== CONT  TestAccRDSCluster_allowMajorVersionUpgradeWithCustomParametersApplyImm
=== CONT  TestAccRDSCluster_allowMajorVersionUpgradeNoApplyImmediately
--- PASS: TestAccRDSCluster_allowMajorVersionUpgradeNoApplyImmediately (950.64s)
--- PASS: TestAccRDSCluster_allowMajorVersionUpgrade (2141.41s)
--- PASS: TestAccRDSCluster_allowMajorVersionUpgradeWithCustomParameters (2259.32s)
--- PASS: TestAccRDSCluster_allowMajorVersionUpgradeWithCustomParametersApplyImm (2409.43s)
PASS
ok      github.com/hashicorp/terraform-provider-aws/internal/service/rds        2412.582s

$ make testacc PKG=rds TESTS=TestAccRDSCluster_
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/rds/... -v -count 1 -parallel 20 -run='TestAccRDSCluster_'  -timeout 180m
<snip>
--- PASS: TestAccRDSCluster_EngineMode_global (152.24s)
=== CONT  TestAccRDSCluster_SnapshotIdentifier_encryptedRestore
--- PASS: TestAccRDSCluster_iamAuth (161.20s)
=== CONT  TestAccRDSCluster_SnapshotIdentifierVPCSecurityGroupIDs_tags
--- PASS: TestAccRDSCluster_identifierGenerated (162.58s)
=== CONT  TestAccRDSCluster_SnapshotIdentifier_vpcSecurityGroupIDs
--- PASS: TestAccRDSCluster_identifierPrefix (162.75s)
=== CONT  TestAccRDSCluster_SnapshotIdentifier_tags
--- PASS: TestAccRDSCluster_basic (166.60s)
=== CONT  TestAccRDSCluster_SnapshotIdentifier_preferredMaintenanceWindow
--- PASS: TestAccRDSCluster_backupsUpdate (191.74s)
=== CONT  TestAccRDSCluster_SnapshotIdentifier_preferredBackupWindow
--- PASS: TestAccRDSCluster_deletionProtection (204.22s)
=== CONT  TestAccRDSCluster_SnapshotIdentifier_masterUsername
--- PASS: TestAccRDSCluster_copyTagsToSnapshot (302.90s)
=== CONT  TestAccRDSCluster_SnapshotIdentifier_masterPassword
--- PASS: TestAccRDSCluster_Scaling_defaultMinCapacity (306.86s)
=== CONT  TestAccRDSCluster_SnapshotIdentifier_kmsKeyID
--- PASS: TestAccRDSCluster_enableHTTPEndpoint (324.78s)
=== CONT  TestAccRDSCluster_SnapshotIdentifierEngineVersion_equal
--- PASS: TestAccRDSCluster_pointInTimeRestore (467.58s)
=== CONT  TestAccRDSCluster_SnapshotIdentifierEngineVersion_different
--- PASS: TestAccRDSCluster_engineMode (498.77s)
=== CONT  TestAccRDSCluster_SnapshotIdentifierEngineMode_provisioned
--- PASS: TestAccRDSCluster_SnapshotIdentifier_preferredMaintenanceWindow (415.64s)
=== CONT  TestAccRDSCluster_SnapshotIdentifierEngineMode_parallelQuery
    cluster_test.go:1756: Step 1/1 error: Error running apply: exit status 1

        Error: creating RDS Cluster (tf-acc-test-2442697265199590370-source): InvalidParameterCombination: Cannot find version null for aurora
                status code: 400, request id: b8fb3c84-4ec4-4b61-8a45-eda467758d15

          with aws_rds_cluster.source,
          on terraform_plugin_test.tf line 2, in resource "aws_rds_cluster" "source":
           2: resource "aws_rds_cluster" "source" {

=== CONT  TestAccRDSCluster_SnapshotIdentifier_deletionProtection
--- FAIL: TestAccRDSCluster_SnapshotIdentifierEngineMode_parallelQuery (4.26s)
--- PASS: TestAccRDSCluster_SnapshotIdentifier_vpcSecurityGroupIDs (450.67s)
=== CONT  TestAccRDSCluster_snapshotIdentifier
--- PASS: TestAccRDSCluster_SnapshotIdentifier_encryptedRestore (476.70s)
=== CONT  TestAccRDSCluster_availabilityZones
--- PASS: TestAccRDSCluster_SnapshotIdentifierVPCSecurityGroupIDs_tags (470.27s)
=== CONT  TestAccRDSCluster_allowMajorVersionUpgradeWithCustomParameters
--- PASS: TestAccRDSCluster_SnapshotIdentifier_preferredBackupWindow (456.09s)
=== CONT  TestAccRDSCluster_GlobalClusterIdentifierEngineModeGlobal_update
--- PASS: TestAccRDSCluster_SnapshotIdentifier_masterUsername (445.91s)
=== CONT  TestAccRDSCluster_tags
--- PASS: TestAccRDSCluster_SnapshotIdentifier_masterPassword (385.53s)
=== CONT  TestAccRDSCluster_serverlessV2ScalingConfiguration
--- PASS: TestAccRDSCluster_SnapshotIdentifierEngineVersion_equal (415.03s)
=== CONT  TestAccRDSCluster_scaling
--- PASS: TestAccRDSCluster_SnapshotIdentifier_kmsKeyID (446.47s)
=== CONT  TestAccRDSCluster_port
    cluster_test.go:1529: Step 1/2 error: Error running apply: exit status 1

        Error: creating RDS Cluster (tf-acc-test-4181258938337893371): InvalidParameterCombination: The Parameter Group default.aurora-postgresql13 with DBParameterGroupFamily aurora-postgresql13 cannot be used for this instance. Please use a Parameter Group with DBParameterGroupFamily aurora-postgresql14
                status code: 400, request id: 9ccc5fdd-7e47-4045-80c4-b90d3c3e3fca

          with aws_rds_cluster.test,
          on terraform_plugin_test.tf line 2, in resource "aws_rds_cluster" "test":
           2: resource "aws_rds_cluster" "test" {

--- FAIL: TestAccRDSCluster_port (5.41s)
=== CONT  TestAccRDSCluster_EnabledCloudWatchLogsExports_postgresql
--- PASS: TestAccRDSCluster_availabilityZones (176.39s)
=== CONT  TestAccRDSCluster_GlobalClusterIdentifier_secondaryClustersWriteForwarding
    cluster_test.go:1501: Step 1/1 error: Error running apply: exit status 1

        Error: creating RDS Global Cluster: InvalidParameterValue: The requested engine version was not found or does not support global functionality
                status code: 400, request id: 8841bd8c-c280-4607-bcc8-70909151616b

          with aws_rds_global_cluster.test,
          on terraform_plugin_test.tf line 18, in resource "aws_rds_global_cluster" "test":
          18: resource "aws_rds_global_cluster" "test" {

--- FAIL: TestAccRDSCluster_GlobalClusterIdentifier_secondaryClustersWriteForwarding (9.44s)
=== CONT  TestAccRDSCluster_encrypted
--- PASS: TestAccRDSCluster_onlyMajorVersion (860.77s)
=== CONT  TestAccRDSCluster_networkType
--- PASS: TestAccRDSCluster_GlobalClusterIdentifierEngineModeGlobal_update (214.30s)
=== CONT  TestAccRDSCluster_kmsKey
--- PASS: TestAccRDSCluster_tags (213.28s)
=== CONT  TestAccRDSCluster_updateIAMRoles
--- PASS: TestAccRDSCluster_SnapshotIdentifierEngineVersion_different (437.60s)
=== CONT  TestAccRDSCluster_engineVersionWithPrimaryInstance
--- PASS: TestAccRDSCluster_serverlessV2ScalingConfiguration (219.72s)
=== CONT  TestAccRDSCluster_GlobalClusterIdentifierEngineModeGlobal_remove
--- PASS: TestAccRDSCluster_SnapshotIdentifier_tags (780.79s)
=== CONT  TestAccRDSCluster_GlobalClusterIdentifierEngineModeGlobal_add
--- PASS: TestAccRDSCluster_allowMajorVersionUpgradeNoApplyImmediately (969.24s)
=== CONT  TestAccRDSCluster_GlobalClusterIdentifierEngineMode_global
--- PASS: TestAccRDSCluster_SnapshotIdentifierEngineMode_provisioned (486.24s)
=== CONT  TestAccRDSCluster_EnabledCloudWatchLogsExports_mySQL
--- PASS: TestAccRDSCluster_encrypted (184.34s)
=== CONT  TestAccRDSCluster_missingUserNameCausesError
--- PASS: TestAccRDSCluster_missingUserNameCausesError (5.16s)
=== CONT  TestAccRDSCluster_takeFinalSnapshot
--- PASS: TestAccRDSCluster_snapshotIdentifier (405.58s)
=== CONT  TestAccRDSCluster_PointInTimeRestore_enabledCloudWatchLogsExports
--- PASS: TestAccRDSCluster_updateIAMRoles (175.31s)
=== CONT  TestAccRDSCluster_backtrackWindow
--- PASS: TestAccRDSCluster_kmsKey (180.37s)
=== CONT  TestAccRDSCluster_dbSubnetGroupName
--- PASS: TestAccRDSCluster_networkType (208.95s)
=== CONT  TestAccRDSCluster_EngineMode_parallelQuery
    cluster_test.go:1183: Step 1/1 error: Error running apply: exit status 1

        Error: creating RDS Cluster (tf-acc-test-6924192451476223119): InvalidParameterCombination: Cannot find version null for aurora
                status code: 400, request id: 745c2f17-093d-4fa1-822a-2e2b32766a5f

          with aws_rds_cluster.test,
          on terraform_plugin_test.tf line 2, in resource "aws_rds_cluster" "test":
           2: resource "aws_rds_cluster" "test" {

--- FAIL: TestAccRDSCluster_EngineMode_parallelQuery (4.37s)
=== CONT  TestAccRDSCluster_engineVersion
--- PASS: TestAccRDSCluster_scaling (336.55s)
=== CONT  TestAccRDSCluster_dbClusterInstanceClass
--- PASS: TestAccRDSCluster_GlobalClusterIdentifierEngineModeGlobal_add (162.41s)
=== CONT  TestAccRDSCluster_EngineMode_multiMaster
--- PASS: TestAccRDSCluster_GlobalClusterIdentifierEngineModeGlobal_remove (198.33s)
=== CONT  TestAccRDSCluster_GlobalClusterIdentifier_primarySecondaryClusters
--- PASS: TestAccRDSCluster_GlobalClusterIdentifierEngineMode_global (137.64s)
=== CONT  TestAccRDSCluster_GlobalClusterIdentifierEngineMode_provisioned
--- PASS: TestAccRDSCluster_SnapshotIdentifier_deletionProtection (567.98s)
=== CONT  TestAccRDSCluster_disappears
--- PASS: TestAccRDSCluster_dbSubnetGroupName (173.48s)
=== CONT  TestAccRDSCluster_GlobalClusterIdentifier_replicationSourceIdentifier
--- PASS: TestAccRDSCluster_EnabledCloudWatchLogsExports_mySQL (243.87s)
--- PASS: TestAccRDSCluster_backtrackWindow (194.27s)
--- PASS: TestAccRDSCluster_GlobalClusterIdentifierEngineMode_provisioned (148.39s)
--- PASS: TestAccRDSCluster_EngineMode_multiMaster (163.55s)
--- PASS: TestAccRDSCluster_takeFinalSnapshot (276.77s)
--- PASS: TestAccRDSCluster_disappears (142.09s)
--- PASS: TestAccRDSCluster_PointInTimeRestore_enabledCloudWatchLogsExports (398.87s)
--- PASS: TestAccRDSCluster_engineVersion (465.10s)
--- PASS: TestAccRDSCluster_iops (1904.38s)
--- PASS: TestAccRDSCluster_allocatedStorage (1944.34s)
--- PASS: TestAccRDSCluster_storageType (1964.65s)
--- PASS: TestAccRDSCluster_ReplicationSourceIdentifier_kmsKeyID (2151.53s)
--- PASS: TestAccRDSCluster_allowMajorVersionUpgradeWithCustomParametersApplyImm (2190.92s)
=== NAME  TestAccRDSCluster_EnabledCloudWatchLogsExports_postgresql
    cluster_test.go:775: Step 2/2 error: Error running post-apply refresh: exit status 1

        Error: reading RDS Global Cluster for RDS Cluster (tf-acc-test-6347732204348353326): RequestError: send request failed
        caused by: Post "https://rds.us-west-2.amazonaws.com/": read tcp 192.168.0.170:61584->54.240.253.127:443: read: connection reset by peer

          with aws_rds_cluster.test,
          on terraform_plugin_test.tf line 2, in resource "aws_rds_cluster" "test":
           2: resource "aws_rds_cluster" "test" {

--- PASS: TestAccRDSCluster_allowMajorVersionUpgrade (2251.78s)
--- PASS: TestAccRDSCluster_dbClusterInstanceClass (1175.51s)
--- PASS: TestAccRDSCluster_engineVersionWithPrimaryInstance (1349.47s)
--- PASS: TestAccRDSCluster_allowMajorVersionUpgradeWithCustomParameters (1667.27s)
--- FAIL: TestAccRDSCluster_EnabledCloudWatchLogsExports_postgresql (1540.10s)
--- PASS: TestAccRDSCluster_GlobalClusterIdentifier_primarySecondaryClusters (1782.34s)
--- PASS: TestAccRDSCluster_GlobalClusterIdentifier_replicationSourceIdentifier (1895.94s)
FAIL
FAIL    github.com/hashicorp/terraform-provider-aws/internal/service/rds        3115.569s
FAIL
make: *** [testacc] Error 1

Failures are unrelated to this change.

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this pull request by adding a 👍 reaction to the original post to help the community and maintainers prioritize this pull request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

For Submitters

• Review the contribution guide relating to the type of change you are making to ensure all of the necessary steps have been taken.
• For new resources and data sources, use skaff to generate scaffolding with comments detailing common expectations.
• Whether or not the branch has been rebased will not impact prioritization, but doing so is always a welcome surprise.

[YakDriver]

I feel good about this, when you do. 👍

[github-actions]

This functionality has been released in v4.62.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.