RDS cluster instance resource doesn't work with aurora-postgres preview

[cwoerner]

Hi there,

Thank you for opening an issue. Please note that we try to keep the Terraform issue tracker reserved for bug reports and feature requests. For general usage questions, please see: https://www.terraform.io/community.html.

Terraform Version

Run terraform -v to show the version. If you are not running the latest version of Terraform, please upgrade because your issue may have already been fixed.

Terraform v0.10.3-dev (a4d182f4e55e90765c12f182e4b14549df10adc1)
terraform-provider-aws 1.0.0 (dde16a0)

Affected Resource(s)

Please list the resources as a list, for example:

• aws_rds_cluster_instance

Terraform Configuration Files

resource "aws_rds_cluster" "aurora_cluster" {
    cluster_identifier                                = "test-aurora-cluster"
    availability_zones                               = ["us-east-1a", "us-east-1b", "us-east-1c"]
    database_name                                  = "test"
    port                                                     = 5432    
    master_username                              = "master"
    master_password                              = "test1234"
    engine                                                = "aurora-postgresql"
    db_cluster_parameter_group_name = "default.aurora-postgresql1"
}

resource "aws_rds_cluster_instance" "aurora_cluster_instance" {
    count                                    = "3"
    identifier                               = "test-instance-${count.index}"
    cluster_identifier                  = "${aws_rds_cluster.aurora_cluster.id}"
    instance_class                      = "db.r4.large"
    publicly_accessible              = false
    db_parameter_group_name = "default.aurora-postgresql1"
}

Debug Output

https://gist.github.com/cwoerner/ab70e1c89ea51e5017a30576c56f6dae

Panic Output

N/A

Expected Behavior

Create an aws aurora postgresql cluster with 3 instances corresponding to the aurora-postgresql engine, version 1.0.

Actual Behavior

The cluster was created with zero instances. An error was reported stating this during apply.

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. terraform apply

Important Factoids

I was able to produce a patch which fixes the issue for me. Using a custom-built version of the terraform aws provider plugin I was able to successfully create an aurora-postgres cluster with 3 instances as expected. Gist with patch below...

https://gist.github.com/cwoerner/021e1e7a8a0b05d590dcdf632a171c8e

After applying this patch, the aws_rds_cluster_instance resource supports an "engine" argument.

References

Nope

[stack72]

Hi @cwoerner

This is being tracked upstream with AWS - aws/aws-sdk-go#1470

This isn't a Terraform issue right now

Paul

[Ninir]

Hey @cwoerner

As per a discussion between @stack72 and one of the Go SDK's member, this is a preview feature that may require your account to be whitelisted.

As this is not related to Terraform, since Paul made the work to allow other engines for cluster, do you mind me closing this issue?
I encourage you to follow the SDK issue to be up-to-date on that :)

Thanks!

[cwoerner]

No, this is a terraform aws provider issue. The terraform aws provider RDS cluster resource does alow you to specify the engine, but the cluster instance resource does not. Consequently, you can create an empty cluster with engine "aurora-postgresql" but the creation of the related writer and reader replicas fail due to prameter mismatch errors. The reason being that terraform aws provider doesn't expose the engine argument on the instance resource as well, causing the aws api default value (aurora mysql) to be substituted at runtime, which fails. The patch I provided demonstrates a fix for the issue. I've tested with a whitelisted account capable of creating postresql aurora clusters in us-east-1 and it works perfectly with the patch, but fails per my report without it.

[cwoerner]

To be clear, I've now created multiple postres aurora clusters in us-east-1 region using my patched local build of the terraform aws provider plugin.

[Ninir]

Oh you're right @cwoerner, misread this one (cluster vs instance) 😅

Do you think you could send a PR for that, along with an acceptance test, documentation update?feel free to implement it, will be happy to review it! :)
Otherwise 'will throw something for that tomorrow using the gist you provided!

[cwoerner]

Yeah, sure. My company (shameless plug - Quantcast) is pretty open source friendly, but I will have to get it reviewed internally first, since technically this is all work-related. Is a couple days to a week ok?

[catsby]

Is a couple days to a week ok?

It absolutely is! We're grateful for all the contributions we can get, and we understand that work takes time especially OSS work. Assuming no one else comes along as does it before you, in which case I hope the reference this issue so we don't have duplicate work 😄

[Ninir]

@cwoerner Yup, would be glad to review it :)
Tell us if you need any help on that! 👍

[ppearcy]

FYI, postgres aurora is now GA:
https://aws.amazon.com/blogs/aws/now-available-amazon-aurora-with-postgresql-compatibility/

[ppearcy]

This looks to be fixed in this merge:
#1591

[Ninir]

Indeed, this has been added both for RDS clusters and RDS instances. Since it is GA, I'm going to close this if you don't mind :)

Happy Terraforming!

[sameer59-saks]

@ALL : We are trying to implement Aurora Postgres RDS and running into the same issue. We would like to know how you guys went about implementing the patch. @cwoerner: can you please put more light on how you went about making this work.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!