aws_vpn_connection: inconsistent remote state data (occasionally)

[n3ph]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.12.17

• provider.aws v2.42.0

Affected Resource(s)

• aws_vpn_connection

Terraform Configuration Files

does not matter

Background

resourceAwsVpnConnectionCreate assembles the connection options [1] and calls AWS API to create the VPN connection with these accordingly. It seems AWS is creating the VPN connections asynchronously as sometimes the order of the VPN connections in the XML config blob does not apply to the order of options[]. Since tunnel1_inside_cidr and tunnel2_inside_cidr are saved directly to the remote state and all other attributes are added later in resourceAwsVpnConnectionRead [2], the data for the resource might be inconsistent.

The code already tries to cover this unsuccessfully via sorting the XML config blob [3]. sort.Sort(vpnConfig) is acting on OutsideAddress of type XmlIpsecTunnel struct [4].

We need to check wether:

• vpnConfig.Tunnels[0].VgwInsideAddress or vpnConfig.Tunnels[0].CgwInsideAddress is a vaild address of tunnel1_inside_cidr
• vpnConfig.Tunnels[1].VgwInsideAddress or vpnConfig.Tunnels[1].CgwInsideAddress is a vaild address of tunnel2_inside_cidr

[1] https://github.com/terraform-providers/terraform-provider-aws/blob/master/aws/resource_aws_vpn_connection.go#L277-L291
[2] https://github.com/terraform-providers/terraform-provider-aws/blob/master/aws/resource_aws_vpn_connection.go#L452-L463
[3] https://github.com/terraform-providers/terraform-provider-aws/blob/master/aws/resource_aws_vpn_connection.go#L596
[4] https://github.com/terraform-providers/terraform-provider-aws/blob/master/aws/resource_aws_vpn_connection.go#L27-L28

Expected Behavior

terraform state show aws_vpn_connection.example | grep -e tunnel1 -e tunnel2 | grep -v -e tunnel1_address -e tunnel2_address  -e bgp -e key
    tunnel1_cgw_inside_address     = "169.254.254.250"
    tunnel1_inside_cidr            = "169.254.254.248/30"
    tunnel1_vgw_inside_address     = "169.254.254.249"
    tunnel2_cgw_inside_address     = "169.254.254.254"
    tunnel2_inside_cidr            = "169.254.254.252/30"
    tunnel2_vgw_inside_address     = "169.254.254.253"

Actual Behavior

terraform state show aws_vpn_connection.example | grep -e tunnel1 -e tunnel2 | grep -v -e tunnel1_address -e tunnel2_address  -e bgp -e key
    tunnel1_cgw_inside_address     = "169.254.254.250"
    tunnel1_inside_cidr            = "169.254.254.252/30"
    tunnel1_vgw_inside_address     = "169.254.254.249"
    tunnel2_cgw_inside_address     = "169.254.254.254"
    tunnel2_inside_cidr            = "169.254.254.248/30"
    tunnel2_vgw_inside_address     = "169.254.254.253"

References

• VPN Connection Tunnels incorrectly ordered #396
• aws_vpn_connection - inside_tunnel_cidr swapped between tunnel interfaces  #5809

[ewbankkit]

Maybe related?

• VPN Connection Tunnels incorrectly ordered #396
• aws_vpn_connection - inside_tunnel_cidr swapped between tunnel interfaces  #5809

[n3ph]

What a hassle. :-( sigh.
I wanted to push an updated version of the test first to show that this current kind of sorting does not work. But I fucked up commiting the right subset of my changes.

All good things come in threes...

[n3ph]

As stated in #10584 (comment) it might be enough to just get rid of sort.Sort(vpnConfig)
As I was trying to reproduce #10584 as a solution I mixed up the custom provider builds..

It is NOT sufficient to just remove sort.Sort(vpnConfig).

[jochen42]

As stated in #10584 (comment) it might be enough to just get rid of sort.Sort(vpnConfig)
As I was trying to reproduce #10584 as a solution I mixed up the custom provider builds..

It is NOT sufficient to just remove sort.Sort(vpnConfig).

hi @n3ph ,

i think you are facing another issue than me. My issue is just the wrong order of the tunnels itself.
With the removed complex-object-sort i just want to have the correct order like aws-api answers.

it is hard to debug for the operators at the client-side if they are wrong ordered.

f.e. the results of the following command should be in the same order as the provided connection-parameter vom resource-output or statefile.

aws ec2 describe-vpn-connections --vpn-connection-id <some_connection_idf> --query 'VpnConnections[*].VgwTelemetry[*]'

the same for aws console.

your issue seems to be much more complex. for me it looks like you are right with the wrong tunnel*_vgw_inside_address, but your solution:

• will also provide the wrong tunnel-order/sorting (hard to debug)
• tries to fix a problem on aws-side with an assumption of their internal-behavior (what if you are wrong or they change the wrong behavior next week?)

Perhaps your issue should be reported to aws support?

[bplunkert]

I am experiencing this issue as well. While I like the idea in #11298 upon testing it does not seem to have any effect at all for me.

My tunnels are still down, with the inside addresses definitely flipped. tunnel1_cgw_inside_address is in the tunnel2_inside_cidr, and vice versa.

This is blocking me from deploying a VPN connection. Any workarounds (aside from removing VPN from Terraform management) would be extremely helpful.

Edit: I think I found a workaround: delete the VPN connection entirely, and re-create it again. Sometimes it comes back correctly. If it isn't correct, keep trying until it is created correctly. (It took me several tries.)

[ghost]

This has been released in version 3.38.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.