EIP scope change incorrectly detected #5330
Labels
bug
Addresses a defect in current functionality.
service/ec2
Issues and PRs that pertain to the ec2 service.
upstream
Addresses functionality related to the cloud provider.
Comments
|
Ran into the same problem this morning, with an interesting difference that I think may (hopefully) shed some light on the issue: Terraform plan outputTerraform ConfigurationCurrent Terraform State"aws_eip.bastion": {
"type": "aws_eip",
"depends_on": [
"aws_instance.bastion"
],
"primary": {
"id": "eipalloc-64c1c253",
"attributes": {
"association_id": "eipassoc-8fc12584",
"domain": "vpc",
"id": "eipalloc-64c1c253",
"instance": "i-0f09aa63bde8abf6d",
"network_interface": "eni-801a3a7b",
"private_ip": "10.110.164.31",
"public_ip": "35.170.90.79",
"tags.%": "0",
"vpc": "true"
},
"meta": {},
"tainted": false
},
"deposed": [],
"provider": "provider.aws"
},
"aws_eip.choose_bastion": {
"type": "aws_eip",
"depends_on": [
"aws_instance.choose_bastion"
],
"primary": {
"id": "eipalloc-a7967aac",
"attributes": {
"association_id": "eipassoc-eb232f31",
"domain": "vpc",
"id": "eipalloc-a7967aac",
"instance": "i-005bbe6d945152a2e",
"network_interface": "eni-f065ca97",
"private_ip": "10.0.0.79",
"public_ip": "18.211.114.115",
"tags.%": "0",
"vpc": "true"
},
"meta": {
"e2bfb730-ecaa-11e6-8f88-34363bc7c4c0": {
"delete": 180000000000,
"read": 900000000000,
"update": 300000000000
}
},
"tainted": false
},
"deposed": [],
"provider": "provider.aws"
},The key thing to notice is that although both eips are different, terraform seems to think they have the exact same starting configuration (except for the ids) that needs to be updated - same public IP, same "vpc: false" status. There are also several other eips in the same account/configuration that are showing the same problem, terraform thinks each one has the same public IP / vpc: false configuration. My guess is that the AWS api is returning things differently than previously (either correctly or incorrectly) that's causing terraform to get the same eip every time it refreshes the state for different eips. additional notes
Update (after looking into it a bit more)After running $ aws ec2 describe-addresses
{
"Addresses": [{
"InstanceId": "",
"PublicIp": "54.163.241.54",
"Domain": "standard"
}, {
// additional eips returned, including those that _should_ be used for the existing ones in configuration
}]
} |
bflad
added
bug
|
Can we determine if this is affecting just one region or multiple? It may be an EC2 API update rolling out. If someone could provide the EC2 API responses from the AWS SDK logging by enabling debug logging, that would be immensely helpful in troubleshooting this issue. e.g. Thanks! |
|
@bflad Here's a plan I did a few minutes ago with trace logging level: Hopefully that helps. |
|
Seems like there's an additional |
|
We ended up fixing our issue by removing the unallocated (i.e. "standard") eips in our account - with debug logging we saw the exact same problem - additional items without the correct allocation id being returned the the AWS DescribeAddresses request. Ideally AWS shouldn't be returning the extra items if you pass the allocation id to the request, but the logic could possibly be changed here to maybe search in all returned items to find the right one? : https://github.com/terraform-providers/terraform-provider-aws/blob/4df232fc4f1b62d0ce7d3a4ce3fb70503574ac60/aws/resource_aws_eip.go#L185-L193 trace: |
|
That sounds like a potential fix we might need to implement if the server side filtering isn't working as expected. For anyone following this issue, I would suggest contacting AWS support if you can about this issue so the EC2 team can potentially rollback/fix this behavior. |
|
@bflad would it be worth making a PR anyways for it? (I am volunteering 😄) seems like it would make the logic in that spot a bit more clear anyways. |
|
@rv-aburdine certainly wouldn't hurt! |
ghost
mentioned this issue
|
Unfortunately we haven't got a support contract currently, so if anyone else has and can report on our behalf that would be much appreciated. In the interim we've had to manually re-create the destroyed resources as we can't remove our Thanks for the assistance thus far everyone! |
|
@zacblazic I just sent in a support request to AWS to hold off on their rollout and pass this bug to their engineering (for whatever that's worth). Given the severity of the issue, would this warrant cutting a release today? Our infra nearly took a nosedive - thankfully, we don't trust the CI to do EIP deallocation ;-) |
Thanks @dafyddcrosby!
This was a scary lesson for us actually. We have the following workflow:
We were unlucky enough to have the In light of what happened today we're updating our workflow to require manual approval between the |
Terraform Plan (DEBUG)RequestNote the allocation ID above: ResponseNote the allocation ID Plan OutputNote the AWS CLIAfter executing the AWS CLI command several times, I was able to get one correct response: This seems to indicate there is a rollout in progress. |
|
I got a response back from AWS support It looks fixed on my end, but we might still want to make this code more robust in case something like this happens again in the future. |
That's great news! Thanks for the update. I'm testing on my end now.
Agreed. |
|
Since it appears this was resolved upstream in the API, going to close this issue out. We'll still followup with the code adjustments of #5331 though 👍 |
|
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks! |
ghost
locked and limited conversation to collaborators
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Community Note
Terraform Version
Affected Resource(s)
Terraform Configuration Files
Debug Output
Expected Behavior
Terraform should not incorrectly detect that the VPC scope has changed when it has not, as per the screenshot below:
Actual Behavior
Terraform incorrectly detects that the VPC scope has changed from
falsetotrue.The example above was not the first incident, but just a more concrete example. Below is the first case we identified:
Error during apply:
This behaviour started today at ~08:45 AM UTC for 1 of 3 availability zones for one environment initially.
However, in subsequent
terraform planexecutions a few minutes later, all of our environments are now affected.Steps to Reproduce
We use the following during our CI runs:
terraform initterraform plan -no-color --out="${environment}.plan" | tee "${environment}.report"terraform apply "${environment}.plan"Don't have an isolated reproducible case yet, but will provide if necessary.
Important Factoids
Running in VPC.
Terraform is run in CI.
The text was updated successfully, but these errors were encountered: