RouteAlreadyExists: The route identified by 0.0.0.0/0 already exists

[ialidzhikov]

We do often see provider-aws terraformer to fail during Shoot creation with error:

Last ErrorFlow "Shoot cluster reconciliation" encountered task errors: [task "Waiting until shoot infrastructure has been reconciled" failed: failed to create infrastructure: retry failed with context deadline exceeded, last error: extension encountered error during reconciliation: Error reconciling infrastructure: Terraform execution job 'foo.infra.tf-job' could not be completed. The following issues have been found in the logs:

-> Pod 'foo.infra.tf-job-8tvwz' reported:
* [0m[0m[1mError creating route: RouteAlreadyExists: The route identified by 0.0.0.0/0 already exists.
    status code: 400, request id: <omitted>[0m
[0m  on tf/main.tf line 221, in resource "aws_route" "private_utility_z0_nat":
 221: resource "aws_route" "private_utility_z0_nat" [4m{[0m
[0m
[0m[0m]

We see that the route is present in the provider, but it is missing in terraform.tfstate which make terraform try to create it and fail.
We see this since several terraform versions (0.12.9 - the current one, 0.11.14, and probably even an older one).

[rfranzke]

Anyone having time to implement a mitigation?

Manual steps with gardenctl:

g aws ec2 -- describe-route-tables --output=json --filters "Name=vpc-id,Values=vpc-123456" // Replace vpc-id by actual id

g aws ec2 -- delete-route --route-table-id=rtb-abcdef --destination-cidr-block=0.0.0.0/0 // Fetch route-table id related to the nat-gateway and replace it.

[vlerenc]

Thx for reporting and indeed, a fix in TF or mitigation in G would be great, otherwise we suffer from it endlessly.

[ialidzhikov]

Actually we found that the initial terraform apply for the described case fails with:

# ...
aws_route.private_utility_z0_nat: Still creating... [1m30s elapsed]
aws_route.private_utility_z0_nat: Still creating... [1m40s elapsed]
aws_route.private_utility_z0_nat: Still creating... [1m50s elapsed]
aws_route.private_utility_z0_nat: Still creating... [2m0s elapsed]

Error finding route after creating it: Unable to find matching route for Route Table (rtb-1234) and destination CIDR block (0.0.0.0/0).
on tf/main.tf line 183, in resource "aws_route" "private_utility_z0_nat":
183: resource "aws_route" "private_utility_z0_nat" 

The aws_route creation times out (the default creation timeout is 2m ). The aws_route is not persistent in the terraform.state and it actually gets created in the cloud provider.
Subsequent terraform apply runs will try to create this aws_route and they will always fail with RouteAlreadyExists .

There are multiple existing issues about Error finding route after creating it: Unable to find matching route for Route Table (rtb-1234) and destination CIDR block (0.0.0.0/0), I linked several ones in hashicorp/terraform-provider-aws#12073. In some of them people report that they increased the creation timeout for the aws_route and that fixed the issue for them.
We also agreed to do that.

[ialidzhikov]

I am closing for now as is merged #38 . Please reopen if it still reproduces.