Skip to content

Commit

Permalink
tests/resource/aws_dx_gateway: Skip when associations (likely from ot…
Browse files Browse the repository at this point in the history
…her regions) remain as the gateway will fail to delete

This sweeper is dependent on the Direct Connect Gateway Association sweeper to run prior, which may skip associations that are associated in a different region than the service client. This will leave lingering associations. Direct Connect Gateways with remaining associations will accept the DeleteDirectConnectGateway API call, enter the deleting state, then after a few minutes revert back to the available state. This causes the sweeper to fail, e.g.

```
[01:36:55][Step 2/4] 2019/09/24 01:36:55 [WARN] WaitForState timeout after 20m0s
[01:36:55][Step 2/4] 2019/09/24 01:36:55 [WARN] WaitForState starting 30s refresh grace period
[01:36:55][Step 2/4] 2019/09/24 01:36:55 [ERR] error running (aws_dx_gateway): error waiting for Direct Connect Gateway (ce06a898-143e-4740-a219-aa32bf564cca) to be deleted: timeout while waiting for state to become 'deleted' (last state: 'available', timeout: 20m0s)
```

Here we skip deleting the Direct Connect Gateway if associations remain as its likely the sweeper will run in the other region and later perform the deletion successfully.

Output from sweeper:

```console
$ go test ./aws -v -sweep=us-east-1,us-west-2 -sweep-run=aws_dx_gateway -timeout 10h
2019/09/23 21:59:16 [DEBUG] Running Sweepers for region (us-east-1):
...
2019/09/23 21:59:18 [DEBUG] Sweeper (aws_dx_gateway_association) has dependency (aws_dx_gateway_association_proposal), running..
2019/09/23 21:59:18 [DEBUG] Sweeper (aws_dx_gateway_association_proposal) already ran in region (us-east-1)
...
2019/09/23 21:59:19 [INFO] Skipping Direct Connect Gateway (ce06a898-143e-4740-a219-aa32bf564cca) Association (vgw-0c30b4b34cfbbeda0) in different home region: us-west-2
2019/09/23 21:59:19 [INFO] Skipping Direct Connect Gateway (ce06a898-143e-4740-a219-aa32bf564cca) Association (vgw-0d6cb1878d8b05231) in different home region: us-west-2
2019/09/23 21:59:19 [INFO] Skipping Direct Connect Gateway (f279f38b-b6a6-4cdf-a376-9f1c5d99a4a4) Association (vgw-0202cd084f386200b) in different home region: us-west-2
2019/09/23 21:59:19 [INFO] Skipping Direct Connect Gateway (f279f38b-b6a6-4cdf-a376-9f1c5d99a4a4) Association (vgw-042795456737cc447) in different home region: us-west-2
2019/09/23 21:59:19 [INFO] Skipping Direct Connect Gateway (09a5ef37-651c-45ac-bab0-1e946dadf958) Association (vgw-087f7eb1fe70dcadb) in different home region: us-west-2
2019/09/23 21:59:19 [INFO] Skipping Direct Connect Gateway (cf2a0cfe-663e-4bb4-bedc-fb5535f46bd9) Association (vgw-06cdd41cfff5fae99) in different home region: us-west-2
2019/09/23 21:59:19 [DEBUG] Sweeper (aws_dx_gateway) has dependency (aws_dx_gateway_association), running..
2019/09/23 21:59:19 [DEBUG] Sweeper (aws_dx_gateway_association) has dependency (aws_dx_gateway_association_proposal), running..
2019/09/23 21:59:19 [DEBUG] Sweeper (aws_dx_gateway_association_proposal) already ran in region (us-east-1)
2019/09/23 21:59:19 [DEBUG] Sweeper (aws_dx_gateway_association) already ran in region (us-east-1)
...
2019/09/23 21:59:20 [INFO] Skipping Direct Connect Gateway with remaining associations: ce06a898-143e-4740-a219-aa32bf564cca
2019/09/23 21:59:20 [INFO] Skipping Direct Connect Gateway with remaining associations: f279f38b-b6a6-4cdf-a376-9f1c5d99a4a4
2019/09/23 21:59:20 [INFO] Skipping Direct Connect Gateway with remaining associations: 09a5ef37-651c-45ac-bab0-1e946dadf958
2019/09/23 21:59:20 [INFO] Skipping Direct Connect Gateway with remaining associations: cf2a0cfe-663e-4bb4-bedc-fb5535f46bd9
2019/09/23 21:59:20 Sweeper Tests ran:
	- aws_dx_gateway_association_proposal
	- aws_dx_gateway_association
	- aws_dx_gateway
```

As for deleting the associations in the correct region, the Direct Connect API is currently throwing server side errors:

```
2019/09/23 22:07:56 [ERR] error running (aws_dx_gateway_association): error deleting Direct Connect Gateway (ce06a898-143e-4740-a219-aa32bf564cca) Association (vgw-0d6cb1878d8b05231): DirectConnectServerException: Unable to process request
```

If they persist, we will need to open an AWS Support case. For now, I have manually deleted the associations via the console, which does not have this issue.
  • Loading branch information
bflad committed Sep 24, 2019
1 parent 0d98f41 commit 48f1b1a
Showing 1 changed file with 33 additions and 0 deletions.
33 changes: 33 additions & 0 deletions aws/resource_aws_dx_gateway_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,39 @@ func testSweepDirectConnectGateways(region string) error {
continue
}

var associations bool
associationInput := &directconnect.DescribeDirectConnectGatewayAssociationsInput{
DirectConnectGatewayId: gateway.DirectConnectGatewayId,
}

for {
associationOutput, err := conn.DescribeDirectConnectGatewayAssociations(associationInput)

if err != nil {
return fmt.Errorf("error retrieving Direct Connect Gateway (%s) Associations: %s", id, err)
}

// If associations still remain, its likely that our region is not the home
// region of those associations and the previous sweepers skipped them.
// When we hit this condition, we skip trying to delete the gateway as it
// will go from deleting -> available after a few minutes and timeout.
if len(associationOutput.DirectConnectGatewayAssociations) > 0 {
associations = true
break
}

if aws.StringValue(associationOutput.NextToken) == "" {
break
}

associationInput.NextToken = associationOutput.NextToken
}

if associations {
log.Printf("[INFO] Skipping Direct Connect Gateway with remaining associations: %s", id)
continue
}

input := &directconnect.DeleteDirectConnectGatewayInput{
DirectConnectGatewayId: aws.String(id),
}
Expand Down

0 comments on commit 48f1b1a

Please sign in to comment.