tests/api_gateway_domain_name: Inconsistent result on Terraform 0.15.0-alpha

[YakDriver]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Acceptance Test

TestAccAWSAPIGatewayDomainName_MutualTlsAuthentication produces inconsistent results after apply on Terraform v0.15.0-alpha:

    resource_aws_api_gateway_domain_name_test.go:299: Step 1/3 error: Error running apply: exit status 1
        
        Error: Provider produced inconsistent result after apply
        
        When applying changes to aws_s3_bucket_object.test, provider
        "provider[\"registry.terraform.io/hashicorp/aws\"]" produced an unexpected
        new value: Root resource was present, but now absent.
        
        This is a bug in the provider, which should be reported in the provider's own
        issue tracker.
        
--- FAIL: TestAccAWSAPIGatewayDomainName_MutualTlsAuthentication (102.25s)

Affected Resources

• aws_api_gateway_domain_name

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

resource "aws_s3_bucket" "test" {
  bucket = "yaket"

  force_destroy = true

  versioning {
    enabled = true
  }
}

resource "aws_s3_bucket_object" "test" {
  bucket = aws_s3_bucket.test.id
  key    = "yaket"
  source = "test-fixtures/apigateway-domain-name-truststore-1.pem"
}

resource "aws_api_gateway_domain_name" "test" {
  domain_name              = aws_acm_certificate.test.domain_name
  regional_certificate_arn = aws_acm_certificate_validation.test.certificate_arn
  security_policy          = "TLS_1_2"

  endpoint_configuration {
    types = ["REGIONAL"]
  }

  mutual_tls_authentication {
    truststore_uri     = "s3://${aws_s3_bucket_object.test.bucket}/${aws_s3_bucket_object.test.key}"
    truststore_version = aws_s3_bucket_object.test.version_id
  }
}

References

• https://github.com/hashicorp/terraform/releases/tag/v0.15.0-alpha20210210

[apparentlymart]

Hi @YakDriver,

In the past when I've seen this error it's been caused by an eventual consistency problem where the provider tries to read back what it just created as part of returning from Create (the typical return resourceS3BucketRead(...) pattern) but the remote API doesn't have read-after-write consistency and so the read ends up producing a "not found", signaled to Terraform Core by returning a null object, which is what in turn causes Terraform to report the "now absent" in this error message.

Do you think that could be a possible explanation in this case? It does seem like this particular resource type is using the aforementioned pattern:

terraform-provider-aws/aws/resource_aws_s3_bucket_object.go

Lines 293 to 294 in 7546a7f

	d.SetId(key)
	return resourceAwsS3BucketObjectRead(d, meta)

Based on the other code I see in there, I think we could confirm it by capturing the provider logs prior to that error to see if we're entering this branch:

terraform-provider-aws/aws/resource_aws_s3_bucket_object.go

Lines 316 to 320 in 7546a7f

	if awsErr, ok := err.(awserr.RequestFailure); ok && awsErr.StatusCode() == 404 {
	d.SetId("")
	log.Printf("[WARN] Error Reading Object (%s), object not found (HTTP status 404)", key)
	return nil
	}

I'm happy to look a bit deeper on the core side if it seems like something different is happening here, of course!

[bflad]

A simple resource change to show a Terraform AWS Provider error instead of the Terraform CLI error in this case is to implement d.IsNewResource() checks before potentially returning d.SetId("") on creation, e.g. #16796

To also mitigate the eventual consistency issues, a pattern such as Resource Lifecycle Retries can be adopted.

[YakDriver]

Cool. Thanks for the input! It looks like straightforward fixes needed on the AWS provider side.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.