Destroying google_redis_instance error with access denied, but does actually destroy it

[jonnyshaw89]

The destroying of a google_redis_instance seems to be throwing access errors.
The instance is actually destroyed when you look in the console, but terraform seem to throw and error when trying to check the state of the destroy.

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment
• If an issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to "hashibot", a community member has claimed the issue already.

Terraform Version

Affected Resource(s)

• google_redis_instance

Terraform Configuration Files

resource "google_project_service" "redis" {
  project = "${google_project.project.project_id}"
  service = "redis.googleapis.com"

  disable_dependent_services = true
  disable_on_destroy         = false
}

resource "google_redis_instance" "memorystore" {
  depends_on           = ["google_project_service.redis"]
  name                      = "memory-cache"
  tier                          = "STANDARD_HA"
  memory_size_gb    = "${var.redis_storage_size}"
  redis_version          = "REDIS_4_0"
  reserved_ip_range = "${var.redis_ip_range}"

  authorized_network = "${google_compute_network.k8s.self_link}"

  region  = "${var.region}"
  project = "${google_project.project.project_id}"
}

Expected Behavior

terraform should have destroyed the resource

Actual Behavior

* google_redis_instance.memorystore (destroy): 1 error(s) occurred:

* google_redis_instance.memorystore: Error waiting for Deleting Instance: error while retrieving operation: googleapi: Error 403: Read access to project 'my-project' was denied

[Chupaka]

What about Debug Log?..

[rileykarson]

Interesting! I suspect the API has implemented async operations in an unconventional fashion. As stated by @Chupaka, debug logs will help us dig into this and resolve your issue.

[jonnyshaw89]

Debug output for the DELETE request is here.
https://gist.github.com/jonnyshaw89/b60f7da9ddb75a1f82168dae1554b306

[rileykarson]

Hmm, it doesn't look like we've received an error yet by that point. Do you mind checking to see if the log was cut off?

[jonnyshaw89]

@rileykarson Other than the errors at the top of this Issue the only other output that looks like an error is

2019-04-08T10:07:35.571+0100 [DEBUG] plugin.terraform-provider-random_v2.1.0_x4: 2019/04/08 10:07:35 [ERR] plugin: plugin server: accept unix /var/folders/vb/z8t76nps3td5nzm26ndyvfgc0000gn/T/plugin102802803: use of closed network connection
2019-04-08T10:07:35.571+0100 [DEBUG] plugin.terraform-provider-google_v2.3.0_x4: 2019/04/08 10:07:35 [ERR] plugin: plugin server: accept unix /var/folders/vb/z8t76nps3td5nzm26ndyvfgc0000gn/T/plugin892296168: use of closed network connection

[rileykarson]

Those errors are a result of how Terraform disconnects from the provider when a real error is encountered; the most useful bit in the debug logs is what happens immediately before / after an error is encountered and I think that's been stripped here?

[MebinAbraham]

Was not successful trying to recreate this issue with a simple project. The Redis memory store instance was successfully destroyed proceeded by the google_project_service Redis API.
Therefore it is possible that the way the project was set up could have resulted in the 403 error?

https://github.com/MebinAbraham/terraform-gcp-redis-example

[rileykarson]

The only situation I can think of offhand is if the service account lost permissions to the project / to redis on the project temporarily. @jonnyshaw89 have you continued to encounter this and are you able to consistently reproduce it?

[MebinAbraham]

The only situation I can think of offhand is if the service account lost permissions to the project / to redis on the project temporarily. @jonnyshaw89 have you continued to encounter this and are you able to consistently reproduce it?.

What I should add to the original issue is that, on a consecutive retry of the destroy command, it states that the API is not enabled. Once the API is manually enabled then the destroy commands works fine.

Error: Error refreshing state: 1 error(s) occurred:

* google_redis_instance.memorystore: 1 error(s) occurred:

* google_redis_instance.memorystore: google_redis_instance.memorystore: Error reading RedisInstance "my-project/europe-west2/memory-cache": googleapi: Error 403: Google Cloud Memorystore for Redis API has not been used in project 123456789876 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/redis.googleapis.com/overview?project=123456789876 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.

[MebinAbraham]

Update: This could be closed. The issue was caused because container.googleapis.com API was being disabled too early and was not implicitly/explicitly added as a dependency of memory store.

Adding it as a dependency to memory store fixed the issue. (Referring to the same repo as OP)

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!