Error enabling bigquery.googleapis.com

[Stono]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment
• If an issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to "hashibot", a community member has claimed the issue already.

Hi,
We are seeing an issue where terraform plans are always trying to enable the biquery api on our project, even though it is already enabled. Even if we push through to an apply, it never completes (with the errors seen below).

Terraform Version

terraform: 0.12.9, provider: 2.16.0

Affected Resource(s)

• google_project_services

Terraform Configuration Files

Debug Output

We see:

2019-10-01T13:14:17.496Z [******] plugin.terraform-provider-google_v2.16.0_x4: 2019/10/01 13:14:17 [******] Creating new batch "Enable Project Services at-infrastructure-sandbox: [bigquery.googleapis.com]" from request "project/at-infrastructure-sandbox/services:batchEnable"

And then we just see it polling:

2019-10-01T13:14:33.360Z [******] plugin.terraform-provider-google_v2.16.0_x4: 2019/10/01 13:14:33 [******] Dismissed an error as retryable based on error code: googleapi: Error 503: The service(s) ["bigquery.googleapis.com"] are still being enabled for project at-infrastructure-omnienv-dev. This isn't a real API error, this is just eventual consistency.
2019-10-01T13:14:33.360Z [******] plugin.terraform-provider-google_v2.16.0_x4: 2019/10/01 13:14:33 [TRACE] Waiting 1s before next try
2019-10-01T13:14:33.362Z [******] plugin.terraform-provider-google_v2.16.0_x4: 2019/10/01 13:14:33 [******] Dismissed an error as retryable based on error code: googleapi: Error 503: The service(s) ["bigquery.googleapis.com"] are still being enabled for project at-signals-platform-dev. This isn't a real API error, this is just eventual consistency.
2019-10-01T13:14:33.362Z [******] plugin.terraform-provider-google_v2.16.0_x4: 2019/10/01 13:14:33 [TRACE] Waiting 1s before next try
2019-10-01T13:14:33.438Z [******] plugin.terraform-provider-google_v2.16.0_x4: 2019/10/01 13:14:33 [******] Dismissed an error as retryable based on error code: googleapi: Error 503: The service(s) ["bigquery.googleapis.com"] are still being enabled for project at-delivery-platform-testing. This isn't a real API error, this is just eventual consistency.
2019-10-01T13:14:33.438Z [******] plugin.terraform-provider-google_v2.16.0_x4: 2019/10/01 13:14:33 [TRACE] Waiting 1s before next try
2019-10-01T13:14:33.445Z [******] plugin.terraform-provider-google_v2.16.0_x4: 2019/10/01 13:14:33 [******] Dismissed an error as retryable based on error code: googleapi: Error 503: The service(s) ["bigquery.googleapis.com"] are still being enabled for project at-data-platform-preprod. This isn't a real API error, this is just eventual consistency.
2019-10-01T13:14:33.445Z [******] plugin.terraform-provider-google_v2.16.0_x4: 2019/10/01 13:14:33 [TRACE] Waiting 1s before next try
2019-10-01T13:14:33.445Z [******] plugin.terraform-provider-google_v2.16.0_x4: 2019/10/01 13:14:33 [******] Dismissed an error as retryable based on error code: googleapi: Error 503: The service(s) ["bigquery.googleapis.com"] are still being enabled for project at-delivery-platform-dev. This isn't a real API error, this is just eventual consistency.
2019-10-01T13:14:33.445Z [******] plugin.terraform-provider-google_v2.16.0_x4: 2019/10/01 13:14:33 [TRACE] Waiting 1s before next try
2019-10-01T13:14:33.560Z [******] plugin.terraform-provider-google_v2.16.0_x4: 2019/10/01 13:14:33 [******] Google API Response Details:

We can see the action was performed in the GCP Audit:

And the BigQuery API is enabled

I find interesting in the debug output too its asking for a list of enabled api's and that doesn't seem to include bigquery (probably explains the reason it's showing in the plan):

2019-10-01T13:16:25.742Z [******] plugin.terraform-provider-google_v2.16.0_x4: -----------------------------------------------------
2019-10-01T13:16:26.110Z [******] plugin.terraform-provider-google_v2.16.0_x4: 2019/10/01 13:16:26 [******] Google API Request Details:
2019-10-01T13:16:26.110Z [******] plugin.terraform-provider-google_v2.16.0_x4: ---[ REQUEST ]---------------------------------------
2019-10-01T13:16:26.110Z [******] plugin.terraform-provider-google_v2.16.0_x4: GET /v1/projects/at-infrastructure-sandbox?alt=json&prettyPrint=false HTTP/1.1
2019-10-01T13:16:26.110Z [******] plugin.terraform-provider-google_v2.16.0_x4: Host: cloudresourcemanager.googleapis.com
2019-10-01T13:16:26.110Z [******] plugin.terraform-provider-google_v2.16.0_x4: User-Agent: google-api-go-client/0.5 HashiCorp Terraform/0.12.9 (+https://www.terraform.io) terraform-provider-google/2.16.0
2019-10-01T13:16:26.110Z [******] plugin.terraform-provider-google_v2.16.0_x4: Accept-Encoding: gzip
2019-10-01T13:16:26.110Z [******] plugin.terraform-provider-google_v2.16.0_x4: 
2019-10-01T13:16:26.110Z [******] plugin.terraform-provider-google_v2.16.0_x4: 
2019-10-01T13:16:26.110Z [******] plugin.terraform-provider-google_v2.16.0_x4: -----------------------------------------------------
2019-10-01T13:16:26.168Z [******] plugin.terraform-provider-google_v2.16.0_x4: 2019/10/01 13:16:26 [******] Google API Response Details:
2019-10-01T13:16:26.169Z [******] plugin.terraform-provider-google_v2.16.0_x4: ---[ RESPONSE ]--------------------------------------
2019-10-01T13:16:26.169Z [******] plugin.terraform-provider-google_v2.16.0_x4: HTTP/2.0 200 OK
2019-10-01T13:16:26.169Z [******] plugin.terraform-provider-google_v2.16.0_x4: Cache-Control: private
2019-10-01T13:16:26.169Z [******] plugin.terraform-provider-google_v2.16.0_x4: Content-Type: application/json; charset=UTF-8
2019-10-01T13:16:26.169Z [******] plugin.terraform-provider-google_v2.16.0_x4: Date: Tue, 01 Oct 2019 13:16:26 GMT
2019-10-01T13:16:26.169Z [******] plugin.terraform-provider-google_v2.16.0_x4: Server: ESF
2019-10-01T13:16:26.169Z [******] plugin.terraform-provider-google_v2.16.0_x4: Vary: Origin
2019-10-01T13:16:26.169Z [******] plugin.terraform-provider-google_v2.16.0_x4: Vary: X-Origin
2019-10-01T13:16:26.169Z [******] plugin.terraform-provider-google_v2.16.0_x4: Vary: Referer
2019-10-01T13:16:26.169Z [******] plugin.terraform-provider-google_v2.16.0_x4: X-Content-Type-Options: nosniff
2019-10-01T13:16:26.169Z [******] plugin.terraform-provider-google_v2.16.0_x4: X-Frame-Options: SAMEORIGIN
2019-10-01T13:16:26.169Z [******] plugin.terraform-provider-google_v2.16.0_x4: X-Xss-Protection: 0
2019-10-01T13:16:26.169Z [******] plugin.terraform-provider-google_v2.16.0_x4: 
2019-10-01T13:16:26.169Z [******] plugin.terraform-provider-google_v2.16.0_x4: {
2019-10-01T13:16:26.169Z [******] plugin.terraform-provider-google_v2.16.0_x4:  "services": [
2019-10-01T13:16:26.169Z [******] plugin.terraform-provider-google_v2.16.0_x4:   {
2019-10-01T13:16:26.169Z [******] plugin.terraform-provider-google_v2.16.0_x4:    "name": "projects/881068955597/services/analytics.googleapis.com"
2019-10-01T13:16:26.169Z [******] plugin.terraform-provider-google_v2.16.0_x4:   },
2019-10-01T13:16:26.169Z [******] plugin.terraform-provider-google_v2.16.0_x4:   {
2019-10-01T13:16:26.169Z [******] plugin.terraform-provider-google_v2.16.0_x4:    "name": "projects/881068955597/services/bigquery-json.googleapis.com"
2019-10-01T13:16:26.169Z [******] plugin.terraform-provider-google_v2.16.0_x4:   },
2019-10-01T13:16:26.169Z [******] plugin.terraform-provider-google_v2.16.0_x4:   {
2019-10-01T13:16:26.169Z [******] plugin.terraform-provider-google_v2.16.0_x4:    "name": "projects/881068955597/services/bigquerydatatransfer.googleapis.com"
2019-10-01T13:16:26.169Z [******] plugin.terraform-provider-google_v2.16.0_x4:   },
2019-10-01T13:16:26.169Z [******] plugin.terraform-provider-google_v2.16.0_x4:   {
2019-10-01T13:16:26.169Z [******] plugin.terraform-provider-google_v2.16.0_x4:    "name": "projects/881068955597/services/bigquerystorage.googleapis.com"
2019-10-01T13:16:26.169Z [******] plugin.terraform-provider-google_v2.16.0_x4:   },
2019-10-01T13:16:26.169Z [******] plugin.terraform-provider-google_v2.16.0_x4:   {
2019-10-01T13:16:26.169Z [******] plugin.terraform-provider-google_v2.16.0_x4:    "name": "projects/881068955597/services/bigtable.googleapis.com"
2019-10-01T13:16:26.169Z [******] plugin.terraform-provider-google_v2.16.0_x4:   },
2019-10-01T13:16:26.169Z [******] plugin.terraform-provider-google_v2.16.0_x4:   {
2019-10-01T13:16:26.169Z [******] plugin.terraform-provider-google_v2.16.0_x4:    "name": "projects/881068955597/services/bigtableadmin.googleapis.com"
2019-10-01T13:16:26.169Z [******] plugin.terraform-provider-google_v2.16.0_x4:   },
2019-10-01T13:16:26.169Z [******] plugin.terraform-provider-google_v2.16.0_x4:   {

Panic Output

Expected Behavior

Terraform should manage the apis deterministically

Actual Behavior

We're unable to complete a terraform apply as the api modification times out after 20 minutes with the following error:

Error: Error authoritatively enabling Project at-admin-project Services: unable to enable Project Services at-admin-project ([bigquerystorage.googleapis.com storage-component.googleapis.com iam.googleapis.com oslogin.googleapis.com bigquery-json.googleapis.com containerregistry.googleapis.com serviceusage.googleapis.com pubsub.googleapis.com cloudkms.googleapis.com compute.googleapis.com servicemanagement.googleapis.com sqladmin.googleapis.com dns.googleapis.com deploymentmanager.googleapis.com replicapool.googleapis.com redis.googleapis.com replicapoolupdater.googleapis.com resourceviews.googleapis.com cloudtrace.googleapis.com monitoring.googleapis.com bigquery.googleapis.com stackdriver.googleapis.com logging.googleapis.com iamcredentials.googleapis.com cloudresourcemanager.googleapis.com container.googleapis.com storage-api.googleapis.com cloudbilling.googleapis.com]): Request project/at-admin-project/services:batchEnable timed out after 20m0s

  on at-admin-project.tf line 169, in resource "google_project_services" "admin-enabled-apis":
 169: resource "google_project_services" "admin-enabled-apis" {

Steps to Reproduce

1. terraform apply

Important Factoids

References

• #0000

[Stono]

OK so this might be a google side problem, as you can see, bigquery isn't even being returned from the gcloud cli:

❯ gcloud services list --project=at-delivery-platform-testing
NAME                                    TITLE
bigquery-json.googleapis.com            BigQuery API
bigquerystorage.googleapis.com          BigQuery Storage API
cloud-file-service-gcp.elastifile.com   Elastifile Cloud File Service
cloudfunctions.googleapis.com           Cloud Functions API
cloudkms.googleapis.com                 Cloud Key Management Service (KMS) API
cloudresourcemanager.googleapis.com     Cloud Resource Manager API
cloudtrace.googleapis.com               Stackdriver Trace API
compute.googleapis.com                  Compute Engine API
container.googleapis.com                Kubernetes Engine API
containerregistry.googleapis.com        Container Registry API
dataflow.googleapis.com                 Dataflow API
deploymentmanager.googleapis.com        Cloud Deployment Manager V2 API
dialogflow.googleapis.com               Dialogflow API
distance-matrix-backend.googleapis.com  Distance Matrix API
dns.googleapis.com                      Google Cloud DNS API
firebaserules.googleapis.com            Firebase Rules API
firestore.googleapis.com                Cloud Firestore API
geocoding-backend.googleapis.com        Geocoding API
geolocation.googleapis.com              Geolocation API
iam.googleapis.com                      Identity and Access Management (IAM) API
iamcredentials.googleapis.com           IAM Service Account Credentials API
logging.googleapis.com                  Stackdriver Logging API
monitoring.googleapis.com               Stackdriver Monitoring API
oslogin.googleapis.com                  Cloud OS Login API
pubsub.googleapis.com                   Cloud Pub/Sub API
redis.googleapis.com                    Google Cloud Memorystore for Redis API
replicapool.googleapis.com              Compute Engine Instance Group Manager API
replicapoolupdater.googleapis.com       Compute Engine Instance Group Updater API
resourceviews.googleapis.com            Compute Engine Instance Groups API
servicemanagement.googleapis.com        Service Management API
servicenetworking.googleapis.com        Service Networking API
serviceusage.googleapis.com             Service Usage API
source.googleapis.com                   Legacy Cloud Source Repositories API
sqladmin.googleapis.com                 Cloud SQL Admin API
stackdriver.googleapis.com              Stackdriver API
stackdriverprovisioning.googleapis.com  Stackdriver Provisioning Service
storage-api.googleapis.com              Google Cloud Storage JSON API
storage-component.googleapis.com        Cloud Storage
websecurityscanner.googleapis.com       Web Security Scanner API

[rileykarson]

Hey @Stono! This happened because of some changes upstream that made bigquery.googleapis.com the new name for bigquery-json.googleapis.com. It's been since rolled back when it caused issues for Terraform users, and we're working with that team to mitigate the impact on Terraform users before they roll the change forward again.

I'm going to use this issue to track that. In the meantime, if you've defined bigquery.googleapis.com in config, you can remove it. bigquery-json.googleapis.com remains the service's canonical name for now. If you're encountering this issue and that value isn't in your config, please let me know.

[Stono]

Thanks a lot @rileykarson - I've removed bigquery.googleapis.com and things seem OK.

We've had a right nightmare because of this, and accidentally disabled bigquery api on some of our projects as a result when the upstream change was made! Do you have any more details with regards to that change or where Google announced the rename so I can use it in a post mortem?

[stepanstipl]

thanks @rileykarson, this caused us issues as well - originally, and now again when it was rolled back :). I was wondering if there are any release notes or more details about this change available anywhere? (I was not able to find any mention)

[rileykarson]

I'm not aware of any release notes being published- I'll update this issue if I'm made aware of any. I'll also update the TPG/TPGB changelogs to note that users may have encountered a disruption.

[rileykarson]

This should be mitigated in release 2.17.0 onwards. Specifying either service (bigquery-json.googleapis.com or bigquery.googleapis.com) in google_project_service and google_project_services should work (or both services for _services).

---

Effectively, Terraform will treat the values as equivalent. It'll retry failed requests using the alternate value in case one of the values no longer works. If there's a real reason for failure and both fail, it'll return the errors.

Upon reading either value from the API, Terraform will record what you've defined in your config in state. If you've defined neither, Terraform will write the bigquery-json.googleapis.com value. Transitioning between the values will show a diff, but Terraform should take no action. Once you apply the update, Terraform will have the correct value recorded in state.

Additionally, I'm going to disallow the bigquery-json.googleapis.com service in google_project_service from 3.0.0 on #4624.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!