Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Terraform crashes while creating Google cloud Endpoint using Cloud build #6840

Closed
mejuhi opened this issue Jul 23, 2020 · 10 comments · Fixed by GoogleCloudPlatform/magic-modules#3781, #6849 or hashicorp/terraform-provider-google-beta#2302
Assignees

Comments

@mejuhi
Copy link

mejuhi commented Jul 23, 2020

Terraform Version

hashicorp/terraform:0.12.24

Affected Resource(s)

  • google_endpoints_service

Terraform Configuration Files

# define GCP project name
variable "gcp_project" {
  type        = string
  description = "GCP project name"
  default = "<my_projectname>"
}

terraform {
  backend "gcs" {
  }
}

# define GCP region
variable "gcp_region" {
  type        = string
  description = "GCP region"
  default = "europe-west1"
}
#Setting variables
provider "google" {
  project     = var.gcp_project
  region      = var.gcp_region
}

resource "google_endpoints_service" "openapi_service" {
    service_name   = "<endpoint_service_name>"
    project        = "<my_projectname>"
    openapi_config = file("openapi-spec1.yaml")
}

Debug Output

starting build "<my-buildno>"

FETCHSOURCE
Initialized empty Git repository in /workspace/.git/
From https://source.developers.google.com/p/<my_projectname>/r/<my_projectrepo>
 * branch            <brinch_hash> -> FETCH_HEAD
HEAD is now at <Head_hash> test
BUILD
Starting Step #0 - "terraform-init"
Step #0 - "terraform-init": Pulling image: hashicorp/terraform:0.12.24
Step #0 - "terraform-init": 0.12.24: Pulling from hashicorp/terraform
Step #0 - "terraform-init": df20fa9351a1: Already exists
Step #0 - "terraform-init": ed8968b2872e: Already exists
Step #0 - "terraform-init": a92cc7c5fd73: Already exists
Step #0 - "terraform-init": 9e0cccf56431: Pulling fs layer
Step #0 - "terraform-init": cbe0275821fc: Pulling fs layer
Step #0 - "terraform-init": 746f3329030a: Pulling fs layer
Step #0 - "terraform-init": 00e5ee36b7a4: Pulling fs layer
Step #0 - "terraform-init": faaffb792239: Pulling fs layer
Step #0 - "terraform-init": 372b6aa41911: Pulling fs layer
Step #0 - "terraform-init": 00e5ee36b7a4: Waiting
Step #0 - "terraform-init": faaffb792239: Waiting
Step #0 - "terraform-init": 372b6aa41911: Waiting
Step #0 - "terraform-init": cbe0275821fc: Verifying Checksum
Step #0 - "terraform-init": cbe0275821fc: Download complete
Step #0 - "terraform-init": 746f3329030a: Verifying Checksum
Step #0 - "terraform-init": 746f3329030a: Download complete
Step #0 - "terraform-init": 00e5ee36b7a4: Verifying Checksum
Step #0 - "terraform-init": 00e5ee36b7a4: Download complete
Step #0 - "terraform-init": 9e0cccf56431: Verifying Checksum
Step #0 - "terraform-init": 9e0cccf56431: Download complete
Step #0 - "terraform-init": 372b6aa41911: Verifying Checksum
Step #0 - "terraform-init": 372b6aa41911: Download complete
Step #0 - "terraform-init": faaffb792239: Verifying Checksum
Step #0 - "terraform-init": faaffb792239: Download complete
Step #0 - "terraform-init": 9e0cccf56431: Pull complete
Step #0 - "terraform-init": cbe0275821fc: Pull complete
Step #0 - "terraform-init": 746f3329030a: Pull complete
Step #0 - "terraform-init": 00e5ee36b7a4: Pull complete
Step #0 - "terraform-init": faaffb792239: Pull complete
Step #0 - "terraform-init": 372b6aa41911: Pull complete
Step #0 - "terraform-init": Digest: sha256:0a9bd4396ed9b835a315ddc0156e3878f155f745001042d1ab996be822a88050
Step #0 - "terraform-init": Status: Downloaded newer image for hashicorp/terraform:0.12.24
Step #0 - "terraform-init": docker.io/hashicorp/terraform:0.12.24
Step #0 - "terraform-init": /workspace/swagger/child
Step #0 - "terraform-init": cloudbuild.yaml
Step #0 - "terraform-init": dev.tfbackend
Step #0 - "terraform-init": gcloud_build_image
Step #0 - "terraform-init": main.tf
Step #0 - "terraform-init": openapi-spec1.yaml
Step #0 - "terraform-init": 
Step #0 - "terraform-init": Initializing the backend...
Step #0 - "terraform-init": 
Step #0 - "terraform-init": Successfully configured the backend "gcs"! Terraform will automatically
Step #0 - "terraform-init": use this backend unless the backend configuration changes.
Step #0 - "terraform-init": 
Step #0 - "terraform-init": Initializing provider plugins...
Step #0 - "terraform-init": 
Step #0 - "terraform-init": The following providers do not have any version constraints in configuration,
Step #0 - "terraform-init": so the latest version was installed.
Step #0 - "terraform-init": 
Step #0 - "terraform-init": To prevent automatic upgrades to new major versions that may contain breaking
Step #0 - "terraform-init": changes, it is recommended to add version = "..." constraints to the
Step #0 - "terraform-init": corresponding provider blocks in configuration, with the constraint strings
Step #0 - "terraform-init": suggested below.
Step #0 - "terraform-init": 
Step #0 - "terraform-init": * provider.google: version = "~> 3.31"
Step #0 - "terraform-init": 
Step #0 - "terraform-init": Terraform has been successfully initialized!
Step #0 - "terraform-init": 
Step #0 - "terraform-init": You may now begin working with Terraform. Try running "terraform plan" to see
Step #0 - "terraform-init": any changes that are required for your infrastructure. All Terraform commands
Step #0 - "terraform-init": should now work.
Step #0 - "terraform-init": 
Step #0 - "terraform-init": If you ever set or change modules or backend configuration for Terraform,
Step #0 - "terraform-init": rerun this command to reinitialize your working directory. If you forget, other
Step #0 - "terraform-init": commands will detect it and remind you to do so if necessary.
Finished Step #0 - "terraform-init"
Starting Step #1 - "terraform-plan"
Step #1 - "terraform-plan": Already have image: hashicorp/terraform:0.12.24
Step #1 - "terraform-plan": Refreshing Terraform state in-memory prior to plan...
Step #1 - "terraform-plan": The refreshed state will be used to calculate this plan, but will not be
Step #1 - "terraform-plan": persisted to local or remote state storage.
Step #1 - "terraform-plan": 
Step #1 - "terraform-plan": 
Step #1 - "terraform-plan": ------------------------------------------------------------------------
Step #1 - "terraform-plan": 
Step #1 - "terraform-plan": Error: rpc error: code = Unavailable desc = transport is closing
Step #1 - "terraform-plan": 
Step #1 - "terraform-plan": 
Step #1 - "terraform-plan": panic: time: missing Location in call to Time.In
Step #1 - "terraform-plan": 2020-07-23T12:58:58.552Z [DEBUG] plugin.terraform-provider-google_v3.31.0_x5: 
Step #1 - "terraform-plan": 2020-07-23T12:58:58.552Z [DEBUG] plugin.terraform-provider-google_v3.31.0_x5: goroutine 38 [running]:
Step #1 - "terraform-plan": 2020-07-23T12:58:58.552Z [DEBUG] plugin.terraform-provider-google_v3.31.0_x5: time.Time.In(...)
Step #1 - "terraform-plan": 2020-07-23T12:58:58.552Z [DEBUG] plugin.terraform-provider-google_v3.31.0_x5: 	/opt/goenv/versions/1.14.3/src/time/time.go:1126
Step #1 - "terraform-plan": 2020-07-23T12:58:58.552Z [DEBUG] plugin.terraform-provider-google_v3.31.0_x5: github.com/terraform-providers/terraform-provider-google/google.predictServiceId(0xc0003d4140, 0x237b500, 0xc000864000, 0xc000919ae0, 0xc0003d4140)
Step #1 - "terraform-plan": 2020-07-23T12:58:58.552Z [DEBUG] plugin.terraform-provider-google_v3.31.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-google/google/resource_endpoints_service.go:157 +0x572
Step #1 - "terraform-plan": 2020-07-23T12:58:58.552Z [DEBUG] plugin.terraform-provider-google_v3.31.0_x5: github.com/hashicorp/terraform-plugin-sdk/helper/schema.schemaMap.Diff(0xc000220ae0, 0xc000dcd5e0, 0xc000dd1620, 0x2830798, 0x237b500, 0xc000864000, 0x2c0cb00, 0x0, 0xffffffffffffffff, 0x0)
Step #1 - "terraform-plan": 2020-07-23T12:58:58.552Z [DEBUG] plugin.terraform-provider-google_v3.31.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk@v1.11.0/helper/schema/schema.go:524 +0xac2
Step #1 - "terraform-plan": 2020-07-23T12:58:58.552Z [DEBUG] plugin.terraform-provider-google_v3.31.0_x5: github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*Resource).simpleDiff(0xc000223440, 0xc000dcd5e0, 0xc000dd1620, 0x237b500, 0xc000864000, 0xc000dd1601, 0xc000803898, 0x40d7dd)
Step #1 - "terraform-plan": 2020-07-23T12:58:58.552Z [DEBUG] plugin.terraform-provider-google_v3.31.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk@v1.11.0/helper/schema/resource.go:356 +0x85
Step #1 - "terraform-plan": 2020-07-23T12:58:58.552Z [DEBUG] plugin.terraform-provider-google_v3.31.0_x5: github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*Provider).SimpleDiff(0xc0000ca800, 0xc000803a80, 0xc000dcd5e0, 0xc000dd1620, 0xc000dd0360, 0xc000dd1620, 0x0)
Step #1 - "terraform-plan": 2020-07-23T12:58:58.552Z [DEBUG] plugin.terraform-provider-google_v3.31.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk@v1.11.0/helper/schema/provider.go:321 +0x99
Step #1 - "terraform-plan": 2020-07-23T12:58:58.552Z [DEBUG] plugin.terraform-provider-google_v3.31.0_x5: github.com/hashicorp/terraform-plugin-sdk/internal/helper/plugin.(*GRPCProviderServer).PlanResourceChange(0xc00000f4f8, 0x2c0be40, 0xc000dd0300, 0xc0003a4000, 0xc00000f4f8, 0xc000dd0300, 0xc00080eb88)
Step #1 - "terraform-plan": 2020-07-23T12:58:58.552Z [DEBUG] plugin.terraform-provider-google_v3.31.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk@v1.11.0/internal/helper/plugin/grpc_provider.go:633 +0x79c
Step #1 - "terraform-plan": 2020-07-23T12:58:58.552Z [DEBUG] plugin.terraform-provider-google_v3.31.0_x5: github.com/hashicorp/terraform-plugin-sdk/internal/tfplugin5._Provider_PlanResourceChange_Handler(0x26ad540, 0xc00000f4f8, 0x2c0be40, 0xc000dd0300, 0xc0007e2a20, 0x0, 0x2c0be40, 0xc000dd0300, 0xc00084b900, 0x1289)
Step #1 - "terraform-plan": 2020-07-23T12:58:58.552Z [DEBUG] plugin.terraform-provider-google_v3.31.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk@v1.11.0/internal/tfplugin5/tfplugin5.pb.go:3287 +0x217
Step #1 - "terraform-plan": 2020-07-23T12:58:58.552Z [DEBUG] plugin.terraform-provider-google_v3.31.0_x5: google.golang.org/grpc.(*Server).processUnaryRPC(0xc0005ce1a0, 0x2c3f180, 0xc000396780, 0xc00079a000, 0xc0005249f0, 0x3b49b68, 0x0, 0x0, 0x0)
Step #1 - "terraform-plan": 2020-07-23T12:58:58.552Z [DEBUG] plugin.terraform-provider-google_v3.31.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.28.0/server.go:1082 +0x50a
Step #1 - "terraform-plan": 2020-07-23T12:58:58.552Z [DEBUG] plugin.terraform-provider-google_v3.31.0_x5: google.golang.org/grpc.(*Server).handleStream(0xc0005ce1a0, 0x2c3f180, 0xc000396780, 0xc00079a000, 0x0)
Step #1 - "terraform-plan": 2020-07-23T12:58:58.552Z [DEBUG] plugin.terraform-provider-google_v3.31.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.28.0/server.go:1405 +0xcc9
Step #1 - "terraform-plan": 2020-07-23T12:58:58.552Z [DEBUG] plugin.terraform-provider-google_v3.31.0_x5: google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc000034b70, 0xc0005ce1a0, 0x2c3f180, 0xc000396780, 0xc00079a000)
Step #1 - "terraform-plan": 2020-07-23T12:58:58.552Z [DEBUG] plugin.terraform-provider-google_v3.31.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.28.0/server.go:746 +0xa1
Step #1 - "terraform-plan": 2020-07-23T12:58:58.552Z [DEBUG] plugin.terraform-provider-google_v3.31.0_x5: created by google.golang.org/grpc.(*Server).serveStreams.func1
Step #1 - "terraform-plan": 2020-07-23T12:58:58.552Z [DEBUG] plugin.terraform-provider-google_v3.31.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.28.0/server.go:744 +0xa1
Step #1 - "terraform-plan": 2020/07/23 12:58:58 [ERROR] <root>: eval: *terraform.EvalDiff, err: rpc error: code = Unavailable desc = transport is closing
Step #1 - "terraform-plan": 2020/07/23 12:58:58 [ERROR] <root>: eval: *terraform.EvalSequence, err: rpc error: code = Unavailable desc = transport is closing
Step #1 - "terraform-plan": 2020/07/23 12:58:58 [TRACE] [walkPlan] Exiting eval tree: google_endpoints_service.openapi_service
Step #1 - "terraform-plan": 2020/07/23 12:58:58 [TRACE] vertex "google_endpoints_service.openapi_service": visit complete
Step #1 - "terraform-plan": 2020/07/23 12:58:58 [TRACE] vertex "google_endpoints_service.openapi_service": dynamic subgraph encountered errors
Step #1 - "terraform-plan": 2020/07/23 12:58:58 [TRACE] vertex "google_endpoints_service.openapi_service": visit complete
Step #1 - "terraform-plan": 2020/07/23 12:58:58 [TRACE] dag/walk: upstream of "provider.google (close)" errored, so skipping
Step #1 - "terraform-plan": 2020/07/23 12:58:58 [TRACE] dag/walk: upstream of "meta.count-boundary (EachMode fixup)" errored, so skipping
Step #1 - "terraform-plan": 2020/07/23 12:58:58 [TRACE] dag/walk: upstream of "root" errored, so skipping
Step #1 - "terraform-plan": 2020/07/23 12:58:58 [INFO] backend/local: plan operation completed
Step #1 - "terraform-plan": 2020-07-23T12:58:58.558Z [DEBUG] plugin: plugin process exited: path=/workspace/swagger/child/.terraform/plugins/linux_amd64/terraform-provider-google_v3.31.0_x5 pid=32 error="exit status 2"
Step #1 - "terraform-plan": 2020-07-23T12:58:58.726Z [DEBUG] plugin: plugin exited
Step #1 - "terraform-plan": 
Step #1 - "terraform-plan": 
Step #1 - "terraform-plan": 
Step #1 - "terraform-plan": !!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!
Step #1 - "terraform-plan": 
Step #1 - "terraform-plan": Terraform crashed! This is always indicative of a bug within Terraform.
Step #1 - "terraform-plan": A crash log has been placed at "crash.log" relative to your current
Step #1 - "terraform-plan": working directory. It would be immensely helpful if you could please
Step #1 - "terraform-plan": report the crash with Terraform[1] so that we can fix this.
Step #1 - "terraform-plan": 
Step #1 - "terraform-plan": When reporting bugs, please include your terraform version. That
Step #1 - "terraform-plan": information is available on the first line of crash.log. You can also
Step #1 - "terraform-plan": get it by running 'terraform --version' on the command line.
Step #1 - "terraform-plan": 
Step #1 - "terraform-plan": SECURITY WARNING: the "crash.log" file that was created may contain 
Step #1 - "terraform-plan": sensitive information that must be redacted before it is safe to share 
Step #1 - "terraform-plan": on the issue tracker.
Step #1 - "terraform-plan": 
Step #1 - "terraform-plan": [1]: https://github.com/hashicorp/terraform/issues
Step #1 - "terraform-plan": 
Step #1 - "terraform-plan": !!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!
Finished Step #1 - "terraform-plan"
ERROR
ERROR: build step 1 "hashicorp/terraform:0.12.24" failed: step exited with non-zero status: 1

Expected Behavior

Create Google Cloud endpoint from the openapi swagger file given as input

Actual Behavior

"hashicorp/terraform:0.12.24" failed: step exited with non-zero status: 1

Steps to Reproduce

Run terraform plan/apply using Cloud build

- id: 'terraform-plan'
  name: hashicorp/terraform:0.12.24
  dir: 'swagger/child'
  entrypoint: 'sh'
  args: 
  - '-c'
  - | 
    chmod 777 openapi-spec1.yaml
    ls -latr
    terraform plan

Important Factoids

  1. Terraform (main.tf) works perfectly fine when i am trying to run terraform apply using my cloud shell (locally)
  2. Issue can be reproduced when trying to perform terraform plan and terraform apply using cloud build
  3. I have added Project editor role on the service account used by cloud build (same level of permission as that of service account i used to run locally)
  4. Also tried to give open-api swagger file directly by
resource "google_endpoints_service" "openapi_service" {
    service_name   = "<endpoint_service_name>"
    project        = "<my_projectname>"
    openapi_config =<<EOF
swagger:'2.0'
info:
  title:Pre-pairing
  description: API on CloudEndpointswith a GoogleCloudFunctions backend...
  version:1.0.0# Same applies to the host. It should be, AFAIK, Cloud Run's URL.
host:"${google_cloud_run_service.cloud-run.status.url}"# <--------[...]
EOF
}  

@ghost ghost added bug crash labels Jul 23, 2020
@edwardmedia edwardmedia self-assigned this Jul 23, 2020
@edwardmedia
Copy link
Contributor

@mejuhi could you post the file crash.log? It is specified in the section of TERRAFORM CRASH. Thanks

@mejuhi
Copy link
Author

mejuhi commented Jul 23, 2020

Yes, following is the crash log received.
crash.log

@mejuhi
Copy link
Author

mejuhi commented Jul 24, 2020

Wanted to know timeline for the updated code.
Image tag to be referred and when can it be available for using it in the cloud build.
@slevenick , @edwardmedia

@slevenick
Copy link
Collaborator

slevenick commented Jul 24, 2020

Wanted to know timeline for the updated code.
Image tag to be referred and when can it be available for using it in the cloud build.
@slevenick , @edwardmedia

It should be available in the 3.33.0 release expected august 3rd

You can compile the provider yourself if a fix is needed prior to then. The fix is in master

@mejuhi
Copy link
Author

mejuhi commented Jul 24, 2020

Wanted to know timeline for the updated code.
Image tag to be referred and when can it be available for using it in the cloud build.
@slevenick , @edwardmedia

It should be available in the 3.33.0 release expected august 3rd

You can compile the provider yourself if a fix is needed prior to then. The fix is in master

Thankyou @slevenick for the quick response. It was extremely helpful.
I am using terraform image from https://hub.docker.com/r/hashicorp/terraform/tags
I will wait for the docker image to be updated with the fixed code. I hope fix will be available on docker image on the date suggested.

@Arconapalus
Copy link

Arconapalus commented Jul 28, 2020

I'm having the same issue when deploying cloud endpoints. Terraform crash when terraform plan is ran in cloudbuild.
Also, I would like to pass in vars to cloud endpoints from cloud run url's. I have tested this locally and works fine, just to clarify that it works in cloudbuild. I'm passing the cloud run url's in root module via outputs.tf then, as vars to cloud endpoints then, to swagger.yml (template_file).
I don't if this would affect the build but would like to get it out there.

Another question, how do you access the crash.log files when running TerraForm in CloudBuild?

@mejuhi
Copy link
Author

mejuhi commented Jul 29, 2020

@Arconapalus To answer your first question, I think they have fixed the issue in master and as suggested by @slevenick we can expect a release on 3rd August. (Not sure when the docker image will be updated, I am waiting for that as well).

To answer the second part of the question, I did just cat on crash.log after terraform plan command.
Hope this helps

@Arconapalus
Copy link

Right on thanks!!

Yes that worked for me thank you

@mejuhi
Copy link
Author

mejuhi commented Aug 6, 2020

@Arconapalus I tried again, issue is fixed, you can use docker image and try it as well

@ghost
Copy link

ghost commented Aug 23, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!

@ghost ghost locked and limited conversation to collaborators Aug 23, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.