Limit of 25 API services saved in remote state for google_project_services resource (Blocker)

[joe-boyce]

Hi,

It looks as though there is a limit of 25 google api services which can be enabled with terraform.
After running a terraform plan & terraform apply the services are enabled but when you then run a terraform plan again there are only 25 services saved within the remote state file causing terraform to output a diff as Google has the full list enabled.

Example output:

~module.project.google_project_services.project
services.#: "25" => "29"
services.1109577435: "cloudbuild.googleapis.com" => "cloudbuild.googleapis.com"
services.133405307: "" => "storage-component.googleapis.com"
services.1462818101: "endpoints.googleapis.com" => "endpoints.googleapis.com"
services.1560437671: "iam.googleapis.com" => "iam.googleapis.com"
services.1610229196: "bigquery-json.googleapis.com" => "bigquery-json.googleapis.com"
services.1712537408: "containerregistry.googleapis.com" => "containerregistry.googleapis.com"
services.1904024597: "runtimeconfig.googleapis.com" => "runtimeconfig.googleapis.com"
services.2117420113: "pubsub.googleapis.com" => "pubsub.googleapis.com"
services.2254174451: "containeranalysis.googleapis.com" => "containeranalysis.googleapis.com"
services.2471815660: "" => "servicemanagement.googleapis.com"
services.2692275209: "dataproc.googleapis.com" => "dataproc.googleapis.com"
services.2928564140: "dns.googleapis.com" => "dns.googleapis.com"
services.2966512281: "deploymentmanager.googleapis.com" => "deploymentmanager.googleapis.com"
services.3010261123: "replicapool.googleapis.com" => "replicapool.googleapis.com"
services.3075019877: "replicapoolupdater.googleapis.com" => "replicapoolupdater.googleapis.com"
services.3077910291: "resourceviews.googleapis.com" => "resourceviews.googleapis.com"
services.323125032: "cloudtrace.googleapis.com" => "cloudtrace.googleapis.com"
services.3237295688: "monitoring.googleapis.com" => "monitoring.googleapis.com"
services.3355193353: "logging.googleapis.com" => "logging.googleapis.com"
services.3445186629: "" => "sourcerepo.googleapis.com"
services.3612242986: "ml.googleapis.com" => "ml.googleapis.com"
services.3644083179: "cloudresourcemanager.googleapis.com" => "cloudresourcemanager.googleapis.com"
services.3731214611: "compute-component.googleapis.com" => "compute-component.googleapis.com"
services.3740470850: "container.googleapis.com" => "container.googleapis.com"
services.3875785048: "" => "storage-api.googleapis.com"
services.3899772697: "cloudfunctions.googleapis.com" => "cloudfunctions.googleapis.com"
services.4240178827: "servicecontrol.googleapis.com" => "servicecontrol.googleapis.com"
services.668598836: "" => "spanner.googleapis.com"
services.729731224: "dataflow.googleapis.com" => "dataflow.googleapis.com"
services.83957402: "dataproc-control.googleapis.com" => ""

Running a terraform show confirms that the remote state only has 25 stored:

module.project.google_project_services.project:
id = XXXXXX
project = XXXX
services.# = 25
services.1109577435 = cloudbuild.googleapis.com
services.1462818101 = endpoints.googleapis.com
services.1560437671 = iam.googleapis.com
services.1610229196 = bigquery-json.googleapis.com
services.1712537408 = containerregistry.googleapis.com
services.1904024597 = runtimeconfig.googleapis.com
services.2117420113 = pubsub.googleapis.com
services.2254174451 = containeranalysis.googleapis.com
services.2692275209 = dataproc.googleapis.com
services.2928564140 = dns.googleapis.com
services.2966512281 = deploymentmanager.googleapis.com
services.3010261123 = replicapool.googleapis.com
services.3075019877 = replicapoolupdater.googleapis.com
services.3077910291 = resourceviews.googleapis.com
services.323125032 = cloudtrace.googleapis.com
services.3237295688 = monitoring.googleapis.com
services.3355193353 = logging.googleapis.com
services.3426235936 = bigtable
services.3612242986 = ml.googleapis.com
services.3644083179 = cloudresourcemanager.googleapis.com
services.3731214611 = compute-component.googleapis.com
services.3740470850 = container.googleapis.com
services.3899772697 = cloudfunctions.googleapis.com
services.729731224 = dataflow.googleapis.com
services.83957402 = dataproc-control.googleapis.com

Terraform Version

Terraform v0.9.2

Affected Resource(s)

google_project
google_project_services
state files

Terraform Configuration Files

  project           = "${google_project.project.id}"
  services          = ["${var.services}"]
}

variable services       { type = "list" default = [
    "cloudresourcemanager.googleapis.com",
    "iam.googleapis.com",
    "container.googleapis.com",
    "sourcerepo.googleapis.com",
    "spanner.googleapis.com",
    "cloudbuild.googleapis.com",
    "compute-component.googleapis.com",
    "deploymentmanager.googleapis.com",
    "dns.googleapis.com",
    "monitoring.googleapis.com",
    "resourceviews.googleapis.com",
    "servicemanagement.googleapis.com",
    "storage-api.googleapis.com",
    "storage-component.googleapis.com",
    "pubsub.googleapis.com",
    "replicapool.googleapis.com",
    "replicapoolupdater.googleapis.com",
    "resourceviews.googleapis.com",
    "dataproc.googleapis.com",
    "dataproc-control.googleapis.com",
    "containeranalysis.googleapis.com",
    "containerregistry.googleapis.com",
    "cloudfunctions.googleapis.com",
    "logging.googleapis.com",
    "source.googleapis.com",
    "runtimeconfig.googleapis.com",
    "bigquery-json.googleapis.com",
    "endpoints.googleapis.com",
    "ml.googleapis.com",
    "cloudtrace.googleapis.com",
    "servicecontrol.googleapis.com",
    "dataflow.googleapis.com"
    ]
}

Expected Behavior

Terraform should enable the google API service specified and save said services into the remote state file

Actual Behavior

Terraform is only showing a maximum of 25 google API services in the remote state file, so any subsequent terraform plan shows a diff

Steps to Reproduce

1. Add more than 25 google API services to your terraform config
2. Run terraform plan and then terraform apply
   3 Re-run terraform plan and you will see that it shows 25 google API services and then see's a diff between what is actually running within Google when it should show the full list

[danawillow]

I can reproduce this. It looks like the default page size for listing services from google is 25 but the request lets you set the page size and paginate. I'll try to take care of that at some point in the next week or two (unless someone gets to it first).

[joe-boyce]

Great thanks Dana, look forward to when this is released! 👍

Cheers,

Joe

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.