ContainerOperationWaiter doesn't support UserProjectOverride

[karlkfi]

Expected

Using Terraform, with a google service account in another project, I should be able to create a GKE cluster (google_container_cluster) using User Project Override (user_project_override) with GKE API enabled in the target project, but not the service account project.

Found

Error:

Error waiting for updating GKE cluster master authorized networks: error while retrieving operation: googleapi: Error 403: Kubernetes Engine API has not been used in project 538440108421 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/container.googleapis.com/overview?project=538440108421 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.

The project number specified is the project the service account is in, not the project the cluster is supposed to be in.

Suspected Root Cause

I did some sleuthing and I'm pretty sure I found the root cause:

resource_container_cluster.go supports UserProjectOverride in 23 places, but it also uses containerOperationWait in 20 places, and containerOperationWait does NOT support UserProjectOverride.

So ultimately, google_container_cluster does not actually fully support user_project_override.

Fix

On https://github.com/hashicorp/terraform-provider-google/blob/master/google/container_operation.go#L78 in the QueryOp method of ContainerOperationWaiter, add the header between the Get() and the Do():

if config.UserProjectOverride {
	clusterCreateCall.Header().Add("X-Goog-User-Project", project)
}

If wouldn't surprise me if there are other operations that also need to be updated for other resources to support UserProjectOverride.

[karlkfi]

Looks like the original UserProjectOverride support for GKE was added with GoogleCloudPlatform/magic-modules#3892

I don't see any unique tests added to verify the feature tho, just a few user_project_override = true thrown in on a few existing tests. I'm not sure the fact that the header is being added is ever being validated. I would expect some test to need to capture or record API calls and loop through them to make sure they all have the header or not.

[karlkfi]

Looks like the code to change is actually in magic-modules v1 (unless there's a newer one in DCL that i can't find):
https://github.com/GoogleCloudPlatform/magic-modules/blob/master/mmv1/third_party/terraform/utils/container_operation.go

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.