Terraform state is not fully applied

[ameukam]

I got when I deployed #1604 for aaa:

  # google_project_iam_binding.readonlymonitoringbinding will be updated in-place
  ~ resource "google_project_iam_binding" "readonlymonitoringbinding" {
        etag    = "BwXAZra+ioU="
        id      = "kubernetes-public/roles/monitoring.viewer"
      ~ members = [
            "group:gke-security-groups@kubernetes.io",
          - "serviceAccount:k8s-infra-monitoring-viewer@kubernetes-public.iam.gserviceaccount.com",
        ]
        project = "kubernetes-public"
        role    = "roles/monitoring.viewer"
    }

  # google_project_iam_member.cluster_node_sa_monitoring_viewer will be created
  + resource "google_project_iam_member" "cluster_node_sa_monitoring_viewer" {
      + etag    = (known after apply)
      + id      = (known after apply)
      + member  = "serviceAccount:gke-nodes-aaa@kubernetes-public.iam.gserviceaccount.com"
      + project = "kubernetes-public"
      + role    = "roles/monitoring.viewer"
    }

Resources introduced in #365

[ameukam]

It's possible those resources were re-created or have been created. Not sure if the investigation is worth it.

[spiffxp]

/assign
Feels pretty likely this is related to something I pruned, possibly elsewhere in bash

[spiffxp]

I bumped into this while deploying changes that were part of #1952. I deployed the changes, so maybe it's resolved?

I suspect if there's still a conflict with bash, it may be with changes in #1974

I'll re-run terraform once that has merged to see if there's still a difference of opinion between terraform and bash about this

[spiffxp]

Added a commit to #1974 to fix this

terraform apply and ./infra/gcp/ensure-main-project.sh were getting in a fight over authoritative IAM bindings for the kubernetes-public project

IMO terraform shouldn't be setting authoritative bindings for a project unless it's also managing the project. Whereas in this case, the module in question is just for managing a GKE cluster within the project. The infra/gcp/ensure-main-project.sh script is the source of truth about the configuration of the kubernetes-public project.

I opted to move the bindings over to ensure-main-project.sh instead of updating the terraform module to non-authoritatively add IAM members, since they were related to a group that is managed outside of terraform vs. service accounts that were created by terraform.

As part of the commit I preemptively ran terraform state rm for the two resources that were removed, so that terraform won't try to destroy them

[ameukam]

IMO terraform shouldn't be setting authoritative bindings for a project unless it's also managing the project. Whereas in this case, the module in question is just for managing a GKE cluster within the project. The infra/gcp/ensure-main-project.sh script is the source of truth about the configuration of the kubernetes-public project.

Totally make sense for me.