Service Account

[omid-verneek]

I have followed this but there is no place to assign the serviceaccount otel-collector to the config. In the example, it's just annotating the namespace. I get permission denied in my deployment.

adding serviceAccount: "otel-collector" doesn't help.

2022/10/21 13:41:11 failed to export to Google Cloud Trace: rpc error: code = PermissionDenied desc = The caller does not have permission

I also changed the role to admin and no luck.

[omid-verneek]

I found the issue. When we use kubectl apply -f ./collector-config.yaml to create the collector, the operator create the serviceaccount and removes the annotation for the GCloud annotations: iam.gke.io/gcp-service-account=otel-collector@${GCLOUD_PROJECT}.iam.gserviceaccount.com. So in the tutorial you should mention that the annotation should happen after.

[damemi]

Thanks for catching this @omid-verneek! Looking into this

[damemi]

Hi @omid-verneek, I tried but wasn't able to reproduce this. Could you confirm these steps or let me know any you did differently/in addition/omitted?

1. Install operator
2. Create a default collector config
3. Create a default Instrumentation
4. Install one of the sample apps (it could also help to know which one, if any)
5. Follow the workload identity steps:
   a. Create the GCP service account (gcloud iam service-accounts create...)
   b. Grant trace permissions to that account (gcloud projects add-iam-policy-binding...)
   c. Bind the GCP service account the the Collector's k8s ServiceAccount (gcloud iam service-accounts add-iam-policy-binding...)
   d. Annotate the k8s ServiceAccount (kubectl annotate serviceaccount otel-collector...)
6. Create the cloud trace collector config

When I do these steps on a GKE autopilot cluster, the k8s ServiceAccount keeps its annotation so it doesn't look like the Operator is always removing that. I'm interested to see if you did anything differently from me that we could document.

[omid-verneek]

@damemi Not sure if it has any effect but I'm using a standard cluster. Also, I'm following the recipe for Cloud-trace.

[damemi]

Sorry for the long update on this.

Possibly related, the operator recently added an update to preserve existing service accounts using the serviceAccount field in the collector CRD (open-telemetry/opentelemetry-operator#1246). I wasn't able to reproduce the issue reported here, but we can probably update our docs to take advantage of this change and avoid the possibility of overwriting the service account altogether. I'll create a PR to do that.

[philipz]

@omid-verneek I have fixed the service account annotations problem. Apply to CRD opentelemetrycollectors, then the service account, otel-collector, will add the iam annotation.

kubectl annotate opentelemetrycollectors otel \
    --namespace $COLLECTOR_NAMESPACE \
    iam.gke.io/gcp-service-account=otel-collector@${GCLOUD_PROJECT}.iam.gserviceaccount.com

[dashpole]

Another customer hit this. We should adopt #25 (comment) in our documentation

[philipz]

@dashpole There is the other solution. 1. create service account first, then apply otel-collector yaml with kubernetes service account.

kubectl annotate serviceaccount \
  --namespace ${K8S_NS} \
  ${K8S_SA} \
iam.gke.io/gcp-service-account=${GSA}@${PROJECT_ID}.iam.gserviceaccount.com

collector-config.yaml

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: otel
spec:
  serviceAccount: ${K8S_SA}
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
    processors:

    exporters:
      logging:
        loglevel: debug

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: []
          exporters: [logging]

[dashpole]

That would also work. Do you think that is a better/easier to understand solution?

[philipz]

@dashpole I think the better solution is Apply to CRD opentelemetrycollectors with annotation. It is simpler and more intuitive.