gcp detector missing CloudRun associated resources "namespace" and "node_id".

[liufuyang]

Description

We see error such as this from time to time:

Points must be written in order. 
One or more of the points specified had an older start time than the most recent point.:
generic_node{namespace:,node_id:,location:europe-west1} timeSeries[0-67]:
workload.googleapis.com/process.cpu.time{state:user}

Looks like we are missing namespace:,node_id: on the CloudRun resources?

We run some service on Google CloudRun and the detectors are setup like this:
https://github.com/einride/cloudrunner-go/blob/17365f7b1348bc6e93296ea4861c47bc105dc00b/cloudotel/resource.go#L16

[liufuyang]

@dashpole Tagging you again as I saw you touched on this part of code recently.

Do you think it is easy to change, perhaps adding something will do?

If easy perhaps I can help as well, just let me know :)

[liufuyang]

By the way, we also noticed that the cloud-run metrics reported by otel was changed from generic_task to generic_node sometime ago (and we had to update all of our metrics configuration to switch to generic_node)

If it is meant to use generic_node, then seems the mapping defined here:

https://github.com/GoogleCloudPlatform/opentelemetry-operations-go/blob/1d581fe1c7d783e2bfcde5d6b1b8422b9dfd479f/internal/resourcemapping/resourcemapping.go#L155-L165

Seems not matching with the setup for CloudRun in detector?

opentelemetry-go-contrib/detectors/gcp/detector.go

Lines 59 to 64 in ba19074

	case gcp.CloudRun:
	b.attrs = append(b.attrs, semconv.CloudPlatformGCPCloudRun)
	b.add(semconv.FaaSNameKey, d.detector.FaaSName)
	b.add(semconv.FaaSVersionKey, d.detector.FaaSVersion)
	b.add(semconv.FaaSInstanceKey, d.detector.FaaSID)
	b.add(semconv.CloudRegionKey, d.detector.FaaSCloudRegion)

Perhaps in detector something such as semconv.HostNameKey or semconv.HostIDKey should be used instead of that FaaSInstanceKey?

[dashpole]

It should be using generic_task. Can you print out the resource you get in https://github.com/einride/cloudrunner-go/blob/17365f7b1348bc6e93296ea4861c47bc105dc00b/cloudotel/resource.go#L16?

[liufuyang]

Sure that resource looks like this when running in CloudRun:

resource: "cloud.account.id=some-service-dev,
cloud.platform=gcp_cloud_run,
cloud.provider=gcp,
cloud.region=europe-west1,
faas.instance=0037d6d5d35ab03f680f1f1abc2dcd4e6495f2f599b1627c617cae5bebeeca74ccd0055be15e6fe4a09cf59a69c2e927bbd260f81c6a4f4faabf2ee48993feaad9,
faas.name=some-service-eu,
faas.version=some-service-eu-00988-whs,
telemetry.sdk.language=go,
telemetry.sdk.name=opentelemetry,
telemetry.sdk.version=1.20.0"

[liufuyang]

So it should be generic_task, the whole metrics reporter with otel were under generic_task before but now they are all under generic_node, perhaps something changed but not in the expected behavior? Wondering which part of the code controls this 🤔

[liufuyang]

Perhaps it is related to GoogleCloudPlatform/opentelemetry-operations-go#735 and also fixed by GoogleCloudPlatform/opentelemetry-operations-go#764 two weeks ago, all I need is to wait for the new version to be released in opentelemetry-operations-go? :)

[dashpole]

It was just released. Can you try updating to v0.45.0 of opentelemetry-operaitons-go?

[liufuyang]

@dashpole Thanks a lot, we updated opentelemetry-operaitons-go and it seems working correctly now. I will close this issue now.