Update to recent version creates memory leak like issue and the byte sent to google continue increases

[liufuyang]

We tried to updated opentelemetry-operations-go to the newest packages see here, but then we noticed this memory leak like issue and the byte sent to google continued increases:

We tried to update to an earlier version last week and saw the same issue.
We thought this could be related to #590, but after that is fixed and we now tried again, seems the issue remains.

Could there still be some batch export-related bugs exist?

Or could it be because we configured the metric exporter wrongly?

Let me know if you need extra information. The package version used can be seen in the our PR doing the upgrades einride/cloudrunner-go#401

Thank you.

[liufuyang]

Or perhaps it is a problem rooted from here? open-telemetry/opentelemetry-go-contrib#3071

[damemi]

Hi @liufuyang, thanks for reporting this. It sounds like the issue you're seeing is related to metrics export, is that right? If so, #590 is a different issue which only affect the logs exporter.

To help with your issue, have you tried isolating it to see if it's a problem with the GCP libraries, or the upstream opentelemetry-go-contrib library? One way to do this would be to use a different exporter like the logging or file exporter and compare the resource profiles.

[liufuyang]

Hey @damemi, thank you so much for the fast feedback. I am not sure if I could have time to play with different exporters at the moment and I have limited knowledge in this domain.

But my guess is that it could be related to the issue from contrib packages, so I am waiting for the next 0.37 to be released and try again on our side as it includes a memory leak issue fix on that version https://github.com/open-telemetry/opentelemetry-go/milestone/40?closed=1

[dashpole]

It definitely could be related to open-telemetry/opentelemetry-go-contrib#3071.

[liufuyang]

Yes, we just verified that:

1. with this No memory leakage in attributes filter open-telemetry/opentelemetry-go#3695 in their next release, the memory issue can be fixed.
2. for the "high metric cardinality" issue (stated in the 3071 issue above) we could sort of get around by using the filters.

So not sure if we want or can do something about 2 here, otherwise I guess we can close this issue here now :)

[dashpole]

I'd say lets close this, and watch 3071. If there is something we need to do in the exporter, we can reopen.

[liufuyang]

Great, thank you very much :)