Improved lambda OTEL Collector

[ghost]

As far as I understand there is currently no workaround for increased lambda execution latency due to synchronous flush even if using Collector extension. That is, we can't use batch processor for the collector in order to avoid missing spans when collector extension gets frozen before flush.

I was thinking about a solution to that, until there is an IDLE event in Lambda API (as mentioned in: https://github.com/open-telemetry/opentelemetry-lambda/blob/main/docs/design_proposal.md). Basically we could take advantage of the following:

• with Fast Invoke Response enhancement to extensions lambda now allows a handler to return results before all extensions have completed processing an event
• execution environment will not get frozen until extensions have completed processing an event
• each and every traced lambda handler invocation will have a "master span" describing execution of the handler itself - and the span will be closed (and send) right before returning handler's result

Altogether, we could implement an OTEL Collector processor that would:

• batch incoming spans (so async receive)
• flush when the master span has been received (we know that at this point in time all other spans are already in batch)
• notify lambda (issuing /next API call) that the processing has been finished only after successful export of the batch

This way we would:

• remove span delivery time impact on handler latency
• improve comm with backend (batches)
• guarantee that no spans are lost (temporarily) due to execution env freeze

[ghost]

@anuraaga any thoughts on this? Perhaps I'm missing something from the current / future landscape that will solve the latency issue rendering this idea useless?

[anuraaga]

Hey @kubawach - I have a feeling you know more about some of these advanced Lambda features than me :P @wangzlei knows more - does this seem like something we should try? I'm happy to have any approach that improves the latency issues.

[ghost]

@wangzlei happy to discuss this "improved lambda collector". Since we still don't have a proper lambda API to notify extensions "you're about to get frozen" I believe there is a room for the proposed solution.