-
Notifications
You must be signed in to change notification settings - Fork 167
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Remove any possibility to record GenAI prompts and completions as attributes once Event API is available #1278
Comments
For me this was tough to grok as I had no idea where to look on a per-language basis, except searching for the word event which well, isn't great. I asked an LLM what the most popular languages for GenAI were and here are the status of the ones it mentioned (besides R which I'm not sure we have a repo for)
Since I personally work in Go and would like to help the uber popular ollama project, I also looked at Go
I don't have karma to edit description, but I would suggest making a checklist so that it is easier for folks to look at this and get a quick gauge. |
also, while it may be too long an issue title, in the description, it helps to clarify the spec is currently about "span event attributes", not the more troubling "span attributes". Basically, linking to my issue comment helps, but maybe after a description of the now state. Example description summary, feel free to use, reword or leave as a comment: The current LLM/GenAI semantics indicate prompt and completion data recorded as (Log) Events, not Span Events. Right now, python is the most dominant programming language in GenAI, but it doesn't yet have support for the Event API. Even languages that do support it, such as java and javascript, caveat it as incubator or experimental. This means instrumentors may not be able to use the (Log) Event API reliably, especially across languages. For this reason, the LLM/GenAI semantics provide for a fallback to using Span Events to record the same prompt/completion data as what should use the Event API. This issue tracks status of the major language SDKs. Once a quorum is met, we should remove the fallback advice so that the Event API, and not Span API is used for prompt/completion data. |
There are two criteria:
Just to add one more piece of context to this conversation since it may not be known / obvious to all readers: the Event API is likely desirable because its built on top of the log signal, which supports an AnyValue body and attribute values. AnyValue allows the modeling of complex types (i.e. maps of maps, heterogeneous arrays, etc) which are not supported in the standard attribute definition used everywhere else attributes appear in the spec (i.e. resource attributes, scope attributes, metric point attribute, span attributes, span event attributes, span link attributes). |
Thanks for the comprehensive reply @jack-berg!
If you feel alone on a PR you feel important and a review by a not-yet-approver would be helpful, feel free to tag me. Sometimes, a beginning review can help momentum, though not a sure thing. |
The best way to know if specific API/feature is available (and where) is spec-compliance matrix - https://github.com/open-telemetry/opentelemetry-specification/blob/main/spec-compliance-matrix.md#events There is a link to it in the #980 |
Thanks, maybe include this in a derivation of this issue desc, unless the norms of this repo is to not have PR descriptions if someone can derive the same from a linked comment. Again, I would do this myself instead of nagging you, but I don't have access. |
I'm closing this issue since #980 does not use span events anymore |
thanks agreed! |
See #980 (review)
How to check that Event API is implemented:
The text was updated successfully, but these errors were encountered: