Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x-pack/metricbeat/module/openai: Add new module #41516

Merged
merged 49 commits into from
Dec 13, 2024
Merged
Show file tree
Hide file tree
Changes from 47 commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
0e1175c
x-pack/metricbeat/module/openai: Add new module
shmsr Nov 4, 2024
bf45bf6
Merge branch 'main' into openai-metrics
shmsr Nov 4, 2024
8725cbb
update module
shmsr Nov 5, 2024
03995f6
update module
shmsr Nov 5, 2024
406c1a4
update module
shmsr Nov 5, 2024
a3c46b7
update module
shmsr Nov 5, 2024
72488f7
update module
shmsr Nov 5, 2024
b5c750a
Merge branch 'main' into openai-metrics
shmsr Nov 5, 2024
2d4027d
fix bug
shmsr Nov 5, 2024
e3f8fe2
update module
shmsr Nov 6, 2024
82cd979
Merge branch 'main' into openai-metrics
shmsr Nov 12, 2024
be628fe
Merge branch 'main' into openai-metrics
shmsr Nov 25, 2024
3259612
Address review comments
shmsr Nov 25, 2024
40b1131
Address review comments
shmsr Nov 25, 2024
2936540
Address review comments
shmsr Nov 25, 2024
850fc43
Merge branch 'main' into openai-metrics
shmsr Nov 25, 2024
5061e46
Address review comments
shmsr Nov 25, 2024
34a7128
Improvements
shmsr Nov 25, 2024
7f4cc75
Fix *.yml
shmsr Nov 26, 2024
7919c81
Address review comments
shmsr Nov 26, 2024
2a4bb56
Improvements
shmsr Nov 26, 2024
4899f3a
Merge branch 'main' into openai-metrics
shmsr Nov 26, 2024
217f7d5
Make linter happy
shmsr Nov 26, 2024
b5dbfb0
Address review comments
shmsr Nov 27, 2024
492eca1
gofumpt'ed
shmsr Nov 28, 2024
413010f
Address review comments
shmsr Nov 28, 2024
92f5062
Address review comments
shmsr Nov 28, 2024
fdb81c9
Merge branch 'main' into openai-metrics
shmsr Nov 28, 2024
78bbd11
Merge branch 'main' into openai-metrics
shmsr Nov 29, 2024
9c28e4d
Address review comments
shmsr Dec 4, 2024
f6067b8
Add more dummy data
shmsr Dec 4, 2024
ab98369
Better prealloc
shmsr Dec 4, 2024
1201845
Address review comments
shmsr Dec 8, 2024
90c2680
make update
shmsr Dec 9, 2024
63703d4
Merge branch 'main' into openai-metrics
shmsr Dec 9, 2024
e1ea8d2
Include OpenAI with Agentbeat
shmsr Dec 9, 2024
e907d98
More changes
shmsr Dec 9, 2024
942454d
make check
shmsr Dec 9, 2024
be9b5f2
make update
shmsr Dec 10, 2024
aff36b5
nitpick
shmsr Dec 10, 2024
549f26e
Logging and other basic improvements
shmsr Dec 10, 2024
e42472c
Merge branch 'main' into openai-metrics
shmsr Dec 10, 2024
33daed7
Update metricbeat/docs/modules/openai.asciidoc
shmsr Dec 12, 2024
33b7267
Merge branch 'main' into openai-metrics
shmsr Dec 12, 2024
2b836c2
Fix docs
shmsr Dec 12, 2024
32a9593
Update CODEOWNER
shmsr Dec 12, 2024
9698cac
Merge branch 'main' into openai-metrics
shmsr Dec 12, 2024
ae85fa8
Make linter happy
shmsr Dec 13, 2024
0b970bd
Merge branch 'main' into openai-metrics
shmsr Dec 13, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
Expand Up @@ -222,6 +222,7 @@ CHANGELOG*
/x-pack/metricbeat/module/iis @elastic/obs-infraobs-integrations
/x-pack/metricbeat/module/istio/ @elastic/obs-cloudnative-monitoring
/x-pack/metricbeat/module/mssql @elastic/obs-infraobs-integrations
/x-pack/metricbeat/module/openai @elastic/obs-infraobs-integrations
/x-pack/metricbeat/module/oracle @elastic/obs-infraobs-integrations
/x-pack/metricbeat/module/panw @elastic/obs-infraobs-integrations
/x-pack/metricbeat/module/prometheus/ @elastic/obs-cloudnative-monitoring
Expand Down
367 changes: 367 additions & 0 deletions metricbeat/docs/fields.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,7 @@ grouped in the following categories:
* <<exported-fields-mysql>>
* <<exported-fields-nats>>
* <<exported-fields-nginx>>
* <<exported-fields-openai>>
* <<exported-fields-openmetrics>>
* <<exported-fields-oracle>>
* <<exported-fields-panw>>
Expand Down Expand Up @@ -56685,6 +56686,372 @@ type: long

--

[[exported-fields-openai]]
== openai fields

openai module



[float]
=== openai




[float]
=== usage

OpenAI API usage metrics and statistics



*`openai.usage.organization_id`*::
+
--
Organization identifier

type: keyword

--

*`openai.usage.organization_name`*::
+
--
Organization name

type: keyword

--

*`openai.usage.api_key_id`*::
+
--
API key identifier

type: keyword

--

*`openai.usage.api_key_name`*::
+
--
API key name

type: keyword

--

*`openai.usage.api_key_redacted`*::
+
--
Redacted API key

type: keyword

--

*`openai.usage.api_key_type`*::
+
--
Type of API key

type: keyword

--

*`openai.usage.project_id`*::
+
--
Project identifier

type: keyword

--

*`openai.usage.project_name`*::
+
--
Project name

type: keyword

--

[float]
=== data

General usage data metrics



*`openai.usage.data.requests_total`*::
+
--
Number of requests made

type: long

--

*`openai.usage.data.operation`*::
+
--
Operation type

type: keyword

--

*`openai.usage.data.snapshot_id`*::
+
--
Snapshot identifier

type: keyword

--

*`openai.usage.data.context_tokens_total`*::
+
--
Total number of context tokens used

type: long

--

*`openai.usage.data.generated_tokens_total`*::
+
--
Total number of generated tokens

type: long

--

*`openai.usage.data.cached_context_tokens_total`*::
+
--
Total number of cached context tokens

type: long

--

*`openai.usage.data.email`*::
+
--
User email

type: keyword

--

*`openai.usage.data.request_type`*::
+
--
Type of request

type: keyword

--

[float]
=== dalle

DALL-E API usage metrics



*`openai.usage.dalle.num_images`*::
+
--
Number of images generated

type: long

--

*`openai.usage.dalle.requests_total`*::
+
--
Number of requests

type: long

--

*`openai.usage.dalle.image_size`*::
+
--
Size of generated images

type: keyword

--

*`openai.usage.dalle.operation`*::
+
--
Operation type

type: keyword

--

*`openai.usage.dalle.user_id`*::
+
--
User identifier

type: keyword

--

*`openai.usage.dalle.model_id`*::
+
--
Model identifier

type: keyword

--

[float]
=== whisper

Whisper API usage metrics



*`openai.usage.whisper.model_id`*::
+
--
Model identifier

type: keyword

--

*`openai.usage.whisper.num_seconds`*::
+
--
Number of seconds processed

type: long

--

*`openai.usage.whisper.requests_total`*::
+
--
Number of requests

type: long

--

*`openai.usage.whisper.user_id`*::
+
--
User identifier

type: keyword

--

[float]
=== tts

Text-to-Speech API usage metrics



*`openai.usage.tts.model_id`*::
+
--
Model identifier

type: keyword

--

*`openai.usage.tts.num_characters`*::
+
--
Number of characters processed

type: long

--

*`openai.usage.tts.requests_total`*::
+
--
Number of requests

type: long

--

*`openai.usage.tts.user_id`*::
+
--
User identifier

type: keyword

--

[float]
=== ft_data

Fine-tuning data metrics



*`openai.usage.ft_data.original`*::
+
--
Raw fine-tuning data

type: object

--

[float]
=== assistant_code_interpreter

Assistant Code Interpreter usage metrics



*`openai.usage.assistant_code_interpreter.original`*::
+
--
Raw assistant code interpreter data

type: object

--

[float]
=== retrieval_storage

Retrieval storage usage metrics



*`openai.usage.retrieval_storage.original`*::
+
--
Raw retrieval storage data

type: object

--

[[exported-fields-openmetrics]]
== Openmetrics fields

Expand Down
Loading
Loading