[Home] Add support for otel sample data - logs, traces and metrics #8587

ps48 · 2024-10-14T15:06:00Z

Description

Add support for otel sample data - logs, traces and metrics

Issues Resolved

OTEL-sample-data.mov

Changelog

feat: Add support for otel sample data - logs, traces and metrics

Check List

All tests pass
- yarn test:jest
- yarn test:jest_integration
New functionality includes testing.
New functionality has been documented.
Update CHANGELOG.md
Commits are signed per the DCO using --signoff

Signed-off-by: Shenoy Pratik <sgguruda@amazon.com>

github-actions · 2024-10-14T15:07:17Z

❌ Empty Changelog Section

The Changelog section in your PR description is empty. Please add a valid changelog entry or entries. If you did add a changelog entry, check to make sure that it was not accidentally included inside the comment block in the Changelog section.

codecov · 2024-10-14T21:09:15Z

Codecov Report

Attention: Patch coverage is 73.17073% with 11 lines in your changes missing coverage. Please review.

Project coverage is 60.83%. Comparing base (0e00abf) to head (28f5548).
Report is 8 commits behind head on main.

Files with missing lines	Patch %	Lines
...ns/home/server/services/sample_data/routes/list.ts	33.33%	5 Missing and 3 partials ⚠️
...ication/components/sample_data_view_data_button.js	87.50%	1 Missing ⚠️
...home/server/services/sample_data/data_sets/util.ts	93.33%	0 Missing and 1 partial ⚠️
...home/server/services/sample_data/routes/install.ts	80.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8587      +/-   ##
==========================================
- Coverage   60.89%   60.83%   -0.06%     
==========================================
  Files        3790     3793       +3     
  Lines       90262    90412     +150     
  Branches    14154    14196      +42     
==========================================
+ Hits        54962    55004      +42     
- Misses      31829    31932     +103     
- Partials     3471     3476       +5

Flag	Coverage Δ
Linux_1	`29.08% <ø> (-0.04%)`	⬇️
Linux_2	`56.39% <ø> (+<0.01%)`	⬆️
Linux_3	`37.67% <ø> (-0.05%)`	⬇️
Linux_4	`29.82% <73.17%> (+0.01%)`	⬆️
Windows_1	`29.09% <ø> (-0.04%)`	⬇️
Windows_2	`56.34% <ø> (+<0.01%)`	⬆️
Windows_3	`37.67% <ø> (-0.05%)`	⬇️
Windows_4	`29.82% <73.17%> (+0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

ruanyl · 2024-10-16T03:48:31Z

Not related to this PR, in the screen record, I noticed that you need to switch data source quite frequently, do you feel it's a bit overwhelming?

src/plugins/home/server/services/sample_data/data_sets/otel/index.ts

src/plugins/home/server/services/sample_data/routes/install.ts

src/plugins/home/server/services/sample_data/routes/list.ts

wanglam · 2024-10-16T08:32:15Z

src/plugins/home/server/services/sample_data/data_sets/otel/index.ts

+});
+const initialAppLinks = [] as AppLinkSchema[];
+
+const DEFAULT_INDEX = 'i254a9691-8a1b-11ef-adb4-9f1097200c20';


What is this default index from? Seems we won't add any saved objects after sample data imported.

Yup, this is not needed anymore. Updated here: c6949c4

wanglam · 2024-10-16T08:35:32Z

src/plugins/home/server/services/sample_data/data_sets/otel/index.ts

+    previewImagePath: '/plugins/home/assets/sample_data_resources/otel/otel_traces.png',
+    darkPreviewImagePath: '/plugins/home/assets/sample_data_resources/otel/otel_traces_dark.png',
+    hasNewThemeImages: true,
+    overviewDashboard: DASHBOARD_ID,


Shall we update SampleDatsetSchema to set field to optional? Since this overviewDashboard value will be used in this line, do we need to hide this button or show another link?

Didn't want to keep this optional, rather just kept it empty for now. Similar to how applinks are kept empty in other sample data objects. Have hidden the button for dashbords if empty and showing app links now.

Signed-off-by: Eric <menwe@amazon.com>

Signed-off-by: Ryan Liang <jiallian@amazon.com>

Signed-off-by: Eric <menwe@amazon.com>

Signed-off-by: Eric Wei <menwe@amazon.com>

…o utils Signed-off-by: Shenoy Pratik <sgguruda@amazon.com>

joshuali925 · 2024-10-18T17:28:06Z

src/plugins/home/server/services/sample_data/data_sets/otel/index.ts

+const initialAppLinks = [] as AppLinkSchema[];
+
+const DEFAULT_INDEX = 'i254a9691-8a1b-11ef-adb4-9f1097200c20';
+const DASHBOARD_ID = '';


why is this empty?

removed the variable, we don't have a default dashboard, users land on to traces page

#8587 (comment)

joshuali925 · 2024-10-18T17:28:26Z

src/plugins/home/server/services/sample_data/data_sets/otel/index.ts

+  defaultMessage:
+    'Sample data including correlated observability signals for an e-commerce application in Open-Telemetry standard.',
+});
+const initialAppLinks = [] as AppLinkSchema[];


nit. avoid assertions if possible

Suggested change

const initialAppLinks = [] as AppLinkSchema[];

const initialAppLinks: AppLinkSchema[] = [];

Updated to add traces and services links c6949c4

joshuali925 · 2024-10-18T17:29:01Z

src/plugins/home/server/services/sample_data/data_sets/otel/index.ts

+        id: 'otel-v1-apm-span-sample',
+        dataPath: path.join(__dirname, './sample_traces.json.gz'),
+        fields: tracesFieldMappings,
+        timeFields: ['startTime', 'endTime', 'traceGroupFields.endTime'], // TODO: add support for 'events.time'


what is needed for events.time?

Parsing array field needs to be added to search for the timefield inside the array and then update it in the doc. My understanding is this is not needed for P0 and can be fine if taken up later and trace queries use startTime and endTime

src/plugins/home/server/services/sample_data/data_sets/otel/logs_field_mappings.ts

Signed-off-by: Shenoy Pratik <sgguruda@amazon.com>

Signed-off-by: Eric <menwe@amazon.com>

src/plugins/home/server/services/sample_data/data_sets/otel/logs_field_mappings.ts

Signed-off-by: Eric <menwe@amazon.com>

joshuali925 · 2024-10-19T02:57:24Z

src/plugins/home/server/services/sample_data/data_sets/otel/logs_field_mappings.ts

+    fields: {
+      keyword: {
+        type: 'keyword',
+        ignore_above: 256,


would be better to remove the .keyword field given it is already keyword type

Sure, will mark this as discussed already with vamsi that we will enhance sample data experience including things like this.

) * add support for otel sample data - traces and services Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * Changeset file for PR #8587 created/updated * upload gz otel data Signed-off-by: Eric <menwe@amazon.com> * add mappings Signed-off-by: Eric <menwe@amazon.com> * correct mappings Signed-off-by: Eric <menwe@amazon.com> * Fix the time mismatch for otel sample data Signed-off-by: Ryan Liang <jiallian@amazon.com> * adjust time maker Signed-off-by: Eric <menwe@amazon.com> * create index name tests Signed-off-by: Eric <menwe@amazon.com> * add indexName field to DataIndexSchema, move nested field functions to utils Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * add button for app redirection, resolve comments Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * adjust id field to be as keyword Signed-off-by: Eric <menwe@amazon.com> * remove redundant keyword declaration Signed-off-by: Eric <menwe@amazon.com> * move to use correct format sample suffix Signed-off-by: Eric <menwe@amazon.com> * update text and one log index name/id Signed-off-by: Eric <menwe@amazon.com> * fix sample data test Signed-off-by: Eric <menwe@amazon.com> --------- Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> Signed-off-by: Eric <menwe@amazon.com> Signed-off-by: Ryan Liang <jiallian@amazon.com> Signed-off-by: Eric Wei <menwe@amazon.com> Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com> Co-authored-by: Eric <menwe@amazon.com> Co-authored-by: Ryan Liang <jiallian@amazon.com> (cherry picked from commit c8b5318) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

) (#8666) * add support for otel sample data - traces and services * Changeset file for PR #8587 created/updated * upload gz otel data * add mappings * correct mappings * Fix the time mismatch for otel sample data * adjust time maker * create index name tests * add indexName field to DataIndexSchema, move nested field functions to utils * add button for app redirection, resolve comments * adjust id field to be as keyword * remove redundant keyword declaration * move to use correct format sample suffix * update text and one log index name/id * fix sample data test --------- (cherry picked from commit c8b5318) Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> Signed-off-by: Eric <menwe@amazon.com> Signed-off-by: Ryan Liang <jiallian@amazon.com> Signed-off-by: Eric Wei <menwe@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com> Co-authored-by: Eric <menwe@amazon.com> Co-authored-by: Ryan Liang <jiallian@amazon.com>

…ensearch-project#8587) * add support for otel sample data - traces and services Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * Changeset file for PR opensearch-project#8587 created/updated * upload gz otel data Signed-off-by: Eric <menwe@amazon.com> * add mappings Signed-off-by: Eric <menwe@amazon.com> * correct mappings Signed-off-by: Eric <menwe@amazon.com> * Fix the time mismatch for otel sample data Signed-off-by: Ryan Liang <jiallian@amazon.com> * adjust time maker Signed-off-by: Eric <menwe@amazon.com> * create index name tests Signed-off-by: Eric <menwe@amazon.com> * add indexName field to DataIndexSchema, move nested field functions to utils Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * add button for app redirection, resolve comments Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * adjust id field to be as keyword Signed-off-by: Eric <menwe@amazon.com> * remove redundant keyword declaration Signed-off-by: Eric <menwe@amazon.com> * move to use correct format sample suffix Signed-off-by: Eric <menwe@amazon.com> * update text and one log index name/id Signed-off-by: Eric <menwe@amazon.com> * fix sample data test Signed-off-by: Eric <menwe@amazon.com> --------- Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> Signed-off-by: Eric <menwe@amazon.com> Signed-off-by: Ryan Liang <jiallian@amazon.com> Signed-off-by: Eric Wei <menwe@amazon.com> Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com> Co-authored-by: Eric <menwe@amazon.com> Co-authored-by: Ryan Liang <jiallian@amazon.com>

add support for otel sample data - traces and services

c7aa30c

Signed-off-by: Shenoy Pratik <sgguruda@amazon.com>

github-actions bot added the repeat-contributor label Oct 14, 2024

github-actions bot added the failed changeset label Oct 14, 2024

Changeset file for PR opensearch-project#8587 created/updated

dbe89bc

github-actions bot removed the failed changeset label Oct 14, 2024

wanglam reviewed Oct 16, 2024

View reviewed changes

mengweieric and others added 8 commits October 17, 2024 19:54

upload gz otel data

4811d32

Signed-off-by: Eric <menwe@amazon.com>

add mappings

bfdbfcd

Signed-off-by: Eric <menwe@amazon.com>

correct mappings

8049e71

Signed-off-by: Eric <menwe@amazon.com>

Fix the time mismatch for otel sample data

b9e9130

Signed-off-by: Ryan Liang <jiallian@amazon.com>

adjust time maker

b55bcdb

Signed-off-by: Eric <menwe@amazon.com>

create index name tests

1a191e3

Signed-off-by: Eric <menwe@amazon.com>

Merge branch 'main' into otel_sample_data

f22be8e

Signed-off-by: Eric Wei <menwe@amazon.com>

add indexName field to DataIndexSchema, move nested field functions t…

b01ec49

…o utils Signed-off-by: Shenoy Pratik <sgguruda@amazon.com>

joshuali925 reviewed Oct 18, 2024

View reviewed changes

ps48 and others added 2 commits October 18, 2024 12:54

add button for app redirection, resolve comments

c6949c4

Signed-off-by: Shenoy Pratik <sgguruda@amazon.com>

adjust id field to be as keyword

66efe4c

Signed-off-by: Eric <menwe@amazon.com>

joshuali925 reviewed Oct 18, 2024

View reviewed changes

src/plugins/home/server/services/sample_data/data_sets/otel/logs_field_mappings.ts Outdated Show resolved Hide resolved

mengweieric added 4 commits October 18, 2024 21:22

remove redundant keyword declaration

6dc07c1

Signed-off-by: Eric <menwe@amazon.com>

move to use correct format sample suffix

cc01c94

Signed-off-by: Eric <menwe@amazon.com>

update text and one log index name/id

6c95400

Signed-off-by: Eric <menwe@amazon.com>

fix sample data test

28f5548

Signed-off-by: Eric <menwe@amazon.com>

mengweieric approved these changes Oct 19, 2024

View reviewed changes

mengweieric added the backport 2.x label Oct 19, 2024

joshuali925 approved these changes Oct 19, 2024

View reviewed changes

mengweieric merged commit c8b5318 into opensearch-project:main Oct 19, 2024
70 checks passed

opensearch-trigger-bot bot mentioned this pull request Oct 19, 2024

[Backport 2.x] [Home] Add support for otel sample data - logs, traces and metrics #8666

Merged

ashwin-pc added the v2.18.0 label Oct 19, 2024

mengweieric changed the title ~~[WIP] Add support for otel sample data - logs, traces and metrics~~ [Home] Add support for otel sample data - logs, traces and metrics Oct 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Home] Add support for otel sample data - logs, traces and metrics #8587

[Home] Add support for otel sample data - logs, traces and metrics #8587

ps48 commented Oct 14, 2024 •

edited

Loading

github-actions bot commented Oct 14, 2024

codecov bot commented Oct 14, 2024 •

edited

Loading

ruanyl commented Oct 16, 2024

wanglam Oct 16, 2024

ps48 Oct 18, 2024

wanglam Oct 16, 2024

ps48 Oct 18, 2024

joshuali925 Oct 18, 2024

ps48 Oct 18, 2024

ps48 Oct 18, 2024

joshuali925 Oct 18, 2024

ps48 Oct 18, 2024

joshuali925 Oct 18, 2024

ps48 Oct 18, 2024

joshuali925 Oct 19, 2024

mengweieric Oct 19, 2024

	const initialAppLinks = [] as AppLinkSchema[];
	const initialAppLinks: AppLinkSchema[] = [];

[Home] Add support for otel sample data - logs, traces and metrics #8587

[Home] Add support for otel sample data - logs, traces and metrics #8587

Conversation

ps48 commented Oct 14, 2024 • edited Loading

Description

Issues Resolved

Changelog

Check List

github-actions bot commented Oct 14, 2024

❌ Empty Changelog Section

codecov bot commented Oct 14, 2024 • edited Loading

Codecov Report

ruanyl commented Oct 16, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ps48 commented Oct 14, 2024 •

edited

Loading

codecov bot commented Oct 14, 2024 •

edited

Loading