Observability namespace #474

aabeshov · 2024-08-07T18:42:12Z

Description

Adding observability namespace specs.

Issues Resolved

Closes #230.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: alen_abeshov <a_abeshov@kbtu.kz>

dblock

Great start! Needs tests.

You can add these specs piecemeal instead of everything at once, maybe start with something simple with tests, changelog, etc.?

Signed-off-by: alen_abeshov <a_abeshov@kbtu.kz>

Signed-off-by: Alen <78027095+aabeshov@users.noreply.github.com>

github-actions · 2024-08-08T19:19:13Z

Changes Analysis

Commit SHA: 8c31adc
Comparing To SHA: 00711ea

API Changes

Summary

├─┬Paths
│ ├──[➕] path (4138:3)
│ ├──[➕] path (4322:3)
│ ├──[➕] path (4074:3)
│ ├──[➕] path (4089:3)
│ └──[➕] path (4273:3)
└─┬Components
  ├──[➕] requestBodies (23452:7)
  ├──[➕] requestBodies (23447:7)
  ├──[➕] requestBodies (23407:7)
  ├──[➕] requestBodies (23412:7)
  ├──[➕] responses (25648:7)
  ├──[➕] responses (25695:7)
  ├──[➕] responses (25606:7)
  ├──[➕] responses (25726:7)
  ├──[➕] responses (25618:7)
  ├──[➕] responses (25708:7)
  ├──[➕] responses (25600:7)
  ├──[➕] responses (25642:7)
  ├──[➕] responses (25636:7)
  ├──[➕] responses (25702:7)
  ├──[➕] responses (25738:7)
  ├──[➕] responses (25732:7)
  ├──[➕] responses (25657:7)
  ├──[➕] responses (25720:7)
  ├──[➕] responses (25630:7)
  ├──[➕] responses (25624:7)
  ├──[➕] responses (25579:7)
  ├──[➕] responses (25588:7)
  ├──[➕] responses (25714:7)
  ├──[➕] parameters (20247:7)
  ├──[➕] parameters (20254:7)
  ├──[➕] parameters (20115:7)
  ├──[➕] parameters (20107:7)
  ├──[➕] parameters (20123:7)
  ├──[➕] parameters (20131:7)
  ├──[➕] parameters (20139:7)
  ├──[➕] schemas (48041:7)
  ├──[➕] schemas (48222:7)
  ├──[➕] schemas (47951:7)
  ├──[➕] schemas (48118:7)
  ├──[➕] schemas (47972:7)
  ├──[➕] schemas (47902:7)
  ├──[➕] schemas (47994:7)
  ├──[➕] schemas (48096:7)
  ├──[➕] schemas (48156:7)
  ├──[➕] schemas (48166:7)
  ├──[➕] schemas (48016:7)
  ├──[➕] schemas (47982:7)
  ├──[➕] schemas (47913:7)
  ├──[➕] schemas (47885:7)
  ├──[➕] schemas (48076:7)
  ├──[➕] schemas (48054:7)
  ├──[➕] schemas (47933:7)
  ├──[➕] schemas (48066:7)
  ├──[➕] schemas (48205:7)
  ├──[➕] schemas (48128:7)
  ├──[➕] schemas (48170:7)
  ├──[➕] schemas (48177:7)
  └──[➕] schemas (48086:7)

Document Element	Total Changes	Breaking Changes
paths	5	0
components	53	0

Total Changes: 58
Additions: 58

Report

The full API changes report is available at: https://github.com/opensearch-project/opensearch-api-specification/actions/runs/10345881955/artifacts/1800221864

API Coverage

	Before	After	Δ
Covered (%)	510 (49.95 %)	522 (51.13 %)	12 (1.18 %)
Uncovered (%)	511 (50.05 %)	499 (48.87 %)	-12 (-1.18 %)
Unknown	24	24	0

github-actions · 2024-08-08T19:19:28Z

Spec Test Coverage Analysis

Total	Tested
546	125 (22.89 %)

nhtruong · 2024-08-09T14:27:07Z

Agree with @dblock. This is a lot in one PR esp once you add the tests. Do break it into smaller PRs and thank you so much for the hard work :)

Signed-off-by: alen_abeshov <a_abeshov@kbtu.kz>

aabeshov · 2024-08-09T15:38:25Z

Agree with @dblock. This is a lot in one PR esp once you add the tests. Do break it into smaller PRs and thank you so much for the hard work :)

It's okay ,thanks, I guess I am close to the finish line with these specs
Check Links is failing, is it connected with test stories that I create ?
Because my test stories are passing locally

dblock · 2024-08-09T17:39:38Z

Looks like only the link checker is failing against http://localhost:9090 which is mentioned in a test.

Add an exclude file like here or change the test?

tests/observability/observability.yaml

Signed-off-by: alen_abeshov <a_abeshov@kbtu.kz>

aabeshov · 2024-08-09T20:23:38Z

I changed the lychee ignore. --exclude-file is deprecated.
BTW, I haven't implemented /_plugins/_query/settings: ["PUT"] this api , because I guess @Tokesh has already made it in this PR [#456 ]

dblock

This looks good! I have some small asks and let's get this merged.

.cspell

.lycheeignore

spec/namespaces/observability.yaml

tests/observability/datasources.yaml

Signed-off-by: alen_abeshov <a_abeshov@kbtu.kz>

aabeshov · 2024-08-09T21:12:29Z

@dblock ,I don't know why but there is a 400 bad request when test story is making DELETE /_plugins/_query/_datasources/{datasource_name}
However locally it's okay

dblock · 2024-08-09T21:22:09Z

@dblock ,I don't know why but there is a 400 bad request when test story is making DELETE /_plugins/_query/_datasources/{datasource_name} However locally it's okay

We should show the detailed output for these errors (please open an issue).

For now, add a --verbose into the GHA workflow to see the details? Note that it fails on older versions, maybe a quick start of a container with 1.3.x locally will reproduce it?

Signed-off-by: alen_abeshov <a_abeshov@kbtu.kz>

aabeshov · 2024-08-10T08:36:26Z

@dblock ,I don't know why but there is a 400 bad request when test story is making DELETE /_plugins/_query/_datasources/{datasource_name} However locally it's okay

We should show the detailed output for these errors (please open an issue).

For now, add a --verbose into the GHA workflow to see the details? Note that it fails on older versions, maybe a quick start of a container with 1.3.x locally will reproduce it?

[INFO] => DELETE /_plugins/_query/_datasources/test_datasource ({}) [application/json]
[INFO] <= 400 (application/json) | "no handler found for uri [/_plugins/_query/_datasources/test_datasource] and method [DELETE]"
[INFO] => DELETE /_plugins/_query/_datasources/test_datasource ({}) [application/json]
[INFO] <= 400 (application/json) | "no handler found for uri [/_plugins/_query/_datasources/test_datasource] and method [DELETE]"

ERROR query/datasources.yaml (/home/runner/work/opensearch-api-specification/opensearch-api-specification/tests/default/query/datasources.yaml)
ERROR PROLOGUES
ERROR DELETE /_plugins/_query/_datasources/{datasource_name} (Bad Request)
SKIPPED CHAPTERS
SKIPPED Create a new query datasource.
SKIPPED Retrieve the list of all query datasources.
SKIPPED Retrieve a specific query datasource by name.
SKIPPED Update an existing query datasource.
SKIPPED Retrieve the updated query datasource by name.
SKIPPED Delete the query datasource by name.
ERROR EPILOGUES
ERROR DELETE /_plugins/_query/_datasources/{datasource_name} (Bad Request)

Here is the details. I am not sure, but maybe there is no such api in old versions

Tokesh · 2024-08-10T18:15:42Z

spec/namespaces/observability.yaml

+    get:
+      operationId: observability.get_localstats.0
+      x-operation-group: observability.get_localstats
+      x-version-added: '1.1'


please recheck x-version-added. i think it can fix your ci issue
especially for APIs where you have failed tests
i tested your branch locally with latest version, everything works fine

@Tokesh committed changes , you can check

Signed-off-by: alen_abeshov <a_abeshov@kbtu.kz>

dblock

Looks good. I have 2 asks below.

.github/workflows/test-spec.yml

.lycheeignore

tests/default/query/datasources.yaml

Signed-off-by: alen_abeshov <a_abeshov@kbtu.kz>

dblock · 2024-08-12T15:22:37Z

Looks like one of the tests here may fail intermittently:

        FAILED  Retrieve list of Observability objects.
            PASSED  REQUEST BODY
            PASSED  RESPONSE STATUS
            FAILED  RESPONSE PAYLOAD BODY (expected totalHits='0', got '1')
            PASSED  RESPONSE PAYLOAD SCHEMA
            SKIPPED RESPONSE OUTPUT VALUES

https://github.com/opensearch-project/opensearch-api-specification/actions/runs/10354531782/job/28660019203?pr=493

@aabeshov care to take a look please?

aabeshov · 2024-08-12T18:40:09Z

Looks like one of the tests here may fail intermittently:
        FAILED  Retrieve list of Observability objects.
            PASSED  REQUEST BODY
            PASSED  RESPONSE STATUS
            FAILED  RESPONSE PAYLOAD BODY (expected totalHits='0', got '1')
            PASSED  RESPONSE PAYLOAD SCHEMA
            SKIPPED RESPONSE OUTPUT VALUES
https://github.com/opensearch-project/opensearch-api-specification/actions/runs/10354531782/job/28660019203?pr=493

@aabeshov care to take a look please?

@dblock I guess I found the location of error, I have no idea why it's not persistent fail in test story.
Should I create new PR ?

dblock · 2024-08-12T18:43:43Z

Thanks! I fixed it in #497 since it was failing often. Does that look right? Next time don't hold back and just make a PR :)

aabeshov · 2024-08-12T18:52:20Z

Thanks! I fixed it in #497 since it was failing often. Does that look right? Next time don't hold back and just make a PR :)

Thanks , but I am really confused because in chapters first I do get request to specific object and it return that It exists then right after this step I do request to get list of all objects and sometimes it returns that list is empty that why it is expected for index to be 0 in some cases.

dblock · 2024-08-12T18:56:41Z

Writing something doesn't guarantee that it will be read right away, it's eventually consistent. Since these options are backed by an index, refreshing means waiting for a full refresh cycle where everything written gets read consistently, making it predictable. Does this help?

https://opensearch.org/docs/latest/api-reference/index-apis/refresh/

aabeshov · 2024-08-14T11:47:44Z

Writing something doesn't guarantee that it will be read right away, it's eventually consistent. Since these options are backed by an index, refreshing means waiting for a full refresh cycle where everything written gets read consistently, making it predictable. Does this help?

https://opensearch.org/docs/latest/api-reference/index-apis/refresh/

aaa okay now I got this, thanks!

Observability api and query api

58d84db

Signed-off-by: alen_abeshov <a_abeshov@kbtu.kz>

aabeshov requested review from dblock, harshavamsi, sachetalva, nhtruong, Xtansia, VachaShah and Tokesh as code owners August 7, 2024 18:42

dblock requested changes Aug 7, 2024

View reviewed changes

aabeshov and others added 4 commits August 8, 2024 23:53

Fixing wrong linting

5b4e1d8

Signed-off-by: alen_abeshov <a_abeshov@kbtu.kz>

added tests

06bbead

Signed-off-by: alen_abeshov <a_abeshov@kbtu.kz>

removed docker file

d853847

Signed-off-by: alen_abeshov <a_abeshov@kbtu.kz>

Merge branch 'main' into observation_api

c94c8f6

Signed-off-by: Alen <78027095+aabeshov@users.noreply.github.com>

Added tests

be75db3

Signed-off-by: alen_abeshov <a_abeshov@kbtu.kz>

dblock reviewed Aug 9, 2024

View reviewed changes

tests/observability/observability.yaml Outdated Show resolved Hide resolved

aabeshov added 2 commits August 10, 2024 01:15

Removed spaces, added exclude file

63306f3

Signed-off-by: alen_abeshov <a_abeshov@kbtu.kz>

Changed lycheeignore

b0a68d6

Signed-off-by: alen_abeshov <a_abeshov@kbtu.kz>

aabeshov requested a review from dblock August 9, 2024 20:21

dblock requested changes Aug 9, 2024

View reviewed changes

.cspell Outdated Show resolved Hide resolved

.lycheeignore Outdated Show resolved Hide resolved

spec/namespaces/observability.yaml Outdated Show resolved Hide resolved

tests/observability/datasources.yaml Outdated Show resolved Hide resolved

aabeshov added 2 commits August 10, 2024 02:03

Fixing path

1cf5569

Signed-off-by: alen_abeshov <a_abeshov@kbtu.kz>

Change URL

be3f0be

Signed-off-by: alen_abeshov <a_abeshov@kbtu.kz>

add verbosity

065bf9c

Signed-off-by: alen_abeshov <a_abeshov@kbtu.kz>

Tokesh reviewed Aug 10, 2024

View reviewed changes

aabeshov added 2 commits August 10, 2024 23:25

Checking version

9e6a635

Signed-off-by: alen_abeshov <a_abeshov@kbtu.kz>

changin operation version

1b29f4b

Signed-off-by: alen_abeshov <a_abeshov@kbtu.kz>

dblock requested changes Aug 11, 2024

View reviewed changes

.github/workflows/test-spec.yml Outdated Show resolved Hide resolved

.lycheeignore Outdated Show resolved Hide resolved

tests/default/query/datasources.yaml Outdated Show resolved Hide resolved

aabeshov and others added 5 commits August 12, 2024 09:18

Replacing http

4ac8c0d

Signed-off-by: alen_abeshov <a_abeshov@kbtu.kz>

Merge branch 'opensearch-project:main' into observation_api

bb83106

Changed desc

76c3cd8

Signed-off-by: alen_abeshov <a_abeshov@kbtu.kz>

Changed desc

40c13da

Signed-off-by: alen_abeshov <a_abeshov@kbtu.kz>

Fixing bug

8c31adc

Signed-off-by: alen_abeshov <a_abeshov@kbtu.kz>

aabeshov requested review from dblock and Tokesh August 12, 2024 04:27

dblock approved these changes Aug 12, 2024

View reviewed changes

dblock merged commit 644a1a5 into opensearch-project:main Aug 12, 2024
17 checks passed

This was referenced Aug 12, 2024

Request sort in rest_total_hits_as_int tests #493

Merged

[BUG] Retrieve list of Observability objects is flaky #495

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Observability namespace #474

Observability namespace #474

aabeshov commented Aug 7, 2024 •

edited by dblock

Loading

dblock left a comment

github-actions bot commented Aug 8, 2024 •

edited

Loading

github-actions bot commented Aug 8, 2024 •

edited

Loading

nhtruong commented Aug 9, 2024

aabeshov commented Aug 9, 2024

dblock commented Aug 9, 2024

aabeshov commented Aug 9, 2024

dblock left a comment

aabeshov commented Aug 9, 2024 •

edited

Loading

dblock commented Aug 9, 2024 •

edited

Loading

aabeshov commented Aug 10, 2024 •

edited

Loading

Tokesh Aug 10, 2024 •

edited

Loading

aabeshov Aug 10, 2024 •

edited

Loading

dblock left a comment

dblock commented Aug 12, 2024

aabeshov commented Aug 12, 2024

dblock commented Aug 12, 2024

aabeshov commented Aug 12, 2024 •

edited

Loading

dblock commented Aug 12, 2024 •

edited by aabeshov

Loading

aabeshov commented Aug 14, 2024

Observability namespace #474

Observability namespace #474

Conversation

aabeshov commented Aug 7, 2024 • edited by dblock Loading

Description

Issues Resolved

dblock left a comment

Choose a reason for hiding this comment

github-actions bot commented Aug 8, 2024 • edited Loading

Changes Analysis

API Changes

Summary

Report

API Coverage

github-actions bot commented Aug 8, 2024 • edited Loading

Spec Test Coverage Analysis

nhtruong commented Aug 9, 2024

aabeshov commented Aug 9, 2024

dblock commented Aug 9, 2024

aabeshov commented Aug 9, 2024

dblock left a comment

Choose a reason for hiding this comment

aabeshov commented Aug 9, 2024 • edited Loading

dblock commented Aug 9, 2024 • edited Loading

aabeshov commented Aug 10, 2024 • edited Loading

Tokesh Aug 10, 2024 • edited Loading

Choose a reason for hiding this comment

aabeshov Aug 10, 2024 • edited Loading

Choose a reason for hiding this comment

dblock left a comment

Choose a reason for hiding this comment

dblock commented Aug 12, 2024

aabeshov commented Aug 12, 2024

dblock commented Aug 12, 2024

aabeshov commented Aug 12, 2024 • edited Loading

dblock commented Aug 12, 2024 • edited by aabeshov Loading

aabeshov commented Aug 14, 2024

aabeshov commented Aug 7, 2024 •

edited by dblock

Loading

github-actions bot commented Aug 8, 2024 •

edited

Loading

github-actions bot commented Aug 8, 2024 •

edited

Loading

aabeshov commented Aug 9, 2024 •

edited

Loading

dblock commented Aug 9, 2024 •

edited

Loading

aabeshov commented Aug 10, 2024 •

edited

Loading

Tokesh Aug 10, 2024 •

edited

Loading

aabeshov Aug 10, 2024 •

edited

Loading

aabeshov commented Aug 12, 2024 •

edited

Loading

dblock commented Aug 12, 2024 •

edited by aabeshov

Loading