[dataset-nav][bug] get connections by cluster and update client namespace #7609

kavilla · 2024-08-01T21:23:25Z

Description

There was a conflict in namespaces when registering a transport client with OpenSearch cluster. The schemas were also conflicting with observability and data source mgmt endpoints.

Modifying the namespace for query enhancements to avoid conflicts and then initialize the external datasource connections on load.

Issues Resolved

n/a

Screenshot

Testing the changes

Changelog

fix: dataset nav to load external connections and update namespace

Check List

All tests pass
- yarn test:jest
- yarn test:jest_integration
New functionality includes testing.
New functionality has been documented.
Update CHANGELOG.md
Commits are signed per the DCO using --signoff

github-actions · 2024-08-01T21:23:59Z

❌ Empty Changelog Section

The Changelog section in your PR description is empty. Please add a valid changelog entry or entries. If you did add a changelog entry, check to make sure that it was not accidentally included inside the comment block in the Changelog section.

codecov · 2024-08-01T21:42:49Z

Codecov Report

Attention: Patch coverage is 18.08511% with 77 lines in your changes missing coverage. Please review.

Project coverage is 63.64%. Comparing base (61d5c49) to head (19aa833).

Files	Patch %	Lines
...taset_navigator/lib/catalog_cache/cache_manager.ts	0.00%	19 Missing ⚠️
...plugins/query_enhancements/server/utils/plugins.ts	5.26%	18 Missing ⚠️
.../public/ui/dataset_navigator/dataset_navigator.tsx	0.00%	11 Missing ⚠️
...taset_navigator/lib/catalog_cache/cache_loader.tsx	0.00%	10 Missing ⚠️
...navigator/lib/utils/fetch_external_data_sources.ts	11.11%	8 Missing ⚠️
...c/plugins/query_enhancements/server/utils/facet.ts	0.00%	6 Missing ⚠️
src/plugins/data/public/ui/settings/settings.ts	28.57%	5 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #7609   +/-   ##
=======================================
  Coverage   63.64%   63.64%           
=======================================
  Files        3634     3632    -2     
  Lines       80099    80043   -56     
  Branches    12682    12677    -5     
=======================================
- Hits        50977    50943   -34     
+ Misses      26015    25993   -22     
  Partials     3107     3107

Flag	Coverage Δ
Linux_1	`30.53% <22.05%> (-0.03%)`	⬇️
Linux_2	`55.58% <ø> (ø)`
Linux_3	`40.55% <22.05%> (-0.03%)`	⬇️
Linux_4	`31.29% <18.08%> (-0.04%)`	⬇️
Windows_1	`30.55% <22.05%> (-0.03%)`	⬇️
Windows_2	`55.53% <ø> (ø)`
Windows_3	`40.56% <22.05%> (-0.03%)`	⬇️
Windows_4	`31.29% <18.08%> (-0.04%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Kawika Avilla <kavilla414@gmail.com>

Signed-off-by: abbyhu2000 <abigailhu2000@gmail.com>

Signed-off-by: Kawika Avilla <kavilla414@gmail.com>

Signed-off-by: Sean Li <lnse@amazon.com>

ashwin-pc

LGTM, small nits but none are blockers

ashwin-pc · 2024-08-01T23:58:30Z

src/plugins/data/public/ui/dataset_navigator/lib/constants.ts

-export const ASYNC_QUERY_SESSION_ID = 'async-query-session-id';
-export const ASYNC_QUERY_EXTERNAL_DATASOURCES_CACHE = 'async_query_external_datasources_cache';
-export const RECENT_DATASET_OPTIONS_CACHE = 'recent_dataset_options_cache';
+export const ASYNC_QUERY = {


I like this structure, nice

ashwin-pc · 2024-08-02T00:00:54Z

src/plugins/query_enhancements/server/plugin.ts

@@ -18,17 +18,16 @@ import { ConfigSchema } from '../common/config';
 import { defineRoutes } from './routes';
 import {
  pplSearchStrategyProvider,
+  pplRawSearchStrategyProvider,


nit: are we using this?

ashwin-pc · 2024-08-02T00:03:11Z

src/plugins/query_enhancements/server/search/index.ts

@@ -4,5 +4,6 @@
 */

 export { pplSearchStrategyProvider } from './ppl_search_strategy';
+export { pplRawSearchStrategyProvider } from './ppl_raw_search_strategy';


nit: I dont see this being referenced anywhere

it probably wasn't required but i just moved it to the right location it was previously being exported differently than the other search strategies.,

…pace (#7609) * changes from abby Signed-off-by: Kawika Avilla <kavilla414@gmail.com> * refactor api Signed-off-by: abbyhu2000 <abigailhu2000@gmail.com> * polling db not populating Signed-off-by: Kawika Avilla <kavilla414@gmail.com> * fixing datasource cache Signed-off-by: Sean Li <lnse@amazon.com> * fixing session id stuff and running async queries Signed-off-by: Sean Li <lnse@amazon.com> * Changeset file for PR #7609 created/updated --------- Signed-off-by: Kawika Avilla <kavilla414@gmail.com> Signed-off-by: abbyhu2000 <abigailhu2000@gmail.com> Signed-off-by: Sean Li <lnse@amazon.com> Co-authored-by: abbyhu2000 <abigailhu2000@gmail.com> Co-authored-by: Sean Li <lnse@amazon.com> Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com> (cherry picked from commit d09895a) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…pace (#7609) (#7611) * changes from abby * refactor api * polling db not populating * fixing datasource cache * fixing session id stuff and running async queries * Changeset file for PR #7609 created/updated --------- (cherry picked from commit d09895a) Signed-off-by: Kawika Avilla <kavilla414@gmail.com> Signed-off-by: abbyhu2000 <abigailhu2000@gmail.com> Signed-off-by: Sean Li <lnse@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: abbyhu2000 <abigailhu2000@gmail.com> Co-authored-by: Sean Li <lnse@amazon.com> Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com>

…pace (#7609) * changes from abby Signed-off-by: Kawika Avilla <kavilla414@gmail.com> * refactor api Signed-off-by: abbyhu2000 <abigailhu2000@gmail.com> * polling db not populating Signed-off-by: Kawika Avilla <kavilla414@gmail.com> * fixing datasource cache Signed-off-by: Sean Li <lnse@amazon.com> * fixing session id stuff and running async queries Signed-off-by: Sean Li <lnse@amazon.com> * Changeset file for PR #7609 created/updated --------- Signed-off-by: Kawika Avilla <kavilla414@gmail.com> Signed-off-by: abbyhu2000 <abigailhu2000@gmail.com> Signed-off-by: Sean Li <lnse@amazon.com> Co-authored-by: abbyhu2000 <abigailhu2000@gmail.com> Co-authored-by: Sean Li <lnse@amazon.com> Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com> (cherry picked from commit d09895a) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…pace (#7609) (#7610) * changes from abby * refactor api * polling db not populating * fixing datasource cache * fixing session id stuff and running async queries * Changeset file for PR #7609 created/updated --------- (cherry picked from commit d09895a) Signed-off-by: Kawika Avilla <kavilla414@gmail.com> Signed-off-by: abbyhu2000 <abigailhu2000@gmail.com> Signed-off-by: Sean Li <lnse@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: abbyhu2000 <abigailhu2000@gmail.com> Co-authored-by: Sean Li <lnse@amazon.com> Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com>

…pace (#7609) (#7610) * changes from abby * refactor api * polling db not populating * fixing datasource cache * fixing session id stuff and running async queries * Changeset file for PR #7609 created/updated --------- (cherry picked from commit d09895a) Signed-off-by: Kawika Avilla <kavilla414@gmail.com> Signed-off-by: abbyhu2000 <abigailhu2000@gmail.com> Signed-off-by: Sean Li <lnse@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: abbyhu2000 <abigailhu2000@gmail.com> Co-authored-by: Sean Li <lnse@amazon.com> Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com> (cherry picked from commit 7605de7) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…pace (#7609) (#7610) (#8258) * changes from abby * refactor api * polling db not populating * fixing datasource cache * fixing session id stuff and running async queries * Changeset file for PR #7609 created/updated --------- (cherry picked from commit d09895a) (cherry picked from commit 7605de7) Signed-off-by: Kawika Avilla <kavilla414@gmail.com> Signed-off-by: abbyhu2000 <abigailhu2000@gmail.com> Signed-off-by: Sean Li <lnse@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: abbyhu2000 <abigailhu2000@gmail.com> Co-authored-by: Sean Li <lnse@amazon.com> Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com>

…nt namespace (opensearch-project#7609) (opensearch-project#7610) (opensearch-project#8258)" This reverts commit 9a2f697.

github-actions bot added the distinguished-contributor label Aug 1, 2024

github-actions bot added the failed changeset label Aug 1, 2024

prudhvigodithi mentioned this pull request Aug 1, 2024

[RELEASE] Release version 2.16.0 opensearch-project/opensearch-build#4771

Closed

74 tasks

kavilla and others added 5 commits August 1, 2024 23:49

changes from abby

53ad67e

Signed-off-by: Kawika Avilla <kavilla414@gmail.com>

refactor api

e816d12

Signed-off-by: abbyhu2000 <abigailhu2000@gmail.com>

polling db not populating

28b3bf4

Signed-off-by: Kawika Avilla <kavilla414@gmail.com>

fixing datasource cache

55ec24f

Signed-off-by: Sean Li <lnse@amazon.com>

fixing session id stuff and running async queries

ffcfd8a

Signed-off-by: Sean Li <lnse@amazon.com>

kavilla force-pushed the kavilla/abbys3nonclean2 branch from a28db12 to ffcfd8a Compare August 1, 2024 23:49

kavilla marked this pull request as ready for review August 1, 2024 23:50

kavilla requested review from ananzh, AMoo-Miki, ashwin-pc, joshuarrrr, abbyhu2000, zengyan-amazon, zhongnansu, manasvinibs, ZilongX, Flyingliuhub, curq, bandinib-amzn, SuZhou-Joe, ruanyl, BionIT, xinruiba, zhyuanqi and mengweieric as code owners August 1, 2024 23:50

kavilla requested review from LDrago27, virajsanghvi, sejli and joshuali925 as code owners August 1, 2024 23:50

kavilla added backport 2.x v2.16.0 backport 2.16 labels Aug 1, 2024

Changeset file for PR opensearch-project#7609 created/updated

19aa833

github-actions bot removed the failed changeset label Aug 1, 2024

kavilla changed the title ~~[dataset-nav] get connections by cluster~~ [dataset-nav][bug] get connections by cluster and update client namespace Aug 1, 2024

ashwin-pc approved these changes Aug 2, 2024

View reviewed changes

abbyhu2000 approved these changes Aug 2, 2024

View reviewed changes

ashwin-pc merged commit d09895a into opensearch-project:main Aug 2, 2024
67 of 68 checks passed

opensearch-trigger-bot bot mentioned this pull request Aug 2, 2024

[Backport 2.x] [dataset-nav][bug] get connections by cluster and update client namespace #7610

Merged

opensearch-trigger-bot bot mentioned this pull request Aug 2, 2024

[Backport 2.16] [dataset-nav][bug] get connections by cluster and update client namespace #7611

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[dataset-nav][bug] get connections by cluster and update client namespace #7609

[dataset-nav][bug] get connections by cluster and update client namespace #7609

kavilla commented Aug 1, 2024 •

edited

Loading

github-actions bot commented Aug 1, 2024

codecov bot commented Aug 1, 2024 •

edited

Loading

ashwin-pc left a comment

ashwin-pc Aug 1, 2024

ashwin-pc Aug 2, 2024

ashwin-pc Aug 2, 2024

kavilla Aug 2, 2024

[dataset-nav][bug] get connections by cluster and update client namespace #7609

[dataset-nav][bug] get connections by cluster and update client namespace #7609

Conversation

kavilla commented Aug 1, 2024 • edited Loading

Description

Issues Resolved

Screenshot

Testing the changes

Changelog

Check List

github-actions bot commented Aug 1, 2024

❌ Empty Changelog Section

codecov bot commented Aug 1, 2024 • edited Loading

Codecov Report

ashwin-pc left a comment

Choose a reason for hiding this comment

ashwin-pc Aug 1, 2024

Choose a reason for hiding this comment

ashwin-pc Aug 2, 2024

Choose a reason for hiding this comment

ashwin-pc Aug 2, 2024

Choose a reason for hiding this comment

kavilla Aug 2, 2024

Choose a reason for hiding this comment

kavilla commented Aug 1, 2024 •

edited

Loading

codecov bot commented Aug 1, 2024 •

edited

Loading