Fix Datasource testing connection don't validate endpoints with path #5663

xinruiba · 2024-01-05T22:47:54Z

Task List

[BUG] Cluster endpoint test connection passed unexpectly #5707
[BUG] Cluster endpoint URL pattern validation for Server and Serverless domain in UX component before sending test connection request (Optional) #5706

Description

Test connection should only success when opensearch-client response status code is 200 and

response body not empty (with serverless collection)
response body have cluster_name field (with server cluster)
Please check this ticket for more details [BUG] Cluster endpoint test connection passed unexpectly #5707

01/19/2024 Update:

Create an initial fix based on discussion, and more details can be found under this issue: #5707

01/09/2024 Update:

Discussed with team (checkout: #5663 (comment))

We are going to take a further investigation and we put the first proposal under the issue description:
#5656

CC @Flyingliuhub @seraphjiang @bandinib-amzn please feel free to leave comments~ Thanks

Issues Resolved

fixes #5656

Fixes #5702

Screenshot

Before:
Test connection with endpoint: "https://google.com" will pass which is unexpected

After:
Test connection with endpoint: "https://google.com" will be failed to connect with expected response message:

Also verified endpoint with path name will fail test connection:

Validate cluster endpoint still able to pass connection testing:

Testing the changes

Check List

All tests pass
- yarn test:jest
- yarn test:jest_integration
New functionality includes testing.
New functionality has been documented.
Update CHANGELOG.md
Commits are signed per the DCO using --signoff

…pensearch-project#5656 Signed-off-by: Xinrui Bai <xinruiba@amazon.com>

src/plugins/data_source_management/public/components/utils.ts

codecov · 2024-01-05T23:01:59Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (e83b7ee) 67.03% compared to head (77b9ce6) 67.03%.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #5663   +/-   ##
=======================================
  Coverage   67.03%   67.03%           
=======================================
  Files        3295     3296    +1     
  Lines       63329    63339   +10     
  Branches    10084    10087    +3     
=======================================
+ Hits        42451    42459    +8     
- Misses      18429    18430    +1     
- Partials     2449     2450    +1

Flag	Coverage Δ
Linux_1	`35.23% <ø> (ø)`
Linux_2	`55.18% <ø> (ø)`
Linux_3	`43.90% <100.00%> (+<0.01%)`	⬆️
Linux_4	`35.33% <ø> (ø)`
Windows_1	`35.26% <ø> (ø)`
Windows_2	`55.15% <ø> (ø)`
Windows_3	`43.92% <100.00%> (+0.01%)`	⬆️
Windows_4	`35.33% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Xinrui Bai <xinruiba@amazon.com>

...ugins/data_source_management/public/components/validation/datasource_form_validation.test.ts

bandinib-amzn · 2024-01-08T19:35:49Z

Also I think it would be great if we can run this fix by UX team to see if it is ok from users perspective.

xinruiba · 2024-01-08T22:00:57Z

Also I think it would be great if we can run this fix by UX team to see if it is ok from users perspective.

Thanks for the comment. I'm reaching out to our UX team for more inputs.

Signed-off-by: Xinrui Bai <xinruiba@amazon.com>

seraphjiang · 2024-01-08T22:23:07Z

Also I think it would be great if we can run this fix by UX team to see if it is ok from users perspective.

@kgcreative who could help to work with us on this.

kgcreative · 2024-01-09T02:06:33Z

I'm happy to review async here. To that end, i have a few questions before i make a recommendation.

When we say "OpenSearch Endpoint" we mean the cluster endpoint, and not an OSD endpoint, correct? If so, then "test connection" should handshake with the endpoint, and there should be some sort of response that validates that this is indeed the correct endpoint.
when it comes to there being a path, is there a legitimate usecase for an endpoint having a path in it? for example: https://www.bank.com/opensearch and if so, how do we differentiate between that and say https://opensearch.bank.com
I don't think we should disable the test connection button, instead, i think if there is an appended path, test connection should return an error, at which point the endpoint URL input should change to failed (red)

...ugins/data_source_management/public/components/validation/datasource_form_validation.test.ts

Flyingliuhub · 2024-01-09T18:44:33Z

I'm happy to review async here. To that end, i have a few questions before i make a recommendation.

1. When we say "OpenSearch Endpoint" we mean the cluster endpoint, and not an OSD endpoint, correct? If so, then "test connection" should handshake with the endpoint, and there should be some sort of response that validates that this is indeed the correct endpoint.

2. when it comes to there being a path, is there a legitimate usecase for an endpoint having a path in it? for example: `https://www.bank.com/opensearch` and if so, how do we differentiate between that and say `https://opensearch.bank.com`

3. I don't think we should disable the `test connection` button, instead, i think if there is an appended path, `test connection` should return an error, at which point the endpoint URL input should change to failed (red)

Thanks, @kgcreative, for the great suggestions. I agree with you.
There are a couple of approaches which we can address this. @xinruiba and I will figure out more detail later on.

Endpoint ULR pattern validation, which includes OpenSearch and Opensearch serverless, we need to have a good endpoint URL pattern validation pattern to validate user's input endpoint URL to quickly provide warning if doesn't match the endpoint pattern.
Test connection validate the response, and see if the response match the OpenSearch/OpenSearch Serverless

xinruiba · 2024-01-09T19:30:40Z

I'm happy to review async here. To that end, i have a few questions before i make a recommendation.

When we say "OpenSearch Endpoint" we mean the cluster endpoint, and not an OSD endpoint, correct? If so, then "test connection" should handshake with the endpoint, and there should be some sort of response that validates that this is indeed the correct endpoint.

Answer:
Yes, it's cluster endpoint.

when it comes to there being a path, is there a legitimate usecase for an endpoint having a path in it? for example: https://www.bank.com/opensearch and if so, how do we differentiate between that and say https://opensearch.bank.com

Answer:
For now, we don't have a user case that supporting an endpoint having a path in it (@bandinib-amzn @Flyingliuhub rectify me if I'm wrong). In the future will we supported endpoint with path? I need to have more discussion with my team.

I don't think we should disable the test connection button, instead, i think if there is an appended path, test connection should return an error, at which point the endpoint URL input should change to failed (red)

Answer:
Thanks for the comment. I think your proposed user experience is:

Before user clicking the "Test Connection" Button:

User fill in the datasource creation form (with an invalidate cluster endpoint)
Test connect button still available
Endpoint URL input still green

After user clicking the "Test Connection" Button:

Banner shows up to say connection failed
Endpoint URL input becomes red

Is that correct?

For this proposal, me and @Flyingliuhub have a discussion and @Flyingliuhub already provide a response.
We will have a further discussion and keep you updated.

Thanks for the comment.

kgcreative · 2024-01-10T02:02:23Z

I think the green underline is just a focus indicator, so when you tab out of the field it should return to it's unfocused state.

… status code is 200 Signed-off-by: Xinrui Bai <xinruiba@amazon.com>

src/plugins/data_source/server/routes/data_source_connection_validator.ts

…e body Signed-off-by: Xinrui Bai <xinruiba@amazon.com>

bandinib-amzn · 2024-01-22T21:57:20Z

Failed cypress test is not related to this change, so we can ignore.

…5663) * fix Datasource testing connection don't validate endpoints with path #5656 Signed-off-by: Xinrui Bai <xinruiba@amazon.com> (cherry picked from commit b5d39b6) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> # Conflicts: # CHANGELOG.md

…5663) (#5727) * fix Datasource testing connection don't validate endpoints with path #5656 Signed-off-by: Xinrui Bai <xinruiba@amazon.com> (cherry picked from commit b5d39b6) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> # Conflicts: # CHANGELOG.md Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…pensearch-project#5663) * fix Datasource testing connection don't validate endpoints with path opensearch-project#5656 Signed-off-by: Xinrui Bai <xinruiba@amazon.com> Signed-off-by: yujin-emma <yujin.emma.work@gmail.com>

fix Datasource testing connection don't validate endpoints with path o…

9b21554

…pensearch-project#5656 Signed-off-by: Xinrui Bai <xinruiba@amazon.com>

xinruiba requested review from ananzh, kavilla, AMoo-Miki, ashwin-pc, joshuarrrr, abbyhu2000, zengyan-amazon, kristenTian, zhongnansu, manasvinibs, ZilongX, Flyingliuhub, BSFishy, curq, bandinib-amzn and SuZhou-Joe as code owners January 5, 2024 22:47

github-actions bot added the first-time-contributor label Jan 5, 2024

Flyingliuhub reviewed Jan 5, 2024

View reviewed changes

src/plugins/data_source_management/public/components/utils.ts Outdated Show resolved Hide resolved

Flyingliuhub previously approved these changes Jan 5, 2024

View reviewed changes

add changelog

9577168

Signed-off-by: Xinrui Bai <xinruiba@amazon.com>

xinruiba dismissed Flyingliuhub’s stale review via 9577168 January 5, 2024 23:12

xinruiba requested a review from Flyingliuhub January 5, 2024 23:56

Flyingliuhub previously approved these changes Jan 8, 2024

View reviewed changes

bandinib-amzn requested changes Jan 8, 2024

View reviewed changes

...ugins/data_source_management/public/components/validation/datasource_form_validation.test.ts Outdated Show resolved Hide resolved

bandinib-amzn added backport 2.x v2.12.0 labels Jan 8, 2024

bandinib-amzn mentioned this pull request Jan 8, 2024

[Meta] Security, bug fix and enhancement for dashboards anywhere #5667

Closed

6 tasks

Update unit test cases

092791b

Signed-off-by: Xinrui Bai <xinruiba@amazon.com>

xinruiba dismissed Flyingliuhub’s stale review via 092791b January 8, 2024 22:18

Merge branch 'opensearch-project:main' into DataSource_URLValidationFix

5eac590

wanglam reviewed Jan 9, 2024

View reviewed changes

...ugins/data_source_management/public/components/validation/datasource_form_validation.test.ts Outdated Show resolved Hide resolved

xinruiba added 2 commits January 19, 2024 23:09

Merge branch 'main' into DataSource_URLValidationFix

ab79f10

Pass datasource test connection only when opensearchclient's response…

d650f3d

… status code is 200 Signed-off-by: Xinrui Bai <xinruiba@amazon.com>

Flyingliuhub reviewed Jan 22, 2024

View reviewed changes

src/plugins/data_source/server/routes/data_source_connection_validator.ts Outdated Show resolved Hide resolved

Update vaclidation logic to all validate on opensearch client respons…

77b9ce6

…e body Signed-off-by: Xinrui Bai <xinruiba@amazon.com>

bandinib-amzn approved these changes Jan 22, 2024

View reviewed changes

Flyingliuhub approved these changes Jan 23, 2024

View reviewed changes

Flyingliuhub merged commit b5d39b6 into opensearch-project:main Jan 23, 2024
70 checks passed

opensearch-trigger-bot bot mentioned this pull request Jan 23, 2024

[Backport 2.x] Fix Datasource testing connection don't validate endpoints with path #5727

Merged

xinruiba mentioned this pull request Jan 23, 2024

[BUG] Cluster endpoint test connection passed unexpectly #5707

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix Datasource testing connection don't validate endpoints with path #5663

Fix Datasource testing connection don't validate endpoints with path #5663

xinruiba commented Jan 5, 2024 •

edited

Loading

codecov bot commented Jan 5, 2024 •

edited

Loading

bandinib-amzn commented Jan 8, 2024

xinruiba commented Jan 8, 2024

seraphjiang commented Jan 8, 2024 •

edited

Loading

kgcreative commented Jan 9, 2024 •

edited

Loading

Flyingliuhub commented Jan 9, 2024

xinruiba commented Jan 9, 2024 •

edited

Loading

kgcreative commented Jan 10, 2024

bandinib-amzn commented Jan 22, 2024

Fix Datasource testing connection don't validate endpoints with path #5663

Fix Datasource testing connection don't validate endpoints with path #5663

Conversation

xinruiba commented Jan 5, 2024 • edited Loading

Task List

Description

01/19/2024 Update:

01/09/2024 Update:

Issues Resolved

Screenshot

Testing the changes

Check List

codecov bot commented Jan 5, 2024 • edited Loading

Codecov Report

bandinib-amzn commented Jan 8, 2024

xinruiba commented Jan 8, 2024

seraphjiang commented Jan 8, 2024 • edited Loading

kgcreative commented Jan 9, 2024 • edited Loading

Flyingliuhub commented Jan 9, 2024

xinruiba commented Jan 9, 2024 • edited Loading

kgcreative commented Jan 10, 2024

bandinib-amzn commented Jan 22, 2024

xinruiba commented Jan 5, 2024 •

edited

Loading

codecov bot commented Jan 5, 2024 •

edited

Loading

seraphjiang commented Jan 8, 2024 •

edited

Loading

kgcreative commented Jan 9, 2024 •

edited

Loading

xinruiba commented Jan 9, 2024 •

edited

Loading