[BUG] Unable to install OpenSearch Dashboards plugins using its IDs in Windows release #4952

Desvelao · 2023-07-04T14:35:25Z

Describe the bug

Hello, we are creating a custom build of OpenSearch Dashboards for Windows ( .zip) that should have some extra plugins as mentioned here: https://opensearch.org/docs/2.6/install-and-configure/install-dashboards/plugins/#available-plugins. We forked this repository and build the .zip release for Windows, then we want to install some OpenSearch Dashboards plugins.

To Reproduce
Steps to reproduce the behavior:

Generate the .zip for Windows.

yarn build-platform --windows --release

Unzip the .zip archive
Change the directory to unzipped archive and use the bin\opensearch-dashboards-plugin.bat binary to install the required OSD plugins using its ID. Example

C:\Users\vagrant\opensearch-dashboards-2.4.1-windows-x64\bin>.\opensearch-dashboards-plugin.bat install reportsDashboards
Attempting to transfer from reportsDashboards
Attempting to transfer from https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/2.4.1/latest/windows/x64/tar/builds/opensearch-dashboards/plugins/reportsDashboards-2.4.1.zip
Plugin installation was unsuccessful due to the error "No valid url specified."

I tried the URL displayed by the opensearch-dashboards-plugin.bat binary in a browser and it displays an error:

https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/2.4.1/latest/windows/x64/tar/builds/opensearch-dashboards/plugins/reportsDashboards-2.4.1.zip

I noticed that the URL contains tar in the path, and I tried to replace it with zip and it seems using this URL instead, the plugin can be installed.

https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/2.4.1/latest/windows/x64/zip/builds/opensearch-dashboards/plugins/reportsDashboards-2.4.1.zip

Note zip in windows/x64/zip/builds

Reviewing the source code, I think that I found where is the problem. The tar in the URL is hardcoded here:

2.4.1: https://github.com/opensearch-project/OpenSearch-Dashboards/blob/2.4.1/src/cli_plugin/install/settings.js#L48
2.6.0: https://github.com/opensearch-project/OpenSearch-Dashboards/blob/2.6.0/src/cli_plugin/install/settings.js#L48
2.8.0 (latest release at the moment): https://github.com/opensearch-project/OpenSearch-Dashboards/blob/2.8.0/src/cli_plugin/install/settings.js#L48
main branch: https://github.com/opensearch-project/OpenSearch-Dashboards/blob/main/src/cli_plugin/install/settings.js#L48

Unfortunately, I didn't find any source where the URL of the plugins is exposed. I don't know if there are other cases that could fail.

The current OpenSearch Dashboards distributable for Windows contains all the plugins available, but fixing this, enables the users to install/reinstall an OpenSearch Dashboards plugin if this is not included (our use case) or if was removed accidentally.

Expected behavior
The plugin should be able to download and install the OpenSearch Dashboard extra plugins using its ID.

OpenSearch Version
Not apply

Dashboards Version
2.4.1
2.6.0
2.8.0 Not checked but it should happen the problem regarding the source code (https://github.com/opensearch-project/OpenSearch-Dashboards/blob/2.8.0/src/cli_plugin/install/settings.js#L48)

Plugins

Please list all plugins currently enabled.

Screenshots

I tried with the official OpenSearch Dashboards .zip distributable and got the same results:

2.4.1
2.6.0

If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

OS: Windows Server 2019 Datacenter Evaluation
Browser and version: no apply

Additional context

Add any other context about the problem here.

The text was updated successfully, but these errors were encountered:

kavilla · 2023-07-05T16:41:40Z

Hey! @Desvelao ! Thanks for opening this!

I don't think it's necessarily related to OpenSearch Dashboards. I am also getting an unauthorized error from that URL.

@opensearch-project/admin can you transfer this to the opensearch-build repo?

bbarani · 2023-07-05T16:52:16Z

@peterzhuamazon Can you please check the plugin installation path used for Windows installation?

peterzhuamazon · 2023-07-05T19:40:05Z

Hi @Desvelao this is expected behavior and the id installation is only suitable for OpenSearch Core Plugins.
https://github.com/opensearch-project/OpenSearch/tree/main/plugins

reportsDashboards is having its own repository and it is OpenSearch-Dashboards, not OpenSearch, not core plugin.

Therefore you would need to provide the full url for it to install.

The url can be found in the manifest.yml file in the bundled version of OSD. The min version of OSD does not include that manifest file because there is no plugin bundled, and we are not recommending min version for production.

PS: @kavilla might be able to answer OSD side of things, but it is never the initial intention to install non-core plugin by the plugin id directly. It is possible that OSD core has changed things like the code you listed, but from release point of view we never have that support officially. Probably this can be a feature request.

Let me know.

Thanks.

Desvelao · 2023-07-06T07:19:06Z

Hi, thank you very much for your quick response!

@peterzhuamazon , if we use the Linux tarball instead of the Windows .zip, it is possible to install the non-core plugins by their id.

Evidence:

vagrant@ubuntu-2204:~$ wget https://artifacts.opensearch.org/releases/bundle/opensearch-dashboards/2.4.1/opensearch-dashboards-2.4.1-linux-x64.tar.gz
--2023-07-06 06:28:40--  https://artifacts.opensearch.org/releases/bundle/opensearch-dashboards/2.4.1/opensearch-dashboards-2.4.1-linux-x64.tar.gz
Resolving artifacts.opensearch.org (artifacts.opensearch.org)... 18.67.240.45, 18.67.240.13, 18.67.240.6, ...
Connecting to artifacts.opensearch.org (artifacts.opensearch.org)|18.67.240.45|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 354289090 (338M) [application/x-gzip]
Saving to: ‘opensearch-dashboards-2.4.1-linux-x64.tar.gz’

opensearch-dashboar 100%[===================>] 337.88M  12.0MB/s    in 29s     

2023-07-06 06:29:09 (11.7 MB/s) - ‘opensearch-dashboards-2.4.1-linux-x64.tar.gz’ saved [354289090/354289090]

vagrant@ubuntu-2204:~$ tar -zxf opensearch-dashboards-2.4.1-linux-x64.tar.gz 
vagrant@ubuntu-2204:~$ cd opensearch-dashboards-2.4.1
vagrant@ubuntu-2204:~/opensearch-dashboards-2.4.1$ ls
assets  bin  config  data  LICENSE.txt  manifest.yml  node  node_modules  NOTICE.txt  package.json  plugins  README.txt  src
vagrant@ubuntu-2204:~/opensearch-dashboards-2.4.1$ bin/opensearch-dashboards-plugin list
alertingDashboards@2.4.1.0
anomalyDetectionDashboards@2.4.1.0
customImportMapDashboards@2.4.1.0
ganttChartDashboards@2.4.1.0
indexManagementDashboards@2.4.1.0
notificationsDashboards@2.4.1.0
observabilityDashboards@2.4.1.0
queryWorkbenchDashboards@2.4.1.0
reportsDashboards@2.4.1.0
searchRelevanceDashboards@2.4.1.0
securityAnalyticsDashboards@2.4.1.0
securityDashboards@2.4.1.0

vagrant@ubuntu-2204:~/opensearch-dashboards-2.4.1$ bin/opensearch-dashboards-plugin remove reportsDashboards
Removing reportsDashboards...
Plugin removal complete
vagrant@ubuntu-2204:~/opensearch-dashboards-2.4.1$ bin/opensearch-dashboards-plugin install reportsDashboards
Attempting to transfer from reportsDashboards
Attempting to transfer from https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/2.4.1/latest/linux/x64/tar/builds/opensearch-dashboards/plugins/reportsDashboards-2.4.1.zip
Transferring 173310420 bytes....................
Transfer complete
Retrieving metadata from plugin archive
Extracting plugin archive
Extraction complete
Plugin installation complete

I don't know if installing non-core OpenSearch Dashboards plugins by their id is expected to be supported officially or not. We have seen that it works for Linux tarball but not for Windows zip.

I reviewed the manifest.yml file included in the OpenSearch Dashboards for Windows (x64/zip), and the plugin URLs are different from the resolved URLs by the bin\opensearch-dashboards-plugin.bat as explained above.

OpenSearch Dashboards 2.4.1 for Windows (x64/zip)'s `manifest.yml`

---
schema-version: '1.1'
build:
  name: OpenSearch Dashboards
  version: 2.4.1
  platform: windows
  architecture: x64
  distribution: zip
  location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/2.4.1/4665/windows/x64/zip/dist/opensearch-dashboards/opensearch-dashboards-2.4.1-windows-x64.zip
  id: '4665'
components:
  - name: OpenSearch-Dashboards
    repository: https://github.com/opensearch-project/OpenSearch-Dashboards.git
    ref: '2.4'
    commit_id: ea36827cdedf1e726e7cb8315ffc49f73f9b4eb7
    location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/2.4.1/4665/windows/x64/zip/builds/opensearch-dashboards/dist\opensearch-dashboards-min-2.4.1-windows-x64.zip
  - name: functionalTestDashboards
    repository: https://github.com/opensearch-project/opensearch-dashboards-functional-test.git
    ref: '2.4'
    commit_id: 8a75cf98f42e40ed4a54b8aa41674ebb7e422c85
  - name: notificationsDashboards
    repository: https://github.com/opensearch-project/notifications.git
    ref: '2.4'
    commit_id: 2fc07b02a1efbba5522eea62a276664c49cfe72c
    location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/2.4.1/4665/windows/x64/zip/builds/opensearch-dashboards/plugins\notificationsDashboards-2.4.1.zip
  - name: observabilityDashboards
    repository: https://github.com/opensearch-project/dashboards-observability.git
    ref: '2.4'
    commit_id: d4e55483b48214fdc34580ac3888718029a6fc6c
    location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/2.4.1/4665/windows/x64/zip/builds/opensearch-dashboards/plugins\observabilityDashboards-2.4.1.zip
  - name: reportsDashboards
    repository: https://github.com/opensearch-project/dashboards-reporting.git
    ref: '2.4'
    commit_id: bc7e5dbafbf193aae925d7b1557e7c897a8f98dc
    location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/2.4.1/4665/windows/x64/zip/builds/opensearch-dashboards/plugins\reportsDashboards-2.4.1.zip
  - name: securityDashboards
    repository: https://github.com/opensearch-project/security-dashboards-plugin.git
    ref: '2.4'
    commit_id: cf65c316c22b7bcbddd517bbd793ba7bf39e5f86
    location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/2.4.1/4665/windows/x64/zip/builds/opensearch-dashboards/plugins\securityDashboards-2.4.1.zip
  - name: anomalyDetectionDashboards
    repository: https://github.com/opensearch-project/anomaly-detection-dashboards-plugin
    ref: '2.4'
    commit_id: 973b2f03ef88c14d5f632efca5ebc7cab96227cd
    location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/2.4.1/4665/windows/x64/zip/builds/opensearch-dashboards/plugins\anomalyDetectionDashboards-2.4.1.zip
  - name: customImportMapDashboards
    repository: https://github.com/opensearch-project/dashboards-maps.git
    ref: '2.4'
    commit_id: b14333aa045225c3983d97d8478f9bdc7b347b81
    location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/2.4.1/4665/windows/x64/zip/builds/opensearch-dashboards/plugins\customImportMapDashboards-2.4.1.zip
  - name: ganttChartDashboards
    repository: https://github.com/opensearch-project/dashboards-visualizations.git
    ref: '2.4'
    commit_id: b218666aebbacd3297b07e6ff1838ddb692a9c71
    location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/2.4.1/4665/windows/x64/zip/builds/opensearch-dashboards/plugins\ganttChartDashboards-2.4.1.zip
  - name: indexManagementDashboards
    repository: https://github.com/opensearch-project/index-management-dashboards-plugin.git
    ref: '2.4'
    commit_id: f85a98d8a4aeb262e70417db3c63ffc5ff6661a3
    location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/2.4.1/4665/windows/x64/zip/builds/opensearch-dashboards/plugins\indexManagementDashboards-2.4.1.zip
  - name: searchRelevanceDashboards
    repository: https://github.com/opensearch-project/dashboards-search-relevance.git
    ref: '2.4'
    commit_id: 16634b7d42c7a4f16f1d0e1b995d13071a6d45b7
    location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/2.4.1/4665/windows/x64/zip/builds/opensearch-dashboards/plugins\searchRelevanceDashboards-2.4.1.zip
  - name: alertingDashboards
    repository: https://github.com/opensearch-project/alerting-dashboards-plugin.git
    ref: '2.4'
    commit_id: cc18d82a751c30f5c46a3eb26c1cc31684e78c70
    location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/2.4.1/4665/windows/x64/zip/builds/opensearch-dashboards/plugins\alertingDashboards-2.4.1.zip
  - name: queryWorkbenchDashboards
    repository: https://github.com/opensearch-project/sql.git
    ref: '2.4'
    commit_id: 7b9c02b5e81b9135cfa8e74c525ac7a3a6da7099
    location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/2.4.1/4665/windows/x64/zip/builds/opensearch-dashboards/plugins\queryWorkbenchDashboards-2.4.1.zip
  - name: securityAnalyticsDashboards
    repository: https://github.com/opensearch-project/security-analytics-dashboards-plugin
    ref: '2.4'
    commit_id: 576d0bc823ace31c38464bf5ee7528e6d3bebd71
    location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/2.4.1/4665/windows/x64/zip/builds/opensearch-dashboards/plugins\securityAnalyticsDashboards-2.4.1.zip

For example, the URL for reportsDashboards non-core plugin:

The difference between these URLs is the usage of zip vs tar in the path.

manifest.yml: /x64/zip/builds
bin\opensearch-dashboards-plugin.bat: /x64/tar/builds

Context:

We forked OpenSearch Dashboards (https://github.com/opensearch-project/OpenSearch-Dashboards) and did some customizations. Then, we build the tarball (Linux) and zip (Windows) files following the documentation. These outputs don't contain neither the non-core plugins nor manifest.yml file. Then we unarchive the output and we want to install some of the non-core OpenSearch Dashboards plugins by their id using bin/opensearch-dashboards-plugin (Linux) or bin\opensearch-dashboards-plugin.bat (Windows). We can add the non-core OpenSearch Dashboards plugins using their id successfully for our custom package for Linux, but the Windows one fails.

I guess that as a workaround we could use the URL instead of the plugin id for the non-core OpenSearch Dashboards plugin.

peterzhuamazon · 2023-07-06T21:09:52Z

Hi, thank you very much for your quick response!

@peterzhuamazon , if we use the Linux tarball instead of the Windows .zip, it is possible to install the non-core plugins by their id.

Hi @Desvelao like I said that might be a behavior that we are not aware of, and that is why I think @kavilla can give some better ideas and possible next step on this from the OSD core team.

It is never intentional for non-core plugin to install directly by id, and if there is one in OSD, then it needs some improvement to make it fully work on all distributions.

Then this issue can move to the OSD core for possible enhancement or even a proper feature request.

Let me know your thoughts on this.

Thanks,
Peter

jordarlu · 2023-08-28T19:52:39Z

Hello, @kavilla, would you also share your feedbacks/comments at here please? Trying to explore a way to move on this case,, appreciate your help !!

bbarani · 2023-08-29T16:58:27Z

@opensearch-project/opensearch-dashboards-core Can you please provide your inputs here?

AMoo-Miki · 2023-09-06T22:32:02Z

Looking at the source code, I see

`${LATEST_PLUGIN_BASE_URL}/${version}/latest/${platform}/${arch}/tar/builds/opensearch-dashboards/plugins/${plugin}-${version}.zip`;

... where irrespective of platform:

the url path has /tar/builds/ hard-coded,
the url points to a zip file, even though the path is under /tar/.

However, experimenting with the URLs, these two worked for me:

This indicates that Windows plugins are not in fact placed in /tar/ but are placed in /zip/.

I will PR the fix on OSD. I would also suggest making an adjustment to the URL to match what it is providing, or even better, drop the /zip and /tar bits; if you do so, please raise an issue on OSD so we can fix the URLs.

bbarani · 2023-09-06T22:37:47Z

Thanks for your update @AMoo-Miki . I will move this issue to OpenSearch dashboards repo.

peterzhuamazon · 2023-09-07T00:36:18Z

Looking at the source code, I see
`${LATEST_PLUGIN_BASE_URL}/${version}/latest/${platform}/${arch}/tar/builds/opensearch-dashboards/plugins/${plugin}-${version}.zip`;
... where irrespective of platform:
* the url path has `/tar/builds/` hard-coded,

* the url points to a zip file, even though the path is under `/tar/`.
However, experimenting with the URLs, these two worked for me:
* https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/2.9.0/latest/windows/x64/zip/builds/opensearch-dashboards/plugins/anomalyDetectionDashboards-2.9.0.zip

* https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/2.9.0/latest/linux/x64/tar/builds/opensearch-dashboards/plugins/anomalyDetectionDashboards-2.9.0.zip
This indicates that Windows plugins are not in fact placed in /tar/ but are placed in /zip/.

I will PR the fix on OSD. I would also suggest making an adjustment to the URL to match what it is providing, or even better, drop the /zip and /tar bits; if you do so, please raise an issue on OSD so we can fix the URLs.

Hi Miki,

Windows is in zip so the url reflect that well.
Same for linux in tarball here.
We can definitely add this change but it was not expected to have this change to start with.

Desvelao added bug Something isn't working untriaged labels Jul 4, 2023

bbarani transferred this issue from opensearch-project/OpenSearch-Dashboards Jul 5, 2023

AlexRuiz7 mentioned this issue Jul 6, 2023

Test distribution packages wazuh/wazuh-dashboard#72

Closed

5 tasks

jordarlu removed the untriaged label Jul 10, 2023

bbarani transferred this issue from opensearch-project/opensearch-build Sep 6, 2023

github-actions bot added the untriaged label Sep 6, 2023

AMoo-Miki self-assigned this Sep 6, 2023

AMoo-Miki removed the untriaged label Sep 6, 2023

AMoo-Miki mentioned this issue Sep 6, 2023

Correct the generated path for downloading plugins by their names on Windows #4953

Merged

8 tasks

joshuarrrr closed this as completed in #4953 Sep 7, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] Unable to install OpenSearch Dashboards plugins using its IDs in Windows release #4952

[BUG] Unable to install OpenSearch Dashboards plugins using its IDs in Windows release #4952

Desvelao commented Jul 4, 2023 •

edited

Loading

kavilla commented Jul 5, 2023

bbarani commented Jul 5, 2023

peterzhuamazon commented Jul 5, 2023 •

edited

Loading

Desvelao commented Jul 6, 2023

peterzhuamazon commented Jul 6, 2023 •

edited

Loading

jordarlu commented Aug 28, 2023

bbarani commented Aug 29, 2023

AMoo-Miki commented Sep 6, 2023

bbarani commented Sep 6, 2023

peterzhuamazon commented Sep 7, 2023

[BUG] Unable to install OpenSearch Dashboards plugins using its IDs in Windows release #4952

[BUG] Unable to install OpenSearch Dashboards plugins using its IDs in Windows release #4952

Comments

Desvelao commented Jul 4, 2023 • edited Loading

kavilla commented Jul 5, 2023

bbarani commented Jul 5, 2023

peterzhuamazon commented Jul 5, 2023 • edited Loading

Desvelao commented Jul 6, 2023

peterzhuamazon commented Jul 6, 2023 • edited Loading

jordarlu commented Aug 28, 2023

bbarani commented Aug 29, 2023

AMoo-Miki commented Sep 6, 2023

bbarani commented Sep 6, 2023

peterzhuamazon commented Sep 7, 2023

Desvelao commented Jul 4, 2023 •

edited

Loading

peterzhuamazon commented Jul 5, 2023 •

edited

Loading

peterzhuamazon commented Jul 6, 2023 •

edited

Loading