[RFC] Plugins Version Decoupling

[Flyingliuhub]

Overview

OpenSearch plugins currently tightly couple their versions with the underlying OpenSearch/OpenSearch Dashboards version. This tight coupling can lead to challenges in maintaining and upgrading plugins independently. This RFC proposes a solution to decouple the plugin versions from the OpenSearch versions, allowing for more flexibility in managing and updating plugins.

Motivation

The motivation behind decoupling OpenSearch plugin versions is to provide a more independent and flexible approach to plugin development and maintenance.

Problem Statement

In Elasticsearch version 7.8, users encounter issues when using the latest 2.11 Dashboards to connect to the 7.8 Elasticsearch cluster. Specifically, they are unable to create the required Index Pattern successfully when adding multiple data sources. The issue has occurred due to changes in the Index Pattern API in Elasticsearch Cluster versions above 7.8. While users can successfully connect to the 7.8 Elasticsearch cluster, they cannot create the necessary Index Pattern. This, in turn, affects the user experience when using several key plugins, especially the Discover plugin, as they cannot utilize the Index Pattern to display the required data on the Discover app. To address this situation, it is recommended to better showcase the supported versions on the Create Index Pattern page. For example, during the Index Pattern creation process, when clicking on "Use external data source connection," only display the versions supported by the current 2.11 version. If it is feasible to implement this functionality, we need to obtain the Manifest of the current connection version to support filtering and display only the versions that support Index Pattern creation.

In another example related to the map app, when users create map visualizations, they need to add multiple layers with existing datasources to Elasticsearch's index pattern. As the index pattern cannot be created during the OSD 2.11 index pattern page, it causes the map plugin to be unable to display content connected to the 7.8 cluster. Additionally, opensearch-geospatial was introduced in OpenSearch 2.3.0. When users use OSD 2.11.0 to connect to OpenSearch versions earlier than 2.3.0, the map app lacks the required functionality.

Proposed Solution

In order to run the plugins connected to different versions of the OpenSearch/Elasticsearch cluster functionally, we would like to propose the following solution to achieve the goal of plugin version decoupling.

Plugin Compatibility Matrix

Establish a compatibility matrix that clearly outlines which versions of the plugin are compatible with different versions of OpenSearch. By collecting plugin versions for selected OpenSearch versions and analyzing the compatibility matrix, we can identify potential issues and gaps. Based on the output of the matrix, we can implement a systematic approach to list the possible contents of each plugin, ensuring comprehensive coverage and documentation of compatibility scenarios. This process will not only aid in understanding the compatibility landscape but also guide developers in making informed decisions when selecting and updating plugins for specific OpenSearch versions.

Get Manifest information

The optimal approach is to obtain all the API information for each version of the plugin. However, gathering API details for every plugin version can be challenging. An alternative approach is to retrieve all plugin versions by calling the _cat/plugins or /_nodes/_local/plugins API during the creation of multiple data sources. we will compare the connected cluster's plugin version with Plugin team's supported version range, to check plugin capabilities, and save those information to a saved object. This information can be leveraged by each plugin team to determine and create a rule to filter. If the customer's OpenSearch is updated, the manifest information could become outdated. We will leverage the multiple datasource update function to retrieve the latest manifest information and update the existing saved object.

For instance, a data-source object may look like:

{
    "applicationId": "xxxxxx",
    "id": "data-source:xxx-xx-xx-xx-xxxx",
    "attributes": {
        "clusterVersion": "2.11.0",
        "capabilities": {
            "discoveryEnabled": true,
            "alertingEnabled": true
        },
        "description": "",
        "endpoint": "https://opensearchendpoint",
        "title": "democonnection"
    },
    "references": [],
    "type": "data-source",
    "updated_at": "2023-11-30T07:06:40.600Z",
    "version": 1
....
}

UI Changes

We propose adding capabilities checks on the "Create Multiple Datasource" page. This includes checkboxes for each plugin and either a dedicated "Check Capabilities" button or utilizing the existing "Test Connection" button. When users click the selected button, the system will retrieve the current cluster version and plugin version range, compare these two inputs, mark the checkboxes accordingly, and then save the multiple-datasource connection with additional information. For instance, capabilities.discoveryEnabled:true will be included. This information will be utilized for the plugin’s UI datasource filters/rules.

Plugin Rule for Version Compatibility Check

After obtaining manifest information and storing it as a saved object, the plugin team could leverage the manifest information to create their rules for filtering supported versions based on their API under specific versions. For example, in the OpenSearch Dashboards discover application, the rule can be used to control how the compatible index pattern is rendered. If the current plugin version isn't compatible with the connected older OpenSearch version, it will filter out the incompatible OpenSearch index pattern and only render the compatible version of the index pattern on the discover page.

For the plugin version range, we can provide an interface for each plugin team to use and register. Each plugin team will use the provided interface, such as datasource.registercompatibilities(versionrange). This way, we will have the support range for each plugin. By comparing the current cluster’s version with the registered version range, we can determine if a plugin is enabled. The current schema includes the addition of clusterVersion to accommodate this feature. Since most plugins currently follow an exact match for the OpenSearch/OpenSearch Dashboards version, having just one OpenSearch version allows us to determine the installed plugin version in most cases.

Scope

In the scope of this proposal, we will focus on selected OpenSearch versions, specifically covering OpenSearch versions ('1.0.0,' '1.1.0,' '1.2.0,' '1.3.0,' '2.3.0,' '2.5.0,' '2.7.0,' '2.9.0', '2.11.0') and Elasticsearch ('7.10'). The plugins included in this analysis are 'alerting,' 'anomaly-detection,' 'index_management,' 'sql,' and 'Machinelearning.'

This focused approach will allow us to thoroughly assess the compatibility matrix for the specified OpenSearch versions and plugins. By concentrating on these specific OpenSearch versions and plugins, we aim to deeply understand how our proposed changes will affect and improve compatibility. The insights gained from this focused analysis will be the basis for developing broader strategies and documentation to ensure plugin compatibility in upcoming OpenSearch versions.

Execution

• Create a matrix for each selected OpenSearch version, as a sample shown in the following table. This matrix will provide detailed information on the current plugin versions. More detailed information has been added Plugins-Version-Decoupling.xlsx

OS Version	Plugins	Plugin Version
2.3.0	opensearch-alerting	opensearch_2.3.0
2.3.0	opensearch-analysis-vietnamese	2.3.0
2.3.0	opensearch-anomaly-detection	2.3.0.0
2.3.0	opensearch-asynchronous-search	2.3.0.0
2.3.0	opensearch-cross-cluster	2.3.0
2.3.0	opensearch-cross-cluster-replication	2.3.0.0-SNAPSHOT
2.3.0	opensearch-geospatial	2.3.0.0
2.3.0	opensearch-index-management	2.3.0.0
2.3.0	opensearch-jetty	2.2.0
2.3.0	opensearch-job-scheduler	2.3.0.0
2.3.0	opensearch-knn	2.3.0.0
2.3.0	opensearch-ml	2.3.0.0
2.3.0	opensearch-notifications	2.3.0.0
2.3.0	opensearch-notifications-core	2.3.0.0
2.3.0	opensearch-observability	2.3.0.0-SNAPSHOT
2.3.0	opensearch-reports-scheduler	2.3.0.0-SNAPSHOT
2.3.0	opensearch-security	2.3.0.0
2.3.0	opensearch-sql	2.3.0.0-SNAPSHOT
2.3.0	performance-analyzer	2.3.0.0
2.3.0	repository-s3	2.3.0
2.3.0	sifi-remediation-plugin	2.3.0.0

FAQ

[seraphjiang]

add original proposal as reference

#1388 (comment)

@Flyingliuhub @BionIT @zengyan-amazon

[elfisher]

I love the idea of decoupling the versions. Right now, even with no code changes versions need bumped in each plugin as the core version increments. Is there anything we need to think about in regards to decoupling for OpenSearch Dashboards plugins versus OpenSearch core plugins?

[kgcreative]

re: When users click the selected button, the system will retrieve the current cluster version and plugin version range, compare these two inputs, mark the checkboxes accordingly, and then save the multiple-datasource connection with additional information. -> I think i'd like to not focus on the individual UI methods at the moment, and focus on either compatibility information, or simply version. Each plugin should be free to do what's right based on that information. For example, if we do know which version a backend is in, we might be able to create a compatible index pattern, rather than disabling it all together.

[Flyingliuhub]

I love the idea of decoupling the versions. Right now, even with no code changes versions need bumped in each plugin as the core version increments. Is there anything we need to think about in regards to decoupling for OpenSearch Dashboards plugins versus OpenSearch core plugins?

Yes, currently, we depend on Dashboards plugins to provide the supported version range of OpenSearch core plugins, which helps us decide whether to hide/show/disable the Dashboards plugins. If we can identify the API gaps for non-supported versions, we can either backport the API changes to unsupported OpenSearch core plugins or modify the current Dashboards plugins to conditionally choose the API based on the version. This approach aims to handle the decoupling in the future.

[SuZhou-Joe]

the system will retrieve the current cluster version and plugin version range, compare these two inputs, mark the checkboxes accordingly, and then save the multiple-datasource connection with additional information.

I like the idea but I do think we may need a more granular capability detection. Using Index management dashboard plugin for example, we introduced Notification for long running operation task feature in 2.8, transform jobs feature in 2.1, meaning that the notifications config menu should be hidden when connecting to a OS cluster under v2.8, and disable the transform jobs menu when connecting to a elasticSearch cluster with v7.10.0 .

[ZilongX]

That's a great call-out @SuZhou-Joe , in this case a more granular capability check is for sure needed.

To simplify , as for the latest version of Index management dashboard plugin (2.14 as we type), the earliest OS version to support the full functionality would be OS 2.8, in another word [2.8, latest], which means the plugin would work as expected with any data source versions within the range and this already expand the support range of Index management dashboards plugin version 2.14 from one (2.14) to seven (2.8, 2.9, 2.10, 2.11, 2.12, 2.13, 2.14) !!

Then with a second layer version filtering logic implemented on plugin side, we can make the plugin compatible with more legacy versions like 2.7 by hiding the notifications config menu in order to expand the compatibility matrix.

So generally we will be dividing the versions into three group as :

1. OS versions fully compatible with the latest version of plugin (in this example, OS 2.8+)
2. OS versions partially compatible with plugins features unsupported (in this example, OS 2.8-)
3. OS versions not compatible with plugins (let's say OS 1.3- assumingly)

Incrementally, to get the compatibility matrix for group 1 would be a good start towards final version decoupling goal.

[derek-ho]

the system will retrieve the current cluster version and plugin version range, compare these two inputs, mark the checkboxes accordingly, and then save the multiple-datasource connection with additional information.

I like the idea but I do think we may need a more granular capability detection. Using Index management dashboard plugin for example, we introduced Notification for long running operation task feature in 2.8, transform jobs feature in 2.1, meaning that the notifications config menu should be hidden when connecting to a OS cluster under v2.8, and disable the transform jobs menu when connecting to a elasticSearch cluster with v7.10.0 .

@SuZhou-Joe @ZilongX if I am not mistaken, since each page has to register the picker anyway (to support things like read only mode vs selectable), a more fine grained way we can deal with this is by changing the supported versions based on UI page. As long as the new feature has its own page which registers its own picker we can filter it differently?

[derek-ho]

Does support for elasticsearch 7.10 mean we should use the legacy client? For example I see usage of non-legacy client here, but maybe this does not apply since ml-commons was introduced after?

[seraphjiang]

some ideas about handling incompatible case.

While we wish to have one single version of OSD (with Plugin) to support as much as versions of OpenSearch (with or without plugin) as possible. In reality, we could only support range of version as each UI has their own version specific dependency of backend feature.

Inspired by the protodb's steam deck compatibility , we could introduce three level of compatibility rating for each version.

1. Verified: All the UI features are tested and function
2. Runnable: Some of feature are working; or there is config/setup/workaround to make the feature working.
3. Unsupported: There is no workaround, unless huge redesign, there is no workaround to use such feature.

When we design each UI page, we design for all three cases. e.g. User is able to see and use all functionality when context match the verified criteria. In the case page is only runnable(partial compatible) or not supported. User will see warning message with link to the help page that explain why.

There could be experience, we base on feature compatibility, we hide some feature. We might go this approach if we could build compatibility rendering root component, this component could will check the compatibility data to determine whether need to show the component or not. Before that, show incompatibility or not supported warning message and provide help link is more straightforward and consistent with very low cost.