Enable BWC tests for plugins

[saratvemulapalli]

Is your feature request related to a problem? Please describe.
As OpenSearch has backward compatibility tests, the framework could be extended to run backwards compatibility tests for plugins.
Ref: https://github.com/opensearch-project/OpenSearch/tree/main/qa

The difference between OpenSearch and plugins is that plugins are built for a specific version of OpenSearch/Elasticsearch.
The same bwc test framework could be used but the framework by default installs the old version of the plugin even though the node is being upgraded to the new version of OpenSearch.

This causes failures while installing old plugin with new version of OpenSearch, bails out of the test.

Failure:

> Task :sarat1 FAILED
Exec output and error:
| Output for ./bin/opensearch-plugin:-> Installing file:/home/ubuntu/anomaly-detection/src/test/resources/job-scheduler/1.13.2/job-scheduler-artifacts.zip
| -> Downloading file:/home/ubuntu/anomaly-detection/src/test/resources/job-scheduler/1.13.2/job-scheduler-artifacts.zip
| -> Failed installing file:/home/ubuntu/anomaly-detection/src/test/resources/job-scheduler/1.13.2/job-scheduler-artifacts.zip
| -> Rolling back file:/home/ubuntu/anomaly-detection/src/test/resources/job-scheduler/1.13.2/job-scheduler-artifacts.zip
| -> Rolled back file:/home/ubuntu/anomaly-detection/src/test/resources/job-scheduler/1.13.2/job-scheduler-artifacts.zip
| Exception in thread "main" java.lang.IllegalArgumentException: property [opensearch.version] is missing for plugin [opendistro-job-scheduler]
|       at org.opensearch.plugins.PluginInfo.readFromProperties(PluginInfo.java:194)
|       at org.opensearch.plugins.InstallPluginCommand.loadPluginInfo(InstallPluginCommand.java:797)
|       at org.opensearch.plugins.InstallPluginCommand.installPlugin(InstallPluginCommand.java:856)
|       at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:263)
|       at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:237)
|       at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:99)
|       at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:140)
|       at org.opensearch.cli.MultiCommand.execute(MultiCommand.java:104)
|       at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:140)
|       at org.opensearch.cli.Command.main(Command.java:103)
|       at org.opensearch.plugins.PluginCli.main(PluginCli.java:60)

You can refer to the issue for more details on how to reproduce the problem: opensearch-project/opensearch-plugins#53

How to reproduce the problem

0. Choose any OpenSearch plugin and setup a test cluster in plugin build.gradle file.
1. Add a test cluster and install the old version of the plugin. Make sure the plugin .zip file exists in the path.

testClusters {
    adMixedCluster {
        testDistribution = "ARCHIVE"
	    versions = ["7.10.2","1.0.0-rc1"]
	    numberOfNodes = 3
        setting 'http.content_type.required', 'true'
        /*plugin(provider(new Callable<RegularFile>(){
            @Override
            RegularFile call() throws Exception {
                return new RegularFile() {
                    @Override
                    File getAsFile() {
                        return fileTree("src/test/resources/job-scheduler/1.13.2").getSingleFile()
                    }
                }
            }
        }))
        plugin(provider(new Callable<RegularFile>(){
            @Override
            RegularFile call() throws Exception {
                return new RegularFile() {
                    @Override
                    File getAsFile() {
                        return fileTree("src/test/resources/anomaly-detection/").getSingleFile()
                    }
                }
            }
        }))*/
        println "Job Scheduler installed!!"
        setting 'repositories.url.allowed_urls', 'http://snapshot.test*'
        setting 'path.repo', "${buildDir}/cluster/shared/repo/adMixedCluster"
        setting 'http.content_type.required', 'true'
    }
}

2. Setup an upgrade task

task oldVersionCluster(type: StandaloneRestIntegTestTask) {
    useCluster testClusters.adMixedCluster
    if (System.getProperty("mixedCluster") != null) {
        filter {
            includeTestsMatching "org.opensearch.ad.bwc.*IT"
        }
    }
    install_plugin(true,0)
    String baseName = "adMixedCluster"
    nonInputProperties.systemProperty('tests.rest.cluster', "${-> testClusters."${baseName}".allHttpSocketURI.join(",")}")
    nonInputProperties.systemProperty('tests.clustername', "${-> testClusters."${baseName}".getName()}")
}

task mixedClusterTask(type: StandaloneRestIntegTestTask) {
    dependsOn oldVersionCluster
    String baseName = "adMixedCluster"
    useCluster testClusters."${baseName}"
    doFirst {
      testClusters.adMixedCluster.nextNodeToNextVersion()
      testClusters.adMixedCluster.nodes[1].plugin(project.tasks.bundlePlugin.archiveFile)
    }
    if (System.getProperty("mixedCluster") != null) {
        filter {
            includeTestsMatching "org.opensearch.ad.bwc.*IT"
        }
    }
    nonInputProperties.systemProperty('tests.rest.cluster', "${-> testClusters."${baseName}".allHttpSocketURI.join(",")}")
    nonInputProperties.systemProperty('tests.clustername', "${-> testClusters."${baseName}".getName()}")
}

3. Run the task ./gradlew mixedClusterTask -DmixedCluster=123 -Dtests.security.manager=false

Describe the solution you'd like
The solution should be able to cleanly upgrade the node with the new plugin version and have the plugin gradle framework define which version to be installed.

Code Pointer:

OpenSearch/buildSrc/src/main/java/org/opensearch/gradle/testclusters/OpenSearchCluster.java

Line 395 in 0ba0e7c

public void nextNodeToNextVersion() {

[dblock]

@saratvemulapalli how does one reproduce this easily?

[saratvemulapalli]

Thats a good point. The other issue on opensearch-plugins should have good information to repro. But let me trim it down and add simple instructions here.

[saratvemulapalli]

@AmiStrn Sure I see there are 2 parts to the question.
1. Why have a dependency on OpenSearch for plugins to test.
For all plugins in the community, there is no way of testing backwards compatibility against its own previous version. For plugin owners to verify its own functionality (APIs, settings, transport, indices etc), this framework will automate the workflows. This is the problem we are trying to solve.
The BWC test framework is already built into OpenSearch and extending the framework is simpler in couple of ways:
a. We would have only one place to manage the cluster for testing and not duplicating code in multiple places, extended via gradle plugin. (Probably eventually we could stub this out of OpenSearch repo).
b. Plugins already have a dependency on opensearch gradle plugin, so this is not a new dependency without which the plugin cannot build.
c. This will allow plugins to run bwc in their CI's without external solution to manage the cluster upgrade.
d. Most of the framework already exists and we do not need to reinvent it.

2. Test plugin interfaces in OpenSearch
Absolutely, if the interfaces do not have bwc tests today we should add them in OpenSearch, verify all plugin interfaces are backwards compatible. This is the best way to do it as it fails fast and run them before any plugin consumes them.
To add on to that we should add this to the CI.
I'll open an issue to track this.

I hope this helps, let me know what you think.

[AmiStrn]

@saratvemulapalli you are faaast! I removed my comment after two minutes bc I realized I had made a mistake assuming that there would be tests in opensearch repo that would literally have classes imported from AD plugin (for example) 😵
I jumped the gun with the comment, and after reading some of the awesome work being done by @VachaShah I realized I was way off 😸
Thanks for the explanation!

[saratvemulapalli]

Haha any discussion is always great :)