Fixing pulling of security plugin to run integ tests in secure opensearch cluster

[vibrantvarun]

Description

The PR fixes the security plugin issue in the neural search build.

Premise:
While building neural search we are pulling security plugin via zip archive.

    secureIntegTestPluginArchive group: 'org.opensearch.plugin', name:'opensearch-security', version: "${opensearch_build}"

But in neural search we do not use security plugin as a compile time dependency.

How zipArchive works?
Zip Archive works by resolving all the dependencies provided in the dependencies section. opensearch-knn we use at compile time dependency and therefore we extract it as a jar in task extractKNN Jar. opensearch-ml-plugin zip we use it in bwc tests.

// In order to add the jar to the classpath, we need to unzip the
// k-NN zip and then copy it into a directory that we specify as a dependency.
task extractKnnJar(type: Copy) {
    mustRunAfter()
    from(zipTree(configurations.zipArchive.find { it.name.startsWith("opensearch-knn")}))
    into knnJarDirectory
}

Now this line(from(zipTree(configurations.zipArchive.find { it.name.startsWith("opensearch-knn")})) ) of code tries to find opensearch-security as well however we don't use it.

The fix:

Segregate the pulling of security plugin in secureIntegTestPluginArchive configuration.

Also,
In this PR, Java 21 has been restricted to 21.0.1 due to ongoing issue in 21.0.2 version.
opensearch-project/OpenSearch#11968 (review).
Once the fix to the issue will be merged, will revert back to pulling of Java 21 to latest version.

Moreover, The bwc tests are disabled in the PR due to there is an ongoing failure of security tests due to zip archive issue. Therefore the latest build has not been published from last few days and bwc tests are failing.

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed as per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (a6841cc) 84.33% compared to head (d5dd7dc) 84.33%.

Additional details and impacted files

@@            Coverage Diff            @@
##               main     #543   +/-   ##
=========================================
  Coverage     84.33%   84.33%           
  Complexity      533      533           
=========================================
  Files            40       40           
  Lines          1564     1564           
  Branches        244      244           
=========================================
  Hits           1319     1319           
  Misses          133      133           
  Partials        112      112           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[vibrantvarun]

BWC tests failure is expected as we do not have latest neural-search 2.12 artifact.