Remove SEARCH_PIPELINE feature flag (#8513)

[noCharger]

Description

Manual backport #8513. See details in #8513 (comment)

Check List

• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20369/
• CommitID: fda23ee
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20370/
• CommitID: 24814cb
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20373/
• CommitID: d09627a
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20374/
• CommitID: 8945bb0
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20375/
• CommitID: 414e123
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[noCharger]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20375/
• CommitID: 414e123
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

Need rerun on the flaky test. Also requesting review since this PR is for search pipelien GA in 2.9. @reta @andrross @saratvemulapalli

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20382/
• CommitID: b66407e
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20384/
• CommitID: 7a3b799
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[dblock]

@noCharger could you please dig through gradle check failures and fix failures or link to existing flaky tests if those are the problem?

[dblock]

am I correct that without the feature flag the feature can be enabled by making a certain query?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20409/
• CommitID: 7a3b799
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[noCharger]

@dblock This is for search pipeline feature GA. The gradle check failed on mixed-cluster test because the node with new code can't find the flag setting from the node with old code and vice versa (see #8513 (comment) for the full evaluation). It's related to this issue.

[noCharger]

am I correct that without the feature flag the feature can be enabled by making a certain query?

Right now you have to enable it by setting opensearch.experimental.feature.search_pipeline.enabled to ture in OpenSearch.yml; The previous commit https://github.com/opensearch-project/OpenSearch/pull/8613/files doesn't work because

OpenSearch/server/src/main/java/org/opensearch/common/util/FeatureFlags.java

Line 96 in 230a06d

return settings != null && settings.getAsBoolean(featureFlagName, false);

is always set to false.

[dblock]

Why did we not remove the experimental flag in #8613 and decided to just flip it? I would like to confirm that this feature is meant to go GA, aka not experimental and not behind a feature flag, in 2.9, yes?

[noCharger]

Why did we not remove the experimental flag in #8613 and decided to just flip it? I would like to confirm that this feature is meant to go GA, aka not experimental and not behind a feature flag, in 2.9, yes?

Yes search pipeline is ready to go GA in 2.9

We started with removing the experimental flag in #8513 and use #8613 as an short term mitigation on this mixed cluster test issue: #8513 (comment).

[dblock]

Thanks @noCharger. There's an order in which these PRs turn 🟢, let's try to push them as such.

[dblock]

so, we've cut 2.9 branch, which makes this a chicken-and-egg problem - I am going to make a call and merge this with failing bcw tests, and we're going to ensure CI / gradle check passes on 2.9 after it's merged, even if we need to retry