[Backport 2.x] Per request phase latency (#10351)

[dzane17]

Signed-off-by: David Zane davizane@amazon.com
(cherry picked from commit daf1350)

Description

2.x Backport for #10351
Documentation: opensearch-project/documentation-website#5154
Also must backport UT fix: #10649

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Compatibility status:

Checks if related components are compatible with change b2b8e2e

Incompatible components

Incompatible components: [https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git]

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/28620/
• CommitID: 8ee8219
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/28622/
• CommitID: 386674a
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/28631/
• CommitID: b2b8e2e

[codecov]

Codecov Report

Merging #10797 (b2b8e2e) into 2.x (8896f58) will increase coverage by 0.00%.
Report is 1 commits behind head on 2.x.
The diff coverage is 72.61%.

@@            Coverage Diff             @@
##                2.x   #10797    +/-   ##
==========================================
  Coverage     70.94%   70.94%            
- Complexity    58705    58732    +27     
==========================================
  Files          4843     4843            
  Lines        277261   277371   +110     
  Branches      40673    40696    +23     
==========================================
+ Hits         196690   196787    +97     
+ Misses        63858    63804    -54     
- Partials      16713    16780    +67     

Files	Coverage Δ
...earch/action/search/AbstractSearchAsyncAction.java	79.16% <100.00%> (-0.45%)	⬇️
...g/opensearch/action/search/MultiSearchRequest.java	80.34% <100.00%> (+0.46%)	⬆️
...opensearch/action/search/SearchResponseMerger.java	78.81% <100.00%> (-3.37%)	⬇️
...rg/opensearch/common/settings/ClusterSettings.java	92.85% <ø> (ø)
.../indices/replication/common/ReplicationTarget.java	72.50% <100.00%> (-5.14%)	⬇️
...va/org/opensearch/action/search/SearchRequest.java	83.20% <81.81%> (-0.54%)	⬇️
...pensearch/rest/action/search/RestSearchAction.java	37.34% <0.00%> (-0.48%)	⬇️
...nsearch/index/store/RemoteStoreFileDownloader.java	92.15% <84.00%> (-0.87%)	⬇️
...ices/replication/RemoteStoreReplicationSource.java	85.24% <75.00%> (-5.83%)	⬇️
...a/org/opensearch/action/search/SearchResponse.java	86.94% <80.00%> (-1.49%)	⬇️
... and 1 more

... and 442 files with indirect coverage changes

[msfroh]

Link to version update on main: #10865