Added integration tests support for Telemetry span validation #8444

Shephalimittal · 2023-07-05T14:22:46Z

Description

Added cluster Integration tests for Otel span validations in plugins. These tests enable and disable tracer setting and performs following checks on emitted spans.

All spans have unique IDs.
Number of requests (index/search) equal to Root Parent spans emitted.
All spans are closed properly.
Number of TraceIDs equal to number of requests.
All spans are in order. (Start time of a particular span is after its parent's and its endtime is before its parent's)

Also, added support for telemetryplugin() in IntegTestClass (All integ tests are inherited from this), so basic validations of spans are called in all the tests by default.

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-07-05T14:27:33Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/19145/
CommitID: 4960d40
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

...internalClusterTest/java/org/opensearch/telemetry/tracing/MockOpenTelemetrySpanExporter.java

...rc/internalClusterTest/java/org/opensearch/telemetry/tracing/RTCTracingDisabledSanityIT.java

server/src/main/java/org/opensearch/node/Node.java

test/telemetry/src/main/java/org/opensearch/test/telemetry/tracing/MockTracingTelemetry.java

test/telemetry/src/main/java/org/opensearch/test/telemetry/tracing/SpanDataValidator.java

.../telemetry/src/main/java/org/opensearch/test/telemetry/tracing/StrictCheckSpanProcessor.java

test/telemetry/src/main/java/org/opensearch/test/telemetry/tracing/TelemetryValidators.java

...src/main/java/org/opensearch/test/telemetry/tracing/validators/AllSpansAreEndedProperly.java

...src/internalClusterTest/java/org/opensearch/telemetry/tracing/RTCTracingEnabledSanityIT.java

github-actions · 2023-07-07T13:05:42Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/19407/
CommitID: c4c77c0
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-07-07T13:13:29Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/19408/
CommitID: 4084b27
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-07-07T13:37:59Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/19411/
CommitID: 0bb858e
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-07-07T13:47:13Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/19412/
CommitID: 75afc17
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

test/telemetry/src/main/java/org/opensearch/test/telemetry/tracing/TelemetryValidators.java

Signed-off-by: Shephali Mittal <shephalm@amazon.com>

opensearch-trigger-bot · 2023-08-04T10:05:53Z

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/security-analytics.git]
Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

BUILD SUCCESSFUL in 29m 39s

...nternalClusterTest/java/org/opensearch/telemetry/tracing/TelemetryTracerEnabledSanityIT.java

...metry/src/main/java/org/opensearch/test/telemetry/tracing/validators/AllSpansAreInOrder.java

Signed-off-by: Shephali Mittal <shephalm@amazon.com>

github-actions · 2023-08-04T11:37:15Z

Gradle Check (Jenkins) Run Completed with:

RESULT: ABORTED ❌
URL: https://build.ci.opensearch.org/job/gradle-check/21870/
CommitID: c6296e0
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

opensearch-trigger-bot · 2023-08-04T12:00:03Z

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/security-analytics.git]
Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

BUILD SUCCESSFUL in 26m 44s

github-actions · 2023-08-04T13:34:46Z

Gradle Check (Jenkins) Run Completed with:

RESULT: ABORTED ❌
URL: https://build.ci.opensearch.org/job/gradle-check/21877/
CommitID: 764f3b2
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Shephali Mittal <shephalm@amazon.com>

github-actions · 2023-08-04T15:55:04Z

Gradle Check (Jenkins) Run Completed with:

RESULT: ABORTED ❌
URL: https://build.ci.opensearch.org/job/gradle-check/21883/
CommitID: 9092d3f
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Shephali Mittal <shephalm@amazon.com>

github-actions · 2023-08-04T20:05:55Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/21905/
CommitID: 1c159d5

github-actions · 2023-08-04T20:13:51Z

Gradle Check (Jenkins) Run Completed with:

RESULT: TIMEOUT ❌
URL: https://build.ci.opensearch.org/job/gradle-check/21903/
CommitID: 2cbe9dd
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Shephali Mittal <shephalm@amazon.com>

opensearch-trigger-bot · 2023-08-07T07:46:15Z

Compatibility status:


> Task :checkCompatibility
Checking compatibility for: https://github.com/opensearch-project/reporting.git with ref: main
Incompatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/ml-commons.git]
Compatible components: [https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git]

BUILD SUCCESSFUL in 27m 25s

github-actions · 2023-08-07T07:59:39Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.action.admin.cluster.node.tasks.ResourceAwareTasksTests.testTaskResourceTrackingDuringTaskCancellation

URL: https://build.ci.opensearch.org/job/gradle-check/21991/
CommitID: e5709a6
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

opensearch-trigger-bot · 2023-08-07T13:02:21Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-8444-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 70e1b2274bf6c1245063442882e561d4fb37742a
# Push it to GitHub
git push --set-upstream origin backport/backport-8444-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-8444-to-2.x.

reta · 2023-08-07T13:20:36Z

@Shephalimittal could you please send a manual backport to 2.x branch? thank you.

…idations (opensearch-project#8444) Signed-off-by: Shephali Mittal <shephalm@amazon.com>

…idations (#8444) (#9188) Signed-off-by: Shephali Mittal <shephalm@amazon.com>

Gaganjuneja · 2023-08-20T16:23:22Z

Resolves #8430

…arch-project#8444) * Added integration tests support for Telemetry span validation Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Added Telemetry validators Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Build failures Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Added changes desciption in changelog.md Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Added telemetry featureflagsetting by default to integtestclass Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Renamed Telemetry IT classes and modfied span exporter Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Removed Logger from InMemorySpanExporter Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Integrated span exporter configurable changes Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Build fixes Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Added wrapper on OTel plugin for Integration tests Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Removed telemetryPlugin() and telemetryFeatureFlagSettings() Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Add support for InMemorySpanExporter class Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Removing Unintended change Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Removing Tracer factory close Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Removed vcs.xml file and few other minor changes Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Removed unintended change Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Changed opentelemetry-sdk-testing dependency scope to test Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Fix integ tests Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Precommit fix Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Added TRACER_EXPORTER_DELAY_SETTING, more logging in TelemetryValidators Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Removed AllSpansAreInOrder validator Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Empty-Commit Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Empty-Commit Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Added comment while closing span Signed-off-by: Shephali Mittal <shephalm@amazon.com> --------- Signed-off-by: Shephali Mittal <shephalm@amazon.com> Co-authored-by: Shephali Mittal <shephalm@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com>

…arch-project#8444) * Added integration tests support for Telemetry span validation Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Added Telemetry validators Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Build failures Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Added changes desciption in changelog.md Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Added telemetry featureflagsetting by default to integtestclass Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Renamed Telemetry IT classes and modfied span exporter Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Removed Logger from InMemorySpanExporter Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Integrated span exporter configurable changes Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Build fixes Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Added wrapper on OTel plugin for Integration tests Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Removed telemetryPlugin() and telemetryFeatureFlagSettings() Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Add support for InMemorySpanExporter class Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Removing Unintended change Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Removing Tracer factory close Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Removed vcs.xml file and few other minor changes Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Removed unintended change Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Changed opentelemetry-sdk-testing dependency scope to test Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Fix integ tests Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Precommit fix Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Added TRACER_EXPORTER_DELAY_SETTING, more logging in TelemetryValidators Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Removed AllSpansAreInOrder validator Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Empty-Commit Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Empty-Commit Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Added comment while closing span Signed-off-by: Shephali Mittal <shephalm@amazon.com> --------- Signed-off-by: Shephali Mittal <shephalm@amazon.com> Co-authored-by: Shephali Mittal <shephalm@amazon.com> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

…arch-project#8444) * Added integration tests support for Telemetry span validation Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Added Telemetry validators Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Build failures Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Added changes desciption in changelog.md Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Added telemetry featureflagsetting by default to integtestclass Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Renamed Telemetry IT classes and modfied span exporter Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Removed Logger from InMemorySpanExporter Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Integrated span exporter configurable changes Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Build fixes Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Added wrapper on OTel plugin for Integration tests Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Removed telemetryPlugin() and telemetryFeatureFlagSettings() Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Add support for InMemorySpanExporter class Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Removing Unintended change Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Removing Tracer factory close Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Removed vcs.xml file and few other minor changes Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Removed unintended change Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Changed opentelemetry-sdk-testing dependency scope to test Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Fix integ tests Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Precommit fix Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Added TRACER_EXPORTER_DELAY_SETTING, more logging in TelemetryValidators Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Removed AllSpansAreInOrder validator Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Empty-Commit Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Empty-Commit Signed-off-by: Shephali Mittal <shephalm@amazon.com> * Added comment while closing span Signed-off-by: Shephali Mittal <shephalm@amazon.com> --------- Signed-off-by: Shephali Mittal <shephalm@amazon.com> Co-authored-by: Shephali Mittal <shephalm@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>