Add trace based testing to the demo

[puckpuck]

Feature Request

The demo is 23 services (and counting). Small changes can have unexpected results, and testing all possibilities is not realistic for committers and approvers of the project.

It would be good to record a complete trace (and metrics) for each defined code path and have that be part of a testing harness. The testing harness should be able to query Jaeger (and Prometheus) to check if the same code paths return similar telemetry signatures.

The testing harness should be callable by a developer using make targets. It should also be integrated into docker-compose and Helm for proper integrated testing.

tracetest should be explored as an option to do this.

[julianocosta89]

Can we also add it as part of our GH actions?

Every time there is a code change, the test runs and makes sure the traces/metrics produced are the expected ones.

[adnanrahic]

Awesome initiative! Tracetest can be used in this case for integration and E2E testing. It can also work as part of the GH actions. We're currently dogfooding by testing Tracetest with Tracetest in our CI pipeline.

Here's a live sample E2E test against the otel demo - http://tracetest-otel-demo.tracetest.io/transaction/79CmirA4g/run/20

I propose the following steps to be:

1. Document exact requirements based on this discussion: https://cloud-native.slack.com/archives/C03B4CWV4DA/p1682994266410579
2. Add Tracetest to the Docker Compose setup
3. Create test cases for known code paths

From there we can discuss further implementing Helm and more intricate E2E testing with transactions and adding automation in GH actions.

[puckpuck]

Can we also add it as part of our GH actions?

Every time there is a code change, the test runs and makes sure the traces/metrics produced are the expected ones.

I want to do this, though I'm unsure what the GH runner cost would be to build and run 23 services to run tests for each PR. At the very least, I would like this to be part of our release process, where a release gets blocked if a test fails.

[danielbdias]

Hi folks! We can do a setup to run Tracetest as a GitHub action. We can build a sample on a fork and build a PR as example.

On Tracetest we have a task that runs on each PR (here) to dogfood our tool. I believe that we can build a simple version to run it on GH actions at least on the release process.

[danielbdias]

Hi folks! I created a draft PR (#877) with some examples of how to run tracetest with the OpenTelemetry demo. The next step that I'm planning to do is to create a GH action executing the testing script and automating these validations.

Feel free to make any comments and give suggestions. Thanks!

[danielbdias]

Hi folks 👋 !

I've updated the PR with more tests migrated from the AVA tests I showed at today's meeting. By running these tests, we noticed some findings:

1. sometimes, the Cart service emits http spans instead of grpc spans, even when we call the endpoints with gRPC. We also got a Tracetest CLI bug that breaks this case's reporting. We will fix it and try pinpointing this behavior on the Cart service to write a bug ticket.
2. the Email service is working but returning HTTP 500 status code on the API call.
3. the Payment service is returning gRPC status code 1 (canceled) instead of 0 (OK) on their calls, even working as intended.

You can reproduce these behaviors by running make run-tracetesting with the PR code.

[danielbdias]

Another point that we discussed today is that Tracetest is mapped on CNCF landscape, but CNCF does not incubate it.
Nowadays, Tracetest is incubated by Kubeshop.io and is intended to be an open-source tool for Trace-based testing.

If you want to see other projects that do Trace-based testing, there are these other three:

• Malabi: open-source tool from Aspect.io that allows you to write Javascript tests against OpenTelemetry Traces;
• Jest OpenTelemetry: a tool from Traceloop to write and run Javascript tests;
• Helios: closed-source tool to visualize and test traces.