Che Tracing

[yarivlifchuk]

Summary

We propose a tracing mechanism that does not require changes to existing Che code. However, we do recommend standardizing the format in which trace are written.
In addition, we propose an option to enable providing additional parameters to trace entries in a standard way, to improve supportability.
To support this, industry-accepted open source components must be deployed to the K8S cluster with special focus on security aspect.

Description

Che epics [Complementary]:
Logging - #10290
Monitoring - #10329
Tracing - #10288
Complementary epic to the opentracing & Jaeger

Che epics [to be reevaluated]:
redhat-developer/rh-che#718
It gives a view on the K8S Tracing best practice and the implication on Che agents and should be complementary epic.

Background

In a distributed system, a trace encapsulates the transaction’s state as it propagates through the system. Tracing helps gather timing data needed to troubleshoot latency problems in microservice architectures, and tells the story of a transaction or workflow as it propagates through a distributed system.
Every transaction might reflect performance anomalies in an early phase when new services are being introduced by independent teams.

Tracers live in the applications. It records timing and metadata about operations that took place. They often instrument libraries, so that their use is transparent to users. For example, an instrumented web server records when it received a request and when it sent a response. The trace data collected is called a Span.
Main benefits of distributed tracing:

• Distributed context propagation: illuminates the request’s path until its final destination.
• Out of the box infrastructure overview: how the interactions between services are done and their dependencies.
• Efficient and fast detection of latency and bottleneck issues.

https://sematext.com/blog/opentracing-distributed-tracing-emerging-industry-standard
http://opentracing.io/documentation/pages/instrumentation

Standard API and Common Tracers

OpenTracing API - offer a consistent, unified and tracer-agnostic instrumentation API for a wide range of frameworks, platforms and programming languages. It abstracts away the differences among numerous tracer implementations, so shifting from an existing one to a new tracer system would only require configuration changes specific to that new tracer.

Common distributed tracers are Zipkin and Jaeger inspired on Google’s Dapper large-scale distributed tracing platform.

1. Zipkin
   Zipkin is an open-source distributed tracing solution implemented in Java and with OpenTracing compatible API. It manages both the collection and lookup of this data.
   Zipkin component is installed per cluster. It collects and display tracing data. Each node is installed with Linkerd as a DeamonSet that export tracing data.
   https://sematext.com/blog/opentracing-zipkin-as-distributed-tracer

2. Jaeger
   Inspired by Dapper and OpenZipkin, is a distributed tracing system released as open source by Uber Technologies. Jaeger extends more complex architecture for larger scale of requests and performance.
   Jaeger has better language coverage of OpenTracing-compatible clients, low memory footprint, and a modern, scalable design compared to Zipkin.
   Jaeger Collector and Query is installed per cluster and require a backing storage. The Jaeger Agent is deployed as a DeamonSet and receive traces via UDP. All pods running on a given node will send data to the same agent. If that's not suitable for workload, an alternative is to deploy the agent as a sidecar.
   https://github.com/jaegertracing/jaeger

Implementation recommendation

1. Write trace based on the relevant library by the Che agents and any relevant Applications.
2. Add custom environment params to the trace’s records, e.g. user’s tenant id.

Implementation

• Implementation plan for Opentracing support for k8s / OpenShift infrastructures in Che Implementation plan for Opentracing support for k8s / OpenShift infrastructures in Che  #11720
• Workspace related operations
  • workspace creation
  • workspace start
  • trace workspace deletion
  • trace volume claim attachment
  • trace creation of che plugins -> moved to Trace ChePlugins and how they are applied to workspaces #12003
  • define meaningful set of tags to apply to the traces. (Postponed)
  • review Made PVC Strategies wait until PVCs are bound during preparing #11920 maybe we need some traces here
• Investigate tracing of wsmaster tomcat server shutdown Investigate tracing of wsmaster tomcat server shutdown #11946
• Investigate tracing of DB migration Investigate tracing of DB migration #11945
• Tracing operation made over JSON-RPC Tracing operation made over JSON-RPC #11923
• [Tracing] Provide more context information about workspace related operation [Tracing] Provide more context information about workspace related operation. #12533
• Trace ChePlugins and how they are applied to workspaces Trace ChePlugins and how they are applied to workspaces #12003
• Trace activity that is happened in workspace??? Do we need this????
• Connect jaeger authentication to keycloak Connect jaeger authentication to keycloak #12544
• Document tracing usecase Document tracing usecase #12545
• Che monitoring/tracing in istio environment Che monitoring/tracing in istio environment. #12546
• [Traces] Machine create/start trace has to be more detailed #12739 [Traces] Machine create/start trace has to be more detailed
• [Traces] Study if there is a way to track container start phases #12740 [Traces] Study if there is a way to track container start phases

[che-bot]

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.