Observability Future Vision
Observability is the ability to measure a system’s current state based on the data it generates, such as logs, metrics, and traces. Observability relies on telemetry derived from instrumentation that comes from the endpoints and services.
Observability telemetry signals (logs, metrics, traces) arriving from the system would contain all the necessary information needed to observe and monitor.
Modern application can have a complicated distributed architecture that combines cloud native and microservices layers. Each layer produces telemetry signals that may have different structure and information.
Using Simple Schema's Observability telemetry schema we can organize, correlate and investigate system behavior in a standard and well-defined manner.
Observability telemetry schema defines the following components - logs, traces and metrics.
Logs provide comprehensive system details, such as a fault and the specific time when the fault occurred. By analyzing the logs, one can troubleshoot code and identify where and why the error occurred.
Traces represent the entire journey of a request or action as it moves through all the layers of a distributed system. Traces allow you to profile and observe systems, especially containerized applications, serverless architectures, or microservices architecture.
Metrics provide a numerical representation of data that can be used to determine a service or component’s overall behaviour over time.
In many occasions, correlation between the logs, traces and metrics is mandatory to be able to monitor and understand how the system is behaving. In addition, the distributed nature of the application produces multiple formats of telemetry signals arriving from different components ( network router, web server, database)
For such correlation to be possible the industry has formulated several protocols (OTEL, ECS, OpenMetrics) for communicating these signals - the Observability schemas.
In order to be able to correlate information across different signal (represented in different indices) we introduced the notion of correlation into the schema. This information is represented explicitly in both the declarative schema file and the physical mapping file
This information will enable the knowledge to be projected and allow for analytic engine to produce a join query that will take advantage of these relationships.
The next ensemble of issues is reflecting our efforts to achieve the data correlation goals for the Observability plugin within OpenSearch
Our vision for data correlations in Observability can be visualized in the following diagrams:
