This library include metric and trace helper functions with opentelemetry.
go get github.com/worldline-go/tell
To close some metrics and trace
# if empty, metrics and trace providers and create noop provider to continue to work same as code perspective.
# default is empty
OTEL_EXPORTER_OTLP_ENDPOINT=otel-collector:4317
# Also TELEMETRY_COLLECTOR can usable for same thing
# TELEMETRY_COLLECTOR=otel-collector:4317
# inteval duration so send new metrics to otel collector (using time.Parseduration)
# default 5s
TELEMETRY_METRIC_PROVIDER_INTERVAL=5s
TELEMETRY_
prefix comes with igconfig!
Metric and trace checking some special environment values for collector. We should fallow to opentelemetry schemas.
Our environment already give these informations not need to do anything.
Local testing more than one service with metrics, you should also provide this informations to prevent mixing.
# OTEL_SERVICE_NAME=transaction_api
OTEL_RESOURCE_ATTRIBUTES=service.name=transaction_api,service.instance.id=xyz123
In our stack this is show like that for swarm:
OTEL_RESOURCE_ATTRIBUTES=service.name={{.Service.Name}},service.instance.id={{.Task.ID}},host.id={{.Node.ID}},host.name={{.Node.Hostname}}
Check much more details of attributes in here opentelemetry-specification
You can also add your own values after that, these are global values for all services need to have.
Add this configuration in your application config struct:
type Config struct {
// Telemetry configurations
Telemetry tell.Config
}
igconfig
can handle our default values, don't need to change configuration.
After that in main of program pass the telemetry config to create new collector which is connection collector and initialize telemetry and trace providers with common attributes.
collector, err := tell.New(ctx, cfg.Telemetry)
if err != nil {
return fmt.Errorf("failed to init telemetry; %w", err)
}
defer collector.Shutdown()
Now you initialized and connected to our collector. You can send some metrics and trace data.
tell.New
function also set the global values so next time when you need you can get from the global
package.
These global get using by third-party libraries.
// to get tracer provider // go.opentelemetry.io/otel
otel.GetTracerProvider()
// to get meter provider // go.opentelemetry.io/otel/metric/global
global.MeterProvider()
To add some metric, use collector's MeterProvider to create a metric entry and add some values to that entry.
Hold this meters in a struct to reach easily. Check example
// to get meter provider in collector
collector.MeterProvider
// to get meter provider in global
otel.GetMeterProvider()
Counter:
successCounter, err = collector.MeterProvider.Meter("").
Int64Counter("request_success", metric.WithDescription("number of success count"))
if err != nil {
log.Panic().Msgf("failed to initialize successCounter; %w", err)
}
// use counter, add attributes here to give much meaning to your counter.
successCounter.Add(c.Request().Context(), 1, attribute.Key("special").String("X"))
Up/Down Counter: this is same as counter but it can also decrese.
counterUpDown, err = collector.MeterProvider.Meter("").
Int64UpDownCounter("request_success", metric.WithDescription("number of success count"))
if err != nil {
log.Panic().Msgf("failed to initialize successCounter; %w", err)
}
// use counter, add attributes here to give much meaning to your counter.
counterUpDown.Add(c.Request().Context(), 1, attribute.Key("special").String("X"))
Histogram:
valuehistogram, err = collector.MeterProvider.Meter("").
Float64Histogram("request_histogram", metric.WithDescription("value histogram"))
if err != nil {
log.Panic().Msgf("failed to initialize valuehistogram; %w", err)
}
// use histogram, add attributes here to give much meaning to your counter
valuehistogram.Record(c.Request().Context(), float64(countInt), attribute.Key("special").String("X"))
Gauge: this is special and it need to be run with async and we need to register to callback. It is like background operation.
meter := collector.MeterProvider.Meter("")
up, err := meter.Int64ObservableGauge("up", metric.WithDescription("application up status"))
if err != nil {
log.Error().Err(err).Msg("failed to set up gauge metric")
}
regUp, err := meter.RegisterCallback(func(ctx context.Context, o metric.Observer) error {
o.ObserveInt64(up, c.isUp) // value to observe
return nil
}, up)
if err != nil {
log.Error().Err(err).Msg("failed to register up gauge metric")
}
// shutdown will deregister
c.AddRegister(regUp)
View is design how to looks like of your metrics. With this view you can setup your histogram bucket's values.
Name
is important, explain which metric we will change.
In here we used *request_duration_seconds
because application name will come as prefix.
customBucketView := metric.NewView(
metric.Instrument{
Name: "*request_duration_seconds",
},
metric.Stream{
Aggregation: metric.AggregationExplicitBucketHistogram{
Boundaries: []float64{.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10},
},
},
)
If you have views, you need to add before to initialize metric provider.
Add to the tglobal
tglobal.MetricViews.Add("echo", GetViews()) // accepts []view.View
Add to the project this package _example/telemetry/metric.go to hold the custom metrics.
package main
//
// config loaded
//
// open telemetry
collector, err := tell.New(ctx, cnf.Telemetry)
if err != nil {
log.Fatal().Err(err).Msg("failed to init telemetry")
}
defer collector.Shutdown()
telemetry.AddGlobalAttr(attribute.Key("channel").String(cnf.Channel))
if err := telemetry.SetGlobalMeter(); err != nil {
log.Fatal().Err(err).Msg("failed to set metric")
}
After that use your metrics
// in somewhere use your metrics
telemetry.GlobalMeter.Success.Add(ctx, 1, telemetry.GlobalAttr...)
go get github.com/worldline-go/tell/metric/metricecho
Use our Echo framework's middleware to share metrics.
// add echo metrics
e.Use(metricecho.HTTPMetrics(nil))
metricecho package has init function and records views always, no need to additional settings.
Trace is not ready for finops, we will add details later.
go get go.opentelemetry.io/contrib/instrumentation/github.com/labstack/echo/otelecho
// add otel tracing
e.Use(otelecho.Middleware(config.LoadConfig.AppName, otelecho.WithTracerProvider(otel.GetTracerProvider())))
go get go.opentelemetry.io/contrib/instrumentation/runtime
if err := runtime.Start(); err != nil {
return fmt.Errorf("failed to start runtime metrics; %w", err)
}
Check the open telemetry's registry page, new instruments can add here.
https://opentelemetry.io/registry/?language=go
Create a collector, also our collector will create trace provider.
collector, err := tell.New(ctx, cfg.Telemetry)
Use to trace provider to create some trace data.
Context is important to trace data, it will give you the parent-child relationship.
But also if you have cancellation in context maybe you need to use new context based on parent without cancellation.ctx := context.WithoutCancel(c.Request().Context())
Use SetStatus before end the span, it will give you more information about the span and good to service graph.
spanCall.SetStatus(codes.Error, err.Error())
Start a trace with using previous context. After the start it will create new context and use that context for next trace.
If you not use context or not give previous one tracer, it will start as root and not good for view.
// set otel.Tracer("") in a value to use again and again
// set always spankind to internal for internal operations
ctx, span := otel.Tracer("").Start(c.Request().Context(), "PostCount", trace.WithSpanKind(trace.SpanKindInternal))
defer span.End()
// add extra values to your trace data
// our collector adds extra fields automatically as servicename, containerid
span.SetAttributes(attribute.Key("request.count.set").Int64(countInt))
Use echo's trace, it will automatically make propagation and starting client type as server.
e.Use(otelecho.Middleware(config.ServiceName))
Create new span to measure http time but don't forget to add span kind as client. This is important for generating service-graph!
ctx, spanCall := tracer.Start(ctx, "GetTransaction", trace.WithSpanKind(trace.SpanKindClient))
defer spanCall.End()
// add context propagation
otel.GetTextMapPropagator().Inject(ctx, propagation.HeaderCarrier(request.Header))
Important to have span kind as client and db.name attribute. It will help to generate service-graph with virtual nodes.
ctx, span := otel.Tracer("").Start(ctx,
"add_product",
trace.WithSpanKind(trace.SpanKindClient),
trace.WithAttributes(attribute.String("db.name", "postgres")),
)
defer span.End()
To test in local machine deploy otel-collector, grafana, prometheus use our telemetry example repo:
https://github.com/worldline-go/telemetry_example
https://github.com/open-telemetry/opentelemetry-go
https://opentelemetry.io/registry/