You can now transform ingested data directly in Uptrace without using OpenTelemetry Collector. The following operations are supported:
rename_attr
allows to change the attribute name.delete_attrs
allows to delete attributes by their exact name or by a regular expression.keep_attrs
allows to keep some attributes and delete the rest.drop
allows to drop matching spans/events/logs/datapoints.sample
allows to sample a fraction of spans/events/logs.script
enables writing simple scripts to parse attributes, reduce attributes cardinality, turn logs into spans, etc.
See documentation for details.
When searching spans and logs, a word filter now matches full words/tokens. If you want to search for logs that contain a certain prefix, just add * to the end of the word.
For example, the query err*
will find the following logs:
err
error
an_error
You can also search over all attribute values using _attrs
scope:
_attrs:error
See documentation for more details.
Uptrace now supports incremental query mode when searching or filtering spans/logs, which allows to search over large periods of time.
It works like this:
-
Incremental mode is enabled when searching or filtering, i.e. it does not work without search or filters.
-
Uptrace incrementally searches spans/logs hour by hour and sends you search results every 5 seconds.
-
The incremental search will stop after 20 seconds, but you can continue searching by pressing the "Continue" button.
-
Search results are stored in a special table and are deleted after 10 days. If you try the same search on another day, it will continue from the last saved point.
For now, this optimization does not work when searching over groups and loading filter facets. It will be added later this month.
Both tracing and metrics now support more complex WHERE such as (service_name = "s1" AND log_severity = "DEBUG") OR (service_name = "s2" AND log_severity = "INFO")
.
Tracing now allows you to use If combinator from ClickHouse, for example, countIf(_status_code = "error")
or sumIf(_count, service_name = "s1")
. With metrics, you can achieve the same with count($spans{_status_code="error"})
.
Filters now behave differently when the attribute does not exist, for example, where attr1 = ""
excludes spans that don't have the attribute. You can get the old behavior by using where attr1 = "" or attr1 does not exist
.
-
You can now tag saved view to organize them into categories.
-
Alert filters are reworked after tracing and metrics filters. You can now pin/unpin attributes.
-
Uptrace now supports all OpenTelemetry severity levels such as
TRACE1
,TRACE2
,TRACE3
, etc. System names remain unchanged, for example,log:trace
,log:info
,log:warn
, etc.
Error monitors have been reworked:
-
You can use all available filters to include/exclude monitored errors. You can also customize default filters to monitor
WARN
logs. -
You can add
group by
clauses to customize the default errors grouping and create a separate alert/notification, for example,group by _group_id, service_name, cloud_region
. -
Uptrace no longer creates alerts for excluded errors.
-
Notifications frequency no longer depends on the error count and instead notifies you every 1/6/24+ hours.
Existing error monitors should be converted automatically. No action is required.
Existing alerts are removed, so you should see an increase in error notifications for a few days.
The quick search now allows to specify the search attribute:
_kind:client
. Include items with "_kind" containing "client".-key:value
. Exclude items with "key" containing "value".~msg:regexp
. Include items with "msg" matching the "regexp".
Besides, the _count
is no longer an alias for sum(_count)
, but instead points to the column holding the adjusted count. Now you should use sum(_count)
and per_min(sum(_count))
to count the number of items.
_error_rate
is still supported and is an alias for sum(_error_count) / sum(_count)
.
-
log_message
andexception_message
attributes are automatically promoted todisplay_name
when used in logs. The original attribute is removed to avoid double indexing.Instead of
log_message contains "something"
you should usedisplay_name contains "something"
. The default search will continue to work as previously. -
Removed
_kind
and_status_code
attributes from the logs and events, because the OpenTelemetry data model does not support such attributes. Previously, these attributes were inherited from spans when possible.
Added support for two-factor authentication (2FA) using a one-time password.
- You can enable 2FA on your profile page.
- On the org page, admins can require 2FA for all org users. Users without 2FA won't be able to access org data until they configure 2FA.
-
Both traces and metrics now support the following functions:
extract(haystack, pattern)
,replace(haystack, substring, replacement)
, andreplace_regexp(haystack, pattern, replacement)
.These functions are most useful in grouping expressions, for example:
group by extract(host_name, `^uptrace-prod-(\w+)$`) as host group by replace_regexp(host_name, `^some-prefix-(\w+)$`, `\1`) as host
See documentation for details.
-
You can use dot notation to access sub-fields in nested structures such as JSON, for example,
data.key1.subkey1
can extract the field value from the following JSON:{ "data": { "key1": { "subkey1": "value1" } } }
This does not work for arrays.
-
Added support for dark mode. Dark mode should be enabled automatically if your operating system uses a dark theme. It can also be enabled manually using the switch in the upper right corner.
-
The
http
system is split intohttpclient
andhttpserver
depending on the span kind. You can learn more about systems here.If the feedback is positive, we're going to do the same for the
rpc
system. -
You can now increase data retention on the Project Settings page for an additional fee.
-
Single sign-on now allows you to specify a regexp to validate user emails, for example, you can only allow users with a certain domain.
-
Navigation is reworked using a navigation drawer.
Uptrace is now mostly compatible with the Prometheus query language, for example, the following PromQL expressions are also valid in Uptrace:
$metric_name{foo="xxx",bar~"yyy"}
increase($metric_name)
anddelta($metric_name)
rate($metric_name[5m])
andirate($metric_name[5m])
avg_over_time($go_goroutines[5m])
avg by (foo)(sum by(foo, bar)($metric_name))
$metric_name offset 1w
- Math between series with automatic many-to-one/one-to-many vector matching, for example,
sum($mem by (type)) / sum($mem) as mem
.
See documentation for more details.
-
Improved metrics UI. Added ability to group items into rows.
-
Added ability to use Uptrace as a Prometheus data source in Grafana. Uptrace uses the original Prometheus engine so all Prometheus queries should be supported and you should be able to use existing Grafana dashboards with the Uptrace data source.
-
Added ability to customize grouping when querying metrics, for example,
sum(irate($cpu_secs by (mode, cpu)) by (mode)) / sum(irate($cpu_secs by (mode, cpu)))
. -
Metric names and attributes are automatically changed to comply with Prometheus/Loki restrictions, for example,
service.name
becomesservice_name
. -
Added ability to view/create monitors from YAML.
-
Added Mattermost notification channel.
-
Added support for service graphs.
Service Graphs provide a visual representation of service interactions, dependencies, and performance metrics. Service graphs are built by analyzing span relationships and require specific span attributes.
The same data is also available as a metrics dashboard:
-
Added Microsoft Teams notification channel.
-
You can now create custom grouping rules for logs and exceptions.
For example, you can configure Uptrace to create a separate error group for each unknown PostgreSQL column:
# Error messages ERROR: column "event.created_at" does not exist (SQLSTATE=42703) ERROR: column "updated_at" does not exist (SQLSTATE=42703) ERROR: column "name" does not exist (SQLSTATE=42703) # Pattern log.severity=ERROR column column_name=<quoted,fingerprint> does not exist sqlstate=<kv>
-
Added ability to specify aliases in the
group by
clause, for example,group by host.name as host
. This allows to join metrics with different attributes:$metric1 + $metric2 | group by $metric1.host.name | group by $metric2.host as host.name
-
Added flapping alerts mitigations to metrics monitors.
-
Migrated some metric attributes to the v1.21 OpenTelemetry schema which introduced some breaking changes to attribute names.
Most notably:
http.method
is renamed tohttp.request.method
.http.status_code
is renamed tohttp.response.status_code
.
- On the billing page, allow to set per-project budget and enable/disable dynamically adjusted sampling.
-
Implemented threading/grouping for email notifications.
-
Migrated to the v1.21 OpenTelemetry schema which introduced some breaking changes to attribute names.
Most notably:
http.method
is renamed tohttp.request.method
.http.status_code
is renamed tohttp.response.status_code
.http.client_ip
is renamed toclient.address
.
Uptrace should automatically handle both variants, but you may need to update your favorite queries.
- Added
uptrace.billing.spans
metric with the number of sampled and dropped spans/logs.
-
The number of dropped bytes metric is removed, because it is not always possible to accurately track this metric. The number of sampled bytes is still maintained.
-
Added support for dynamically adjusted sampling to stay within the budget. When enabled, sampling is adjusted every 5 minutes.
Added more user roles:
- Outside Collaborator. Can be added to a team to access team projects.
- Member. Can be added to a team to access team projects. Can create new projects.
- Any Viewer. Can view any project in the org. Can create new projects.
- Any Admin. Can admin any project in the org. Can create new projects.
- Billing Manager. Can manage billing details. Can create new projects.
- Owner. Has full administrative access to the entire organization.
Added more notification channels:
- Opsgenie
- Telegram
Added Single Sign-On support using OpenID Connect (OIDC) and SAML 2.0, for example, Google, Okta, and Keycloak.
Single Sign-On allows to manage users using OIDC or SAML providers. After logging in, such users are automatically added to a team and can access team projects. When users are removed by the provider, they automatically lose granted access.
Add support for Chart annotations.
Add Prometheus remote write support.
Improve spans querying language, for example, per_min(sum(http.read_bytes)) as read_per_min
is now supported.
Add Vercel Logs integration.
-
Added Spans, Logs, and Events system groups.
-
Uptrace now uses display.name instead of
.name
and.event_name
attributes. -
You can now use
display.name contains "get|post"
to search for spans, logs, and events at the same time. -
Exceptions are now grouped together with
log.error
logs. You can still distinguish logs and exceptions usingwhere exception.type exists
filter.
- Span attributes now start with a dot, for example,
.name
instead ofspan.name
. In metrics, this can lead to names like$spans..name
and$spans..status_code
.
-
Add a separate page to each organization that displays number or ingested bytes, spans, and datapoints.
On that page you can also change Paddle subscription that is used to process payments.
To view org billing info, click Organizations -> Org Name -> Billing and usage.
-
Add AWS CloudWatch intergration to collect metrics and logs from AWS CloudWatch.
-
Add FluentBit integration to ingest logs into Uptrace.
-
Histogram metric changed from
per_min($metric_name)
toper_min(count($metric_name))
for clarity. Existing dashboards should be automatically updated.You can now also use
per_min
as a general function, for example,per_min(p50($metric_name))
which is the equivalent ofp50($metric_name) / _minutes
. -
Documented Uptrace Enterprise features.
- Accept errors and spans from Sentry SDK. The integration was created so you can use Sentry for browsers with Uptrace.
- Added metrics pricing.
The cost of metrics is $2 for 1000 timeseries. Uptrace stores 1 datapoint per minute for each timeseries and uses the number of active datapoints to calculate the bill.
Also, for every $5 spent on tracing or logs, you get 1000 timeseries for free. For example, if your tracing bill is $100, you get 20k timeseries for free.
A timeseries is a metric with an unique set of attributes/labels. For example, if you collect 10 metrics on each host and you have 10 hosts, that is 10*10=100 timeseries.
A datapoint is a single measured occurrence of a metric within a timeseries, consisting of a value and timestamp.
-
Allow to edit dashboards using YAML format.
-
Added new notification channel: webhooks. Webhooks allow receiving notifications about alerts managed by Uptrace via HTTP POST requests.
For example, you will receive the following JSON message when an alert is created:
{ "id": "1676471814931265794", "eventName": "created", "payload": { "custom_key": "custom_value" }, "createdAt": "2023-02-15T14:36:54.931265914Z", "alert": { "id": "123", "url": "https://app.uptrace.dev/alerting/1/alerts/123", "name": "Test message", "type": "metric", "state": "open", "createdAt": "2023-02-15T14:36:54.931265914Z" } }
-
Allow to create "Text gauges" at the top of dashboards.
-
Allow to quickly change grouping in the Table dashboards. Pinned attributes are displayed first.
-
Improved
uptrace.tracing.spans
anduptrace.tracing.events
to fetch data directly from the spans table. That means that you can now group and filter by all available span attributes. -
Improved metrics UI to provide attribute facets just like with spans.
-
Added a view to visualize spans query results as a chart. You can save the resulting view using "Saved views" feature.
-
Added an ability to create errors monitors in addition to metrics monitors. Using errors monitors, you can include/exclude certain errors by span attributes, for example, to receive notifications only for certain environments or services.
-
You can now configure per-project sampling to sample a fraction of spans and drop the rest.
You can also adjust sampling fraction depending on span attributes, for example, you can have different sampling fraction depending on
deployment.environment
attribute.Visit the Billing page to learn more.
-
You can now monitor number of sampled and dropped bytes to get notified when you receive more data than usually.
-
To ease navigation, all alerting-related pages are grouped under the "Alerts" tab.
-
You can now configure multiple Slack and PagerDuty notification channels. Channels are associated with projects, not users.
-
You can specify notification channels immediately when creating metrics monitors.
-
Span groups are no longer automatically monitored. Instead, Uptrace converts spans to metrics and you can monitor spans metrics using monitors.
By default, the following metrics are available:
uptrace.tracing.spans
,uptrace.tracing.events
,uptrace.billing.bytes
. In future, you will be able to create custom metrics from spans.Using
uptrace.tracing.spans
metric, you can monitor number of spans, errors, error rate, and p50/p75/p90/p99 duration. -
You can quickly create metrics monitors from span systems:
From services and hostnames:
And metrics:
- Improve logs and errors grouping.
- Allow to filter span facets by attribute key.
- Added ability to save and restore views:
- Added ability to pin attributes/facets to the top:
- Added ability to retry payments on the billing page.
- You can now change payment details in Paddle without recreating a subscription.
Pricing is updated with the following changes:
-
Added more volume-based discounts.
-
Removed minimum $30 payment. If your monthly usage is 70 gigabytes, you will be billed $7 instead of $30.
-
Uptrace now bills for the amount of ingested gigabytes monthly or every $100, whichever happens first. All payments with receipts are available on your billing page.
-
Tail-based sampling is now billed separately at $0.02 per gigabyte.
-
You can disable tail-based sampling to get a discount, but Uptrace will stop processing new spans when your monthly budget is fully spent. In that case, you need to increase the budget or wait until the next billing cycle.
When tail-based sampling is enabled, Uptrace makes sure your budget is spent uniformly across the month so you always have fresh statistics.
Tail-based sampling is disabled by default.
-
Starting from December 1st, users on the free tier won't be able to use tail-based sampling.
-
Added quick filters by
deployment.environment
andservice.name
attributes. -
Added faceted filters for spans and events.
-
Added a project setting to group spans by
deployment.environment
attribute. -
Added a project setting to group spans with
funcs
system byservice.name
attribute.
- Released an initial version of uptrace-php