v2.6.0

Breaking Changes

• autocomplete_filtering_enabled feature flag has been removed and always enabled.
• use_v2_sdk in the Azure backend has been removed. Azure v2 sdk has been promoted to GA/dfeault.
• Removed completedfilepath and blocksfilepath from the wal config.
• TraceQL metrics now bases historical requests off of a set of RF1 blocks written to the backend by the metrics-generators. Set flush_to_storage to true to run metrics queries on historical data.

  metrics_generator:
    processor:
      local_blocks:
        flush_to_storage: true
  

Changes

• [CHANGE] BREAKING CHANGE Remove autocomplete_filtering_enabled feature flag #3729 (@mapno)
• [CHANGE] Bump opentelemetry-collector to 0.102.1 #3784 (@debasishbsws)
• [CHANGE] Bump Jaeger query docker image to 1.57.0 #3652 (@iblancasa)
• [CHANGE] Update Go to 1.22.4 #3757 #3793 (@joe-elliott, @mapno)
• [CHANGE] Make vParquet4 the default block encoding #3810 (@ie-pham)
• [CHANGE] Azure v2 backend becomes the only and primary Azure backend #3875 (@zalegrala)
  BREAKING CHANGE The use_v2_sdk configuration option has been removed.

Features

• [FEATURE] TraceQL support for link scope and link:traceID and link:spanID #3741 (@stoewer)
• [FEATURE] TraceQL support for link attribute querying #3814 (@ie-pham)
• [FEATURE] TraceQL support for event scope and event:name intrinsic #3708 (@stoewer)
• [FEATURE] TraecQL support for event attributes #3708 (@ie-pham)
• [FEATURE] TraceQL support for event:timeSinceStart #3908 (@ie-pham)
• [FEATURE] Autocomplete support for events and links #3846 (@ie-pham)
• [FEATURE] Flush and query RF1 blocks for TraceQL metric queries #3628 #3691 #3723 (@mapno)
• [FEATURE] Add new compare() metrics function #3695 (@mdisibio)
• [FEATURE] Add new api /api/metrics/query for instant metrics queries #3859 (@mdisibio)
• [FEATURE] Add a q parameter to /api/v2/search/tags for tag name filtering #3822 (@joe-elliott)
• [FEATURE] Add exemplars to TraceQL metrics #3824 (@mapno)
• [FEATURE] Added new middleware to block matching urls #3963 (@javiermolinar)
• [FEATURE] Flush and query RF1 blocks for TraceQL metric queries #3628 #3691 #3723 (@mapno) #3995 (@mdisibio)
• [FEATURE] Add support for generating native histograms from metrics-generator #3789 (@zalegrala, @kvrhdn)

Enhancements

• [ENHANCEMENT] Implement arrays for traceql.Static with reused fields #3827 (@stoewer)
• [ENHANCEMENT] Tag value lookup use protobuf internally for improved latency #3731 (@mdisibio)
• [ENHANCEMENT] TraceQL metrics queries use protobuf internally for improved latency #3745 (@mdisibio)
• [ENHANCEMENT] TraceQL search and other endpoints use protobuf internally for improved latency and resource usage #3944 (@mdisibio)
• [ENHANCEMENT] Add local disk caching of metrics queries in local-blocks processor #3799 (@mdisibio)
• [ENHANCEMENT] Improve use of OTEL semantic conventions on the service graph #3711 (@zalegrala)
• [ENHANCEMENT] Performance improvement for rate() by () queries #3719 (@mapno)
• [ENHANCEMENT] Performance improvement for queries using trace-level intrinsics #3920 (@mdisibio)
• [ENHANCEMENT] Use multiple goroutines to unmarshal responses in parallel in the query frontend. #3713 (@joe-elliott)
• [ENHANCEMENT] Protect ingesters from panics by adding defer/recover to all read path methods. #3790 (@joe-elliott)
• [ENHANCEMENT] Added a boolean flag to enable or disable dualstack mode on Storage block config for S3 #3721 (@sid-jar, @mapno)
• [ENHANCEMENT] Add caching to query range queries #3796 (@mapno)
• [ENHANCEMENT] Only stream diffs on metrics queries #3808 (@joe-elliott)
• [ENHANCEMENT] Add data quality metric to measure traces without a root #3812 (@mapno)
• [ENHANCEMENT] Added an example for running Tempo vulture #3829 (@javiermolinar)
• [ENHANCEMENT] Add a new helper method to allow debugging e2e tests #3836 (@javiermolinar)
• [ENHANCEMENT] Refactor Tempo Vulture to reduce code complexity #3850 (@javiermolinar)
• [ENHANCEMENT] Self document makefile #3844 (@javiermolinar)
• [ENHANCEMENT] Added a Tempo CLI command to drop traces by id by rewriting blocks. #3856 (@joe-elliott)
• [ENHANCEMENT] Mixin, make recording rule range interval configurable and increase range interval in alert to support scrape interval of 1 minute #3851 (@jmichalek132)
• [ENHANCEMENT] Add vParquet4 support to the tempo-cli analyse blocks command #3868 (@stoewer)
• [ENHANCEMENT] Improve trace id lookup from Tempo Vulture by selecting a date range #3874 (@javiermolinar)
• [ENHANCEMENT] Add native histograms for internal metrics#3870 (@zalegrala)
• [ENHANCEMENT] Expose availability-zone as a cli flag in ingester #3881
• [ENHANCEMENT] Rename batches property of Trace to ResourceSpans to be OTEL compatible #3895
• [ENHANCEMENT] Reduce memory consumption of query-frontend#3888 (@joe-elliott)
• [ENHANCEMENT] Reduce log level verbosity for e2e tests#3900 (@javiermolinar)
• [ENHANCEMENT] Added new Traces api V2#3912 (@javiermolinar)
• [ENHANCEMENT] Update to the latest dskit #3915 (@andreasgerstmayr)
• [ENHANCEMENT] Reduce allocs building queriers sharded requests #3932 (@javiermolinar)
• [ENHANCEMENT] Allow compaction disablement per-tenant #3965 (@zalegrala)
• [ENHANCEMENT] Implement polling tenants concurrently #3647 (@zalegrala)
• [ENHANCEMENT] Reduce allocs of caching middleware #3976 (@joe-elliott)
• [ENHANCEMENT] BlockMeta improvements to reduce the size #3950 #3951 #3952(@zalegrala)
• [ENHANCEMENT] Add bytes and spans received to usage stats #3983 (@joe-elliott)

Bugfixes

• [BUGFIX] Fix panic in certain metrics queries using rate() with by #3847 (@stoewer)
• [BUGFIX] Fix double appending the primary iterator on second pass with event iterator #3903 (@ie-pham)
• [BUGFIX] Fix metrics queries when grouping by attributes that may not exist #3734 (@mdisibio)
• [BUGFIX] Fix frontend parsing error on cached responses #3759 (@mdisibio)
• [BUGFIX] Fix autocomplete of a query using scoped instrinsics #3865 (@mdisibio)
• [BUGFIX] Fix metrics query histograms and quantiles on traceDuration #3879 (@mdisibio)
• [BUGFIX] Fix divide by 0 bug in query frontend exemplar calculations #3936 (@mdisibio)
• [BUGFIX] max_global_traces_per_user: take into account ingestion.tenant_shard_size when converting to local limit #3618 (@kvrhdn)
• [BUGFIX] Fix http connection reuse on GCP and AWS by reading io.EOF through the http body. #3760 (@bmteller)
• [BUGFIX] Improved handling of complete blocks in localblocks processor after enabling flusing #3805 (@mapno)
• [BUGFIX] ...

v2.6.0-rc.1

• [FEATURE] Flush and query RF1 blocks for TraceQL metric queries #3628 #3691 #3723 (@mapno) #3995 (@mdisibio)
• [ENHANCEMENT] Add bytes and spans received to usage stats #3983 (@joe-elliott)

v2.6.0-rc.0

Breaking Changes

• autocomplete_filtering_enabled feature flag has been removed and always enabled.
• use_v2_sdk in the Azure backend has been removed. Azure v2 sdk has been promoted to GA/dfeault.
• Removed completedfilepath and blocksfilepath from the wal config.

Changes

• [CHANGE] BREAKING CHANGE Remove autocomplete_filtering_enabled feature flag #3729 (@mapno)
• [CHANGE] Bump opentelemetry-collector to 0.102.1 #3784 (@debasishbsws)
• [CHANGE] Bump Jaeger query docker image to 1.57.0 #3652 (@iblancasa)
• [CHANGE] Update Go to 1.22.4 #3757 #3793 (@joe-elliott, @mapno)
• [CHANGE] Make vParquet4 the default block encoding #3810 (@ie-pham)
• [CHANGE] Azure v2 backend becomes the only and primary Azure backend #3875 (@zalegrala)
  BREAKING CHANGE The use_v2_sdk configuration option has been removed.

Features

• [FEATURE] TraceQL support for link scope and link:traceID and link:spanID #3741 (@stoewer)
• [FEATURE] TraceQL support for link attribute querying #3814 (@ie-pham)
• [FEATURE] TraceQL support for event scope and event:name intrinsic #3708 (@stoewer)
• [FEATURE] TraecQL support for event attributes #3708 (@ie-pham)
• [FEATURE] TraceQL support for event:timeSinceStart #3908 (@ie-pham)
• [FEATURE] Autocomplete support for events and links #3846 (@ie-pham)
• [FEATURE] Flush and query RF1 blocks for TraceQL metric queries #3628 #3691 #3723 (@mapno)
• [FEATURE] Add new compare() metrics function #3695 (@mdisibio)
• [FEATURE] Add new api /api/metrics/query for instant metrics queries #3859 (@mdisibio)
• [FEATURE] Add a q parameter to /api/v2/search/tags for tag name filtering #3822 (@joe-elliott)
• [FEATURE] Add exemplars to TraceQL metrics #3824 (@mapno)
• [FEATURE] Added new middleware to block matching urls #3963 (@javiermolinar)

Enhancements

• [ENHANCEMENT] Implement arrays for traceql.Static with reused fields #3827 (@stoewer)
• [ENHANCEMENT] Tag value lookup use protobuf internally for improved latency #3731 (@mdisibio)
• [ENHANCEMENT] TraceQL metrics queries use protobuf internally for improved latency #3745 (@mdisibio)
• [ENHANCEMENT] TraceQL search and other endpoints use protobuf internally for improved latency and resource usage #3944 (@mdisibio)
• [ENHANCEMENT] Add local disk caching of metrics queries in local-blocks processor #3799 (@mdisibio)
• [ENHANCEMENT] Improve use of OTEL semantic conventions on the service graph #3711 (@zalegrala)
• [ENHANCEMENT] Performance improvement for rate() by () queries #3719 (@mapno)
• [ENHANCEMENT] Performance improvement for queries using trace-level intrinsics #3920 (@mdisibio)
• [ENHANCEMENT] Use multiple goroutines to unmarshal responses in parallel in the query frontend. #3713 (@joe-elliott)
• [ENHANCEMENT] Protect ingesters from panics by adding defer/recover to all read path methods. #3790 (@joe-elliott)
• [ENHANCEMENT] Added a boolean flag to enable or disable dualstack mode on Storage block config for S3 #3721 (@sid-jar, @mapno)
• [ENHANCEMENT] Add caching to query range queries #3796 (@mapno)
• [ENHANCEMENT] Only stream diffs on metrics queries #3808 (@joe-elliott)
• [ENHANCEMENT] Add data quality metric to measure traces without a root #3812 (@mapno)
• [ENHANCEMENT] Added an example for running Tempo vulture #3829 (@javiermolinar)
• [ENHANCEMENT] Add a new helper method to allow debugging e2e tests #3836 (@javiermolinar)
• [ENHANCEMENT] Refactor Tempo Vulture to reduce code complexity #3850 (@javiermolinar)
• [ENHANCEMENT] Self document makefile #3844 (@javiermolinar)
• [ENHANCEMENT] Added a Tempo CLI command to drop traces by id by rewriting blocks. #3856 (@joe-elliott)
• [ENHANCEMENT] Mixin, make recording rule range interval configurable and increase range interval in alert to support scrape interval of 1 minute #3851 (@jmichalek132)
• [ENHANCEMENT] Add vParquet4 support to the tempo-cli analyse blocks command #3868 (@stoewer)
• [ENHANCEMENT] Improve trace id lookup from Tempo Vulture by selecting a date range #3874 (@javiermolinar)
• [ENHANCEMENT] Add native histograms for internal metrics#3870 (@zalegrala)
• [ENHANCEMENT] Expose availability-zone as a cli flag in ingester #3881
• [ENHANCEMENT] Rename batches property of Trace to ResourceSpans to be OTEL compatible #3895
• [ENHANCEMENT] Reduce memory consumption of query-frontend#3888 (@joe-elliott)
• [ENHANCEMENT] Reduce log level verbosity for e2e tests#3900 (@javiermolinar)
• [ENHANCEMENT] Added new Traces api V2#3912 (@javiermolinar)
• [ENHANCEMENT] Update to the latest dskit #3915 (@andreasgerstmayr)
• [ENHANCEMENT] Reduce allocs building queriers sharded requests #3932 (@javiermolinar)
• [ENHANCEMENT] Allow compaction disablement per-tenant #3965 (@zalegrala)
• [ENHANCEMENT] Implement polling tenants concurrently #3647 (@zalegrala)
• [ENHANCEMENT] Reduce allocs of caching middleware #3976 (@joe-elliott)
• [ENHANCEMENT] BlockMeta improvements to reduce the size #3950 #3951 #3952(@zalegrala)

Bugfixes

• [BUGFIX] Fix panic in certain metrics queries using rate() with by #3847 (@stoewer)
• [BUGFIX] Fix double appending the primary iterator on second pass with event iterator #3903 (@ie-pham)
• [BUGFIX] Fix metrics queries when grouping by attributes that may not exist #3734 (@mdisibio)
• [BUGFIX] Fix frontend parsing error on cached responses #3759 (@mdisibio)
• [BUGFIX] Fix autocomplete of a query using scoped instrinsics #3865 (@mdisibio)
• [BUGFIX] Fix metrics query histograms and quantiles on traceDuration #3879 (@mdisibio)
• [BUGFIX] Fix divide by 0 bug in query frontend exemplar calculations #3936 (@mdisibio)
• [BUGFIX] max_global_traces_per_user: take into account ingestion.tenant_shard_size when converting to local limit #3618 (@kvrhdn)
• [BUGFIX] Fix http connection reuse on GCP and AWS by reading io.EOF through the http body. #3760 (@bmteller)
• [BUGFIX] Improved handling of complete blocks in localblocks processor after enabling flusing #3805 (@mapno)
• [BUGFIX] Handle out of boundaries spans kinds #3861 (@javiermolinar)
• [BUGFIX] Maintain previous tenant blocklist on tenant errors #3860 (@zalegrala)
• [BUGFIX] Fix prefix handling in Azure backend Find() call #3875 (@zalegrala)
• [BUGFIX] BREAKING CHANGE Remove unused properties from the WAL configuration #3911 (@javiermolinar)
• [BUGFIX] Bring back OTEL receiver metrics #3917 (@javiermolinar)
• [BUGFIX] Correct block end time when the ingested traces are outside the ingestion slack #3954 (@javiermolinar)
• [BUGFIX] Fix race condition where a streaming response could be marshalled while being modi...

v2.5.0

Breaking Changes

• Ownership of /var/tempo is changing. Historically this has been owned by root:root, and with this change it will now be owned by tempo:tempo with the UID/GID of 10001. The ingester and metrics-generator statefulsets may need to be chown'd in order to start properly. A jsonnet example of an init container is included with the PR. This impacts impacts all users of the grafana/tempo Docker image.
  #2265

• In the last release vParquet (the first version) was deprecated and blocked from writes. Now it is removed entirely. It will no longer be recognized as a valid encoding and cannot read any remaining vParquet blocks. Installations running with historical defaults should not require any changes as the default has been migrated for several releases. Installations with storage settings pinned to vParquet must run a previous release configured for vParquet2 or higher until all existing vParquet (1) blocks have expired and been deleted from the backend, or else will encounter read errors after upgrading to this release.
  #3663

• Remove trace by id hedging from the frontend. We consider this configuration an anti-pattern. Enabling it only caused additional issues with Tempo operation. The following have been removed:

  query_frontend:
      trace_by_id:
          hedge_requests_at: 2s
          hedge_requests_up_to: 2
  

  #3522

• Dropped meta-tag for tenant from trace by id multitenant. #3522

• The update to OTLP 1.3.0 removes the deprecated InstrumentationLibrary and InstrumentationLibrarySpan from the OTLP receivers #3649

Changes

• [CHANGE] Align metrics query time ranges to the step parameter #3490 (@mdisibio)
• [CHANGE] Change the UID and GID of the tempo user to avoid root #2265 (@zalegrala)
  BREAKING CHANGE Ownership of /var/tempo is changing. Historically this
  has been owned by root:root, and with this change it will now be owned by
  tempo:tempo with the UID/GID of 10001. The ingester and
  metrics-generator statefulsets may need to be chown'd in order to start
  properly. A jsonnet example of an init container is included with the PR.
  This impacts impacts all users of the grafana/tempo Docker image.
• [CHANGE] Remove vParquet encoding #3663 (@mdisibio)
  BREAKING CHANGE In the last release vParquet (the first version) was deprecated and blocked from writes. Now it is
  removed entirely. It will no longer be recognized as a valid encoding and cannot read any remaining vParquet blocks. Installations
  running with historical defaults should not require any changes as the default has been migrated for several releases. Installations
  with storage settings pinned to vParquet must run a previous release configured for vParquet2 or higher until all existing vParquet (1) blocks
  have expired and been deleted from the backend, or else will encounter read errors after upgrading to this release.
• [CHANGE] Return a less confusing error message to the client when refusing spans due to ingestion rates. #3485 (@ie-pham)
• [CHANGE] Clean Metrics Generator's Prometheus wal before creating instance #3548 (@ie-pham)
• [CHANGE] Update docker examples for permissions, deprecations, and clean-up #3603 (@zalegrala)
• [CHANGE] Update debian and rpm packages to grant required permissions to default storage path after installation #3657 (@mdisibio)
• [CHANGE] Delete any remaining objects for empty tenants after a configurable duration, requires config enable #3611 (@zalegrala)
• [CHANGE] Add golangci to the tools image and update lint make target #3610 (@zalegrala)
• [CHANGE] Update Alpine image version to 3.20 #3710 (@joe-elliott)

Features

• [FEATURE] Add TLS support for Memcached Client #3585 (@sonisr)
• [FEATURE] TraceQL metrics queries: add quantile_over_time #3605 #3633 (@mdisibio)
• [FEATURE] TraceQL metrics queries: add histogram_over_time #3644 (@mdisibio)
• [FEATURE] Added gRPC streaming endpoints for Tempo APIs.
  • Added gRPC streaming endpoints for all tag queries. #3460 (@joe-elliott)
  • Added gRPC streaming endpoints for metrics. #3584 (@joe-elliott)
  • Reduced memory consumption in the frontend for large traces. #3522 (@joe-elliott)
  • Breaking Change Remove trace by id hedging from the frontend. #3522 (@joe-elliott)
  • Breaking Change Dropped meta-tag for tenant from trace by id multitenant. #3522 (@joe-elliott)
• [FEATURE] New block encoding vParquet4 with support for links, events, and arrays #3368 (@stoewer @ie-pham @andreasgerstmayr)

Enhancements

• [ENHANCEMENT] Remove hardcoded delay in distributor shutdown #3687 (@chodges15)
• [ENHANCEMENT] Tempo CLI - add percentage support for query blocks #3697 #3697 (@edgarkz)
• [ENHANCEMENT] Update OTLP and add attributes to instrumentation scope in vParquet4 #3649 (@stoewer)
  Breaking Change The update to OTLP 1.3.0 removes the deprecated InstrumentationLibrary
  and InstrumentationLibrarySpan from the OTLP receivers
• [ENHANCEMENT] Surface new labels for uninstrumented services and systems #3543 (@t00mas)
• [ENHANCEMENT] Add querier metrics for requests executed #3524 (@electron0zero)
• [ENHANCEMENT] Add messaging-system latency histogram to service-graph #3453 (@adirmatzkin)
• [ENHANCEMENT] Add string interning to TraceQL queries #3411 (@mapno)
• [ENHANCEMENT] Add new (unsafe) query hints for metrics queries #3396 (@mdisibio)
• [ENHANCEMENT] Add nestedSetLeft/Right/Parent instrinsics to TraceQL. #3497 (@joe-elliott)
• [ENHANCEMENT] Add tenant to frontend job cache key. #3527 (@joe-elliott)
• [ENHANCEMENT] Better compaction throughput and memory usage #3579 (@mdisibio)
• [ENHANCEMENT] Add support for sharded ingester queries #3574 (@zalegrala)
• [ENHANCEMENT] TraceQL - Add support for scoped intrinsics using : #3629 (@ie-pham)
  available scoped intrinsics: trace:duration, trace:rootName, trace:rootService, span:duration, span:kind, span:name, span:status, span:statusMessage
• [ENHANCEMENT] Performance improvements on TraceQL and tag value search. #3650,#3667 (@joe-elliott)
• [ENHANCEMENT] TraceQL - Add support for trace:id and span:id #3670 (@ie-pham)
• [ENHANCEMENT] Add toggle to inject the tenant ID to generated metrics #3638 (@kvrhdn)

Bugfixes

• [BUGFIX] Fix handling of regex matchers in autocomplete endpoints #3641 (@sd2k)
• [BUGFIX] Update golang.org/x/net package to 0.24.0 to fix CVE-2023-45288 #3613 (@pavolloffay)
• [BUGFIX] Fix metrics query results when filtering and rating on the same attribute #3428 (@mdisibio)
• [BUGFIX] Fix metrics query results when series contain empty strings or nil values #3429 (@mdisibio)
• [BUGFIX] Fix metrics query duration check, add per-tenant override for max metrics query duration #3479 (@mdisibio)
• [BUGFIX] Fix metrics query panic "index out of range [-1]" when a trace has zero-length ID (@mdisibio)
• [BUGFIX] Return unfiltered results when a bad TraceQL query is provided in autocomplete. #3426 (@mapno)
• [BUGFIX] Add support for dashes, quotes and spaces in attribute names in autocomplete #3458 (@mapno)
• [BUGFIX] Correctly handle 429s in GRPC search streaming. #3469 (@joe-ellitot)
• [BUGFIX] Correctly cancel GRPC and HTTP contexts in the frontend to prevent having to rely on http write timeout. #3443 (@joe-elliott)
• [BUGFIX] Add spss and limit to the frontend cache key to prevent the return of incorrect results. #3557 (@joe-elliott)
• [BUGFIX] Use os path separator to split blocks path. [#3552](https://github.co...

v2.5.0-rc.1

Changes

• [CHANGE] Update Alpine image version to 3.20 #3710 (@joe-elliott)

Enhancements

• [ENHANCEMENT] TraceQL - Add support for trace:id and span:id #3670 (@ie-pham)

Bugfixes

• [BUGFIX] Fix TraceQL queries involving non boolean operations between statics and attributes. #3698 (@joe-elliott)

v2.5.0-rc.0

Breaking Changes

• Ownership of /var/tempo is changing. Historically this has been owned by root:root, and with this change it will now be owned by tempo:tempo with the UID/GID of 10001. The ingester and metrics-generator statefulsets may need to be chown'd in order to start properly. A jsonnet example of an init container is included with the PR. This impacts impacts all users of the grafana/tempo Docker image.
  #2265

• In the last release vParquet (the first version) was deprecated and blocked from writes. Now it is removed entirely. It will no longer be recognized as a valid encoding and cannot read any remaining vParquet blocks. Installations running with historical defaults should not require any changes as the default has been migrated for several releases. Installations with storage settings pinned to vParquet must run a previous release configured for vParquet2 or higher until all existing vParquet (1) blocks have expired and been deleted from the backend, or else will encounter read errors after upgrading to this release.
  #3663

• Remove trace by id hedging from the frontend. We consider this configuration an anti-pattern. Enabling it only caused additional issues with Tempo operation. The following have been removed:

  query_frontend:
      trace_by_id:
          hedge_requests_at: 2s
          hedge_requests_up_to: 2
  

  #3522

• Dropped meta-tag for tenant from trace by id multitenant. #3522

• The update to OTLP 1.3.0 removes the deprecated InstrumentationLibrary and InstrumentationLibrarySpan from the OTLP receivers #3649

Changes

• [CHANGE] Align metrics query time ranges to the step parameter #3490 (@mdisibio)
• [CHANGE] Change the UID and GID of the tempo user to avoid root #2265 (@zalegrala)
  BREAKING CHANGE Ownership of /var/tempo is changing. Historically this
  has been owned by root:root, and with this change it will now be owned by
  tempo:tempo with the UID/GID of 10001. The ingester and
  metrics-generator statefulsets may need to be chown'd in order to start
  properly. A jsonnet example of an init container is included with the PR.
  This impacts impacts all users of the grafana/tempo Docker image.
• [CHANGE] Remove vParquet encoding #3663 (@mdisibio)
  BREAKING CHANGE In the last release vParquet (the first version) was deprecated and blocked from writes. Now it is
  removed entirely. It will no longer be recognized as a valid encoding and cannot read any remaining vParquet blocks. Installations
  running with historical defaults should not require any changes as the default has been migrated for several releases. Installations
  with storage settings pinned to vParquet must run a previous release configured for vParquet2 or higher until all existing vParquet (1) blocks
  have expired and been deleted from the backend, or else will encounter read errors after upgrading to this release.
• [CHANGE] Return a less confusing error message to the client when refusing spans due to ingestion rates. #3485 (@ie-pham)
• [CHANGE] Clean Metrics Generator's Prometheus wal before creating instance #3548 (@ie-pham)
• [CHANGE] Update docker examples for permissions, deprecations, and clean-up #3603 (@zalegrala)
• [CHANGE] Update debian and rpm packages to grant required permissions to default storage path after installation #3657 (@mdisibio)
• [CHANGE] Delete any remaining objects for empty tenants after a configurable duration, requires config enable #3611 (@zalegrala)
• [CHANGE] Add golangci to the tools image and update lint make target #3610 (@zalegrala)

Features

• [FEATURE] Add TLS support for Memcached Client #3585 (@sonisr)
• [FEATURE] TraceQL metrics queries: add quantile_over_time #3605 #3633 (@mdisibio)
• [FEATURE] TraceQL metrics queries: add histogram_over_time #3644 (@mdisibio)
• [FEATURE] Added gRPC streaming endpoints for Tempo APIs.
  • Added gRPC streaming endpoints for all tag queries. #3460 (@joe-elliott)
  • Added gRPC streaming endpoints for metrics. #3584 (@joe-elliott)
  • Reduced memory consumption in the frontend for large traces. #3522 (@joe-elliott)
  • Breaking Change Remove trace by id hedging from the frontend. #3522 (@joe-elliott)
  • Breaking Change Dropped meta-tag for tenant from trace by id multitenant. #3522 (@joe-elliott)
• [FEATURE] New block encoding vParquet4 with support for links, events, and arrays #3368 (@stoewer @ie-pham @andreasgerstmayr)

Enhancements

• [ENHANCEMENT] Remove hardcoded delay in distributor shutdown #3687 (@chodges15)
• [ENHANCEMENT] Tempo CLI - add percentage support for query blocks #3697 #3697 (@edgarkz)
• [ENHANCEMENT] Update OTLP and add attributes to instrumentation scope in vParquet4 #3649 (@stoewer)
  Breaking Change The update to OTLP 1.3.0 removes the deprecated InstrumentationLibrary
  and InstrumentationLibrarySpan from the OTLP receivers
• [ENHANCEMENT] Surface new labels for uninstrumented services and systems #3543 (@t00mas)
• [ENHANCEMENT] Add querier metrics for requests executed #3524 (@electron0zero)
• [ENHANCEMENT] Add messaging-system latency histogram to service-graph #3453 (@adirmatzkin)
• [ENHANCEMENT] Add string interning to TraceQL queries #3411 (@mapno)
• [ENHANCEMENT] Add new (unsafe) query hints for metrics queries #3396 (@mdisibio)
• [ENHANCEMENT] Add nestedSetLeft/Right/Parent instrinsics to TraceQL. #3497 (@joe-elliott)
• [ENHANCEMENT] Add tenant to frontend job cache key. #3527 (@joe-elliott)
• [ENHANCEMENT] Better compaction throughput and memory usage #3579 (@mdisibio)
• [ENHANCEMENT] Add support for sharded ingester queries #3574 (@zalegrala)
• [ENHANCEMENT] TraceQL - Add support for scoped intrinsics using : #3629 (@ie-pham)
  available scoped intrinsics: trace:duration, trace:rootName, trace:rootService, span:duration, span:kind, span:name, span:status, span:statusMessage
• [ENHANCEMENT] Performance improvements on TraceQL and tag value search. #3650,#3667 (@joe-elliott)

Bugfixes

• [BUGFIX] Fix handling of regex matchers in autocomplete endpoints #3641 (@sd2k)
• [BUGFIX] Update golang.org/x/net package to 0.24.0 to fix CVE-2023-45288 #3613 (@pavolloffay)
• [BUGFIX] Fix metrics query results when filtering and rating on the same attribute #3428 (@mdisibio)
• [BUGFIX] Fix metrics query results when series contain empty strings or nil values #3429 (@mdisibio)
• [BUGFIX] Fix metrics query duration check, add per-tenant override for max metrics query duration #3479 (@mdisibio)
• [BUGFIX] Fix metrics query panic "index out of range [-1]" when a trace has zero-length ID (@mdisibio)
• [BUGFIX] Return unfiltered results when a bad TraceQL query is provided in autocomplete. #3426 (@mapno)
• [BUGFIX] Add support for dashes, quotes and spaces in attribute names in autocomplete #3458 (@mapno)
• [BUGFIX] Correctly handle 429s in GRPC search streaming. #3469 (@joe-ellitot)
• [BUGFIX] Correctly cancel GRPC and HTTP contexts in the frontend to prevent having to rely on http write timeout. #3443 (@joe-elliott)
• [BUGFIX] Add spss and limit to the frontend cache key to prevent the return of incorrect results. #3557 (@joe-elliott)
• [BUGFIX] Use os path separator to split blocks path. #3552 (@teyyubismayil)
• [BUGFIX] Correctly parse traceql queries with > 1024 character attribute names or static values. #3571 (@joe-elliott)
• [BUGFIX] Fix span-metrics' subprocessors bug that applied wrong configs when running multiple tenants. [#3612](#3...

v2.4.2

• [BUGFIX] Update golang.org/x/net package to 0.24.0 to fix CVE-2023-45288 #3613 (@pavolloffay)

v2.4.1

• [BUGFIX] Fix compaction/retention in AWS S3 and GCS when a prefix is configured. #3465 (@bpfoster)

v2.4.0

Breaking Changes

• Fix issue where tempo drops the entire batch if one trace triggers an error #2571 (@ie-pham)
  Distributor now returns 200 for any batch containing only trace_too_large and max_live_traces errors
  The number of discarded spans are still reflected in the tempo_discarded_spans_total metrics
• Deprecating vParquet v1 #3377 (@ie-pham)
  vParquet can no longer be explicitly configured as a block version. Tempo 2.4 will continue to read these blocks. Support will be dropped entirely in 2.5. The current/default block version is vParquet3.
• Major cache refactor to allow multiple role based caches to be configured #3166 (@joe-elliott).
  Deprecate the following fields. These have all been migrated to a top level "cache:" field.

  storage:
    trace:
      cache:
      search:
        cache_control:
      background_cache:
      memcached:
      redis:
  

Changes

• [CHANGE] Merge the processors overrides set through runtime overrides and user-configurable overrides #3125 (@kvrhdn)
• [CHANGE] Make vParquet3 the default block encoding #2526 (@stoewer)
• [CHANGE] Set autocomplete_filtering_enabled to true by default #3178 (@mapno)
• [CHANGE] Update Alpine image version to 3.19 #3289 (@zalegrala)
• [CHANGE] Breaking Change Fix issue where tempo drops the entire batch if one trace triggers an error #2571 (@ie-pham)
  Distributor now returns 200 for any batch containing only trace_too_large and max_live_traces errors
  The number of discarded spans are still reflected in the tempo_discarded_spans_total metrics
• [CHANGE] Remove experimental websockets support for search streaming. GRPC is the supported method of streaming results #3307 (@joe-elliott)
• [CHANGE] Breaking Change Deprecating vParquet v1 #3377 (@ie-pham)

Features

• [FEATURE] TraceQL metrics queries #3227 #3252 #3258 (@mdisibio @zalegrala)
• [FEATURE] Add support for multi-tenant queries. #3087 (@electron0zero)
• [FEATURE] Major cache refactor to allow multiple role based caches to be configured #3166.
  BREAKING CHANGE Deprecate the following fields. These have all been migrated to a top level "cache:" field.

  storage:
    trace:
      cache:
      search:
        cache_control:
      background_cache:
      memcached:
      redis:
  

Enhancements

• [ENHANCEMENT] Add support for multi-tenant queries in streaming search #3262 (@electron0zero)
• [ENHANCEMENT] Add configuration on tempo-query plugin for fetch services older than complete_block_timeout #3262 (@rubenvp8510)
• [ENHANCEMENT] Add tracing integration to profiling endpoints #3276 (@cyriltovena)
• [ENHANCEMENT] Introduced AttributePolicyMatch & IntrinsicPolicyMatch structures to match span attributes based on strongly typed values & precompiled regexp #3025 (@andriusluk)
• [ENHANCEMENT] Make the trace ID label name configurable for remote written exemplars #3074
• [ENHANCEMENT] Update poller to make use of previous results and reduce backend load. #2652 (@zalegrala)
• [ENHANCEMENT] Improve TraceQL regex performance in certain queries. #3139 (@joe-elliott)
• [ENHANCEMENT] Improve TraceQL performance in complex queries. #3113 (@joe-elliott)
• [ENHANCEMENT] Added a frontend-search cache role for job search caching. #3225 (@joe-elliott)
• [ENHANCEMENT] Added a parquet-page cache role for page level caching. #3196 (@joe-elliott)
• [ENHANCEMENT] Update opentelemetry-collector-contrib dependency to the latest version, v0.89.0 #3148 (@gebn)
• [ENHANCEMENT] Add --max-start-time and --min-start-time flag to tempo-cli command analyse blocks #3250 (@mapno)
• [ENHANCEMENT] Add per-tenant configurable remote_write headers to metrics-generator #3175 (@mapno)
• [ENHANCEMENT] Add variable expansion support to overrides configuration #3175 (@mapno)
• [ENHANCEMENT] Update memcached default image in jsonnet for multiple CVE #3310 (@zalegrala)
• [ENHANCEMENT] Add HTML pages /status/overrides and /status/overrides/{tenant} #3244 #3332 (@kvrhdn)
• [ENHANCEMENT] Precalculate and reuse the vParquet3 schema before opening blocks #3367 (@stoewer)
• [ENHANCEMENT] Add --shutdown-delay to allow Tempo to cleanly drain connections. #3395 (@joe-elliott)
• [ENHANCEMENT] Introduce localblocks process config option to select only server spans 3303#3303 (@zalegrala)
• [ENHANCEMENT] TraceQL/Structural operators performance improvement. #3088 (@joe-elliott)
• [ENHANCEMENT] Localblocks processor honor tenant max trace size limit 3305 (@mdisibio)
• [ENHANCEMENT] Introduce list_blocks_concurrency on GCS and S3 backends to control backend load and performance. #2652 (@zalegrala)
• [ENHANCEMENT] Add per-tenant compaction window #3129 (@zalegrala)
• [ENHANCEMENT] Config: Adds query-frontend.log-query-request-headers to enable logging of request headers in query logs. #3383 (@jmichalek132)

Bugfixes

• [BUGFIX] Fix parsing of span.resource.xyz attributes in TraceQL. #3284 (@mghildiy)
• [BUGFIX] Change exit code if config is successfully verified #3174 (@am3o @agrib-01)
• [BUGFIX] The tempo-cli analyse blocks command no longer fails on compacted blocks #3183 (@stoewer)
• [BUGFIX] Move waitgroup handling for poller error condition #3224 (@zalegrala)
• [BUGFIX] Fix head block excessive locking in ingester search #3328 (@mdisibio)
• [BUGFIX] Fix issue with ingester failed to cut traces no such file or directory #3346 (@mdisibio)
• [BUGFIX] Restore tempo_request_duration_seconds metrics for querier_api_* requests #3403 (@kvrhdn)
• [BUGFIX] Prevent building parquet iterators that would loop forever. #3159 (@mapno)
• [BUGFIX] Sanitize name in mapped dimensions in span-metrics processor #3171 (@mapno)
• [BUGFIX] Fixed an issue where cached footers were requested then ignored. #3196 (@joe-elliott)
• [BUGFIX] Fix panic in autocomplete when query condition had wrong type #3277 (@mapno)
• [BUGFIX] Fix TLS when GRPC is enabled on HTTP #3300 (@joe-elliott)
• [BUGFIX] Correctly return 400 when max limit is requested on search. #3340 (@joe-elliott)
• [BUGFIX] Fix autocomplete filters sometimes returning erroneous results. #3339 (@joe-elliott)
• [BUGFIX] Fixes trace context propagation between query-frontend and querier. #3387 (@mapno)
• [BUGFIX] Fix some instances where spanmetrics histograms could be inconsistent #3412 (@mdisibio)

v2.4.0-rc.0

Breaking Changes

• Fix issue where tempo drops the entire batch if one trace triggers an error #2571 (@ie-pham)
  Distributor now returns 200 for any batch containing only trace_too_large and max_live_traces errors
  The number of discarded spans are still reflected in the tempo_discarded_spans_total metrics
• Deprecating vParquet v1 #3377 (@ie-pham)
  vParquet can no longer be explicitly configured as a block version. Tempo 2.4 will continue to read these blocks. Support will be dropped entirely in 2.5. The current/default block version is vParquet3.
• Major cache refactor to allow multiple role based caches to be configured #3166 (@joe-elliott).
  Deprecate the following fields. These have all been migrated to a top level "cache:" field.

  storage:
    trace:
      cache:
      search:
        cache_control:
      background_cache:
      memcached:
      redis:
  

Changes

• [CHANGE] Merge the processors overrides set through runtime overrides and user-configurable overrides #3125 (@kvrhdn)
• [CHANGE] Make vParquet3 the default block encoding #2526 (@stoewer)
• [CHANGE] Set autocomplete_filtering_enabled to true by default #3178 (@mapno)
• [CHANGE] Update Alpine image version to 3.19 #3289 (@zalegrala)
• [CHANGE] Breaking Change Fix issue where tempo drops the entire batch if one trace triggers an error #2571 (@ie-pham)
  Distributor now returns 200 for any batch containing only trace_too_large and max_live_traces errors
  The number of discarded spans are still reflected in the tempo_discarded_spans_total metrics
• [CHANGE] Remove experimental websockets support for search streaming. GRPC is the supported method of streaming results #3307 (@joe-elliott)
• [CHANGE] Breaking Change Deprecating vParquet v1 #3377 (@ie-pham)

Features

• [FEATURE] TraceQL metrics queries #3227 #3252 #3258 (@mdisibio @zalegrala)
• [FEATURE] Add support for multi-tenant queries. #3087 (@electron0zero)
• [FEATURE] Major cache refactor to allow multiple role based caches to be configured #3166.
  BREAKING CHANGE Deprecate the following fields. These have all been migrated to a top level "cache:" field.

  storage:
    trace:
      cache:
      search:
        cache_control:
      background_cache:
      memcached:
      redis:
  

Enhancements

• [ENHANCEMENT] Add support for multi-tenant queries in streaming search #3262 (@electron0zero)
• [ENHANCEMENT] Add configuration on tempo-query plugin for fetch services older than complete_block_timeout #3262 (@rubenvp8510)
• [ENHANCEMENT] Add tracing integration to profiling endpoints #3276 (@cyriltovena)
• [ENHANCEMENT] Introduced AttributePolicyMatch & IntrinsicPolicyMatch structures to match span attributes based on strongly typed values & precompiled regexp #3025 (@andriusluk)
• [ENHANCEMENT] Make the trace ID label name configurable for remote written exemplars #3074
• [ENHANCEMENT] Update poller to make use of previous results and reduce backend load. #2652 (@zalegrala)
• [ENHANCEMENT] Improve TraceQL regex performance in certain queries. #3139 (@joe-elliott)
• [ENHANCEMENT] Improve TraceQL performance in complex queries. #3113 (@joe-elliott)
• [ENHANCEMENT] Added a frontend-search cache role for job search caching. #3225 (@joe-elliott)
• [ENHANCEMENT] Added a parquet-page cache role for page level caching. #3196 (@joe-elliott)
• [ENHANCEMENT] Update opentelemetry-collector-contrib dependency to the latest version, v0.89.0 #3148 (@gebn)
• [ENHANCEMENT] Add --max-start-time and --min-start-time flag to tempo-cli command analyse blocks #3250 (@mapno)
• [ENHANCEMENT] Add per-tenant configurable remote_write headers to metrics-generator #3175 (@mapno)
• [ENHANCEMENT] Add variable expansion support to overrides configuration #3175 (@mapno)
• [ENHANCEMENT] Update memcached default image in jsonnet for multiple CVE #3310 (@zalegrala)
• [ENHANCEMENT] Add HTML pages /status/overrides and /status/overrides/{tenant} #3244 #3332 (@kvrhdn)
• [ENHANCEMENT] Precalculate and reuse the vParquet3 schema before opening blocks #3367 (@stoewer)
• [ENHANCEMENT] Add --shutdown-delay to allow Tempo to cleanly drain connections. #3395 (@joe-elliott)
• [ENHANCEMENT] Introduce localblocks process config option to select only server spans 3303#3303 (@zalegrala)
• [ENHANCEMENT] TraceQL/Structural operators performance improvement. #3088 (@joe-elliott)
• [ENHANCEMENT] Localblocks processor honor tenant max trace size limit 3305 (@mdisibio)
• [ENHANCEMENT] Introduce list_blocks_concurrency on GCS and S3 backends to control backend load and performance. #2652 (@zalegrala)
• [ENHANCEMENT] Add per-tenant compaction window #3129 (@zalegrala)

Bugfixes

• [BUGFIX] Fix parsing of span.resource.xyz attributes in TraceQL. #3284 (@mghildiy)
• [BUGFIX] Change exit code if config is successfully verified #3174 (@am3o @agrib-01)
• [BUGFIX] The tempo-cli analyse blocks command no longer fails on compacted blocks #3183 (@stoewer)
• [BUGFIX] Move waitgroup handling for poller error condition #3224 (@zalegrala)
• [BUGFIX] Fix head block excessive locking in ingester search #3328 (@mdisibio)
• [BUGFIX] Fix issue with ingester failed to cut traces no such file or directory #3346 (@mdisibio)
• [BUGFIX] Restore tempo_request_duration_seconds metrics for querier_api_* requests #3403 (@kvrhdn)
• [BUGFIX] Prevent building parquet iterators that would loop forever. #3159 (@mapno)
• [BUGFIX] Sanitize name in mapped dimensions in span-metrics processor #3171 (@mapno)
• [BUGFIX] Fixed an issue where cached footers were requested then ignored. #3196 (@joe-elliott)
• [BUGFIX] Fix panic in autocomplete when query condition had wrong type #3277 (@mapno)
• [BUGFIX] Fix TLS when GRPC is enabled on HTTP #3300 (@joe-elliott)
• [BUGFIX] Correctly return 400 when max limit is requested on search. #3340 (@joe-elliott)
• [BUGFIX] Fix autocomplete filters sometimes returning erroneous results. #3339 (@joe-elliott)
• [BUGFIX] Fixes trace context propagation between query-frontend and querier. #3387 (@mapno)