Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cortex/querier: fix analysis merging #7179

Merged
merged 1 commit into from
Mar 5, 2024
Merged

Conversation

GiedriusS
Copy link
Member

We were not merging analysis properly - mergo was overwriting data. Instead of using a whole library for this, just write two small functions and use them. Add test to cover this.

@GiedriusS GiedriusS force-pushed the fix_analysis_merging branch from aede7c8 to c4d97b2 Compare March 4, 2024 16:20
We were not merging analysis properly - mergo was overwriting data.
Instead of using a whole library for this, just write two small
functions and use them. Add test to cover this.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
@GiedriusS GiedriusS force-pushed the fix_analysis_merging branch from c4d97b2 to 49e8bfe Compare March 5, 2024 07:13
}
}

func AnalyzesMerge(analysis ...*Analysis) *Analysis {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be a slightly more appropriate name

Suggested change
func AnalyzesMerge(analysis ...*Analysis) *Analysis {
func MergeAnalyses(analysis ...*Analysis) *Analysis {

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All other merge methods have the form of XMerge(). For instance:

matrixMerge
StatsMerge

So, I'd like to keep them the same

@GiedriusS GiedriusS merged commit c06d55d into main Mar 5, 2024
20 checks passed
@GiedriusS GiedriusS deleted the fix_analysis_merging branch March 5, 2024 07:52
jnyi pushed a commit to jnyi/thanos that referenced this pull request Apr 4, 2024
We were not merging analysis properly - mergo was overwriting data.
Instead of using a whole library for this, just write two small
functions and use them. Add test to cover this.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
coleenquadros added a commit to stolostron/thanos that referenced this pull request Jul 24, 2024
* CI: increase e2e test parallelism (#6875)

Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>

* compact: hook nodownsamplemarkfilter into filters chain (#6893)

We have a NoDownsampleMarkFilter that we were not using before in the
compactor for some reason. Hook it into the filters chain if
downsampling is enabled and then trim matching ULIDs from the
downsampling process. Add a test to cover this scenario.

Fixes https://github.com/thanos-io/thanos/issues/6179.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* e2e/compact: fix assertion values (#6909)

I think the original values were simply too small there - if the runner
is slow then it can see smaller metrics values. If the runner works well
then the values can go to these bigger values and then the test fails.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* exposes asyncOperationProcessor (#6908)

expose struct



fix lint

Signed-off-by: Ben Ye <benye@amazon.com>

* Removed the gitpod from the contribution.md (#6907)

Signed-off-by: MeenuyD <meenu.coninja@gmail.com>

* .circleci: cache tool deps (#6910)

I have noticed that downloading tool deps takes ~5 min in CI each time.
We should be able to also cache those tool deps.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* MAINTAINERS.md: add myself as maintainer (#6911)

Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>

* blog: Introducing ThanosCon in Paris KubeCon EU 2024 (#6912)

* blog: Introducing ThanosCon in Paris KubeCon EU 2024

Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com>

* Add image

Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com>

* make docs

Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com>

* Apply suggestions

Co-authored-by: Matej Gera <38492574+matej-g@users.noreply.github.com>
Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com>

* Apply suggestions

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com>

* Add link to slack channel

Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com>

---------

Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com>
Co-authored-by: Matej Gera <38492574+matej-g@users.noreply.github.com>
Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Update 2023-20-11-thanoscon.md (typo) (#6915)

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Added info about ThanosCon to website and Readme. (#6917)

* Added info about ThanosCon to website and Readme.

Signed-off-by: bwplotka <bwplotka@gmail.com>

* Update website/layouts/index.html

Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

---------

Signed-off-by: bwplotka <bwplotka@gmail.com>
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com>

* Objstore: Bump Objstore for Azure Workload Identity support (#6891)

* Bump objstore for Azure Workload Identity support

Signed-off-by: Rikhil Shah <rikhil.shah@metaswitch.com>

* make check-docs

Signed-off-by: Rikhil Shah <rikhil.shah@metaswitch.com>

* Add changelog entry

Signed-off-by: Rikhil Shah <rikhil.shah@metaswitch.com>

* Update Azure client docs

Signed-off-by: Rikhil Shah <rikhil.shah@metaswitch.com>

* make check-docs

Signed-off-by: Rikhil Shah <rikhil.shah@metaswitch.com>

* Move changelog entry to 0.33.0 release

Signed-off-by: Rikhil Shah <rikhil.shah@metaswitch.com>

* Move changelog entry

Signed-off-by: Rikhil Shah <rikhil.shah@metaswitch.com>

---------

Signed-off-by: Rikhil Shah <rikhil.shah@metaswitch.com>
Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>

* update compactor backlog doc for checking halt (#6906)

Signed-off-by: Ben Ye <benye@amazon.com>

* Update LabyrintLabs logo

Signed-off-by: Martin Hauskrecht <hauskrechtmartin@gmail.com>

* Support reload using signal (#6453)

* Support reload using signal

Signed-off-by: Simon Pasquier <spasquie@redhat.com>

* Add --reloader.method option to sidecar

This option allows to tell the sidecar to send a SIGHUP signal to the
monitored process to reload its configuration instead of the default
HTTP-based method.

Signed-off-by: Simon Pasquier <spasquie@redhat.com>

* Update docs and CHANGELOG.md

Signed-off-by: Simon Pasquier <spasquie@redhat.com>

---------

Signed-off-by: Simon Pasquier <spasquie@redhat.com>

* Support float histogram in store gateway (#6925)

* support float histogram in store gateway

Signed-off-by: Ben Ye <benye@amazon.com>

* fix lint

Signed-off-by: Ben Ye <benye@amazon.com>

* fix e2e test

Signed-off-by: Ben Ye <benye@amazon.com>

* update prompb

Signed-off-by: Ben Ye <benye@amazon.com>

* add changelog

Signed-off-by: Ben Ye <benye@amazon.com>

---------

Signed-off-by: Ben Ye <benye@amazon.com>

* chore: add truelayer to the adopters (#6933)

Signed-off-by: Matteo Martellini <matteo.martellini@truelayer.com>

* Fetcher: Add a BlockIDsFetcher Interface to BaseFetcher (#6902)

* add BlockIDsFetcher to BaseFetcher

Signed-off-by: Wen Xu <wenxuamz@amazon.com>

* fix lint

Signed-off-by: Wen Xu <wenxuamz@amazon.com>

* use chan in the interface method to accept active block ids

Signed-off-by: Wen Xu <wenxuamz@amazon.com>

* fix comments

Signed-off-by: Wen Xu <wenxuamz@amazon.com>

* fix lint

Signed-off-by: Wen Xu <wenxuamz@amazon.com>

* add description of active and parital blocks and modify changelog

Signed-off-by: Wen Xu <wenxuamz@amazon.com>

* fix interface description

Signed-off-by: Wen Xu <wenxuamz@amazon.com>

* remove entry in changelog

Signed-off-by: Wen Xu <wenxuamz@amazon.com>

---------

Signed-off-by: Wen Xu <wenxuamz@amazon.com>

* Update hugo file (#6927)

* Update hugo file #6889

Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in>

* Version 0.33 removed

Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in>

---------

Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in>

* Updates busybox SHA (#6897) (#6937)

Signed-off-by: GitHub <noreply@github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com>

* Fix hotstar logo (#6938)

* Fix hotstar logo

Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in>

* Delete website/.hugo_build.lock

Signed-off-by: Kartikay <120778728+kartikaysaxena@users.noreply.github.com>

---------

Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in>
Signed-off-by: Kartikay <120778728+kartikaysaxena@users.noreply.github.com>

* errutil: fix deadlock

Fix deadlock in the Add() function - it accidentally was calling itself
recursively. This struct wraps a `MultiError` so we need to call `Add()`
on that `MultiError` inside.

This deadlock can manifest in Receive - million+ leaking goroutines, all
stuck on the MultiTSDB lock.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* update runtime/CloseWithErrCapture comment (#6952)

* update runtime/CloseWithErrCapture comment

Signed-off-by: Sagnik Das <116512372+sagnik3788@users.noreply.github.com>

* fix lint

Signed-off-by: Sagnik Das <116512372+sagnik3788@users.noreply.github.com>

---------

Signed-off-by: Sagnik Das <116512372+sagnik3788@users.noreply.github.com>

* Added a new flag for maximum retention bytes for thanos (#6944)

* changes

Signed-off-by: Kartikay <120778728+kartikaysaxena@users.noreply.github.com>

* Changed Int64 to BytesVar

Signed-off-by: Kartikay <120778728+kartikaysaxena@users.noreply.github.com>

* Updated Changelog for new flag in receive

Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in>

---------

Signed-off-by: Kartikay <120778728+kartikaysaxena@users.noreply.github.com>
Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in>

* fix store gateway span name (#6953)

Signed-off-by: Ben Ye <benye@amazon.com>

* Upgrade prometheus to latest main (#6950)

* upgrade prometheus to latest main

Signed-off-by: Ben Ye <benye@amazon.com>

* upgrade again

Signed-off-by: Ben Ye <benye@amazon.com>

* fix test

Signed-off-by: Ben Ye <benye@amazon.com>

---------

Signed-off-by: Ben Ye <benye@amazon.com>

* Improve filtered index cache filtering (#6955)

* improve filtered index cache filtering

Signed-off-by: Ben Ye <benye@amazon.com>

* make inline

Signed-off-by: Ben Ye <benye@amazon.com>

---------

Signed-off-by: Ben Ye <benye@amazon.com>

* store: fix misc label_value issues related to external labels (#6879)

Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>

* receive: fix adding fields to logger

Fix adding fields to the logger because otherwise it looks like
"[tenant foo]: (UNDEFINED)".

Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com>

* Copy labels coming from remote engines (#6957)

* Copy labels coming from remote engines

When running in distributed mode, the remote engine will use an unsafe
cast from ZLabels to Prometheus labels to avoid making new allocations.
This makes it hard to use the new gRPC shared buffer pool for receiving
and decompressing messages since memory gets retained beyond the scope
of a Recv() call.

This commit removes the unsafe cast and makes an explicit memory copy
of received series labels. Since remote queries are already aggregated
series, the amount of data we receive should be small anyway, and the copies
on average should have a small impact.

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>

* Use clone on strings

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>

---------

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>

* Add tracing index cache (#6954)

* add tracing index cache

Signed-off-by: Ben Ye <benye@amazon.com>

* changelog

Signed-off-by: Ben Ye <benye@amazon.com>

---------

Signed-off-by: Ben Ye <benye@amazon.com>

* Change order of log for finished compacting blocks (#6966)

Signed-off-by: Daniel Deluiggi <ddeluigg@amazon.com>

* Update MAINTAINERS.md (#6967)

Signed-off-by: Matej Gera <38492574+matej-g@users.noreply.github.com>

* Add `keep_firing_for` support to alerting rule (#6943)

* Add keep_firing_for to alert rules

Signed-off-by: JHeil <jheil@coveo.com>

* Updating changelog

Signed-off-by: JHeil <jheil@coveo.com>

* Re-order keepFiringFor at the end of the struct

Signed-off-by: JHeil <jheil@coveo.com>

---------

Signed-off-by: JHeil <jheil@coveo.com>
Signed-off-by: JHeil <52839010+JHeilCoveo@users.noreply.github.com>

* Use registerer for gRPC opts and endpoints (#6971)

The EndpointSet and StoreGRPC options accept concrete *prometheus.Registry type.
This makes it hard to create multiple instances of them because they will try
to register the same metrics and the registry will panic.

This commit changes the type to a prometheus.Registerer which allows clients
to inject a prefixed registry and get metrics for each instance.

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>

* receive: upload compacted blocks if OOO enabled (#6974)

* Add streaming series limit at block series client (#6972)

* add series limit that is applied when streaming using block series client

Signed-off-by: Ben Ye <benye@amazon.com>

* changelog

Signed-off-by: Ben Ye <benye@amazon.com>

* add unit tests

Signed-off-by: Ben Ye <benye@amazon.com>

* address comments

Signed-off-by: Ben Ye <benye@amazon.com>

* fix comment

Signed-off-by: Ben Ye <benye@amazon.com>

---------

Signed-off-by: Ben Ye <benye@amazon.com>

* Updated the defaultChecked property in checkbox element (#6976)

Signed-off-by: lpreethvika-samsung <l.preethvika@samsung.com>

* Update prometheus to d0c2d9c (#6978)

* Update prometheus to d0c2d9c

This commit updates Prometheus to commit d0c2d9c which contains
the loser-tree based postings merge. Note that we cannot update to
latest main since there is currently a bug in PromQL when querying
native histograms introduced by https://github.com/prometheus/prometheus/pull/13276.
This issue should be fixed by https://github.com/prometheus/prometheus/pull/13289.

This commit also updates the Thanos PromQL engine to latest main.

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>

* Fix handler_test.go

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>

* Fix manager_test.go

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>

* Use empty registry for file discovery

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>

---------

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>

* Updates busybox SHA (#6982)

Signed-off-by: GitHub <noreply@github.com>
Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com>

* Updates busybox SHA (#6982)

Signed-off-by: GitHub <noreply@github.com>
Co-authored-by: fpetkovski <fpetkovski@users.noreply.github.com>

* VERSION: Bump version for release

Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>

* rule: native histogram support (#6390)

* Added native histogram support for ruler

Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>

Formatted imports

Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>

Fixed imports

Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>

Formated imports

Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>

* Fixed native histogram tests

Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>

Fixed receiver type

Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>

* Fix for rebase

Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>

* Added docs for query endpoints differences

Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>

* Fixed comments and naming

Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>

* made HTTPConfig optional

Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>

* made HTTPConfig optional

Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>

* Reverted and added check

Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>

* Fixes from comments

Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>

* renamed queryconfig to clientconfig

Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>

* common prepareEndpointSet

Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>

* fixed lint

Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>

* Fixed sidecar

Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>

* Fixed tests

Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>

---------

Signed-off-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>

* mdox ignore checking twitter urls (#7001)

Signed-off-by: Ben Ye <benye@amazon.com>

* Updated Grofers logo (#7006)

Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in>

* TraceID : Fetching TraceID (#6973)

* docs: add promcon 2023 thanos talks

Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>

* Added website page for companies who offer consultancy and enterprise… (#7000)

* Added website page for companies who offer consultancy and enterprise support for Thanos

Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in>

* adopters.yml revert

Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in>

* retrigger checks

Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in>

* added a new line in welcome.md

Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in>

* retrigger checks

Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in>

---------

Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in>

* Lazy downloaded index header (#6984)

* lazy downloaded index header

Signed-off-by: Ben Ye <benye@amazon.com>

* update tests

Signed-off-by: Ben Ye <benye@amazon.com>

* address comments

Signed-off-by: Ben Ye <benye@amazon.com>

* address comments

Signed-off-by: Ben Ye <benye@amazon.com>

* changelog

Signed-off-by: Ben Ye <benye@amazon.com>

---------

Signed-off-by: Ben Ye <benye@amazon.com>

* tests: use remote write in query frontend tests (#6998)

* query-frontend: Added support of auto_discovery for memcached (#7004)

* query-frontend: Added support of auto_discovery for memcached

Signed-off-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com>

* adjustments to build on main branch

Signed-off-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com>

* CHANGELOG.md

Signed-off-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com>

* typo fixed

Signed-off-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com>

* minor fixex after review

Signed-off-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com>

---------

Signed-off-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com>

* Fix URI encoding of strings

Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in>

* ui: enable partial response strategy by default

Rebuild

Signed-off-by: Vanshika <vanshikav928@gmail.com>

* Added negative offset check for caching queries (#7011)

Signed-off-by: pawarpranav83 <pawarpranav@gmail.com>
Co-authored-by: pawarpranav83 <pawarpranav@gmail.com>

* Update MAINTAINERS.md (Bartek's company)

Should be done in January last year...

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* store: add chunksize tests to acceptance tests

* add chunk size tests to acceptance tests
* refactor acceptance tests slightly

Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>

* added tasrie it services as support partner (#7023)

* added tasrie it services as support partner

Signed-off-by: Tasrie IT Services <info@tasrieit.com>

* added the link in the list.html page

Signed-off-by: Tasrie IT Services <info@tasrieit.com>

* fixed the logo resolution to 190x70

Signed-off-by: Tasrie IT Services <info@tasrieit.com>

* Remove extraneous text

Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com>

---------

Signed-off-by: Tasrie IT Services <info@tasrieit.com>
Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com>
Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com>

* Query Frontend: Add tenant label to metrics (#6887)

This commit adds a tenant label to the HTTP metrics which are exported
by the Query Frontend.

Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>

* tests: use remote write in query frontend tests (#7017)

* resolved issue #7024

Signed-off-by: Pratham Agarwal <agarwalpratham1812@gmail.com>

* QueryFrontend|Query: Create new arg to enable extended functions (#7028)

* Adding new parameter for extended functions in querier

Signed-off-by: Pedro Tanaka <pedro.stanaka@gmail.com>

* Adding new flag for QFE

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* improve argument passing in query side

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* Adding changelog

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* Adding e2e test for query

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* undoing uneeded changes

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* fixing docs

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* fixing e2e tests

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* Fixing backward compat test

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* changes from CR

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

---------

Signed-off-by: Pedro Tanaka <pedro.stanaka@gmail.com>
Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* receive: respect forward timeout in http handler buckets (#7030)

Forward timeout is what ultimately decides how long a HTTP request might
take so it doesn't make sense to hard-code max value of 5. In this pull
request I propose respecting the configured forward timeout and adding
extra buckets if the default buckets don't cover it completely.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* receive: do not leak grpc connections (#7031)

* receive: do not leak grpc connections

Prevent a leak in gRPC connections by garbage collecting old ones when
the hashring changes. For that purpose, I propose adding a `Nodes()
string` method so that it would be possible to know what nodes do not
exist in the hashring anymore.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* receive: change order of operations

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

---------

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* fix multi-tenancy link (#7032)

Signed-off-by: Alec Rajeev <ccarajeev342@gmail.com>

* removing todo comments from query docs

Signed-off-by: Harsh Pratap Singh <harshpratapsingh8210@gmail.com>

* Query: add optional tenancy enforcement (#6756)

* Query: add optional tenancy enforcement

With this commit it's now possible to enable enforcement of tenancy. If
tenancy is enabled, a tenant label will be added to queries based on the
tenant information provided by the tenant header, and the
tenant-label-name.

The implementation for query APIs are done by using prom-label-proxy as
library, while the implementation for non-query APIs are written from
scratch.

Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>

* Add changelog entry

Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>

* Query: Add non-default tenant testcase

Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>

* Test: make query a constant to make linter happy

Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>

* Address review comments

- Remove empty lines
- If multiple tenant matchers are found in the original query, we only
  replace the first one with the header provided tenant, and remove any
  subsequent ones.

Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>

* Address review comments

- `--enable-tenancy` -> `--enforce-tenancy`
- Create `RewritePromQL` and `RewriteLabelMatchers` to clean up code in
  query api. Also move getLabelMatchers to tenancy pkg.
- Use prom-label-proxys `EnforceMatchers` to rewrite labels on non-query
  APIs instead of own solution
- Don't specifically handle `illegalLabelMatcherError`

Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>

* Re-arrage go.mod to make linter happy.

Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>

* Address review comments

Minor changes to CLI docs, code-comments and changelog.

Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>

* Query: Add tenancy docs

This commit adds documentation for the tenancy features.

Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>

* Update docs/components/query.md

Review comment

Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com>
Signed-off-by: Jacob Baungård Hansen <jacobbaungard@redhat.com>

---------

Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>
Signed-off-by: Jacob Baungård Hansen <jacobbaungard@redhat.com>
Co-authored-by: Saswata Mukherjee <saswataminsta@yahoo.com>

* Tests: unique docker env name for tenant tests (#7038)

The e2e tests would occasionally fail due to non-unqiue docker environment
names. With this commit the tests are environments are given unique names
to avoid these failures.

Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>

* changed default 3m to 15m

Signed-off-by: Player256 <dattucodes@gmail.com>

* Fix docs (#7039)

Fix docs post #6539 merge.

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>

* Revert "receive: upload compacted blocks if OOO enabled (#6974)" (#7053)

This reverts commit 7b8eb86c0ff3b1144aa1cea392806afa40a4cdf8.

Proper way to handle this is to disable vertical compaction. I am trying
to add this functionality here:
https://github.com/prometheus/prometheus/pull/13393

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* [CHORE] adding thanos upload-snapshot command (#6884)

Signed-off-by: Nicolas Takashi <nicolas.tcs@hotmail.com>

* fix docs error

Signed-off-by: Kartikay <120778728+kartikaysaxena@users.noreply.github.com>

* cmd: make bucket upload command take lset from flags (#7059)

Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>

* CHANGELOG: mark 0.34 as in progress

Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>

* VERSION: cut release 0.34.0-rc.0

Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>

* UI: Don't always force tracing (#7062)

Forced tracing was.. Forced true always, even if the checkbox in the UI
to enable tracing was not actually checked.

Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>

* go.mod: update Prometheus version (#7047)

Update Prometheus version to include
https://github.com/prometheus/prometheus/pull/13242 which is important
for me - it unblocks further postings work.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* Make RetryError and HaltError able to be fetched for root cause (#7043)

* Make RetryError and HaltError able to be fetched for root cause

Signed-off-by: Alex Le <leqiyue@amazon.com>

* Added unit test

Signed-off-by: Alex Le <leqiyue@amazon.com>

* fix lint

Signed-off-by: Alex Le <leqiyue@amazon.com>

* fixed IsRetryError and IsHaltError functions

Signed-off-by: Alex Le <leqiyue@amazon.com>

---------

Signed-off-by: Alex Le <leqiyue@amazon.com>

* receive: disable overlapping compaction

Use the new TSDB flag to disable overlapping compaction to fix OOO
samples handling in the Receive component.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* CI: Ensure static react-app is checked in (#7063)

* CI: Ensure static react-app is checked in

With this commit the CI system should fail if changes to the react-app
has been made without checking in the changes.

Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>

* Add `react-app` as dependency `check-react-app`

To ensure the react-app is rebuilt before checking for changes.

Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>

---------

Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>

* Receive: refactor handler for improved readability and organization (#6898)

* [wip] First checkpoint

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* [wip] Second checkpoint

All tests passing, unit and e2e.

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Small random refactors

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add some useful trace tags

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Concurrent and traced local writes

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Improve variable names in remote writes

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Rename `newFanoutForward` function

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* More refactors

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix linting issue

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add a quorum test with sloppy quorum

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* [wip] Try to make retries work

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* [wip] Checkpoint: wait group still hanging

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Some refactors

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add some commented code so I don't lose it

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Adapt tests

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove sloppy quorum code

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Move some code around

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove even more leftover of sloppy quorum

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Extract a type to hold function params

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove unused struct field

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove useless variable

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove type that wasn't used enough

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Delete function to tighten up max buffered responses

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add comments to some functions

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix peer up check

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix size of replication tracking slices

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Rename context

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Don't do local writes concurrently

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove extra error logging

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix syntax after merge

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add missing methods to peersContainer

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix handler test

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Reset peers state on hashring changes

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Handle PR comment regarding waitgroup

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Set span tags to help debug

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix concurrency issue

We close the request as soon as quorum is reached and leave a few Go routines running to finish replication and so cleanups.

This means that the context from the HTTP request is cancelled... which ends up also cancelling the pending replication requests.

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix request ID middleware

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix `distributeTimeseriesToReplicas` comment

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Extract var with 1-indexed replication index

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Rename methods in peersContainer interface

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Make peerGroup `getConnection` check if peers are up

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove yet one more not useful log

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove logger from `h.sendWrites`

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

---------

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Upgrade grpc to 1.57.2 (#7078)

1、In the replace of go.mod, due to weaveworks/common#239, The grpc version is 1.45.0, but there are vulnerabilities in this version. In order to fix CVE-2023-44478, the grpc version needs to be upgraded to 1.57.2
2、In order to upgrade GRPC, the version of weaveworks/common also needs to be upgraded, otherwise the build will fail

Signed-off-by: hanyuting8 <hytxidian@163.com>

* Store: acceptance test for proxy store (#7084)

* Add basic acceptance tests for proxy store
* Fix bug where invalid requests got ignored because of partial response
  strategy

Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>

* Fix lazy postings with zero length (#7083)

* fix lazy postings with zero length

Signed-off-by: Ben Ye <benye@amazon.com>

* changelog

Signed-off-by: Ben Ye <benye@amazon.com>

* unit tests

Signed-off-by: Ben Ye <benye@amazon.com>

* fix doc

Signed-off-by: Ben Ye <benye@amazon.com>

---------

Signed-off-by: Ben Ye <benye@amazon.com>

* Store: fix label values edge case (#7082)

If the requested label is an external label and we have series matchers
we should only return results if the series matchers actually match a
series.

Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>

* VERSION: cut release 0.34.0-rc.1

Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>

* receive: race condition in handler Close() when stopped early (#7087)

Receiver hangs waiting for the HTTP Hander to shutdown if an error occurs
before Handler is initialized. This might happen, for example, if the hashring
is too small for a given replication factor.

Signed-off-by: Mikhail Nozdrachev <mikhail.nozdrachev@aiven.io>

* receive: use async remote writing (#7045)

* CHANGELOG: cut release 0.34 (#7095)

Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>

* Stores: convert tests to not rely on slice labels (#7098)

Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>

* all: get rid of query pushdown to simplify query path (#7014)

Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>

* Store: dont rely on slice labels continued

Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>

* Update prometheus/prometheus (#7096)

* Update prometheus/prometheus

This commit updates prometheus/prometheus to latest main (60b6266e).

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>

* Fix file discovery

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>

---------

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>

* receive/handler: fix locking twice (#7112)

Fix bug introduced in https://github.com/thanos-io/thanos/pull/6898: we
were RLock()ing twice. This leads to a deadlock in some situations.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* fix minio store gateway err (#7114)

Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in>

* receive/handler: do not double lock (#7124)

markPeerUnavailable was always taking a lock and in one case we were
calling it with a lock already taken. Fix this.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* default to alertmanager v2 api (#7123)

Signed-off-by: Jake Keeys <jake@keeys.org>

* Receive: dont rely on slice labels (#7100)

Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>

* Upgrade grpc to 1.57.2 (#7078)

1、In the replace of go.mod, due to weaveworks/common#239, The grpc version is 1.45.0, but there are vulnerabilities in this version. In order to fix CVE-2023-44478, the grpc version needs to be upgraded to 1.57.2
2、In order to upgrade GRPC, the version of weaveworks/common also needs to be upgraded, otherwise the build will fail

Signed-off-by: hanyuting8 <hytxidian@163.com>
Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>

* VERSION: cut release 0.34.1

Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>

* docs: update helm installation instruction

the prometheus helm chart is a community maintained chart since a few
years. With that, the old example pointed to an old chart and the
provided example values aren't also working anymore.

This update the documentation.

Signed-off-by: Mario Constanti <github@constanti.de>

* docs: fix link (#7129)

The link has moved to another since Cisco bought Banzai Cloud.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* docs: run make docs for helm installation instruction

Signed-off-by: Mario Constanti <github@constanti.de>

* Fixing log line for remote engine in debug mode (#7133)

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* Adding new method on BucketedBytes to expose used memory (#7137)

* Adding new method on bucketed bytes to expose used

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* Removing interface, using RWMutex

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

---------

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* Adding InfraCloud as Enterprise support partner (#7141)

* adding InfraCloud as Enterprise support partner

Signed-off-by: Chetan Deshmukh <cdeshmukh@infracloud.io>

* replaced svg file to match layout

Signed-off-by: Chetan Deshmukh <cdeshmukh@infracloud.io>

* added alt-text and horizontal image

Signed-off-by: Chetan Deshmukh <cdeshmukh@infracloud.io>

---------

Signed-off-by: Chetan Deshmukh <cdeshmukh@infracloud.io>

* docs: fix link (#7129)

The link has moved to another since Cisco bought Banzai Cloud.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>

* cache: attach object storage hash to iter key (#6880)

Attach object storage hash to the iter key so that it would be possible
to reuse the same cache storage e.g. Redis for different buckets.
Without this, the results are funny to say the least if you accidentally
attempt to do that. Thus, let's add the hash to reduce the possibility
of an accident for our users.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* Add support for extended promql functions in rule (#7105)

Adds a flag to register the extended promql functions supported by the thanos
query engine when running the rule component.  This will allow rule config
files containing query expressions with (xrate / xincrease / xdelta) to pass
validation.  This will only work if the query endpoint in use is running the
thanos engine.

Signed-off-by: Samuel Dufel <samuel.dufel@shopify.com>

* Extended func support - doc update (#7161)

* Add support for extended promql functions in rule

Adds a flag to register the extended promql functions supported by the thanos
query engine when running the rule component.  This will allow rule config
files containing query expressions with (xrate / xincrease / xdelta) to pass
validation.  This will only work if the query endpoint in use is running the
thanos engine.

Signed-off-by: Samuel Dufel <samuel.dufel@shopify.com>

* Update rendered docs with added flag

Signed-off-by: Samuel Dufel <samuel.dufel@shopify.com>

---------

Signed-off-by: Samuel Dufel <samuel.dufel@shopify.com>

* Copy labels from remote instant queries (#7151)

Similar to https://github.com/thanos-io/thanos/pull/6957, we should copy
labels from remote instant queries so that memory does not get overwritten
when processing series in a central engine.

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>

* e2e/query_frontend: add tests for explain/analyze (#7160)

Adding tests for explain/analyze with QFE. Will add fixes as separate
PR.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* cache/caching_bucket: add path to hash (#7158)

Add path to the hash. This allows identifying difference instances by
different config paths.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* Allow using different listing strategies (#7134)

* Allow using different listing strategies

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>

* Expose flags for block list strategy

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>

* Run make docs

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>

* Fix whitespace

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>

* Add CHANGELOG entry

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>

---------

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>

* cache: implement the circuit breaker pattern for asynchronous set operations in the cache client (#7010)

* Implement the circuit breaker pattern for asynchronous set operations in the cache client

Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com>

* Add feature flag for circuitbreaker

Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com>

* Sync docs

Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com>

* Skip configuration validation if the circuit breaker is disabled

Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com>

* Make lint happy

Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com>

* Abstract the logic of the circuit breaker

Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com>

---------

Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com>

* queryfrontend: fix analysis after API changes

Fix the analysis functionality with query-frontend after the recent
changes. Added tests for this.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* bugfix: lazy posting optimization with wrong cardinality for estimation (#7122)

* bugfix: catch lazy posting optimization using wrong cardinality for estimation

Signed-off-by: Ben Ye <benye@amazon.com>

* update changelog

Signed-off-by: Ben Ye <benye@amazon.com>

---------

Signed-off-by: Ben Ye <benye@amazon.com>

* *: properly treat native histogram deduplication in chunk series merger

We have detected a problem in the chunk seriers merger where it will
panic in case it encounters native histogram chunks.
I am using thanos as a library for a project and wanted to use the
penalty function to dedup blocks from Prometheus instances.

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* Query UI: Add tenant box (#6867)

* Query UI: Add tenant box

With this commit as tenant box is added to the query UI. It can be used
to specify which tenant to use when making a query.

Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>

* Re-compile static react app

Recompiles the static react app as now needed following:
https://github.com/thanos-io/thanos/pull/6900

Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>

* Move changelog item to appropiate future release

After merging it was under the 0.34 release.

Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>

* Move query path tenancy proposal to done

Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>

---------

Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>

* receive: add support for globbing tenant specifiers (#7155)

We want to be able to route all tenants which begin with certain letters
to some receivers so we need to have some kind of globbing/regex support
in the hashring. This PR adds that functionality. We've been using this
in prod successfully.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* .circleci: bump setup_docker_version version (#7177)

The current image is deprecated. See
https://discuss.circleci.com/t/remote-docker-image-deprecations-and-eol-for-2024/50176.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* Unhide distributed execution mode

This commit exposes the distributed query execution mode to end-users by unhiding the
flag used to toggle this feature.

The commit also adds documentation on when the mode is appropriate to be used.

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>

* Add changelog entry

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>

* cortex/querier: fix analysis merging (#7179)

We were not merging analysis properly - mergo was overwriting data.
Instead of using a whole library for this, just write two small
functions and use them. Add test to cover this.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* Fix docs

Fixes docs formatting and updates the distributed execution link to the done proposal.

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>

* Added platformengineers

Signed-off-by: Munir Khakhi <munir@improwised.com>

* Added platformengineers (#7181)

Signed-off-by: Munir Khakhi <munir@improwised.com>

* Bump google.golang.org/protobuf to v1.33.0 (#7191)

This PR bumps the version of google.golang.org/protobuf to v1.33.0 fix a
potential vulnerability in the protojson.Unmarhsl function [1] that can
occure when unmarshaling a message with a protobuf value.

Even if the function isn't used directly in Thanos it would be safer to
just bump it directly.

[1] https://pkg.go.dev/vuln/GO-2024-2611

Signed-off-by: Daniel Mellado <dmellado@redhat.com>

* fix: make responsive support page and change size of platform-engineer logo

fix: add anchor tag to all images
Signed-off-by: Payal17122000 <raviyapayal17@gmail.com>

* downsample: retry objstore related errors

Signed-off-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com>

* downsample: retry objstore related errors

Signed-off-by: Vasiliy Rumyantsev <4119114+xBazilio@users.noreply.github.com>

* rule: do not turn off if resolving fails (#7192)

Do not turn off Ruler if resolving fails. We can still (try to) evaluate
rules even if Alertmanager is not available.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* Query UI: Only show tenant box with enforcement on (#7186)

With this commit we only show the tenant-ui box when enforcement of
tenancy is on, as it is not needed otherwise.

Signed-off-by: Jacob Baungard Hansen <jacobbaungard@redhat.com>

* Reloader: Add support for watching and decompressing Prometheus configuration directories (#7199)

Signed-off-by: Daniel Hrabovcak <thespiritxiii@gmail.com>

* [FEAT] Adding blog post (#7202)

Signed-off-by: Helia Barroso <helia.barroso@hotmail.com>
Co-authored-by: Helia Barroso <helia.barroso@hotmail.com>

* storepb: make ServerAsClient channels unbuffered

Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>

* Add support for TSDB selector in querier (#7200)

* Add support for TSDB selector in querier

This PR allows using the query distributed mode against a set of multi-tenant receivers
as described in https://github.com/thanos-io/thanos/blob/main/docs/proposals-done/202301-distributed-query-execution.md#distributed-execution-against-receive-components.

The feature is enabled by a selector.relabel-config flag in the Query component
which allows it to select a subset of TSDBs to query based on their external labels.

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>

* Add CHANGELOG entry and fix docs

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>

* Fix tests

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>

* Add comments

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>

* Add test case for MatchersForLabelSets

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>

* Fix failing test

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>

* Use an unbuffered channel

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>

* Change flag description

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>

* Remove parameter from ServerAsClient

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>

---------

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>

* Update thanos-io/promql-engine (#7215)

* Update thanos-io/promql-engine

This commit updates the promql-engine module to latest main and modifies
to remote engine based on the breaking change.

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>

* Fix lint

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>

---------

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>

* expose NewPromSeriesSet (#7214)

Signed-off-by: Ben Ye <benye@amazon.com>

* [CHORE] adding auto GOMEMLIMIT flag (#7223)

Co-authored-by: Filip Petkovski <filip.petkovsky@gmail.com>

* bump objstore package version to latest main (#7228)

Signed-off-by: Ben Ye <benye@amazon.com>

* Query-frontend: fix missing redis username config (#7224)

* add username cfg to rueidis client

Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>

* update changelog

Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>

---------

Signed-off-by: Thibault Mange <22740367+thibaultmg@users.noreply.github.com>

* UI: Showing Block Size Stats (#7233)

* feat(ui): added BlockSizeStats calculation to blocks page

A block can have a list of contained files set in `.thanos.files`.
If the `files` array is set, all referenced files with `size_bytes` set are counted:
- sum of all `chunk/*` file sizes
- size of index file
- total size (sum of both)

Shows statistics about the selected block in the block details view:
- Total size of block
- Size of index (and percentage of total)
- Size of all chunks (and percentage of total)
- Daily growth, based on total size and block duration

Output is humanized up to Pebibytes and fixed to two decimal places;
raw bytes are accessible through mouse over / title text.

Signed-off-by: Markus Möslinger <markus.moeslinger@socra.dev>

* feat(ui): added aggregated BlockSizeStats to blocks row title

Added total size of all blocks from a source to the row title, beneath the source name.

The shown total size is humanized up to pebibytes and fixed to two decimal places;
raw bytes value is accessible through mouse over / title text.

The shown value will refresh with selected compaction levels, but doesn't take block filter into account.

I thought about showing daily growth as well, but just summing all milliseconds of all blocks doesn't work with overlapping blocks / multiple resolutions.

Signed-off-by: Markus Möslinger <markus.moeslinger@socra.dev>

* chore(docs): added UI block size PR to CHANGELOG.md

Signed-off-by: Markus Möslinger <markus.moeslinger@socra.dev>

* chore(ui): removed comments

Automatic code formatting duplicated some comments near import statements.

Signed-off-by: Markus Möslinger <markus.moeslinger@socra.dev>

---------

Signed-off-by: Markus Möslinger <markus.moeslinger@socra.dev>

* Fix lazy expanded postings cache and bug of non equal matcher (#7220)

* fix lazy expanded postings cache and bug of non equal matcher with non existent values

Signed-off-by: Ben Ye <benye@amazon.com>

* test case for remove keys noop

Signed-off-by: Ben Ye <benye@amazon.com>

* add promqlsmith fuzz test

Signed-off-by: Ben Ye <benye@amazon.com>

* update

Signed-off-by: Ben Ye <benye@amazon.com>

* changelog

Signed-off-by: Ben Ye <benye@amazon.com>

* fix go mod

Signed-off-by: Ben Ye <benye@amazon.com>

* rename test

Signed-off-by: Ben Ye <benye@amazon.com>

* fix series request timestamp

Signed-off-by: Ben Ye <benye@amazon.com>

* skip e2e test

Signed-off-by: Ben Ye <benye@amazon.com>

* handle non lazy expanded case

Signed-off-by: Ben Ye <benye@amazon.com>

* update comment

Signed-off-by: Ben Ye <benye@amazon.com>

---------

Signed-off-by: Ben Ye <benye@amazon.com>

* Bump Prometheus to include new label regex optimization (#7232)

* bump Prometheus version to include new label matcher regex value optimization

Signed-off-by: Ben Ye <benye@amazon.com>

* update

Signed-off-by: Ben Ye <benye@amazon.com>

* fix again

Signed-off-by: Ben Ye <benye@amazon.com>

* include latest fix

Signed-off-by: Ben Ye <benye@amazon.com>

* update go mod

Signed-off-by: Ben Ye <benye@amazon.com>

* fix explain test

Signed-off-by: Ben Ye <benye@amazon.com>

* fix test again

Signed-off-by: Ben Ye <benye@amazon.com>

* update again

Signed-off-by: Ben Ye <benye@amazon.com>

* update

Signed-off-by: Ben Ye <benye@amazon.com>

* fix tests so far

Signed-off-by: Ben Ye <benye@amazon.com>

* fix compactor tests

Signed-off-by: Ben Ye <benye@amazon.com>

* use own out of order chunk index

Signed-off-by: Ben Ye <benye@amazon.com>

---------

Signed-off-by: Ben Ye <benye@amazon.com>

* update docs for receive routing only with limits (#7241)

Signed-off-by: Alec Rajeev <13004609+alecrajeev@users.noreply.github.com>

* docs: add thanoscon 2024 talks (#7243)

Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>

* remove write method from Compactor interface (#7246)

Signed-off-by: Ben Ye <benye@amazon.com>

* change shipper to not overwrite all external labels (#7247)

Signed-off-by: Ben Ye <benye@amazon.com>

* fix(changelog): fix GOMEMLIMIT pull request reference

Signed-off-by: roth-wine <philipp.roth@hetzner.com>

* Update kakkoyun's affiliation (#7251)

* Added Shield in adopters (#7254)

* Added Shield in adopters

Signed-off-by: suhas.chikkanna.shield <suhas.chikkanna@shield.com>

* Upload compatible image 

Signed-off-by: suhas-chikkanna <162577490+suhas-chikkanna@users.noreply.github.com>

---------

Signed-off-by: suhas.chikkanna.shield <suhas.chikkanna@shield.com>
Signed-off-by: suhas-chikkanna <162577490+suhas-chikkanna@users.noreply.github.com>
Co-authored-by: suhas.chikkanna.shield <suhas.chikkanna@shield.com>

* Tracing: added missing sampler types (#7231)

* added missing sampler types

Signed-off-by: Neeraj Nagure <nagureneeraj@gmail.com>

* added changelog entry

Signed-off-by: Neeraj Nagure <nagureneeraj@gmail.com>

* fixed changelog entry

Signed-off-by: Neeraj Nagure <nagureneeraj@gmail.com>

* Fixed changelog entry conflict

Signed-off-by: Neeraj Nagure <nagureneeraj@gmail.com>

---------

Signed-off-by: Neeraj Nagure <nagureneeraj@gmail.com>

* fix query_test when --race enabled (#7258)

Signed-off-by: Yi Jin <yi.jin@databricks.com>

* go.mod: bump promql-engine (#7263)

Bump promql-engine version to include samples counting.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* Fix 7244 error targets page (#7245)

* added UNKNOWN to TargetHealth_value at targets proto

Signed-off-by: Tidhar Klein Orbach <tizkiko@gmail.com>

* added TargetHealth_value UNKNOWN to rpc.pb.go

Signed-off-by: Tidhar Klein Orbach <tizkiko@gmail.com>

---------

Signed-off-by: Tidhar Klein Orbach <tizkiko@gmail.com>

* clarify documentation around selecor.relabel-config option

Signed-off-by: notafile <nota@notafile.com>

* clarify writing around sharding

Signed-off-by: notafile <nota@notafile.com>

* don't halt compaction due to overlapping sources when vertical compaction is enabled (#7225)

Signed-off-by: Ben Ye <benye@amazon.com>

* Propagate the query plan

* Serialize the plan for remote executions

latest engine

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

Propagate marshaled plan and introduce optimizer

Propagating the query plan in the remote engine requests and introduce new SetProjectionColumns optimizer

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* Fixing passing down of plan

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* go mod tidy

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* avoid panics

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* delete dev file

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* undo small refactor

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* improve test

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

---------

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

generating protos

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

fixing v1

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

delete unused method

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

Set projection labels after distributing queries

* removing second precision engine, upstream already truncates

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* Passing the plan along as the query in remote executions

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* Using proper constructors passing the query plan

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* Refactor query creation from plan

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* fallback in case we cant use plan

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* refactor, add tests

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* Refactor to method

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* last nits

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* api/ui: show peak/total samples in analyze (#7269)

Show the new peak/total fields in analyze output next to each operator.
Add tooltips to explain what is the meaning of each field.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* receive/handler: implement tenant label splitting (#7256)

* receive/handler: implement tenant label splitting

Implement splitting incoming HTTP requests along some label inside of
the timeseries themselves. This functionality is useful when you have
one big application exposing lots of series and, for instance, you have
a label `team` that identifies different owners of metrics in that
application. Then using this you can use that `team` label to have
different tenants in Thanos.

Only negative thing that I could spot is that if after splitting one of
the requests fails then that code is used for all tenants and that skews
the Receiver metrics a little bit. I think that can be left as a TODO
task.

Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com>

* test/e2e: add more receiver tests

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* thanos/receive: note that splitting takes precendence over HTTP

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* thanos/receive: fix typo

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

---------

Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com>
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* query: fixing dedup iterator when working on mixed sample types (#7271)

* query: fixing dedup iterator when working on mixed sample types

There was a panic in case the dedupiterator worked on two chunks with both Native Histograms and Float (XOR encoded).

Co-authored-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>
Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* Adding changelog

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* fixing lint

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* removing comments

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* Fixing repro test case

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* fixing initialization

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* fixing changelog

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* adding header to new file

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* using t.Run

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

* fixing ordering of samples in tests

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>

---------

Signed-off-by: Pedro Tanaka <pedro.tanaka@shopify.com>
Co-authored-by: Sebastian Rabenhorst <sebastian.rabenhorst@shopify.com>

* change the reflect package to an unsafe package (#7143)

- as 'reflect.String.Header' is deprecated, it is replaced with an unsafe package.

Signed-off-by: Youngjun <yj.yoo@okestro.com>

* Receive: fix issue-7248 with parallel receive_forward (#7267)

* Receive: fix issue-7248 by introducing a worker pool

Signed-off-by: Yi Jin <yi.jin@databricks.com>

* fix unit test bug

Signed-off-by: Yi Jin <yi.jin@databricks.com>

* fix CLI flags not pass into the receive handler

Signed-off-by: Yi Jin <yi.jin@databricks.com>

* address comments

Signed-off-by: Yi Jin <yi.jin@databricks.com>

* init context in constructor

Signed-off-by: Yi Jin <yi.jin@databricks.com>

---------

Signed-off-by: Yi Jin <yi.jin@databricks.com>

* Fix incorrect comments (#7268)

Signed-off-by: Magiceses <magiceses0118@gmail.com>

* [CHORE] adding user agent (#7281)

Signed-off-by: Nicolas Takashi <nicolas.tcs@hotmail.com>

* receive/multitsdb: do not delete not uploaded blocks (#7166)

* receive/multitsdb: do not delete not uploaded blocks

If a block hasn't been uploaded yet then tell the TSDB layer not to
delete them. This prevents a nasty race where the TSDB layer can delete
a block before the shipper gets to it. I saw this happen with a very
small block.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* receive/multitsdb: change order

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* shipper/receive: just use a single lock

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

---------

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

* Compact: Replace group with resolution in ownsample metrics (#7283)

Compaction dowsnample metrics have too high a cardinality, causing metric
bloat on large installations. The group information is better suited to logs.
* Replace with a resolution label to reduce cardinality.

Fixes: #5841

Signed-off-by: SuperQ <superq@gmail.com>

* Propagate warnings from instant queries

Warnings from remote instant queries get turned into errors, which
is a bug. It should be up to the root client to decide whether warnings
should be show as such, or converted to errors.

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>

* Add CHANGELOG entry

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>

* Receive: stop relying on grpc server config to set grpc client secure/skipVerify

Signed-off-by: Guillaume Lecerf <guillaume.lecerf@iziwork.com>

* Show warnings in query frontend (#7289)

* Show warnings in query frontend

QFE currently does not parse warnings from downstream queriers.
This commit fixes that by adding the field to proto messages and
modifies the merge function to take warnings into account.

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>

* Add CHANGELOG entry

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>

* Omit empty warnings

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>

---------

Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>

* Optimize empty posting check in lazy posting (#7298)

* change lazy postings empty posting check to use cardinality

Signed-off-by: Ben Ye <benye@amazon.com>

* update lazy posting test

Signed-off-by: Ben Ye <benye@amazon.com>

---------

Signed-off-by: Ben Ye <benye@amazon.com>

* Sidecar: mark as unqueryable if prometheus is down (#7297)

If the prometheus that belongs to a sidecar is down we dont need to
query the sidecar. This PR makes it so that we take the sidecar out of
the endpoint set then. We do the same for all other store APIs by
retuning an error in the info/Info gRPC call if they are marked as not
ready.

Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>

* [CHORE] considering X-Forwarded-For on HTTP Logging (#7303)

Signed-off-by: Nicolas Takashi <nicolas.tcs@hotmail.com>

* Query|Receiver: Do not log full request on ProxyStore by default (#7305)

* Query|Receiver|Store: Do not log full request on ProxyStore by default

We had a problem on our production where a sudden increase in requests with long matchers was putting our receivers under a lot of pressure.
Upon checking profiles we saw that the problem was calls to Log()

S…
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants