Support histograms in pkg/storage and update other breakages #4354

codesome · 2023-03-02T14:34:14Z

What this PR does

Main aim was to sync pkg/storage from sparsehistogram branch. Since it changes some function signature, I pulled in those changes from sparsehistogram branch as well.

Checklist

Tests updated
Documentation added
CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

pkg/storage/series/series_set.go

pkg/util/chunkcompat/compat.go

pkg/querier/matrix.go

pkg/storage/series/series_set.go

codesome · 2023-03-06T07:39:45Z

pkg/storage/series/series_set.go

+	if int64(c.series.samples[c.curFloat+1].Timestamp) < c.series.histograms[c.curHisto+1].Timestamp {
+		c.curFloat++
+		c.atHisto = false
+		return chunkenc.ValFloat
+	}
+	c.curHisto++
+	c.atHisto = true
+	return c.atTypeHisto()


IIUC, we are assuming that the timestamps will never clash between float and histogram, right? Is there any case where we can get clashing timestamps here?

Seems very unlikely. We merge data when reading from store-gateways, ingesters or when doing sharded query. Store gateways would load the same data, so practically no chance of clash. Ingesters store replicated data so should not clash either. Sharding will read promql results and merge them. That should produce distinct sets.

In any case, it's not like we lose the data, we'll return both values.

pkg/storage/series/series_set.go

pkg/util/modelutil/merger.go

pkg/util/modelutil/merger_test.go

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

krajorama

LGTM

* Helm: nginx HPA and tests kubeversion fixes (#4299) * Helm: fix Kubernetes override for nginx HPA The template did not take into account the override "kubeVersionOverride". Fix by using the mimir template implemented for this reason. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> * Helm: fix missing Kubernetes version overrides in tests The golden record tests need a fixed version because helm uses the version of the default context and can produce different results between contributor's machine and the CI environment. Add logic to test build to inject the minimal version if not found in the values file. Mainly because we cannot have a version override in the small and large values files. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> Co-authored-by: Jon Kartago Lamida <lamidaj@gmail.com> * Ruler: load more tenants in parallel during startup (#4258) * Ruler: load more tenants in parallel during startup * add more tests * fix lint * Apply suggestions from code review Co-authored-by: Marco Pracucci <marco@pracucci.com> * Ingester: fix OOO blocks labelling (#4297) * Ingester: fix OOO blocks labelling This fixes a bug where the OutOfOrderExternalLabel was being added to all blocks instead of the ones coming from OOO data, when the feature flag was enabled. * Changelog * PR number to changelog * Update previous changelog entry instead * Ruler: load more tenants in parallel during startup * fix context * improve unittest --------- Co-authored-by: Marco Pracucci <marco@pracucci.com> Co-authored-by: Nicolás Pazos <32206519+npazosmendez@users.noreply.github.com> * Change language to match the math. (#4356) * Upgrade mimir-prometheus to get a fast regexp path optimization (#4357) * Upgrade mimir-prometheus to get a fast regexp path optimization Signed-off-by: Marco Pracucci <marco@pracucci.com> * Added CHANGELOG entry Signed-off-by: Marco Pracucci <marco@pracucci.com> --------- Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fix typo in the docs URL for migrating from Cortex (#4358) * Remove forced paragraph break. (#4359) * Bump actions/setup-go to v3 to resolve Node.js 12 deprecation warning. (#4361) * Improve flaky `TestIngesterWithShippingDisabledDeletesBlocksOnlyAfterRetentionExpires` (#4362) * Use more specific assertion to include more information in test failures. See #4198. * Reduce flakiness of test by extending retention period. This gives the rest of the test more time to retrieve `oldBlocks` before any of the blocks is removed. * Add asynchronous validation scaffolding for block upload (#3411) * Add asynchronous validation scaffolding for block upload * addressed lint errors * Update pkg/compactor/block_upload.go Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com> * Update pkg/compactor/block_upload.go Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com> * Update pkg/compactor/block_upload.go Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com> * Update pkg/compactor/block_upload.go Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com> * Update pkg/compactor/block_upload.go Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com> * Update pkg/compactor/block_upload.go Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com> * enable block upload for dev testing * fixed validation errors, added debug log messages * fixed cancelled context issue * changed name of flag to disable complete block upload * addressed reviewer feedback * addressed reviewer feedback * Address some review comments, WIP * Small spacing cleanup * Transition to in-memory bucket for block finish test * Async validation test coordination, adding configuration flags * Small comment and flag fix * Swap config strategy, test still needs separation * Docs + lint * Review comments, begin separating tests * Finish validateAndComplete test * Update docs * Remove docker compose arguments * Regenerate rather than modify * Review, add test for periodicValidationUpdater * Make validateAndComplete test clearer, add upload meta check * Set missing cancelContext * Add sleep as suggested * Configure data directory * fixed compactor data dir in e2e test * Add changelog entry * Split into two entries * Missing entry number * Update CHANGELOG.md --------- Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com> Co-authored-by: Andy Asp <andy.asp@grafana.com> Co-authored-by: Andy Asp <90626759+andyasp@users.noreply.github.com> * Jsonnet: honor the minimum shard size configured (#4363) Signed-off-by: Marco Pracucci <marco@pracucci.com> * [CHANGE] Ruler: set default `evaluation-delay-duration` to 1m (#4250) * change the default evaluation delay of ruler to 1m * revert the changes in ruler test * change integration test to set ruler default value * fix integration tests * Update integration/configs.go Co-authored-by: Marco Pracucci <marco@pracucci.com> * Update CHANGELOG.md Co-authored-by: Peter Štibraný <pstibrany@gmail.com> --------- Co-authored-by: Marco Pracucci <marco@pracucci.com> Co-authored-by: Peter Štibraný <pstibrany@gmail.com> * [Chore] Update jsonnet manifest create query frontend discovery only when it is necessary (#4353) * [Chore] update jsonnet manifest, avoid setting querier.frontend-address or create query-frontend-discovery when deployement mode is microserivces or query-scheduler is enabled * linter and changelog * Helm: fix parity with jsonnet on query frontend headless service Do not generate query-frontend-headless service if query scheduler is enabled Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> * Apply suggestions from code review Co-authored-by: Marco Pracucci <marco@pracucci.com> * correct changelog * regenerate helm golden files * Update CHANGELOG.md --------- Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> Co-authored-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> Co-authored-by: Marco Pracucci <marco@pracucci.com> * Remove block validation mimirtool changelog entry (#4369) * Spread TSDB head compaction over the configured interval (#4364) * Spread TSDB head compaction over the configured interval Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fixed unit test Signed-off-by: Marco Pracucci <marco@pracucci.com> * Apply suggestion from code review Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fix typo in CHANGELOG entry Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fix typo in CHANGELOG entry Signed-off-by: Marco Pracucci <marco@pracucci.com> --------- Signed-off-by: Marco Pracucci <marco@pracucci.com> * Fix port number values. (#4368) * Ruler: change deployment max surge and max unavailable to reduce ownership spillover (#4381) * Ruler: change deployment max surge and max unavailable to reduce ownership spillover Signed-off-by: Marco Pracucci <marco@pracucci.com> * Apply suggestions from code review Co-authored-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com> --------- Signed-off-by: Marco Pracucci <marco@pracucci.com> Co-authored-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com> * Move "Note:" about cross-zone costs to "Costs" (#4370) This note was in an unrelated section. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * Change default -blocks-storage.tsdb.retention-period from 24h to 13h (#4382) Signed-off-by: Marco Pracucci <marco@pracucci.com> * Support histograms in pkg/storage and update other breakages (#4354) * Support histograms in pkg/storage and update other breakages Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> --------- Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com> Signed-off-by: Marco Pracucci <marco@pracucci.com> Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com> Co-authored-by: Jon Kartago Lamida <lamidaj@gmail.com> Co-authored-by: ying-jeanne <74549700+ying-jeanne@users.noreply.github.com> Co-authored-by: Marco Pracucci <marco@pracucci.com> Co-authored-by: Nicolás Pazos <32206519+npazosmendez@users.noreply.github.com> Co-authored-by: Ursula Kallio <ursula.kallio@grafana.com> Co-authored-by: l3ioo <122443155+l3ioo@users.noreply.github.com> Co-authored-by: Charles Korn <charleskorn@users.noreply.github.com> Co-authored-by: Vernon Miller <96601789+aldernero@users.noreply.github.com> Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com> Co-authored-by: Andy Asp <andy.asp@grafana.com> Co-authored-by: Andy Asp <90626759+andyasp@users.noreply.github.com> Co-authored-by: Peter Štibraný <pstibrany@gmail.com> Co-authored-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com> Co-authored-by: Oleg Zaytsev <mail@olegzaytsev.com> Co-authored-by: Ganesh Vernekar <ganeshvern@gmail.com>

Support histograms in pkg/storage and update other breakages

88dc435

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

codesome requested a review from a team as a code owner March 2, 2023 14:34

codesome requested a review from krajorama March 2, 2023 14:34

This was referenced Mar 6, 2023

Update distributor and compactor tests for native histograms #4352

Merged

Support histograms in pkg/querier #4376

Merged

Support histograms in pkg/frontend #4377

Merged

krajorama reviewed Mar 6, 2023

View reviewed changes

pkg/storage/series/series_set.go Outdated Show resolved Hide resolved

krajorama reviewed Mar 6, 2023

View reviewed changes

pkg/util/chunkcompat/compat.go Outdated Show resolved Hide resolved

codesome commented Mar 6, 2023

View reviewed changes

codesome added 2 commits March 6, 2023 14:36

Fix Krajo's comments

ffcc7f3

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

Fix Ganesh's comments

a0b26a9

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

codesome force-pushed the codesome/sync-storage branch from 2cd1c71 to a0b26a9 Compare March 6, 2023 09:35

krajorama approved these changes Mar 6, 2023

View reviewed changes

krajorama enabled auto-merge (squash) March 6, 2023 09:41

krajorama mentioned this pull request Mar 6, 2023

Prometheus Native Histogram support phase1 #3478

Closed

krajorama merged commit a033254 into main Mar 6, 2023

krajorama deleted the codesome/sync-storage branch March 6, 2023 09:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support histograms in pkg/storage and update other breakages #4354

Support histograms in pkg/storage and update other breakages #4354

codesome commented Mar 2, 2023 •

edited by krajorama

Loading

codesome Mar 6, 2023

krajorama Mar 6, 2023

krajorama left a comment

Support histograms in pkg/storage and update other breakages #4354

Support histograms in pkg/storage and update other breakages #4354

Conversation

codesome commented Mar 2, 2023 • edited by krajorama Loading

What this PR does

Checklist

codesome Mar 6, 2023

Choose a reason for hiding this comment

krajorama Mar 6, 2023

Choose a reason for hiding this comment

krajorama left a comment

Choose a reason for hiding this comment

codesome commented Mar 2, 2023 •

edited by krajorama

Loading