Skip to content

Commit

Permalink
Merge branch 'main' into sparsehistogram (#4027)
Browse files Browse the repository at this point in the history
* Update test

* Add missing changelog entries for commits since Mimir 2.5 (#4006)

All other commits weren't user-facing or were helm-chart specific.

See #3979

Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>

* Add --concurrency support to 'mimirtool rules sync' command (#3996)

* Add --concurrency support to 'mimirtool rules sync' command

Signed-off-by: Marco Pracucci <marco@pracucci.com>

* Update pkg/mimirtool/commands/rules.go

Co-authored-by: Patrick Oyarzun <patrick.oyarzun@grafana.com>

Signed-off-by: Marco Pracucci <marco@pracucci.com>
Co-authored-by: Patrick Oyarzun <patrick.oyarzun@grafana.com>

* store-gateway: ExpandedPostings shortcut: avoid LabelValues unless necessary (#3872)

* Return `Canceled` rather than `Aborted` when a `Series` request to a store-gateway is cancelled by the calling querier. (#4007)

* Return Canceled rather than Aborted when a Series request to a store-gateway is cancelled.

* Add changelog entry.

* Update mimir-prometheus, add support for align_evaluation_time_on_interval. (#4013)

Signed-off-by: Peter Štibraný <pstibrany@gmail.com>

* Fix title of guide in link text; reword phrase. (#4008)

* Fix ExampleInitLogger to work in UTC (#4016)

The test didn't pass in my time zone (tm).

--- FAIL: ExampleInitLogger (0.00s)
got:
ts=1970-01-01T01:00:00+01:00 caller=log_test.go:31 level=info test=1
ts=1970-01-01T01:00:00+01:00 caller=log_test.go:33 level=info msg="test 3"
want:
ts=1970-01-01T00:00:00Z caller=log_test.go:31 level=info test=1
ts=1970-01-01T00:00:00Z caller=log_test.go:33 level=info msg="test 3"

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>

Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>

* Create outline of Mimir 2.6 release notes (#4002)

Includes notable features and bugfixes based on the CHANGELOG. Helm changes
to be filled out later by product and engineering.

See #3979

Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>

* Fix post-merge comments from PR #4013. (#4014)

Signed-off-by: Peter Štibraný <pstibrany@gmail.com>

* Update CODEOWNERS to include mimir-ruler-and-alertmanager-maintainers (#4019)

For those who only want notifications re the ruler or Alertmanager.

* Remove internal use of store.max-query-length (#4017)

Make deprecation of the option more obvious and attempt to remove
any use of store.max-query-length in our documentation, jsonnet, helm,
and integration tests.

See #2793
See #3825

Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>

Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>

* [otlp] Update OTel Collector to latest release (#3852)

* [otlp] Update otel collector dependecy to latest
* Update code to deal with deprecated functions

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* [otlp] Docs: Highlight common issues with OTLP --> Prometheus (#3629)

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
Co-authored-by: Ursula Kallio <ursula.kallio@grafana.com>

* make it possible to inject memberlist kv codecs (#4018)

* make it possible to inject memberlist kv codecs

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* add comment

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* improve comment wording

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* Limits and errors for ephemeral storage (#4004)

* Add limits for ephemeral storage.
* Add new reason when ingestion of ephemeral metrics fails.
* Add tests for max ephemeral series limit.
* Introduce new discard reasons when ingesting ephemeral series.

Signed-off-by: Peter Štibraný <pstibrany@gmail.com>

* Reduce maintainership and step down as team member. (#4023)

* Reduce maintainership and step down as team member.

My future priorities will be on the alerting aspects of Mimir, so I think it is
right to reduce my maintainership accordingly and allow others to take my place.
Similarly, remove myself as a team member.

* Sort previous team members.

Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com>
Signed-off-by: Marco Pracucci <marco@pracucci.com>
Signed-off-by: Peter Štibraný <pstibrany@gmail.com>
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>
Co-authored-by: Nick Pillitteri <56quarters@users.noreply.github.com>
Co-authored-by: Marco Pracucci <marco@pracucci.com>
Co-authored-by: Patrick Oyarzun <patrick.oyarzun@grafana.com>
Co-authored-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
Co-authored-by: Charles Korn <charleskorn@users.noreply.github.com>
Co-authored-by: Peter Štibraný <pstibrany@gmail.com>
Co-authored-by: Ursula Kallio <ursula.kallio@grafana.com>
Co-authored-by: Oleg Zaytsev <mail@olegzaytsev.com>
Co-authored-by: gotjosh <josue.abreu@gmail.com>
Co-authored-by: Goutham Veeramachaneni <gouthamve@gmail.com>
Co-authored-by: Mauro Stettler <mauro.stettler@gmail.com>
Co-authored-by: Steve Simpson <steve.simpson@grafana.com>
  • Loading branch information
13 people authored Jan 20, 2023
1 parent 8e6d704 commit 3fde520
Show file tree
Hide file tree
Showing 254 changed files with 14,140 additions and 10,220 deletions.
8 changes: 7 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

### Grafana Mimir

* [CHANGE] Querier: Introduce `-querier.max-partial-query-length` to limit the time range for partial queries at the querier level and deprecate `-store.max-query-length`. #3825
* [CHANGE] Querier: Introduce `-querier.max-partial-query-length` to limit the time range for partial queries at the querier level and deprecate `-store.max-query-length`. #3825 #4017
* [CHANGE] Store-gateway: Remove experimental `-blocks-storage.bucket-store.max-concurrent-reject-over-limit` flag. #3706
* [CHANGE] Ingester: If shipping is enabled block retention will now be relative to the upload time to cloud storage. If shipping is disabled block retention will be relative to the creation time of the block instead of the mintime of the last block created. #3816
* [CHANGE] Query-frontend: Deprecated CLI flag `-query-frontend.align-querier-with-step` has been removed. #3982
Expand Down Expand Up @@ -33,6 +33,8 @@
* [ENHANCEMENT] Reduce overhead of debug logging when filtered out. #3875
* [ENHANCEMENT] Update Docker base images from `alpine:3.16.2` to `alpine:3.17.1`. #3898
* [ENHANCEMENT] Ingester: Add new `/ingester/tsdb_metrics` endpoint to return tenant-specific TSDB metrics. #3923
* [ENHANCEMENT] Query-frontend: CLI flag `-query-frontend.max-total-query-length` and its associated YAML configuration is now stable. #3882
* [ENHANCEMENT] Ruler: rule groups now support optional and experimental `align_evaluation_time_on_interval` field, which causes all evaluations to happen on interval-aligned timestamp. #4013
* [BUGFIX] Log the names of services that are not yet running rather than `unsupported value type` when calling `/ready` and some services are not running. #3625
* [BUGFIX] Alertmanager: Fix template spurious deletion with relative data dir. #3604
* [BUGFIX] Security: update prometheus/exporter-toolkit for CVE-2022-46146. #3675
Expand All @@ -44,7 +46,9 @@
* [BUGFIX] Distributor, Query-scheduler: Make sure ring metrics include a `cortex_` prefix as expected by dashboards. #3809
* [BUGFIX] Querier: canceled requests are no longer reported as "consistency check" failures. #3837 #3927
* [BUGFIX] Distributor: don't panic when `metric_relabel_configs` in overrides contains null element. #3868
* [BUGFIX] Distributor: don't panic when OTLP histograms don't have any buckets. #3853
* [BUGFIX] Ingester, Compactor: fix panic that can occur when compaction fails. #3955
* [BUGFIX] Store-gateway: return `Canceled` rather than `Aborted` error when the calling querier cancels the request. #4007

### Mixin

Expand Down Expand Up @@ -80,7 +84,9 @@

### Mimirtool

* [ENHANCEMENT] Update `mimirtool config convert` to work with Mimir 2.4, 2.5, 2.6 changes. #3952
* [ENHANCEMENT] Mimirtool is now available to install through Homebrew with `brew install mimirtool`. #3776
* [ENHANCEMENT] Added `--concurrency` to `mimirtool rules sync` command. #3996
* [BUGFIX] Fix summary output from `mimirtool rules sync` to display correct number of groups created and updated. #3918

### Documentation
Expand Down
3 changes: 3 additions & 0 deletions CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,6 @@

# Documentation.
/docs/ @grafana/docs-metrics @grafana/mimir-maintainers

/pkg/alertmanager/ @grafana/mimir-ruler-and-alertmanager-maintainers @grafana/mimir-maintainers
/pkg/ruler/ @grafana/mimir-ruler-and-alertmanager-maintainers @grafana/mimir-maintainers
2 changes: 1 addition & 1 deletion GOVERNANCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,13 +58,13 @@ The current team members are:
- Oleg Zaytsev — [@colega](https://github.com/colega) ([Grafana Labs](https://grafana.com/))
- Patrick Oyarzun - [@Logiraptor](https://github.com/Logiraptor) ([Grafana Labs](https://grafana.com/))
- Peter Štibraný — [@pstibrany](https://github.com/pstibrany) ([Grafana Labs](https://grafana.com/))
- Steve Simpson - [@stevesg](https://github.com/stevesg) ([Grafana Labs](https://grafana.com/))
- Tyler Reid — [@treid314](https://github.com/treid314) ([Grafana Labs](https://grafana.com/))

Previous team members:

- Goutham Veeramachaneni — [@gouthamve](https://github.com/gouthamve)
- Jacob Lisi — [@jtlisi](https://github.com/jtlisi)
- Steve Simpson - [@stevesg](https://github.com/stevesg)
- Tom Wilkie — [@tomwilkie](https://github.com/tomwilkie)

### Maintainers
Expand Down
3 changes: 2 additions & 1 deletion MAINTAINERS.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@ The following are the main/default maintainers:
- Nick Pillitteri — [@56quarters](https://github.com/56quarters) ([Grafana Labs](https://grafana.com/))
- Oleg Zaytsev — [@colega](https://github.com/colega) ([Grafana Labs](https://grafana.com/))
- Peter Štibraný — [@pstibrany](https://github.com/pstibrany) ([Grafana Labs](https://grafana.com/))
- Steve Simpson - [@stevesg](https://github.com/stevesg) ([Grafana Labs](https://grafana.com/))
- Tyler Reid — [@treid314](https://github.com/treid314) ([Grafana Labs](https://grafana.com/))
- Patrick Oyarzun — [@Logiraptor](https://github.com/Logiraptor) ([Grafana Labs](https://grafana.com/))

Expand All @@ -21,7 +20,9 @@ Some parts of the codebase have other maintainers, the package paths also includ
- Ursula Kallio — [@osg-grafana](https://github.com/osg-grafana) ([Grafana Labs](https://grafana.com/))
- `pkg/alertmanager`:
- Josh Abreu Mesa — [@gotjosh](https://github.com/gotjosh) ([Grafana Labs](https://grafana.com/))
- Steve Simpson - [@stevesg](https://github.com/stevesg) ([Grafana Labs](https://grafana.com/))
- `pkg/frontend`:
- Cyril Tovena — [@cyriltovena](https://github.com/cyriltovena) ([Grafana Labs](https://grafana.com/)) # limited to reviews
- `pkg/ruler`:
- Josh Abreu Mesa — [@gotjosh](https://github.com/gotjosh) ([Grafana Labs](https://grafana.com/))
- Steve Simpson - [@stevesg](https://github.com/stevesg) ([Grafana Labs](https://grafana.com/))
22 changes: 22 additions & 0 deletions cmd/mimir/config-descriptor.json
Original file line number Diff line number Diff line change
Expand Up @@ -2780,6 +2780,17 @@
"fieldType": "int",
"fieldCategory": "advanced"
},
{
"kind": "field",
"name": "max_ephemeral_series",
"required": false,
"desc": "Max ephemeral series that this ingester can hold (across all tenants). Requests to create additional ephemeral series will be rejected. 0 = unlimited.",
"fieldValue": null,
"fieldDefaultValue": 0,
"fieldFlag": "ingester.instance-limits.max-ephemeral-series",
"fieldType": "int",
"fieldCategory": "experimental"
},
{
"kind": "field",
"name": "max_inflight_push_requests",
Expand Down Expand Up @@ -3032,6 +3043,17 @@
"fieldFlag": "ingester.max-global-series-per-metric",
"fieldType": "int"
},
{
"kind": "field",
"name": "max_ephemeral_series_per_user",
"required": false,
"desc": "The maximum number of in-memory ephemeral series per tenant, across the cluster before replication. 0 to disable ephemeral storage.",
"fieldValue": null,
"fieldDefaultValue": 0,
"fieldFlag": "ingester.max-ephemeral-series-per-user",
"fieldType": "int",
"fieldCategory": "experimental"
},
{
"kind": "field",
"name": "max_global_metadata_per_user",
Expand Down
6 changes: 5 additions & 1 deletion cmd/mimir/help-all.txt.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -1027,6 +1027,8 @@ Usage of ./cmd/mimir/mimir:
Override the expected name on the server certificate.
-ingester.ignore-series-limit-for-metric-names string
Comma-separated list of metric names, for which the -ingester.max-global-series-per-metric limit will be ignored. Does not affect the -ingester.max-global-series-per-user limit.
-ingester.instance-limits.max-ephemeral-series int
[experimental] Max ephemeral series that this ingester can hold (across all tenants). Requests to create additional ephemeral series will be rejected. 0 = unlimited.
-ingester.instance-limits.max-inflight-push-requests int
Max inflight push requests that this ingester can handle (across all tenants). Additional requests will be rejected. 0 = unlimited. (default 30000)
-ingester.instance-limits.max-ingestion-rate float
Expand All @@ -1035,6 +1037,8 @@ Usage of ./cmd/mimir/mimir:
Max series that this ingester can hold (across all tenants). Requests to create additional series will be rejected. 0 = unlimited.
-ingester.instance-limits.max-tenants int
Max tenants that this ingester can hold. Requests from additional tenants will be rejected. 0 = unlimited.
-ingester.max-ephemeral-series-per-user int
[experimental] The maximum number of in-memory ephemeral series per tenant, across the cluster before replication. 0 to disable ephemeral storage.
-ingester.max-global-exemplars-per-user int
[experimental] The maximum number of exemplars in memory, across the cluster. 0 to disable exemplars ingestion.
-ingester.max-global-metadata-per-metric int
Expand Down Expand Up @@ -2062,7 +2066,7 @@ Usage of ./cmd/mimir/mimir:
-store.max-labels-query-length duration
Limit the time range (end - start time) of series, label names and values queries. This limit is enforced in the querier. If the requested time range is outside the allowed range, the request will not fail but will be manipulated to only query data within the allowed time range. 0 to disable.
-store.max-query-length duration
Limit the query time range (end - start time). This limit is enforced in the querier (on the query possibly split by the query-frontend) and ruler. 0 to disable.
Deprecated: Limit the query time range (end - start time). This limit is enforced in the querier (on the query possibly split by the query-frontend) and ruler. 0 to disable. This option is deprecated, use -querier.max-partial-query-length or -query-frontend.max-total-query-length instead.
-target comma-separated-list-of-strings
Comma-separated list of components to include in the instantiated process. The default value 'all' includes all components that are required to form a functional Grafana Mimir instance in single-binary mode. Use the '-modules' command line flag to get a list of available components, and to see which components are included with 'all'. (default all)
-tenant-federation.enabled
Expand Down
2 changes: 1 addition & 1 deletion cmd/mimir/help.txt.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -594,7 +594,7 @@ Usage of ./cmd/mimir/mimir:
-store.max-labels-query-length duration
Limit the time range (end - start time) of series, label names and values queries. This limit is enforced in the querier. If the requested time range is outside the allowed range, the request will not fail but will be manipulated to only query data within the allowed time range. 0 to disable.
-store.max-query-length duration
Limit the query time range (end - start time). This limit is enforced in the querier (on the query possibly split by the query-frontend) and ruler. 0 to disable.
Deprecated: Limit the query time range (end - start time). This limit is enforced in the querier (on the query possibly split by the query-frontend) and ruler. 0 to disable. This option is deprecated, use -querier.max-partial-query-length or -query-frontend.max-total-query-length instead.
-target comma-separated-list-of-strings
Comma-separated list of components to include in the instantiated process. The default value 'all' includes all components that are required to form a functional Grafana Mimir instance in single-binary mode. Use the '-modules' command line flag to get a list of available components, and to see which components are included with 'all'. (default all)
-tenant-federation.enabled
Expand Down
4 changes: 2 additions & 2 deletions development/mimir-microservices-mode/docker-compose.jsonnet
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ std.manifestYamlDoc({
httpPort: 8007,
jaegerApp: 'query-frontend',
extraArguments:
'-store.max-query-length=8760h' +
'-query-frontend.max-total-query-length=8760h' +
// Use of scheduler is activated by `-query-frontend.scheduler-address` option.
(if $._config.use_query_scheduler then ' -query-frontend.scheduler-address=query-scheduler:9011' else ''),
}),
Expand All @@ -87,7 +87,7 @@ std.manifestYamlDoc({
'query-scheduler': mimirService({
target: 'query-scheduler',
httpPort: 8011,
extraArguments: '-store.max-query-length=8760h',
extraArguments: '-query-frontend.max-total-query-length=8760h',
}),
} else {}
),
Expand Down
4 changes: 2 additions & 2 deletions development/mimir-microservices-mode/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -260,7 +260,7 @@
"command":
- "sh"
- "-c"
- "sleep 3 && exec ./mimir -config.file=./config/mimir.yaml -target=query-frontend -server.http-listen-port=8007 -server.grpc-listen-port=9007 -activity-tracker.filepath=/activity/query-frontend-8007 -store.max-query-length=8760h -query-frontend.scheduler-address=query-scheduler:9011 -memberlist.nodename=query-frontend -memberlist.bind-port=10007 -ingester.ring.store=memberlist -distributor.ring.store=memberlist -compactor.ring.store=memberlist -store-gateway.sharding-ring.store=memberlist -ruler.ring.store=memberlist -alertmanager.sharding-ring.store=memberlist"
- "sleep 3 && exec ./mimir -config.file=./config/mimir.yaml -target=query-frontend -server.http-listen-port=8007 -server.grpc-listen-port=9007 -activity-tracker.filepath=/activity/query-frontend-8007 -query-frontend.max-total-query-length=8760h -query-frontend.scheduler-address=query-scheduler:9011 -memberlist.nodename=query-frontend -memberlist.bind-port=10007 -ingester.ring.store=memberlist -distributor.ring.store=memberlist -compactor.ring.store=memberlist -store-gateway.sharding-ring.store=memberlist -ruler.ring.store=memberlist -alertmanager.sharding-ring.store=memberlist"
"depends_on":
- "minio"
- "distributor-1"
Expand All @@ -283,7 +283,7 @@
"command":
- "sh"
- "-c"
- "sleep 3 && exec ./mimir -config.file=./config/mimir.yaml -target=query-scheduler -server.http-listen-port=8011 -server.grpc-listen-port=9011 -activity-tracker.filepath=/activity/query-scheduler-8011 -store.max-query-length=8760h -memberlist.nodename=query-scheduler -memberlist.bind-port=10011 -ingester.ring.store=memberlist -distributor.ring.store=memberlist -compactor.ring.store=memberlist -store-gateway.sharding-ring.store=memberlist -ruler.ring.store=memberlist -alertmanager.sharding-ring.store=memberlist"
- "sleep 3 && exec ./mimir -config.file=./config/mimir.yaml -target=query-scheduler -server.http-listen-port=8011 -server.grpc-listen-port=9011 -activity-tracker.filepath=/activity/query-scheduler-8011 -query-frontend.max-total-query-length=8760h -memberlist.nodename=query-scheduler -memberlist.bind-port=10011 -ingester.ring.store=memberlist -distributor.ring.store=memberlist -compactor.ring.store=memberlist -store-gateway.sharding-ring.store=memberlist -ruler.ring.store=memberlist -alertmanager.sharding-ring.store=memberlist"
"depends_on":
- "minio"
- "distributor-1"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ weight: 40

# Run Grafana Mimir in production using the Helm chart

Beyond [Getting started with Grafana Mimir using the Helm chart]({{< relref "../get-started-helm-charts/" >}}),
In addition to the guide [Get started with Grafana Mimir using the Helm chart]({{< relref "../get-started-helm-charts/" >}}),
which covers setting up Grafana Mimir on a local Kubernetes cluster or
within a low-risk development environment, you can prepare Grafana Mimir
for production.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ The following features are currently experimental:
- Disable alerting and recording rules evaluation on a per-tenant basis
- `-ruler.recording-rules-evaluation-enabled`
- `-ruler.alerting-rules-evaluation-enabled`
- Aligning of evaluation timestamp on interval (`align_evaluation_time_on_interval`)
- Distributor
- Metrics relabeling
- OTLP ingestion path
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -102,3 +102,25 @@ service:
processors: [...]
exporters: [..., otlphttp]
```

## Format considerations

We follow the official [OTLP Metric points to Prometheus](https://opentelemetry.io/docs/reference/specification/compatibility/prometheus_and_openmetrics/#otlp-metric-points-to-prometheus) specification.

You might experience the following common issues:

- Dots (.) are converted to \_

Prometheus metrics do not support `.` and `-` characters in metric or label names. Prometheus converts these characters to `_`.

For example:

`requests.duration{http.status_code=500, cloud.region=us-central1}` in OTLP

`requests_duration{http_status_code=”500”, cloud_region=”us-central1”}` in Prometheus

- Resource attributes are added to the `target_info` metric.

However, `<service.namespace>/<service.name>` or `<service.name>` (if the namespace is empty), is added as the label `job`, and `service.instance.id` is added as the label `instance` to every metric.

For details, see the [OpenTelemetry Resource Attributes](https://opentelemetry.io/docs/reference/specification/compatibility/prometheus_and_openmetrics/#resource-attributes) specification.
Loading

0 comments on commit 3fde520

Please sign in to comment.