metrics-generator: use Prometheus Agent WAL and remote storage #1323
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
TODO
|
yvrhdn
force-pushed
the
kvrhdn/metrics-generator-prometheus-agent-wal
branch
from
ea81280 to
afa9c01
Compare
yvrhdn
commented
Mar 7, 2022
| val float64 | ||
| } | ||
|
|
||
| func TestGenerator(t *testing.T) { |
There was a problem hiding this comment.
I've deleted these tests in favour of the e2e tests. These tests have to do a lot of config management/mocking to get the generator running and verify the metrics emitted are correctly. This is exactly the same as what the e2e tests do, but their code is a bit simpler.
If anyone feels strong about keeping them, I don't mind reinstating them.
yvrhdn
requested review from
joe-elliott,
annanay25,
mdisibio,
dgzlopes,
mapno and
zalegrala
as code owners
26 tasks
mapno
reviewed
Mar 8, 2022
mapno
approved these changes
Mar 8, 2022
|
Hmm, |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does:
Replaces our homegrown remote write implementation with the WAL and remote storage implementation from the Prometheus Agent.
We use two components from Prometheus:
agent.DB: this is the WAL optimised for remote writing metrics only. It's a Prometheus TSDB without the querying, alerting,... capabilities. Whenever we scrape/collect metrics we append them toagent.DBwhich will store the samples on disk.remote.Storage(only the remote write functionality): this tails the WAL and writes data to the configured remote write endpoint(s). It has retry logic and can scale up queues as necessary. It supports a wide range of authorization options and can rewrite labels, see https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_writeMulti-tenancy: Prometheus is not multi-tenant, to support multi-tenancy we create a WAL and remote-writer for every tenant sending data. Each WAL will be stored in
<WAL path>/<tenant ID>.Resilience: even though the WAL stores samples on disk, it does not make the metrics-generator resilient to crashes. After a restart the remote writer will start from the end of the WAL, even if older data was sent yet. See prometheus/prometheus#8809.
But it will make the metrics-generator more resilient against an outage of the downstream TSDB. Pending samples is stored on disk so we do not lose them or risk running out of memory. This should allow the metrics-generator to overcome short outages.
Which issue(s) this PR fixes:
Related to #1303
Checklist
Documentation addedWill be done in a later PRCHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]