-
Notifications
You must be signed in to change notification settings - Fork 308
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: adding victoriametrics remote write #3641
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #3641 +/- ##
==========================================
- Coverage 85.14% 84.81% -0.33%
==========================================
Files 943 943
Lines 157070 157091 +21
==========================================
- Hits 133731 133233 -498
- Misses 23339 23858 +519 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
* feat: adding victoria metrics remote write * test: add e2e tests for prom and vm remote writes
* fix: columns table in information_schema misses some columns * fix: test_information_schema_dot_columns * fix: fuzz test * feat: adds srs_id and refactor some columns with constant vector * fix: test_information_schema_dot_columns * chore: update comment Co-authored-by: JeremyHi <jiachun_feng@proton.me> * build(deps): bump h2 from 0.3.24 to 0.3.26 (#3642) Bumps [h2](https://github.com/hyperium/h2) from 0.3.24 to 0.3.26. - [Release notes](https://github.com/hyperium/h2/releases) - [Changelog](https://github.com/hyperium/h2/blob/v0.3.26/CHANGELOG.md) - [Commits](hyperium/h2@v0.3.24...v0.3.26) --- updated-dependencies: - dependency-name: h2 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump whoami from 1.4.1 to 1.5.1 (#3643) Bumps [whoami](https://github.com/ardaku/whoami) from 1.4.1 to 1.5.1. - [Changelog](https://github.com/ardaku/whoami/blob/v1/CHANGELOG.md) - [Commits](ardaku/whoami@v1.4.1...v1.5.1) --- updated-dependencies: - dependency-name: whoami dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat: adding victoriametrics remote write (#3641) * feat: adding victoria metrics remote write * test: add e2e tests for prom and vm remote writes * fix: construct correct pk list with pre-existing pk (#3614) * fix: construct correct pk list with pre-existing pk Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * update UT Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * test(sqlness): release databases after tests (#3648) * refactor: rename Greptime_Type to Greptime_type --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Ruihang Xia <waynestxia@gmail.com> Co-authored-by: JeremyHi <jiachun_feng@proton.me> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ning Sun <sunng@protonmail.com> Co-authored-by: Ruihang Xia <waynestxia@gmail.com> Co-authored-by: Weny Xu <wenymedia@gmail.com>
I came across this PR. Just a small heads up:
In the fourth step, I mentioned this issue in https://jiekun.dev/posts/vmagent-data-structures (思考题2). VictoriaMetrics will handle such situations with fallback logic. It will try I'm not familiar with Rust, so I haven't read the code and I'm not sure if this is also implemented in GreptimeDB. This could potentially cause Edit: It seems the related source codes is: greptimedb/src/servers/src/http/prom_store.rs Line 250 in 6ab3aeb
It could be fixed like this (generated by ChatGPT): let buf = match is_zstd {
true => {
match zstd_decompress(&body[..]) {
Ok(result) => Bytes::from(result),
Err(_) => snappy_decompress(&body[..]).map_err(|_| "Both decompression methods failed")?,
}
}
false => {
match snappy_decompress(&body[..]) {
Ok(result) => Bytes::from(result),
Err(_) => zstd_decompress(&body[..]).map_err(|_| "Both decompression methods failed")?,
}
}
}; |
@jiekun, thank you for your hint. It seems like a bug in vmagent. Can you open the issue, and we can discuss it? |
It should be handled more gracefully by Remote-write clients won't know what protocol data is compressed in. |
I hereby agree to the terms of the GreptimeDB CLA.
Refer to a related PR or issue link (optional)
What's changed and what's your intention?
This patch enhances our remote write interface to support victoriametrics' remote write variant.
Basically, the protocol variant is a change from snappy encoding to zstd encoding. For compatibility check, it introduces a new handshake process.
The client sends an empty write request with
get_vm_proto_version
Compatible server will reply 200 and body of a single
1
Then client starts to send remote write protobuf in zstd encoding:
The server will reply same
204
.Learn more about VictoriaMetrics remote write: https://victoriametrics.com/blog/victoriametrics-remote-write/
The patch also includes e2e tests for both prometheus and vm remote write.
Checklist