feat: add proxy for Prometheus remote write #2065
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
pmalek-sumo
reviewed
Feb 1, 2022
deploy/helm/sumologic/templates/metrics/remote-write-proxy/deployment.yaml
Outdated
Show resolved
Hide resolved
pmalek-sumo
reviewed
Feb 1, 2022
kasia-kujawa
reviewed
Feb 1, 2022
deploy/helm/sumologic/templates/metrics/remote-write-proxy/deployment.yaml
Outdated
Show resolved
Hide resolved
|
autoscalling will probably end with poor loadbalancing like we observe for current architecture, but I think it will be improvement anyway |
swiatekm
force-pushed
the
feat/remote-write-proxy
branch
2 times, most recently
from
f2fa6b1 to
61c1e63
Compare
In the real world, I'm not convinced anyone will actually need more than three Nginx instances - the three replicas are only here for availability. In a real-world use case, a customer is pushing ~13M samples per minute through nginx, and the total CPU usage is around 500 mCPU. Nginx also eats so few resources doing this, that I think the additional complexity of the HPA isn't worth it. |
swiatekm
force-pushed
the
feat/remote-write-proxy
branch
3 times, most recently
from
a1cc929 to
e5d56de
Compare
perk-sumo
reviewed
Feb 3, 2022
perk-sumo
reviewed
Feb 3, 2022
deploy/helm/sumologic/templates/metrics/remote-write-proxy/deployment.yaml
Outdated
Show resolved
Hide resolved
perk-sumo
approved these changes
Feb 3, 2022
pmalek-sumo
reviewed
Feb 3, 2022
sumo-drosiek
reviewed
Feb 3, 2022
added 2 commits
February 3, 2022 15:41
Prometheus remote write uses a single persistent HTTP connection per target, which interacts poorly with TCP load balancing with iptables that K8s Services do. Use a real HTTP load balancer for this instead - nginx with a very simple configuration.
swiatekm
force-pushed
the
feat/remote-write-proxy
branch
from
e533333 to
de2242f
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Prometheus remote write uses a single persistent HTTP connection per target, which interacts poorly with TCP load balancing with iptables that K8s Services do. Use a real HTTP load balancer for this instead - nginx with a very simple configuration.
I manually ran integration tests and they passed. I don't want to add another test to run for all K8s versions, so I'll add the integration test after we redo the test matrix in the CI do allow tests to only run in a single K8s environment.
Checklist
Remove items which don't apply to your PR.
Testing performed