Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sort seed hosts to avoid unecessary configmap updates #2171

Merged
merged 3 commits into from
Nov 26, 2019

Conversation

pebrc
Copy link
Collaborator

@pebrc pebrc commented Nov 25, 2019

{"level":"info","@timestamp":"2019-11-25T01:27:00.726Z","logger":"settings","message":"Seed hosts updated","ver":"1.0.0-beta1-b559ec38","namespace":"e2e-m9cld-mercury","es_name":"test-es-keystore-drf6","hosts":["10.72.2.38:9300","10.72.1.31:9300","10.72.0.25:9300"]}
{"level":"info","@timestamp":"2019-11-25T01:27:00.781Z","logger":"zen2","message":"Ensuring no voting exclusions are set","ver":"1.0.0-beta1-b559ec38","namespace":"e2e-m9cld-mercury","es_name":"test-es-keystore-drf6"}
{"level":"info","@timestamp":"2019-11-25T01:27:00.849Z","logger":"elasticsearch-controller","message":"Updating status","ver":"1.0.0-beta1-b559ec38","iteration":1232,"namespace":"e2e-m9cld-mercury","es_name":"test-es-keystore-drf6"}
{"level":"info","@timestamp":"2019-11-25T01:27:00.849Z","logger":"elasticsearch-controller","message":"Ending reconciliation run","ver":"1.0.0-beta1-b559ec38","iteration":1232,"namespace":"e2e-m9cld-mercury","name":"test-es-keystore-drf6","took":0.8772917}
{"level":"info","@timestamp":"2019-11-25T01:27:00.849Z","logger":"elasticsearch-controller","message":"Starting reconciliation run","ver":"1.0.0-beta1-b559ec38","iteration":1233,"namespace":"e2e-m9cld-mercury","name":"test-es-keystore-drf6"}
{"level":"info","@timestamp":"2019-11-25T01:27:01.587Z","logger":"generic-reconciler","message":"Updating resource","ver":"1.0.0-beta1-b559ec38","kind":"ConfigMap","namespace":"e2e-m9cld-mercury","name":"test-es-keystore-drf6-es-unicast-hosts"}
{"level":"info","@timestamp":"2019-11-25T01:27:01.595Z","logger":"settings","message":"Seed hosts updated","ver":"1.0.0-beta1-b559ec38","namespace":"e2e-m9cld-mercury","es_name":"test-es-keystore-drf6","hosts":["10.72.1.31:9300","10.72.0.25:9300","10.72.2.38:9300"]}
{"level":"info","@timestamp":"2019-11-25T01:27:01.671Z","logger":"zen2","message":"Ensuring no voting exclusions are set","ver":"1.0.0-beta1-b559ec38","namespace":"e2e-m9cld-mercury","es_name":"test-es-keystore-drf6"}
{"level":"info","@timestamp":"2019-11-25T01:27:01.736Z","logger":"elasticsearch-controller","message":"Updating status","ver":"1.0.0-beta1-b559ec38","iteration":1233,"namespace":"e2e-m9cld-mercury","es_name":"test-es-keystore-drf6"}
{"level":"info","@timestamp":"2019-11-25T01:27:01.736Z","logger":"elasticsearch-controller","message":"Ending reconciliation run","ver":"1.0.0-beta1-b559ec38","iteration":1233,"namespace":"e2e-m9cld-mercury","name":"test-es-keystore-drf6","took":0.886652754}
{"level":"info","@timestamp":"2019-11-25T01:27:01.736Z","logger":"elasticsearch-controller","message":"Starting reconciliation run","ver":"1.0.0-beta1-b559ec38","iteration":1234,"namespace":"e2e-m9cld-mercury","name":"test-es-keystore-drf6"}
{"level":"info","@timestamp":"2019-11-25T01:27:02.466Z","logger":"generic-reconciler","message":"Updating resource","ver":"1.0.0-beta1-b559ec38","kind":"ConfigMap","namespace":"e2e-m9cld-mercury","name":"test-es-keystore-drf6-es-unicast-hosts"}
{"level":"info","@timestamp":"2019-11-25T01:27:02.475Z","logger":"settings","message":"Seed hosts updated","ver":"1.0.0-beta1-b559ec38","namespace":"e2e-m9cld-mercury","es_name":"test-es-keystore-drf6","hosts":["10.72.0.25:9300","10.72.2.38:9300","10.72.1.31:9300"]}

While trying to understand why our e2e tests fail I noticed a lot of seed host updates going on that seemed purely caused by non-deterministic pod order and consequently updates of the configmap we use for the unicast hosts file.

This PR sorts the seed nodes to avoid these configmap updates and adjusts the relevant unit test accordingly.

@pebrc pebrc added the >enhancement Enhancement of existing functionality label Nov 25, 2019
@pebrc pebrc closed this Nov 25, 2019
@pebrc pebrc changed the title Sort seed hosts to avoid unecessary pod recreations Sort seed hosts to avoid unecessary secret updates Nov 26, 2019
@pebrc pebrc changed the title Sort seed hosts to avoid unecessary secret updates Sort seed hosts to avoid unecessary configmap updates Nov 26, 2019
@pebrc pebrc reopened this Nov 26, 2019
Copy link
Contributor

@thbkrkr thbkrkr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@sebgl sebgl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch :)

@pebrc pebrc merged commit 9a16f8d into elastic:master Nov 26, 2019
@anyasabo anyasabo added the v1.0.0 label Jan 8, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>enhancement Enhancement of existing functionality v1.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants