Add Elasticsearch client cache #2875
Conversation
There was a problem hiding this comment.
Implementationwise this looks good to me. But I am having second thoughts as to whether the complexity of caching ES requests is actually worth it. If I see it correctly we could to a GET before PUT/POST pattern for all of these with the exception of the voting exclusions. GET requests are comparatively cheap compared to the cluster state changing PUT/POST requests iirc and we would side step any cache invalidation/sync issues. Basically: GET the currrent allocation filter, compare and update it only if it's not what we expect. Same for mininum_master_nodes.
| type CachedClientBuilder interface { | ||
|
|
||
| // NewElasticsearchCachedClient returns an Elasticsearch client that relies on a cache for some operations. | ||
| // Do not use the resulting client concurrently for a same operation. Concurrent calls of a given operation is not thread-safe. |
There was a problem hiding this comment.
| // Do not use the resulting client concurrently for a same operation. Concurrent calls of a given operation is not thread-safe. | |
| // Do not use the resulting client concurrently for the same operation. Concurrent calls of a given operation are not thread-safe. |
| steps := []struct { | ||
| es1Value, es2Value *string // value to set when calling the api | ||
| es1ExpectedValue, es2ExpectedValue *string // expected value stored in the fake client | ||
| es1ExpectedCalled, es2ExpectedCalled int // number of time an api is called |
There was a problem hiding this comment.
| es1ExpectedCalled, es2ExpectedCalled int // number of time an api is called | |
| es1ExpectedCalled, es2ExpectedCalled int // number of times an api is called |
|
Discussed out of band: we decided to not rely on any form of cache and always perform sync. calls to Elasticsearch API |
ECK attempts to reduce the number of API calls to Elasticsearch by storing the api call arguments in annotations.
Unfortunately when ECK fails to update the annotation it may lead to some consistencies: see #2788 and #2864
This PR adds an in memory cache to keep track of the last arguments used when calling the ES API for the following operations:
still a draft but ready for review re. the overall architecture