[Autoscaling] Add Elasticsearch autoscaling controller

[barkbay]

This PR introduces the Elasticsearch autoscaling controller.

This new controller manages the autoscaling policies, periodically polls the _autoscaling/capacity API and adjusts automatically the resources in the node sets.

Some parts of the code are not optimized, this PR is large, I tried to keep something which is easy to understand.
I also tried to add comments where needed to help the reader/reviewer.

Starting by the reconcileInternal function in pkg/controller/autoscaling/elasticsearch/driver.go might be a good starting point for a review.

The following improvements will be added in dedicated PRs:

• Add a warning in case of overlapping roles.
• Manage limits using ratios, only resource requests are supported for now.
• Add a stabilization period
• Make the poll period a setting, current default is 60s

Testing

This PR relies on the ES autoscaling client and requires to activate a trial (or a valid license)

Deciders are documented here.

Data tier

Unfortunately I don't have strong advices for testing storage deciders.
You can try to use Rally, but note that it will only exercise the reactive decider:

esrally race --pipeline=benchmark-only --track=eventdata --track-repository=eventdata --challenge=index-logs-fixed-daily-volume \
             --track-params="bulk_size:100,number_of_replicas:1,bulk_indexing_clients:1,daily_logging_volume:1GB" \
             --target-hosts=https://${ES}:9200 \
             --client-options="use_ssl:true,verify_certs:false,basic_auth_user:'elastic',basic_auth_password:$PASSWORD"

You can also deploy some agents to create datastreams.

ML

You can use the following API call to create a ML job (it requires the sample data flights). Then do a POST _ml/data_frame/analytics/mljob/_start to start the job.

Logs and status

The expected resources for the node sets and some useful messages are printed in the logs but also stored in the elasticsearch.alpha.elastic.co/autoscaling-status annotation. You can use the following command to retrieve it:kubectl get -n elasticsearch.elasticsearch.k8s.elastic.co/mycluster -o jsonpath='{.metadata.annotations.elasticsearch\.alpha\.elastic\.co/autoscaling-status}' | jq

[pebrc]

I am still working through this but I thought I share a first set of comments. All minor.

[charith-elastic]

It's a lot to digest and I won't claim to have understood everything. But, overall LGTM 👍🏼

[pebrc]

I think I am 2/3 through ...

[pebrc]

Should we allow CPU capacity in smaller increments than one core?

[barkbay]

I don't have a strong opinion. I always have preferred rounded values because it makes it easier to resonate about the performance model (e.g. how to size thread pools in the JVM with fractional number ?) But I agree that the user might want to have more accurate resource requests.
@elastic/cloud-k8s I'm curious if there are other opinions.

[plaformsre]

For Elasticsearch or Kibana, we would round up CPU capacity to an integer, but might be too much for lighter components, for example beats.
Also, if you just want to run the whole ECK stack on a laptop, users would prefer to have less granual option to define resource limits, just to enable them to test locally.

[pebrc]

Impressive work! I think I went through everything once. But I haven't tested as this is depending on the API PR IIUC

[barkbay]

Thanks all for the time you spent on this PR, I know it is not an easy one 🙇

Quick update:

• I have been working on a PR to support request/limit ratio but I think it requires this one to be merged before I raise it.
• Also this PR is blocked by this one.

[pebrc]

LGTM! I did only a very few tests over night and storage autoscaling worked as expected. But I hoped to be able to do some more tests. But I believe we can merge this as is and iterate if necessary.