support historical analysis for HC detector

[ylwu-amzn]

Signed-off-by: Yaliang Wu ylwu@amazon.com

Description

Support historical analysis for HC detector. For single entity detector, we just need to create one AD task as it has only one ML model. For HC detector, AD will run separate ML model for each entity. So we will create individual task for each entity. And we will have one detector level task to track overall state like progress. Will save detector level task id as parent task id in entity task.

The main steps for HC detector historical analysis:

1. Get top entities
2. Calculate how many entities can run concurrently based on settings and cluster resources. Then start N task lanes to run entities. Each lane will run 1 entity. For every entity task will get all data node's state and dispatch entity task to the node with least node.
3. If one entity task finished, worker node will send entity task done message to coordinating node to poll next entity task.
4. If entity task failed, will handle exception. Check java doc of ADTaskManager#getAdEntityTaskAction for details.
5. Once all entities done. Will mark the HC detector level task as done.

Check List

• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov-commenter]

Codecov Report

Merging #104 (e3c4299) into single-flow-development-rc1 (d5e5084) will decrease coverage by 9.05%.
The diff coverage is 18.75%.

@@                        Coverage Diff                        @@
##             single-flow-development-rc1     #104      +/-   ##
=================================================================
- Coverage                          79.12%   70.07%   -9.06%     
+ Complexity                          2700     2523     -177     
=================================================================
  Files                                243      244       +1     
  Lines                              11244    11895     +651     
  Branches                            1033     1089      +56     
=================================================================
- Hits                                8897     8335     -562     
- Misses                              1917     3137    +1220     
+ Partials                             430      423       -7     

Flag	Coverage Δ
plugin	70.07% <18.75%> (-9.06%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...rg/opensearch/ad/AnomalyDetectorProfileRunner.java	76.47% <ø> (-0.79%)	⬇️
...java/org/opensearch/ad/feature/FeatureManager.java	93.96% <ø> (-2.27%)	⬇️
...ensearch/ad/rest/RestAnomalyDetectorJobAction.java	41.37% <0.00%> (-15.77%)	⬇️
...arch/ad/rest/RestExecuteAnomalyDetectorAction.java	26.19% <0.00%> (ø)
...est/handler/IndexAnomalyDetectorActionHandler.java	43.08% <0.00%> (-6.31%)	⬇️
...ava/org/opensearch/ad/task/ADHCBatchTaskCache.java	0.00% <0.00%> (ø)
...d/transport/GetAnomalyDetectorTransportAction.java	53.95% <ø> (-3.60%)	⬇️
...rch/ad/transport/ForwardADTaskTransportAction.java	10.16% <4.34%> (-75.55%)	⬇️
...ava/org/opensearch/ad/task/ADTaskCacheManager.java	42.77% <7.46%> (-50.81%)	⬇️
...ain/java/org/opensearch/ad/task/ADTaskManager.java	21.09% <10.03%> (-52.13%)	⬇️
... and 57 more

[kaituo]

New comments based on our discussions on Friday.

[kaituo]

resolved all of open comments from me. Please address the left over comments from the old PR: #58

[ylwu-amzn]

resolved all of open comments from me. Please address the left over comments from the old PR: #58

Thanks Kaituo. To save some effort, I just replied most other comments on the old PR directly. You can check my replies on the old PR and comment more on this PR if have new questions.

[zhanghg08]

No concerns from my side. Thanks for the changes.