Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add delete anomaly results API #131

Conversation

ylwu-amzn
Copy link
Collaborator

@ylwu-amzn ylwu-amzn commented Jul 15, 2021

Description

With historical analysis, user can generate much more anomaly result in a short time, especially for HC detector. AD plugin will roll over anomaly result index if its doc count reaches some limit, and will delete roll-over index after some period (30 days by default). So if user run historical analysis, it may generate too many AD results and speed up the rollover and AD result index cleanup.

This PR adds delete anomaly result API. User can use this API to delete anomaly results with specific query like delete all anomaly results of one detector/task.

Sample request

DELETE _plugins/_anomaly_detection/detectors/results
{
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "detector_id": {
              "value": "6tZminoBaXxjTubPk0k6"
            }
          }
        }
      ]
    }
  }
}

Issues Resolved

[List any issues this PR will resolve]

Check List

  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Yaliang Wu <ylwu@amazon.com>
@ylwu-amzn ylwu-amzn force-pushed the single-flow-development-rc1-deleteresults-pr branch from ec5e872 to 63822a4 Compare July 15, 2021 06:55
Copy link

@zhanghg08 zhanghg08 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A high level comment, do we need to consider FGAC with delete api?

@ylwu-amzn
Copy link
Collaborator Author

ylwu-amzn commented Jul 15, 2021

A high level comment, do we need to consider FGAC with delete api?

Yes, check DeleteAnomalyResultsTransportAction.java line 79

addUserBackendRolesFilter(user, request.getSearchRequest().source());

This method adds user backend role filter, so users can only delete anomaly results which they have permission.

Copy link

@zhanghg08 zhanghg08 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Comment on lines +75 to +80
client.execute(DeleteByQueryAction.INSTANCE, request, listener);
} else {
// Security is enabled and backend role filter is enabled
try {
addUserBackendRolesFilter(user, request.getSearchRequest().source());
client.execute(DeleteByQueryAction.INSTANCE, request, listener);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

client.execute() will be called anyway?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, split the code branch clearly to make it more readable.

Copy link
Contributor

@spbjss spbjss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me.

@ylwu-amzn ylwu-amzn merged commit 64335e0 into opensearch-project:single-flow-development-rc1 Jul 20, 2021
@ylwu-amzn ylwu-amzn mentioned this pull request Jul 27, 2021
1 task
ylwu-amzn added a commit that referenced this pull request Jul 27, 2021
* tune data model for unified flow

Signed-off-by: Yaliang Wu <ylwu@amazon.com>

* support unified flow for single entity detector

Signed-off-by: Yaliang Wu <ylwu@amazon.com>

* add realtime task

Signed-off-by: Yaliang Wu <ylwu@amazon.com>

* support preview with detector directly

Signed-off-by: Yaliang Wu <ylwu@amazon.com>

* support historical analysis for HC detector (#104)

* support run historical analysis for HC detector

Signed-off-by: Yaliang Wu <ylwu@dev-dsk-ylwu-2c-e500d1cf.us-west-2.amazon.com>

* add more javadoc

Signed-off-by: Yaliang Wu <ylwu@dev-dsk-ylwu-2c-e500d1cf.us-west-2.amazon.com>

* address comments: rename method, remove comments etc

Signed-off-by: Yaliang Wu <ylwu@dev-dsk-ylwu-2c-e500d1cf.us-west-2.amazon.com>

* add more comments for task action

Signed-off-by: Yaliang Wu <ylwu@dev-dsk-ylwu-2c-e500d1cf.us-west-2.amazon.com>

* address comments from old CR: mainly renaming and add more comments

Signed-off-by: Yaliang Wu <ylwu@dev-dsk-ylwu-2c-e500d1cf.us-west-2.amazon.com>

Co-authored-by: Yaliang Wu <ylwu@dev-dsk-ylwu-2c-e500d1cf.us-west-2.amazon.com>

* support HC task profiles in profile API;error handling (#123)

* support HC task profiles in profile API;error handling

Signed-off-by: Yaliang Wu <ylwu@amazon.com>

* add init progress of realtime task (#130)

* add init progress of realtime task

* add realtime task cache

* remove DetectionStateHandler, will track error in realtime task

* add more comments

* add delete anomaly results API (#131)

Signed-off-by: Yaliang Wu <ylwu@amazon.com>

* maintain running historical task in hourly cron (#132)

* maintain running historical task in hourly cron

Signed-off-by: Yaliang Wu <ylwu@amazon.com>

* maintain realtime tasks

* address comments

Co-authored-by: Yaliang Wu <ylwu@dev-dsk-ylwu-2c-e500d1cf.us-west-2.amazon.com>
ohltyler pushed a commit to ohltyler/anomaly-detection-2 that referenced this pull request Sep 1, 2021
* tune data model for unified flow

Signed-off-by: Yaliang Wu <ylwu@amazon.com>

* support unified flow for single entity detector

Signed-off-by: Yaliang Wu <ylwu@amazon.com>

* add realtime task

Signed-off-by: Yaliang Wu <ylwu@amazon.com>

* support preview with detector directly

Signed-off-by: Yaliang Wu <ylwu@amazon.com>

* support historical analysis for HC detector (opensearch-project#104)

* support run historical analysis for HC detector

Signed-off-by: Yaliang Wu <ylwu@dev-dsk-ylwu-2c-e500d1cf.us-west-2.amazon.com>

* add more javadoc

Signed-off-by: Yaliang Wu <ylwu@dev-dsk-ylwu-2c-e500d1cf.us-west-2.amazon.com>

* address comments: rename method, remove comments etc

Signed-off-by: Yaliang Wu <ylwu@dev-dsk-ylwu-2c-e500d1cf.us-west-2.amazon.com>

* add more comments for task action

Signed-off-by: Yaliang Wu <ylwu@dev-dsk-ylwu-2c-e500d1cf.us-west-2.amazon.com>

* address comments from old CR: mainly renaming and add more comments

Signed-off-by: Yaliang Wu <ylwu@dev-dsk-ylwu-2c-e500d1cf.us-west-2.amazon.com>

Co-authored-by: Yaliang Wu <ylwu@dev-dsk-ylwu-2c-e500d1cf.us-west-2.amazon.com>

* support HC task profiles in profile API;error handling (opensearch-project#123)

* support HC task profiles in profile API;error handling

Signed-off-by: Yaliang Wu <ylwu@amazon.com>

* add init progress of realtime task (opensearch-project#130)

* add init progress of realtime task

* add realtime task cache

* remove DetectionStateHandler, will track error in realtime task

* add more comments

* add delete anomaly results API (opensearch-project#131)

Signed-off-by: Yaliang Wu <ylwu@amazon.com>

* maintain running historical task in hourly cron (opensearch-project#132)

* maintain running historical task in hourly cron

Signed-off-by: Yaliang Wu <ylwu@amazon.com>

* maintain realtime tasks

* address comments

Co-authored-by: Yaliang Wu <ylwu@dev-dsk-ylwu-2c-e500d1cf.us-west-2.amazon.com>
ohltyler pushed a commit to ohltyler/anomaly-detection-2 that referenced this pull request Sep 1, 2021
* tune data model for unified flow

Signed-off-by: Yaliang Wu <ylwu@amazon.com>

* support unified flow for single entity detector

Signed-off-by: Yaliang Wu <ylwu@amazon.com>

* add realtime task

Signed-off-by: Yaliang Wu <ylwu@amazon.com>

* support preview with detector directly

Signed-off-by: Yaliang Wu <ylwu@amazon.com>

* support historical analysis for HC detector (opensearch-project#104)

* support run historical analysis for HC detector

Signed-off-by: Yaliang Wu <ylwu@dev-dsk-ylwu-2c-e500d1cf.us-west-2.amazon.com>

* add more javadoc

Signed-off-by: Yaliang Wu <ylwu@dev-dsk-ylwu-2c-e500d1cf.us-west-2.amazon.com>

* address comments: rename method, remove comments etc

Signed-off-by: Yaliang Wu <ylwu@dev-dsk-ylwu-2c-e500d1cf.us-west-2.amazon.com>

* add more comments for task action

Signed-off-by: Yaliang Wu <ylwu@dev-dsk-ylwu-2c-e500d1cf.us-west-2.amazon.com>

* address comments from old CR: mainly renaming and add more comments

Signed-off-by: Yaliang Wu <ylwu@dev-dsk-ylwu-2c-e500d1cf.us-west-2.amazon.com>

Co-authored-by: Yaliang Wu <ylwu@dev-dsk-ylwu-2c-e500d1cf.us-west-2.amazon.com>

* support HC task profiles in profile API;error handling (opensearch-project#123)

* support HC task profiles in profile API;error handling

Signed-off-by: Yaliang Wu <ylwu@amazon.com>

* add init progress of realtime task (opensearch-project#130)

* add init progress of realtime task

* add realtime task cache

* remove DetectionStateHandler, will track error in realtime task

* add more comments

* add delete anomaly results API (opensearch-project#131)

Signed-off-by: Yaliang Wu <ylwu@amazon.com>

* maintain running historical task in hourly cron (opensearch-project#132)

* maintain running historical task in hourly cron

Signed-off-by: Yaliang Wu <ylwu@amazon.com>

* maintain realtime tasks

* address comments

Co-authored-by: Yaliang Wu <ylwu@dev-dsk-ylwu-2c-e500d1cf.us-west-2.amazon.com>
ohltyler pushed a commit that referenced this pull request Sep 1, 2021
* tune data model for unified flow

Signed-off-by: Yaliang Wu <ylwu@amazon.com>

* support unified flow for single entity detector

Signed-off-by: Yaliang Wu <ylwu@amazon.com>

* add realtime task

Signed-off-by: Yaliang Wu <ylwu@amazon.com>

* support preview with detector directly

Signed-off-by: Yaliang Wu <ylwu@amazon.com>

* support historical analysis for HC detector (#104)

* support run historical analysis for HC detector

Signed-off-by: Yaliang Wu <ylwu@dev-dsk-ylwu-2c-e500d1cf.us-west-2.amazon.com>

* add more javadoc

Signed-off-by: Yaliang Wu <ylwu@dev-dsk-ylwu-2c-e500d1cf.us-west-2.amazon.com>

* address comments: rename method, remove comments etc

Signed-off-by: Yaliang Wu <ylwu@dev-dsk-ylwu-2c-e500d1cf.us-west-2.amazon.com>

* add more comments for task action

Signed-off-by: Yaliang Wu <ylwu@dev-dsk-ylwu-2c-e500d1cf.us-west-2.amazon.com>

* address comments from old CR: mainly renaming and add more comments

Signed-off-by: Yaliang Wu <ylwu@dev-dsk-ylwu-2c-e500d1cf.us-west-2.amazon.com>

Co-authored-by: Yaliang Wu <ylwu@dev-dsk-ylwu-2c-e500d1cf.us-west-2.amazon.com>

* support HC task profiles in profile API;error handling (#123)

* support HC task profiles in profile API;error handling

Signed-off-by: Yaliang Wu <ylwu@amazon.com>

* add init progress of realtime task (#130)

* add init progress of realtime task

* add realtime task cache

* remove DetectionStateHandler, will track error in realtime task

* add more comments

* add delete anomaly results API (#131)

Signed-off-by: Yaliang Wu <ylwu@amazon.com>

* maintain running historical task in hourly cron (#132)

* maintain running historical task in hourly cron

Signed-off-by: Yaliang Wu <ylwu@amazon.com>

* maintain realtime tasks

* address comments

Co-authored-by: Yaliang Wu <ylwu@dev-dsk-ylwu-2c-e500d1cf.us-west-2.amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants