Removing default integtest.sh (1.x). #217
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Tyler Ohlsen <ohltyler@amazon.com>
Signed-off-by: Tyler Ohlsen <ohltyler@amazon.com>
Signed-off-by: Tyler Ohlsen <ohltyler@amazon.com>
Signed-off-by: Yaliang Wu <ylwu@dev-dsk-ylwu-2c-e500d1cf.us-west-2.amazon.com> Co-authored-by: Yaliang Wu <ylwu@dev-dsk-ylwu-2c-e500d1cf.us-west-2.amazon.com>
…demo Add Integtest.sh for OpenSearch integtest setups
This PR is a conglomerate of the following PRs. #60 #64 #65 #67 #68 #69 #70 #71 #74 #75 #76 #77 #78 #79 #82 #83 #84 #92 #94 #93 #95 kaituo#1 kaituo#2 kaituo#3 kaituo#4 kaituo#5 kaituo#6 kaituo#7 kaituo#8 kaituo#9 kaituo#10 This spreadsheet contains the mappings from files to PR number (bug fix in my AD fork and tests are not included): https://gist.github.com/kaituo/9e1592c4ac4f2f449356cb93d0591167
* don't replace detector user when update Signed-off-by: Yaliang Wu <ylwu@amazon.com> * fix wrong doc link
* tune data model for unified flow Signed-off-by: Yaliang Wu <ylwu@amazon.com> * support unified flow for single entity detector Signed-off-by: Yaliang Wu <ylwu@amazon.com> * add realtime task Signed-off-by: Yaliang Wu <ylwu@amazon.com> * support preview with detector directly Signed-off-by: Yaliang Wu <ylwu@amazon.com> * support historical analysis for HC detector (#104) * support run historical analysis for HC detector Signed-off-by: Yaliang Wu <ylwu@dev-dsk-ylwu-2c-e500d1cf.us-west-2.amazon.com> * add more javadoc Signed-off-by: Yaliang Wu <ylwu@dev-dsk-ylwu-2c-e500d1cf.us-west-2.amazon.com> * address comments: rename method, remove comments etc Signed-off-by: Yaliang Wu <ylwu@dev-dsk-ylwu-2c-e500d1cf.us-west-2.amazon.com> * add more comments for task action Signed-off-by: Yaliang Wu <ylwu@dev-dsk-ylwu-2c-e500d1cf.us-west-2.amazon.com> * address comments from old CR: mainly renaming and add more comments Signed-off-by: Yaliang Wu <ylwu@dev-dsk-ylwu-2c-e500d1cf.us-west-2.amazon.com> Co-authored-by: Yaliang Wu <ylwu@dev-dsk-ylwu-2c-e500d1cf.us-west-2.amazon.com> * support HC task profiles in profile API;error handling (#123) * support HC task profiles in profile API;error handling Signed-off-by: Yaliang Wu <ylwu@amazon.com> * add init progress of realtime task (#130) * add init progress of realtime task * add realtime task cache * remove DetectionStateHandler, will track error in realtime task * add more comments * add delete anomaly results API (#131) Signed-off-by: Yaliang Wu <ylwu@amazon.com> * maintain running historical task in hourly cron (#132) * maintain running historical task in hourly cron Signed-off-by: Yaliang Wu <ylwu@amazon.com> * maintain realtime tasks * address comments Co-authored-by: Yaliang Wu <ylwu@dev-dsk-ylwu-2c-e500d1cf.us-west-2.amazon.com>
* compact rcf
… fix hard code query size of historical analysis (#150) * avoid sending back verbose error message and wrong 500 error to user Signed-off-by: Yaliang Wu <ylwu@amazon.com> * put general error message into constants
Signed-off-by: lai <laijiang@amazon.com>
Disable model splitting in single-stream detectors We split and distribute models to different nodes to avoid large models on a single node. The splitting is unnecessary after introducing compact rcf as the model is smaller (at least 4x smaller). Splitting also undoes the shared point store optimization among trees. Also, splitting brings complications when computing expected values. Thus, this PR disables splitting by increasing the desired model size. We won't split a model whose size is less than the desired size. This PR adjusts max features and shingle size accordingly to avoid huge models without explicit benefits. This PR also adjusts the model size formula due to the change aws/random-cut-forest-by-aws#265. I will update the rcf version once rcf 2.0 is released in maven. Testing done: 1. tested single-stream models won't be split after the change. 2. Updated unit tests.
* Bug fixes and disable BWC This PR did the following: 1)Fixed a bug in ModelPartitioner that we should use estimated rcf instead of total model size (including rcf and threshold models). 2)Disabled backward-compatibility check so that we can run multi-category detectors in OpenSearch 1.0. 3)remove unused fields in MemoryTracker 4)add null check when deserializing the sample vector Testing done: 1)added unit tests 2)did e2e tests for the basic single-stream detector workflow.
* Handle more AD exceptions thrown over the wire OpenSearch restricts the kind of exceptions that can be thrown over the wire (Read OpenSearchException.OpenSearchExceptionHandle). Since we cannot add our exception like ResourceNotFoundException without modifying OpenSearch's code, we must unwrap the NotSerializableExceptionWrapper and check its root cause message. This PR adds checks all AD exceptions in such cases. Previously, we only checked a couple of them. Previously, we wrap all exceptions thrown in AD using one of AnomalyDetectionExceptions. The wrap brings a complication to NotSerializableExceptionWrapper thrown over the wire as NotSerializableExceptionWrapper won't keep the original AnomalyDetectionExceptions object and only keeps the cause message. Therefore, we won't be able to restore the original exception encapsulated inside AnomalyDetectionExceptions. This PR stops wrapping the original exception inside AnomalyDetectionExceptions. This PR also adds a null check inside EntityModel in case of a potential null pointer exception. Testing done: 1. Added unit tests to check if exceptions wrapped inside NotSerializableExceptionWrapper can be decoded. 2. Did e2e testing for basic single-stream and HCAD workflow.
* Adding mixed cluster bwc tests Signed-off-by: Vacha <vachshah@amazon.com> * Adding anomaly detector assertions for bwc Signed-off-by: Vacha <vachshah@amazon.com> * Adding rolling upgrade and full restart upgrade bwc tests Signed-off-by: Vacha <vachshah@amazon.com>
Signed-off-by: Miki <mehranb@amazon.com>
* support historical analysis for multi-category HC Signed-off-by: Yaliang Wu <ylwu@amazon.com> * add todo for tuning timeout * add delete AD result setting; tune minimum doc count for historical analysis * fix minmum doc count for historical analysis * fix wrong min doc threshold
* Adding support for BWC tests to run in CI Signed-off-by: Vacha <vachshah@amazon.com> * Adding bwc test commands in developer guide Signed-off-by: Vacha <vachshah@amazon.com>
First, this PR adds an empty string check in PriorityCache. Second, in EntityColdStarter, this PR clears samples after using them for training. Third, this PR removes unused fields and local variables in ModelManager and BulkUtil. Fourth, this PR decreases the concurrent preview from 5 to 2, with a conservative purpose to reduce potential GC. Fifth, this PR finishes unit tests owed due to my previous PRs. 99% code of the PR is about tests. Testing done: 1. Added unit tests. 2. made sure the basic e2e workflow still works.
Signed-off-by: Vacha <vachshah@amazon.com>
* Limit the max models shown on the stats and profile API Previously, the number of models we show on the stats/profile API is unbounded. The unbounded models can cause potential memory/network issues if a vast number of models exist. This PR sets an upper bound and makes the bound a dynamic setting. Due to the above change, the models may not be complete, and thus I added a model_count to the stats/profile API to show the actual models present. This PR also fixed bugs: First, AnomalyDetectorProfileRunner used a listener instead of delegateListener, which can cause the profile API to return early before collecting all outputs. Second, I forgot to support the multi-category fields in AnomalyDetectorProfileRunner.profileEntityStats. This PR adds support there. In detail, profile API needs to report total entities. We can use cardinality aggregation for a single-category field. But we cannot do that for multi-category fields as it requires scripting to generate run time fields, which is expensive. We work around the problem by using a composite query to find the first 10_000 buckets. Generally, traversing all buckets/combinations can't be done without visiting all matches, which is costly for data with many entities. Given that it is often enough to have a lower bound of the number of entities, such as "there are at least 10000 entities", the default is set to 10,000. That is, requests will count the total entities up to 10,000. Third, I fixed a potential null pointer exception in ADTaskManager. Testing done: 1. e2e workflow tests by verifying profile/stats API shows correct output, and the limit is in effect. 2. backward compatibility tests to make sure the newly added model_count does not cause issues. 3. added unit tests.
Previously, we hardcoded HCAD (including real-time and historical) shingle size to be one due to time constraints when releasing HCAD. This PR enabled shingle. By default, the shingle size is eight but can be customized. I removed shingling-related code (even though I didn't allow shingling, but there is some half-finished shingling code) in HCAD as RCF can create shingles by itself (turned on via the internalShinglingEnabled configuration). This PR also: 1. upgrade RCF to 2.0.1. 2. Removed MULTI_ENTITY_NUM_TREES with NUM_TREES as both HCAD and single-stream detectors have the same number of trees. 3.Fixed a bug in ModelManager.score method: I should compute anomaly grade and confidence before updating rcf and threshold model. Testing done: 1. added unit tests for the bug found. 2. Manual testing to verify HCAD single stream and historical workflow still work, and the RCF models have desired shingle size.
Signed-off-by: lai <laijiang@amazon.com>
…sk (#170) * support backward compatibility of historical analysis and realtime task * scale task lane automatically * fix reset task state and update task * tune scale task slots logic and move it to task manager * use AD version hash ring for realtime job; reserve 1 task slot when scale down * rollback single flow task type as single entity to save frontend effort * remove request id from cron request; address comments * support BWC of RCF result response based on AD version * remove BWC check for anomaly result response * address comments: fix log; add more comments * enhance stop logic;address comments * add more comments;tune function name * add AD version hash ring with cooldown period for realtime job * tune hash ring * tune hash ring;add more ut * use new PluginInfo constrctor as it's changed in OpenSearch PR#848 * tune error message of null AD version
* block updating detector category field Signed-off-by: Yaliang Wu <ylwu@amazon.com> * remove debug code * ignore category field order when compare
* Adding support to run BWC tests for nightly and release builds Signed-off-by: Sarat Vemulapalli <vemulapallisarat@gmail.com> * Updating test suite for AD Signed-off-by: Sarat Vemulapalli <vemulapallisarat@gmail.com>
Signed-off-by: Tyler Ohlsen <ohltyler@amazon.com>
Signed-off-by: dblock <dblock@dblock.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Coming from opensearch-project/opensearch-build#497, removing default integtest.sh on the 1.x brach.