Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[AUTOCUT] Integration Test failed for flow-framework: 2.17.0 #856

Closed
opensearch-ci-bot opened this issue Sep 5, 2024 · 27 comments
Closed
Labels
autocut integ-test-failure Issues related to integration test failure untriaged v2.17.0 Issues targeting release v2.17.0

Comments

@opensearch-ci-bot
Copy link
Collaborator

The integration test failed at distribution level for component flow-framework
Version: 2.17.0
Distribution: tar
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8609/display/redirect

* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test/2.17.0/10268/linux/arm64/tar/test-results/8609/integ-test/test-report.yml

Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here.

@opensearch-ci-bot opensearch-ci-bot added autocut untriaged v2.17.0 Issues targeting release v2.17.0 labels Sep 5, 2024
@opensearch-ci-bot
Copy link
Collaborator Author

The integration test failed at distribution level for component flow-framework
Version: 2.17.0
Distribution: tar
Architecture: x64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8610/display/redirect

* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test/2.17.0/10268/linux/x64/tar/test-results/8610/integ-test/test-report.yml

Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here.

@opensearch-ci-bot opensearch-ci-bot added integ-test-failure Issues related to integration test failure linux:tar:x64 labels Sep 5, 2024
@opensearch-ci-bot
Copy link
Collaborator Author

The integration test failed at distribution level for component flow-framework
Version: 2.17.0
Distribution: tar
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8611/display/redirect

* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test/2.17.0/10269/linux/arm64/tar/test-results/8611/integ-test/test-report.yml

Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here.

@opensearch-ci-bot
Copy link
Collaborator Author

The integration test failed at distribution level for component flow-framework
Version: 2.17.0
Distribution: tar
Architecture: x64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8614/display/redirect

* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test/2.17.0/10269/linux/x64/tar/test-results/8614/integ-test/test-report.yml

Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here.

@opensearch-ci-bot
Copy link
Collaborator Author

The integration test failed at distribution level for component flow-framework
Version: 2.17.0
Distribution: rpm
Architecture: x64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8613/display/redirect

* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test/2.17.0/10269/linux/x64/rpm/test-results/8613/integ-test/test-report.yml

Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here.

@opensearch-ci-bot
Copy link
Collaborator Author

The integration test failed at distribution level for component flow-framework
Version: 2.17.0
Distribution: rpm
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8616/display/redirect

* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test/2.17.0/10269/linux/arm64/rpm/test-results/8616/integ-test/test-report.yml

Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here.

@opensearch-ci-bot
Copy link
Collaborator Author

The integration test failed at distribution level for component flow-framework
Version: 2.17.0
Distribution: deb
Architecture: x64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8617/display/redirect

* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test/2.17.0/10269/linux/x64/deb/test-results/8617/integ-test/test-report.yml

Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here.

@opensearch-ci-bot
Copy link
Collaborator Author

The integration test failed at distribution level for component flow-framework
Version: 2.17.0
Distribution: deb
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8615/display/redirect

* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test/2.17.0/10269/linux/arm64/deb/test-results/8615/integ-test/test-report.yml

Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here.

@opensearch-ci-bot
Copy link
Collaborator Author

The integration test failed at distribution level for component flow-framework
Version: 2.17.0
Distribution: tar
Architecture: x64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8619/display/redirect

* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test/2.17.0/10271/linux/x64/tar/test-results/8619/integ-test/test-report.yml

Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here.

@opensearch-ci-bot
Copy link
Collaborator Author

The integration test failed at distribution level for component flow-framework
Version: 2.17.0
Distribution: deb
Architecture: x64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8622/display/redirect

* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test/2.17.0/10271/linux/x64/deb/test-results/8622/integ-test/test-report.yml

Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here.

@opensearch-ci-bot
Copy link
Collaborator Author

The integration test failed at distribution level for component flow-framework
Version: 2.17.0
Distribution: rpm
Architecture: x64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8621/display/redirect

* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test/2.17.0/10271/linux/x64/rpm/test-results/8621/integ-test/test-report.yml

Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here.

@opensearch-ci-bot
Copy link
Collaborator Author

The integration test failed at distribution level for component flow-framework
Version: 2.17.0
Distribution: deb
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8624/display/redirect

* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test/2.17.0/10271/linux/arm64/deb/test-results/8624/integ-test/test-report.yml

Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here.

@opensearch-ci-bot
Copy link
Collaborator Author

The integration test failed at distribution level for component flow-framework
Version: 2.17.0
Distribution: tar
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8620/display/redirect

* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test/2.17.0/10271/linux/arm64/tar/test-results/8620/integ-test/test-report.yml

Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here.

@opensearch-ci-bot
Copy link
Collaborator Author

The integration test failed at distribution level for component flow-framework
Version: 2.17.0
Distribution: rpm
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8623/display/redirect

* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test/2.17.0/10271/linux/arm64/rpm/test-results/8623/integ-test/test-report.yml

Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here.

@opensearch-ci-bot
Copy link
Collaborator Author

The integration test failed at distribution level for component flow-framework
Version: 2.17.0
Distribution: tar
Architecture: arm64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8627/display/redirect

* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test/2.17.0/10271/linux/arm64/tar/test-results/8627/integ-test/test-report.yml

Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here.

@opensearch-ci-bot
Copy link
Collaborator Author

The integration test failed at distribution level for component flow-framework
Version: 2.17.0
Distribution: zip
Architecture: x64
Platform: windows

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8626/display/redirect

* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test/2.17.0/10271/windows/x64/zip/test-results/8626/integ-test/test-report.yml

Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here.

@opensearch-ci-bot
Copy link
Collaborator Author

The integration test failed at distribution level for component flow-framework
Version: 2.17.0
Distribution: tar
Architecture: x64
Platform: linux

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8630/display/redirect

* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test/2.17.0/10271/linux/x64/tar/test-results/8630/integ-test/test-report.yml

Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here.

@dbwiddis
Copy link
Member

dbwiddis commented Sep 6, 2024

src/test/java/org/opensearch/flowframework/workflow/RegisterLocalPretrainedModelStepTests.java:140: error: no suitable constructor found for MLTask(String,String,,,MLTaskState,,,,,,,,,boolean)

Looks like opensearch-project/ml-commons#2825 is a breaking change in a minor version bump. @rbhavna is this intentional?

CC @Zhangxunmt @ylwu-amzn @minalsha

@dbwiddis
Copy link
Member

Closing as this has passed on all platforms.

@opensearch-ci-bot
Copy link
Collaborator Author

The integration test failed at distribution level for component flow-framework
Version: 2.17.0
Distribution: zip
Architecture: x64
Platform: windows

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8670/display/redirect

* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test/2.17.0/10289/windows/x64/zip/test-results/8670/integ-test/test-report.yml

Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here.

@dbwiddis
Copy link
Member

Failure looks flaky, has passed multiple previous times.

@opensearch-ci-bot
Copy link
Collaborator Author

The integration test failed at distribution level for component flow-framework
Version: 2.17.0
Distribution: zip
Architecture: x64
Platform: windows

Please check the logs: https://build.ci.opensearch.org/job/integ-test/8673/display/redirect

* Test-report manifest:*
- https://ci.opensearch.org/ci/dbc/integ-test/2.17.0/10289/windows/x64/zip/test-results/8673/integ-test/test-report.yml

Note: Steps to reproduce, additional logs and other files can be found within the above test-report manifest.
Instructions of this test-report manifest can be found here.

@dbwiddis
Copy link
Member

@gaiksaya
Copy link
Member

@dbwiddis For future releases can you check why flow-framework is being flaky for windows platform?
Had to run multiple times to get it to pass.
Thank you!

@dbwiddis
Copy link
Member

@dbwiddis For future releases can you check why flow-framework is being flaky for windows platform? Had to run multiple times to get it to pass. Thank you!

Our integ tests load local models into memory, which often causes memory issues in CI test environments. It's much more common on macOS github runners, but occasionally happens for WIndows.

We could probably try to refactor and do more cleanup between test runs... will create an issue.

@dbwiddis
Copy link
Member

(FYI, ML Commons integ tests often trip memory circuit breakers for similar reasons. Any suggestions welcome.)

@ylwu-amzn
Copy link

Tune the memory circuit breaker setting ?

@dbwiddis
Copy link
Member

OK, so it turns out this isn't memory related. I suspect a race condition somewhere due to how the test is written. Opened #870, we'll fix before 2.18.

CC @gaiksaya @minalsha

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
autocut integ-test-failure Issues related to integration test failure untriaged v2.17.0 Issues targeting release v2.17.0
Projects
None yet
Development

No branches or pull requests

4 participants