Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Perf framework hangs while executing async scenario with insufficient heap. #21629

Closed
kasobol-msft opened this issue May 18, 2021 · 1 comment · Fixed by #21690
Closed

[BUG] Perf framework hangs while executing async scenario with insufficient heap. #21629

kasobol-msft opened this issue May 18, 2021 · 1 comment · Fixed by #21690
Assignees
@mikeharder
Labels
Central-EngSys This issue is owned by the Engineering System team. pillar-performance The issue is related to performance, one of our core engineering pillars. test bug Problem in test source code (most likely) test-reliability Issue that causes tests to be unreliable

Comments

@kasobol-msft
Copy link
Contributor

Try the following scenario:

java -Xms100m -Xmx100m -jar azure-storage-perf-1.0.0-beta.1-jar-with-dependencies.jar uploadblob --warmup 0 --duration 1 --size 1048576000

The program keeps reporting zero progress. It is expected to crash with OOM in that case.

It repros easily on local machime.
Test run where this happened: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=899330&view=logs&j=1ebff73e-a26c-5a81-5f48-b56b69f60e6e&t=3f94ee88-7ff1-5b4e-3fcb-67bf1ac93b37
image

In order to triage whether it's SDK or perf framework I executed similar scenario in simple console app. That one failed immediately with OOM. That suggest problem is in perf framework.
image
@kasobol-msft kasobol-msft added test bug Problem in test source code (most likely) pillar-performance The issue is related to performance, one of our core engineering pillars. test-reliability Issue that causes tests to be unreliable labels May 18, 2021
@kasobol-msft kasobol-msft mentioned this issue May 18, 2021
Merged
@mikeharder mikeharder added the Central-EngSys This issue is owned by the Engineering System team. label May 20, 2021
@mikeharder mikeharder mentioned this issue May 20, 2021
Merged
@mikeharder
Copy link
Member

The root cause is exceptions like OutOfMemoryError are handled differently by the default Reactor schedulers. Instead of terminating the Flux, the Flux will hang and the exception is only sent to the thread's uncaughtExceptionHandler and the Reactor Schedulers.onHandleError.

reactor/reactor-core#2521

https://github.com/reactor/reactor-core/blob/84e921fbfd84754d519b5e3c78262495dcf48b87/reactor-core/src/main/java/reactor/core/Exceptions.java#L443

https://github.com/reactor/reactor-core/blob/84e921fbfd84754d519b5e3c78262495dcf48b87/reactor-core/src/main/java/reactor/core/Exceptions.java#L443

The perf framework will be fixed by this PR: #21690

This issue could also impact any SDK code which uses the default Reactor schedulers.

@mikeharder mikeharder mentioned this issue May 20, 2021
Closed
mikeharder added a commit that referenced this issue May 20, 2021
@mikeharder
[Perf] Fail fast on unhandled exceptions in Reactor threads (#21690)
c5b4288 
- Fixes #21629
azure-sdk pushed a commit to azure-sdk/azure-sdk-for-java that referenced this issue Jan 16, 2023
@satumkurmsft @mattgotteiner
CodeGen from PR 21964 in Azure/azure-rest-api-specs
c416002 
Dev search microsoft.search 2022 09 01 (Azure#21964)

* Adds base for updating Microsoft.Search from version stable/2020-08-01 to version 2022-09-01

* Updates readme

* Updates API version in new specs and examples

* Add missing private endpoint connection properties (Azure#21629)

* add missing private endpoint connection properties

* fix errors

* Adding delegated identities to data plane

* Revert "Adding delegated identities to data plane"

This reverts commit def1bd3ba296458c19b8f04cb1a7a5b800d002f9.

* Fix Lint ProvisioningStateValidation Error

* Applying prettier changes
npm run prettier -- --write

Co-authored-by: Enrique Jaimes <enjaimes@microsoft.com>

* Satumkur/new api ver 20220901 (Azure#21922)

* semantic updates

* Add Dataplane RBAC related changes + examples

* Prettier fixes

* Fix LintDiff errors

* Update SearchUpdateServiceDisableLocalAuth.json

update erroneous api version param

Co-authored-by: Samartha Tumkur Vani <satumkur@microsoft.com>
Co-authored-by: Enrique Jaimes <enjaimes@microsoft.com>

* Add CMK Enforcement policy support (Azure#21996)

* Add CMK Enforcement policy support

* Update api-version

* Remove unused properties

* Remove bypass property

Co-authored-by: satumkurmsft <72990508+satumkurmsft@users.noreply.github.com>
Co-authored-by: Matt <57731498+mattmsft@users.noreply.github.com>
Co-authored-by: Samartha Tumkur Vani <satumkur@microsoft.com>
@github-actions github-actions bot locked and limited conversation to collaborators Apr 12, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@mikeharder mikeharder

Labels
Central-EngSys This issue is owned by the Engineering System team. pillar-performance The issue is related to performance, one of our core engineering pillars. test bug Problem in test source code (most likely) test-reliability Issue that causes tests to be unreliable
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Perf] Fail fast on unhandled exceptions in Reactor threads mikeharder/azure-sdk-for-java
2 participants
@mikeharder @kasobol-msft