Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Service Bus] Message loss with receiveMessages in receiveAndDelete mode when maxWaitTime is under 2 seconds #15115

Closed
HarshaNalluru opened this issue May 3, 2021 · 1 comment
Closed

[Service Bus] Message loss with receiveMessages in receiveAndDelete mode when maxWaitTime is under 2 seconds #15115

HarshaNalluru opened this issue May 3, 2021 · 1 comment
Assignees
@richardpark-msft
Labels
bug This issue requires a change to an existing behavior in the product in order to be resolved. Client This issue points to a problem in the data-plane of the library. Service Bus
Milestone
[2021] July

Comments

@HarshaNalluru
Copy link
Member

HarshaNalluru commented May 3, 2021 •
edited
Loading

Message loss with receiveMessages in receiveAndDelete mode

When the maxWaitTime is smaller such as 1500ms, calling await receiver.receiveMessages(50, { maxWaitTimeInMs: 1500 }) repeatedly in a while loop causes message loss.

Reason could be the placement of drain call.

  • Not seen with larger maxWaitTime.
  • Not seen with subscribe (more reason to believe the issue is related to draining credits)
  • Behavior is seen better with partitioned queues than unpartitioned (since receiving from partitioned queues is relatively slower when compared).

Code to repro - https://github.com/HarshaNalluru/sb-2048-issues/blob/main/message-loss-repro.ts

Observed while working on #14060
@ghost ghost added the needs-triage Workflow: This is a new issue that needs to be triaged to the appropriate team. label May 3, 2021
@HarshaNalluru HarshaNalluru mentioned this issue May 3, 2021
Closed
4 tasks
@ramya-rao-a ramya-rao-a added bug This issue requires a change to an existing behavior in the product in order to be resolved. Client This issue points to a problem in the data-plane of the library. Service Bus labels May 3, 2021
@ghost ghost removed the needs-triage Workflow: This is a new issue that needs to be triaged to the appropriate team. label May 3, 2021
@ramya-rao-a ramya-rao-a added this to the [2021] June milestone May 3, 2021
@ramya-rao-a ramya-rao-a changed the title [Service Bus] Message loss with receiveMessages in receiveAndDelete mode [Service Bus] Message loss with receiveMessages in receiveAndDelete mode when maxWaitTime is under 2 seconds May 3, 2021
@richardpark-msft richardpark-msft mentioned this issue Jun 25, 2021
Merged
richardpark-msft added a commit that referenced this issue Jul 1, 2021
@richardpark-msft
[service-bus] Fix message loss issues with peekLock and receiveAndDel…
d75f119 
…ete (#15989)

Fixing an issue where we could lose messages or provoke an alarming message from rhea (`Received transfer when credit was 0`)
    
The message loss issue is related to how we trigger 'drain' using 'addCredit(1)'. Our 'receiver.drain; receiver.addCredit(1)' pattern actually does add a credit, which shows up in the flow frame that gets sent for our drain. This has led to occasionally receiving more messages than we intended.
    
The second part of this was that we were masking this error because we had code that specifically threw out messages if more arrived than were requested. If the message was being auto-renewed it's possible for the message to appear to be missing, and if we were in receiveAndDelete the message is effectively lost at that point. That code is now removed (we defer to just allowing the extrra message, should a bug arise that causes that) and we log an error indicating it did happen.
    
The rhea error message appeared to be triggered by our accidentally allowing multiple overlapping 'drain's to occur (finalAction did not check to see if we were _already_ draining and would allow it to happen multiple times). Removing the concurrent drains fixed this issue but I didn't fully investigate why.

Fixes #15606, #15115
@richardpark-msft
Copy link
Member

This has been fixed with #15989 and will be released with the release of Service Bus next week.

openapi-sdkautomation bot pushed a commit to AzureSDKAutomation/azure-sdk-for-js that referenced this issue Sep 23, 2021
@azaslonov @yingru97
@JatinSanghvi @atharos1 @VitaliyKurokhtin @RupengLiu @msyyc
CodeGen from PR 15832 in Azure/azure-rest-api-specs
0c4bb6e 
Microsoft.ApiManagement : Release api-version 2021-04-01-preview (Azure#15832)

* Adds base for updating Microsoft.ApiManagement from version preview/2021-01-01-preview to version 2021-04-01-preview

* Updates readme

* Updates API version in new specs and examples

* Added support for GraphQL API type (Azure#14851)

* Added support for GraphQL API type

* Added missing enum value for GraphQL APIs.

* add schema definition fix from 2021-01-01-preview (Azure#14883)

Co-authored-by: Alan Feng <alanfeng@microsoft.com>

* Vanguard: Get Outbound Network Dependency (Azure#14900)

* outbound dependency

* fix resource

* Update backup/restore API for managed identities (Azure#14973)

* Update backup/restore API for system-assigned and user-assigned identities

* Fix formatting issue

* Rename user-assigned-msi-client-id with client-id

* SystemData implemenation on API Management Control Plane (Azure#14899)

* system-data

* fix spec

* remove unreferenced file

* APIM - Network Watcher Connectivity Check integration specs (Azure#15056)

* Connectivity Check API Specs

* Fix

* Fix errors

* style fixs

* Fix 202 code

* HTTPConnect request parameters

* Fix errors

Co-authored-by: Nicolás Barrera <t-nbarrera@microsoft.com>

* Parameter examples added (Azure#14836)

* platformversion (Azure#15114)

* fix update service (Azure#15478)

* HttpConnect example (Azure#15493)

* HttpConnect example

* Prettifier fix

Co-authored-by: Nicolás Barrera <t-nbarrera@microsoft.com>

* Added properties of difference API spec format for "SchemaDocumentProperties" (Azure#15703)

* add schema definition fix from 2021-01-01-preview

* added properties for SchemaDocumentProperties

* updated the fix

* updated fix

* [APIM]Add private endpoint connection APIs (Azure#15115)

* Add private endpoint connection apis

* add readme

* quick fixes

* fix to PE contract

* Small fixes

* small fixes

* small fixes

* Small fixes

* small fixes

* small fix

* small fixes

* small fixes

* small fixes

* Update readme.md

* fix error

* small fix

* fix format

* address comments

* small fixes

* Change to lower case

* small fixes

Co-authored-by: msyyc <70930885+msyyc@users.noreply.github.com>

* Adds base for updating Microsoft.ApiManagement from version preview/2021-01-01-preview to version 2021-04-01-preview

* Updates readme

* Updates API version in new specs and examples

* Added support for GraphQL API type (Azure#14851)

* Added support for GraphQL API type

* Added missing enum value for GraphQL APIs.

* add schema definition fix from 2021-01-01-preview (Azure#14883)

Co-authored-by: Alan Feng <alanfeng@microsoft.com>

* Vanguard: Get Outbound Network Dependency (Azure#14900)

* outbound dependency

* fix resource

* Update backup/restore API for managed identities (Azure#14973)

* Update backup/restore API for system-assigned and user-assigned identities

* Fix formatting issue

* Rename user-assigned-msi-client-id with client-id

* SystemData implemenation on API Management Control Plane (Azure#14899)

* system-data

* fix spec

* remove unreferenced file

* APIM - Network Watcher Connectivity Check integration specs (Azure#15056)

* Connectivity Check API Specs

* Fix

* Fix errors

* style fixs

* Fix 202 code

* HTTPConnect request parameters

* Fix errors

Co-authored-by: Nicolás Barrera <t-nbarrera@microsoft.com>

* Parameter examples added (Azure#14836)

* platformversion (Azure#15114)

* fix update service (Azure#15478)

* HttpConnect example (Azure#15493)

* HttpConnect example

* Prettifier fix

Co-authored-by: Nicolás Barrera <t-nbarrera@microsoft.com>

* Added properties of difference API spec format for "SchemaDocumentProperties" (Azure#15703)

* add schema definition fix from 2021-01-01-preview

* added properties for SchemaDocumentProperties

* updated the fix

* updated fix

* [APIM]Add private endpoint connection APIs (Azure#15115)

* Add private endpoint connection apis

* add readme

* quick fixes

* fix to PE contract

* Small fixes

* small fixes

* small fixes

* Small fixes

* small fixes

* small fix

* small fixes

* small fixes

* small fixes

* Update readme.md

* fix error

* small fix

* fix format

* address comments

* small fixes

* Change to lower case

* small fixes

Co-authored-by: msyyc <70930885+msyyc@users.noreply.github.com>

* fix tenant sync contract

* fix linter issues

* examples for managed identity backup

* prettier

* Fix Apis-Get and updated comments for api-version 2021-04-01-preview (#3)

* add schema definition fix from 2021-01-01-preview

* added fix

* small fix (#6)

* 'examples' added to RepresentationContract (#7)

Co-authored-by: Alexander Zaslonov <zaslonov.alexander@gmail.com>
Co-authored-by: DreamlessA <DreamlessA@users.noreply.github.com>
Co-authored-by: Alan Feng <alanfeng@microsoft.com>
Co-authored-by: Jatin Sanghvi <20547963+JatinSanghvi@users.noreply.github.com>
Co-authored-by: Nicolás Barrera <nicolasbarrera@outlook.com>
Co-authored-by: Nicolás Barrera <t-nbarrera@microsoft.com>
Co-authored-by: VitaliyKurokhtin <vitaliik@microsoft.com>
Co-authored-by: RupengLiu <rliu1211@terpmail.umd.edu>
Co-authored-by: msyyc <70930885+msyyc@users.noreply.github.com>
Co-authored-by: RupengLiu <rupliu@microsoft.com>
Co-authored-by: VitaliyKurokhtin <Vitaliy.Kurokhtin@gmail.com>
openapi-sdkautomation bot pushed a commit to AzureSDKAutomation/azure-sdk-for-js that referenced this issue Nov 10, 2021
@solankisamir @azaslonov
@yingru97 @JatinSanghvi @atharos1 @VitaliyKurokhtin @RupengLiu @msyyc
CodeGen from PR 15667 in Azure/azure-rest-api-specs
26202e4 
Schema resource added in REST API Spec for api-version 2021-04-01-preview (Azure#15667)

* Adds base for updating Microsoft.ApiManagement from version preview/2021-01-01-preview to version 2021-04-01-preview

* Updates readme

* Updates API version in new specs and examples

* Added support for GraphQL API type (Azure#14851)

* Added support for GraphQL API type

* Added missing enum value for GraphQL APIs.

* add schema definition fix from 2021-01-01-preview (Azure#14883)

Co-authored-by: Alan Feng <alanfeng@microsoft.com>

* Vanguard: Get Outbound Network Dependency (Azure#14900)

* outbound dependency

* fix resource

* Update backup/restore API for managed identities (Azure#14973)

* Update backup/restore API for system-assigned and user-assigned identities

* Fix formatting issue

* Rename user-assigned-msi-client-id with client-id

* SystemData implemenation on API Management Control Plane (Azure#14899)

* system-data

* fix spec

* remove unreferenced file

* APIM - Network Watcher Connectivity Check integration specs (Azure#15056)

* Connectivity Check API Specs

* Fix

* Fix errors

* style fixs

* Fix 202 code

* HTTPConnect request parameters

* Fix errors

Co-authored-by: Nicolás Barrera <t-nbarrera@microsoft.com>

* Parameter examples added (Azure#14836)

* platformversion (Azure#15114)

* fix update service (Azure#15478)

* HttpConnect example (Azure#15493)

* HttpConnect example

* Prettifier fix

Co-authored-by: Nicolás Barrera <t-nbarrera@microsoft.com>

* Rename from SchemaContract to ApiSchemaContract

* apimschema.json added to last apim preview version

* Completing specification for Schema resource.
Fixing changes in SchemaContract.

* Value field added to Schema resource specification.

* path fixed.
Renamed from ApiSchema to Schema.
Adding suppression rule to readme.md
Custom word added for spellcheck

* Fixing PR observations.

* Added properties of difference API spec format for "SchemaDocumentProperties" (Azure#15703)

* add schema definition fix from 2021-01-01-preview

* added properties for SchemaDocumentProperties

* updated the fix

* updated fix

* [APIM]Add private endpoint connection APIs (Azure#15115)

* Add private endpoint connection apis

* add readme

* quick fixes

* fix to PE contract

* Small fixes

* small fixes

* small fixes

* Small fixes

* small fixes

* small fix

* small fixes

* small fixes

* small fixes

* Update readme.md

* fix error

* small fix

* fix format

* address comments

* small fixes

* Change to lower case

* small fixes

Co-authored-by: msyyc <70930885+msyyc@users.noreply.github.com>

* Merge main

* Modifications to Schema definition to add "description" field.
Updating examples of Schema.

* Fixing apimanagement.json

* Prettier applied in files

* Fixing PR comments

* fixing x-ms-long-running-operation.

* x-ms-long-running-operation : true in apischema.

* Removing "Resource" type from apimanagement.json

* Apimschema.json updated to point to types.json for ErrorResponse.

* undoing change of responseBody

* Updating ErrorResponse.

* Fixing definitions merge.

Co-authored-by: Samir Solanki <samirsolanki@outlook.com>
Co-authored-by: Alexander Zaslonov <zaslonov.alexander@gmail.com>
Co-authored-by: DreamlessA <DreamlessA@users.noreply.github.com>
Co-authored-by: Alan Feng <alanfeng@microsoft.com>
Co-authored-by: Jatin Sanghvi <20547963+JatinSanghvi@users.noreply.github.com>
Co-authored-by: Nicolás Barrera <nicolasbarrera@outlook.com>
Co-authored-by: Nicolás Barrera <t-nbarrera@microsoft.com>
Co-authored-by: VitaliyKurokhtin <vitaliik@microsoft.com>
Co-authored-by: Christian Palomares <cpalomar@microsoft.com>
Co-authored-by: RupengLiu <rliu1211@terpmail.umd.edu>
Co-authored-by: msyyc <70930885+msyyc@users.noreply.github.com>
openapi-sdkautomation bot pushed a commit to AzureSDKAutomation/azure-sdk-for-js that referenced this issue Dec 14, 2021
@solankisamir @azaslonov
@yingru97 @JatinSanghvi @atharos1 @VitaliyKurokhtin @RupengLiu @msyyc
CodeGen from PR 16715 in Azure/azure-rest-api-specs
778a29f 
Schema resource added in REST API Spec for api-version 2021-08-01 (Azure#16715)

* Adds base for updating Microsoft.ApiManagement from version preview/2021-01-01-preview to version 2021-04-01-preview

* Updates readme

* Updates API version in new specs and examples

* Added support for GraphQL API type (Azure#14851)

* Added support for GraphQL API type

* Added missing enum value for GraphQL APIs.

* add schema definition fix from 2021-01-01-preview (Azure#14883)

Co-authored-by: Alan Feng <alanfeng@microsoft.com>

* Vanguard: Get Outbound Network Dependency (Azure#14900)

* outbound dependency

* fix resource

* Update backup/restore API for managed identities (Azure#14973)

* Update backup/restore API for system-assigned and user-assigned identities

* Fix formatting issue

* Rename user-assigned-msi-client-id with client-id

* SystemData implemenation on API Management Control Plane (Azure#14899)

* system-data

* fix spec

* remove unreferenced file

* APIM - Network Watcher Connectivity Check integration specs (Azure#15056)

* Connectivity Check API Specs

* Fix

* Fix errors

* style fixs

* Fix 202 code

* HTTPConnect request parameters

* Fix errors

Co-authored-by: Nicolás Barrera <t-nbarrera@microsoft.com>

* Parameter examples added (Azure#14836)

* platformversion (Azure#15114)

* fix update service (Azure#15478)

* HttpConnect example (Azure#15493)

* HttpConnect example

* Prettifier fix

Co-authored-by: Nicolás Barrera <t-nbarrera@microsoft.com>

* Rename from SchemaContract to ApiSchemaContract

* apimschema.json added to last apim preview version

* Completing specification for Schema resource.
Fixing changes in SchemaContract.

* Value field added to Schema resource specification.

* path fixed.
Renamed from ApiSchema to Schema.
Adding suppression rule to readme.md
Custom word added for spellcheck

* Fixing PR observations.

* Added properties of difference API spec format for "SchemaDocumentProperties" (Azure#15703)

* add schema definition fix from 2021-01-01-preview

* added properties for SchemaDocumentProperties

* updated the fix

* updated fix

* [APIM]Add private endpoint connection APIs (Azure#15115)

* Add private endpoint connection apis

* add readme

* quick fixes

* fix to PE contract

* Small fixes

* small fixes

* small fixes

* Small fixes

* small fixes

* small fix

* small fixes

* small fixes

* small fixes

* Update readme.md

* fix error

* small fix

* fix format

* address comments

* small fixes

* Change to lower case

* small fixes

Co-authored-by: msyyc <70930885+msyyc@users.noreply.github.com>

* Adds base for updating Microsoft.ApiManagement from version preview/2021-04-01-preview to version 2021-08-01

* Updates readme

* Updates API version in new specs and examples

* Merge main

* Modifications to Schema definition to add "description" field.
Updating examples of Schema.

* Fixing apimanagement.json

* Prettier applied in files

* Fixing PR comments

* fixing x-ms-long-running-operation.

* x-ms-long-running-operation : true in apischema.

* Removing "Resource" type from apimanagement.json

* Apimschema.json updated to point to types.json for ErrorResponse.

* Merging from 2021-04-01-preview to 2021-08-01 stable version.

* undoing change of responseBody

* Updating ErrorResponse.

* Fixing definitions merge.

* Changing definitions to fix output of examples.

* Changes in definitions

* Updating response in some examples.

* Rolling back
ApiSchema -> Schema
Schema -> GlobalSchema

* Changes to undo apimapis.json and adding prefix for Global example Schemas.

* Setting x-ms-long-running-operation-options to location.

Co-authored-by: Samir Solanki <samirsolanki@outlook.com>
Co-authored-by: Alexander Zaslonov <zaslonov.alexander@gmail.com>
Co-authored-by: DreamlessA <DreamlessA@users.noreply.github.com>
Co-authored-by: Alan Feng <alanfeng@microsoft.com>
Co-authored-by: Jatin Sanghvi <20547963+JatinSanghvi@users.noreply.github.com>
Co-authored-by: Nicolás Barrera <nicolasbarrera@outlook.com>
Co-authored-by: Nicolás Barrera <t-nbarrera@microsoft.com>
Co-authored-by: VitaliyKurokhtin <vitaliik@microsoft.com>
Co-authored-by: Christian Palomares <cpalomar@microsoft.com>
Co-authored-by: RupengLiu <rliu1211@terpmail.umd.edu>
Co-authored-by: msyyc <70930885+msyyc@users.noreply.github.com>
openapi-sdkautomation bot pushed a commit to AzureSDKAutomation/azure-sdk-for-js that referenced this issue Jan 5, 2022
@solankisamir @azaslonov
@yingru97 @JatinSanghvi @atharos1 @VitaliyKurokhtin @RupengLiu @msyyc
CodeGen from PR 16715 in Azure/azure-rest-api-specs
33569a3 
Schema resource added in REST API Spec for api-version 2021-08-01 (Azure#16715)

* Adds base for updating Microsoft.ApiManagement from version preview/2021-01-01-preview to version 2021-04-01-preview

* Updates readme

* Updates API version in new specs and examples

* Added support for GraphQL API type (Azure#14851)

* Added support for GraphQL API type

* Added missing enum value for GraphQL APIs.

* add schema definition fix from 2021-01-01-preview (Azure#14883)

Co-authored-by: Alan Feng <alanfeng@microsoft.com>

* Vanguard: Get Outbound Network Dependency (Azure#14900)

* outbound dependency

* fix resource

* Update backup/restore API for managed identities (Azure#14973)

* Update backup/restore API for system-assigned and user-assigned identities

* Fix formatting issue

* Rename user-assigned-msi-client-id with client-id

* SystemData implemenation on API Management Control Plane (Azure#14899)

* system-data

* fix spec

* remove unreferenced file

* APIM - Network Watcher Connectivity Check integration specs (Azure#15056)

* Connectivity Check API Specs

* Fix

* Fix errors

* style fixs

* Fix 202 code

* HTTPConnect request parameters

* Fix errors

Co-authored-by: Nicolás Barrera <t-nbarrera@microsoft.com>

* Parameter examples added (Azure#14836)

* platformversion (Azure#15114)

* fix update service (Azure#15478)

* HttpConnect example (Azure#15493)

* HttpConnect example

* Prettifier fix

Co-authored-by: Nicolás Barrera <t-nbarrera@microsoft.com>

* Rename from SchemaContract to ApiSchemaContract

* apimschema.json added to last apim preview version

* Completing specification for Schema resource.
Fixing changes in SchemaContract.

* Value field added to Schema resource specification.

* path fixed.
Renamed from ApiSchema to Schema.
Adding suppression rule to readme.md
Custom word added for spellcheck

* Fixing PR observations.

* Added properties of difference API spec format for "SchemaDocumentProperties" (Azure#15703)

* add schema definition fix from 2021-01-01-preview

* added properties for SchemaDocumentProperties

* updated the fix

* updated fix

* [APIM]Add private endpoint connection APIs (Azure#15115)

* Add private endpoint connection apis

* add readme

* quick fixes

* fix to PE contract

* Small fixes

* small fixes

* small fixes

* Small fixes

* small fixes

* small fix

* small fixes

* small fixes

* small fixes

* Update readme.md

* fix error

* small fix

* fix format

* address comments

* small fixes

* Change to lower case

* small fixes

Co-authored-by: msyyc <70930885+msyyc@users.noreply.github.com>

* Adds base for updating Microsoft.ApiManagement from version preview/2021-04-01-preview to version 2021-08-01

* Updates readme

* Updates API version in new specs and examples

* Merge main

* Modifications to Schema definition to add "description" field.
Updating examples of Schema.

* Fixing apimanagement.json

* Prettier applied in files

* Fixing PR comments

* fixing x-ms-long-running-operation.

* x-ms-long-running-operation : true in apischema.

* Removing "Resource" type from apimanagement.json

* Apimschema.json updated to point to types.json for ErrorResponse.

* Merging from 2021-04-01-preview to 2021-08-01 stable version.

* undoing change of responseBody

* Updating ErrorResponse.

* Fixing definitions merge.

* Changing definitions to fix output of examples.

* Changes in definitions

* Updating response in some examples.

* Rolling back
ApiSchema -> Schema
Schema -> GlobalSchema

* Changes to undo apimapis.json and adding prefix for Global example Schemas.

* Setting x-ms-long-running-operation-options to location.

Co-authored-by: Samir Solanki <samirsolanki@outlook.com>
Co-authored-by: Alexander Zaslonov <zaslonov.alexander@gmail.com>
Co-authored-by: DreamlessA <DreamlessA@users.noreply.github.com>
Co-authored-by: Alan Feng <alanfeng@microsoft.com>
Co-authored-by: Jatin Sanghvi <20547963+JatinSanghvi@users.noreply.github.com>
Co-authored-by: Nicolás Barrera <nicolasbarrera@outlook.com>
Co-authored-by: Nicolás Barrera <t-nbarrera@microsoft.com>
Co-authored-by: VitaliyKurokhtin <vitaliik@microsoft.com>
Co-authored-by: Christian Palomares <cpalomar@microsoft.com>
Co-authored-by: RupengLiu <rliu1211@terpmail.umd.edu>
Co-authored-by: msyyc <70930885+msyyc@users.noreply.github.com>
@github-actions github-actions bot locked and limited conversation to collaborators Apr 12, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@richardpark-msft richardpark-msft

Labels
bug This issue requires a change to an existing behavior in the product in order to be resolved. Client This issue points to a problem in the data-plane of the library. Service Bus
Projects
None yet
Milestone
[2021] July
Development

No branches or pull requests
3 participants
@HarshaNalluru @ramya-rao-a @richardpark-msft