Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(comms/messaging): fix possible deadlock in outbound pipeline #4657

Merged

Conversation

sdbondi
Copy link
Member

@sdbondi sdbondi commented Sep 9, 2022

Description

  • Fixes possible rare deadlock when broadcasting many messages due to internal channel
  • Reduce number of inbound and outbound pipeline workers
  • Greatly reduce buffer size between inbound messaging and the inbound pipeline to allow for substream backpressure
  • Adds "last resort" timeout in outbound pipeline

Motivation and Context

The outbound pipeline could deadlock when all pipeline workers are busy, and the outbound sink service is full, causing the pipeline to wait for both a free executor slot and a free slot to send on the channel

How Has This Been Tested?

Memorynet, Manually: wallet stress tests (2 x wallets, 2 x base nodes), checked SAF message exchange

Copy link
Collaborator

@SWvheerden SWvheerden left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good

base_layer/p2p/src/config.rs Show resolved Hide resolved
Copy link
Contributor

@hansieodendaal hansieodendaal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice - please see the comments below.

comms/core/src/pipeline/outbound.rs Show resolved Hide resolved
base_layer/p2p/src/config.rs Show resolved Hide resolved
base_layer/p2p/src/config.rs Show resolved Hide resolved
@sdbondi sdbondi force-pushed the comms-fix-messaging-deadlock branch from 5f0ca06 to 2153f74 Compare September 12, 2022 09:00
Copy link
Collaborator

@stringhandler stringhandler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice

@stringhandler stringhandler merged commit 3fcc6a0 into tari-project:development Sep 12, 2022
@sdbondi sdbondi deleted the comms-fix-messaging-deadlock branch September 12, 2022 12:42
sdbondi added a commit to sdbondi/tari that referenced this pull request Sep 16, 2022
* development: (72 commits)
  fix: reinsert transactions from failed block (tari-project#4675)
  fix: stray clippy error (tari-project#4685)
  fix(wallet): mark mined_height as null when pending outputs are cancelled (tari-project#4686)
  chore: updated dependancies (tari-project#4684)
  fix(p2p): remove DETACH flag usage (tari-project#4682)
  fix(comms): simplify and remove possibility of deadlock from pipelines and substream close (tari-project#4676)
  feat(ci): add default CI and FFI testing with custom dispatch (tari-project#4672)
  chore: remove broken test (tari-project#4678)
  fix: fix potential race condition between add_block and sync (tari-project#4677)
  fix deadlock (tari-project#4674)
  fix: add burn funds command to console wallet (see issue tari-project#4547) (tari-project#4655)
  v0.38.3
  fix: fee estimate (tari-project#4656)
  fix(comms/messaging): fix possible deadlock in outbound pipeline (tari-project#4657)
  fix: replace Luhn checksum with DammSum (tari-project#4639)
  fix(core/sync): handle deadline timeouts by changing peer (tari-project#4649)
  fix(ci): libtor build on Ubuntu (tari-project#4644)
  chore: fix log (tari-project#4634)
  v0.38.2
  fix(comms/rpc): detect early close in all cases (tari-project#4647)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants