feat: enhanced the retry logic by adding a random noise #4320
Conversation
WalkthroughThe recent changes introduce the Changes
Poem
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (invoked as PR comments)
Additionally, you can add CodeRabbit Configration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files ignored due to path filters (1)
Cargo.lockis excluded by!**/*.lock
Files selected for processing (2)
- src/common/procedure/Cargo.toml (1 hunks)
- src/common/procedure/src/local/runner.rs (2 hunks)
Files skipped from review due to trivial changes (1)
- src/common/procedure/Cargo.toml
Additional comments not posted (2)
src/common/procedure/src/local/runner.rs (2)
21-21: LGTM! The import is necessary for the new functionality.The addition of
rand::Rngis appropriate for introducing randomness into the retry intervals.
203-206: LGTM! Ensure the noise is appropriately bounded.The changes correctly introduce randomness into the retry delay. Verify that the noise, which is bounded by a quarter of the retry interval, is appropriate for your use case.
…delay to avoid retry storms
WenyXu
force-pushed
the
feat/random-noise-on-retry
branch
from
4071bf9
to
c7075e3
Compare
There was a problem hiding this comment.
Actionable comments posted: 0
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files ignored due to path filters (1)
Cargo.lockis excluded by!**/*.lock
Files selected for processing (2)
- src/common/procedure/Cargo.toml (1 hunks)
- src/common/procedure/src/local/runner.rs (2 hunks)
Files skipped from review as they are similar to previous changes (1)
- src/common/procedure/Cargo.toml
Additional comments not posted (2)
src/common/procedure/src/local/runner.rs (2)
21-21: Importedrand::Rngfor random noise in retry logic.The import of
rand::Rngis necessary for adding random noise to the retry delay.
203-207: Enhanced retry logic with random noise.The retry logic now includes random noise to avoid retry storms. This is a good enhancement to prevent simultaneous retries.
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #4320 +/- ##
==========================================
- Coverage 85.15% 84.89% -0.27%
==========================================
Files 1061 1061
Lines 188255 188260 +5
==========================================
- Hits 160313 159818 -495
- Misses 27942 28442 +500 |
| @@ -198,6 +200,11 @@ impl Runner { | |||
| ProcedureState::Retrying { error } => { | |||
| retry_times += 1; | |||
| if let Some(d) = retry.next() { | |||
| let millis = d.as_millis() as u64; | |||
| // Add random noise to the retry delay to avoid retry storms. | |||
| let noise = rand::thread_rng().gen_range(0..(millis / 4) + 1); | |||
There was a problem hiding this comment.
this one is typically called jitter
I hereby agree to the terms of the GreptimeDB CLA.
Refer to a related PR or issue link (optional)
What's changed and what's your intention?
Enhanced the retry logic by adding a random noise to the retry delay to avoid retry storms
Checklist
Summary by CodeRabbit