forward port workarround for queue populator batch getting stuck #2544

Kerkesni · 2024-09-24T14:58:37Z

7.x fix was not forward ported to avoid introducing memory leaks into the code. Instead of just unblocking the next batch processing call and leaving the stuck one alive, now we shutdown the whole process.

7.x fix commit: dfcc2f5

Issue: BB-526

bert-e · 2024-09-24T14:58:42Z

Hello kerkesni,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request `TBA`
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

codecov · 2024-09-24T15:01:14Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 69.50%. Comparing base (b8bd575) to head (3779062).
Report is 13 commits behind head on development/8.7.

Additional details and impacted files

Files with missing lines	Coverage Δ
lib/constants.js	`100.00% <ø> (ø)`
lib/queuePopulator/LogReader.js	`40.80% <100.00%> (+2.02%)`	⬆️
lib/queuePopulator/QueuePopulator.js	`32.87% <100.00%> (+0.61%)`	⬆️

... and 2 files with indirect coverage changes

Components	Coverage Δ
Bucket Notification	`66.66% <ø> (ø)`
Core Library	`74.17% <100.00%> (-0.16%)`	⬇️
Ingestion	`69.24% <ø> (ø)`
Lifecycle	`75.07% <ø> (ø)`
Oplog Populator	`82.95% <ø> (ø)`
Replication	`57.42% <ø> (ø)`
Bucket Scanner	`85.76% <ø> (ø)`

@@                 Coverage Diff                 @@
##           development/8.7    #2544      +/-   ##
===================================================
- Coverage            69.56%   69.50%   -0.06%     
===================================================
  Files                  194      194              
  Lines                12792    12803      +11     
===================================================
  Hits                  8899     8899              
- Misses                3883     3894      +11     
  Partials                10       10

Flag	Coverage Δ
api:retry	`9.65% <0.00%> (-0.01%)`	⬇️
api:routes	`9.54% <0.00%> (-0.01%)`	⬇️
bucket-scanner	`85.76% <ø> (ø)`
ingestion	`12.53% <36.36%> (+0.02%)`	⬆️
lib	`7.53% <0.00%> (-0.01%)`	⬇️
lifecycle	`19.41% <0.00%> (-0.02%)`	⬇️
notification	`0.88% <0.00%> (-0.01%)`	⬇️
replication	`18.94% <0.00%> (-0.11%)`	⬇️
unit	`43.53% <100.00%> (+0.20%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

bert-e · 2024-09-26T10:21:07Z

Request integration branches

Waiting for integration branch creation to be requested by the user.

To request integration branches, please comment on this pull request with the following command:

/create_integration_branches

Alternatively, the /approve and /create_pull_requests commands will automatically
create the integration branches.

bert-e · 2024-10-10T14:44:22Z

Incorrect fix version

The Fix Version/s in issue BB-526 contains:

8.6.50
8.7.0

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

8.7.0

Please check the Fix Version/s of BB-526, or the target
branch of this pull request.

bert-e · 2024-10-10T14:47:37Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

francoisferrand · 2024-10-11T14:58:01Z

lib/queuePopulator/LogReader.js

+            this.log.error('queue populator batch timeout', {
+                logStats: batchState.logStats,
+            });
+            process.emit('SIGTERM');


should we really "crash" ?
could we not just fail healthcheck? (may not be supported at the moment in s3c, but will need to anyway...)

As far as i know, S3C (precisely supervisord) doesn't support restarting a process on healthcheck failure.
I added logic for failing the healthcheck, but i'm still keeping the crashing behaviour.

7.x fix was not forward ported to avoid introducing memory leaks into the code. Instead of just unblocking the next batch processing call and leaving the stuck one alive, now we shutdown the whole process. 7.x fix commit: dfcc2f5 Issue: BB-526

Issue: BB-526

Kerkesni · 2024-10-22T09:41:37Z

/approve

bert-e · 2024-10-22T09:41:48Z

In the queue

The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.

The changeset will be merged in:

✔️ development/8.7

The following branches will NOT be impacted:

development/7.10
development/7.4
development/7.70
development/8.5
development/8.6

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of the admin team will
contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

Any commit you add on the source branch will trigger a new cycle after the
current queue is merged.
Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of the admin team now.

The following options are set: approve

bert-e · 2024-10-22T10:17:34Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/8.7

The following branches have NOT changed:

development/7.10
development/7.4
development/7.70
development/8.5
development/8.6

Please check the status of the associated issue BB-526.

Goodbye kerkesni.

Kerkesni force-pushed the improvement/BB-526 branch 2 times, most recently from f605ea3 to 85f130b Compare September 26, 2024 10:19

scality deleted a comment from bert-e Sep 26, 2024

Kerkesni force-pushed the improvement/BB-526 branch from 85f130b to 5a630a0 Compare September 26, 2024 10:41

Kerkesni force-pushed the improvement/BB-526 branch from 00b882b to 9c67ba8 Compare October 10, 2024 14:43

Kerkesni changed the base branch from development/8.6 to development/8.7 October 10, 2024 14:44

Kerkesni force-pushed the improvement/BB-526 branch from 9c67ba8 to 29b5481 Compare October 10, 2024 14:45

Kerkesni marked this pull request as ready for review October 10, 2024 14:46

Kerkesni force-pushed the improvement/BB-526 branch 2 times, most recently from d1b659e to 29b5481 Compare October 10, 2024 15:49

Kerkesni requested review from francoisferrand, williamlardier, KillianG and benzekrimaha October 10, 2024 15:51

KillianG approved these changes Oct 11, 2024

View reviewed changes

francoisferrand reviewed Oct 11, 2024

View reviewed changes

Kerkesni force-pushed the improvement/BB-526 branch 2 times, most recently from f5fcc9b to fa9f680 Compare October 14, 2024 13:27

Kerkesni added 2 commits October 14, 2024 15:35

fail healthcheck when batch processing gets stuck

9d1905c

Issue: BB-526

Kerkesni force-pushed the improvement/BB-526 branch from fa9f680 to 9d1905c Compare October 14, 2024 13:35

Kerkesni requested a review from francoisferrand October 14, 2024 13:40

williamlardier approved these changes Oct 15, 2024

View reviewed changes

only crash on S3C

3779062

Issue: BB-526

francoisferrand approved these changes Oct 21, 2024

View reviewed changes

bert-e merged commit 7692b3a into development/8.7 Oct 22, 2024
9 checks passed

bert-e deleted the improvement/BB-526 branch October 22, 2024 10:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

forward port workarround for queue populator batch getting stuck #2544

forward port workarround for queue populator batch getting stuck #2544

Kerkesni commented Sep 24, 2024 •

edited

Loading

bert-e commented Sep 24, 2024

codecov bot commented Sep 24, 2024 •

edited

Loading

bert-e commented Sep 26, 2024

bert-e commented Oct 10, 2024

bert-e commented Oct 10, 2024

francoisferrand Oct 11, 2024 •

edited

Loading

Kerkesni Oct 14, 2024

Kerkesni commented Oct 22, 2024

bert-e commented Oct 22, 2024

bert-e commented Oct 22, 2024

forward port workarround for queue populator batch getting stuck #2544

forward port workarround for queue populator batch getting stuck #2544

Conversation

Kerkesni commented Sep 24, 2024 • edited Loading

bert-e commented Sep 24, 2024

Hello kerkesni,

codecov bot commented Sep 24, 2024 • edited Loading

Codecov Report

bert-e commented Sep 26, 2024

Request integration branches

bert-e commented Oct 10, 2024

Incorrect fix version

bert-e commented Oct 10, 2024

Waiting for approval

francoisferrand Oct 11, 2024 • edited Loading

Choose a reason for hiding this comment

Kerkesni Oct 14, 2024

Choose a reason for hiding this comment

Kerkesni commented Oct 22, 2024

bert-e commented Oct 22, 2024

In the queue

bert-e commented Oct 22, 2024

Kerkesni commented Sep 24, 2024 •

edited

Loading

codecov bot commented Sep 24, 2024 •

edited

Loading

francoisferrand Oct 11, 2024 •

edited

Loading