feat: support scale SourceBackfill executor #16825

xxchan · 2024-05-20T05:25:24Z

Signed-off-by: xxchan xxchan22f@gmail.com

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

part of #16003

Their are 2 main things to consider:

Like Source, SourceBackfill need to migrate the connector splits. And the new assignment need to be aligned with the upstream. This part is relatively trivial.
SourceBackfill is connected to the upstream Source with a NoShuffle. This makes it trickier, as the logic of scaling NoShuffle is relatively complex (scaling upstream vs scaling downstream is different). I'm not sure I've done this part correct.

Added sim tests (low-level reschedule) and manual e2e tests (high-level ALTER PARALLELISM).

TODO: think whether we need to change this after Problem of online scaling for source backfill #18300. We will only need to consider scaling during recovery, but not online scaling.
- nothing to change, because it's handled by reschedule_lock

Checklist

I have written necessary rustdoc comments
I have added necessary unit tests and integration tests
I have added test labels as necessary. See details.
I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
All checks passed in ./risedev check (or alias, ./risedev c)
My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

xxchan · 2024-06-10T09:29:33Z

feat: support scale SourceBackfill executor #16825 👈
feat: track progress for SourceBackfill (blocking DDL) #18112 : 1 other dependent PR (#18355 )
refactor: use high watermark to finish backfill faster #18342 : 1 other dependent PR (#18368 )
refactor: add target_offsets to determinine if source backfill finished #18297
refactor: simplify SourceBackfill's memory state #18296
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @xxchan and the rest of your teammates on Graphite

src/meta/src/stream/scale.rs

xxchan · 2024-09-04T02:09:19Z

Test failed. This is the problem described in #18300, and can be solved after #18112

After rebasing, it passed.

shanicky

Scale part lgtm

We may need to conduct actual tests on the behavior of sources with split reduction like Kinesis during scaling (accompanied by split reduction). #9961

Signed-off-by: xxchan <xxchan22f@gmail.com> add notes Signed-off-by: xxchan <xxchan22f@gmail.com> Merge remote-tracking branch 'origin/main' into xxchan/scale-source add sim test discard minor changes Signed-off-by: xxchan <xxchan22f@gmail.com>

xxchan · 2024-09-06T07:20:46Z

We may need to conduct actual tests on the behavior of sources with split reduction like Kinesis during scaling (accompanied by split reduction). #9961

#15994

BugenZhao · 2024-09-06T08:18:49Z

src/meta/src/rpc/ddl_controller.rs

    pub async fn alter_parallelism(
        &self,
        table_id: u32,
        parallelism: PbTableParallelism,
        mut deferred: bool,
    ) -> MetaResult<()> {
+        tracing::info!("alter parallelism");


Suggested change

tracing::info!("alter parallelism");

BugenZhao · 2024-09-06T08:24:02Z

e2e_test/source_inline/kafka/shared_source.slt

+# # Note: the parallelism depends on the risedev profile.
+# # So scale tests below are commented out.


I think these unenforced tests can be stale quickly. Shall we remove this as we've implemented madsim tests?

github-actions bot added Invalid PR Title ci/run-e2e-single-node-tests labels May 20, 2024

xxchan mentioned this pull request May 20, 2024

Tracking issue for shared source #16003

Closed

21 tasks

xxchan force-pushed the xxchan/scale-source branch from 75c65af to f7f957f Compare June 10, 2024 09:29

xxchan force-pushed the xxchan/scale-source branch from f7f957f to 710adf4 Compare July 2, 2024 06:21

xxchan changed the title ~~support scaling source backfill~~ feat[DRAFT]: support scaling source backfill Jul 2, 2024

github-actions bot added the type/feature label Jul 2, 2024

xxchan force-pushed the xxchan/scale-source branch from 3493eaa to 3311bdc Compare July 10, 2024 05:58

shanicky reviewed Jul 17, 2024

View reviewed changes

src/meta/src/stream/scale.rs Outdated Show resolved Hide resolved

xxchan changed the title ~~feat[DRAFT]: support scaling source backfill~~ feat[DRAFT]: support scale source backfill Jul 31, 2024

xxchan force-pushed the xxchan/scale-source branch 3 times, most recently from 9509e3f to 3cc05b4 Compare July 31, 2024 06:56

xxchan changed the base branch from main to 07-31-refactor_add_some_comments_for_scaling July 31, 2024 06:56

xxchan mentioned this pull request Jul 31, 2024

refactor: add some comments for scaling #17879

Merged

9 tasks

xxchan force-pushed the 07-31-refactor_add_some_comments_for_scaling branch from 46b90b9 to 441460b Compare July 31, 2024 07:05

xxchan force-pushed the xxchan/scale-source branch from 3cc05b4 to c14100d Compare July 31, 2024 07:05

Base automatically changed from 07-31-refactor_add_some_comments_for_scaling to main July 31, 2024 08:00

xxchan force-pushed the xxchan/scale-source branch 6 times, most recently from a681dc0 to ff480ff Compare August 6, 2024 03:32

xxchan changed the title ~~feat[DRAFT]: support scale source backfill~~ feat: support scale source backfill Aug 6, 2024

github-actions bot removed the Invalid PR Title label Aug 6, 2024

xxchan marked this pull request as ready for review August 6, 2024 03:33

xxchan requested a review from a team as a code owner August 6, 2024 03:33

xxchan requested a review from lmatz August 6, 2024 03:33

xxchan force-pushed the xxchan/drop-db branch from 323feeb to 15d8287 Compare September 2, 2024 06:04

xxchan force-pushed the xxchan/scale-source branch 3 times, most recently from 32c62c6 to f430814 Compare September 2, 2024 07:10

xxchan force-pushed the xxchan/drop-db branch from 8f0f545 to 1854392 Compare September 3, 2024 06:30

xxchan force-pushed the xxchan/scale-source branch from f430814 to 2391b38 Compare September 3, 2024 06:31

xxchan requested a review from shanicky September 3, 2024 06:34

Base automatically changed from xxchan/drop-db to main September 3, 2024 08:29

xxchan force-pushed the xxchan/scale-source branch from 2391b38 to 878e494 Compare September 3, 2024 10:08

xxchan force-pushed the xxchan/scale-source branch from 878e494 to c0a2cc1 Compare September 4, 2024 02:10

xxchan changed the base branch from main to 08-16-feat_track_progress_for_sourcebackfill September 4, 2024 02:11

This was referenced Sep 4, 2024

feat: track progress for SourceBackfill (blocking DDL) #18112

Merged

feat: disallow online scaling for shared source #18355

Closed

xxchan force-pushed the 08-16-feat_track_progress_for_sourcebackfill branch from e2fd423 to a54a89f Compare September 4, 2024 02:32

xxchan force-pushed the xxchan/scale-source branch 2 times, most recently from be7279c to e4499af Compare September 4, 2024 02:35

Base automatically changed from 08-16-feat_track_progress_for_sourcebackfill to main September 6, 2024 05:28

shanicky approved these changes Sep 6, 2024

View reviewed changes

xxchan added 2 commits September 6, 2024 14:54

support scaling source backfill

d5f5d1a

Signed-off-by: xxchan <xxchan22f@gmail.com> add notes Signed-off-by: xxchan <xxchan22f@gmail.com> Merge remote-tracking branch 'origin/main' into xxchan/scale-source add sim test discard minor changes Signed-off-by: xxchan <xxchan22f@gmail.com>

nit

f825c33

xxchan force-pushed the xxchan/scale-source branch from e4499af to f825c33 Compare September 6, 2024 06:54

xxchan enabled auto-merge September 6, 2024 07:22

BugenZhao approved these changes Sep 6, 2024

View reviewed changes

xxchan added this pull request to the merge queue Sep 6, 2024

Merged via the queue into main with commit e2c89f4 Sep 6, 2024
31 of 32 checks passed

xxchan deleted the xxchan/scale-source branch September 6, 2024 08:53

xxchan mentioned this pull request Nov 21, 2024

Support ALTER SOURCE SET PARALLELISM #19508

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: support scale SourceBackfill executor #16825

feat: support scale SourceBackfill executor #16825

xxchan commented May 20, 2024 •

edited

Loading

xxchan commented Jun 10, 2024 •

edited

Loading

xxchan commented Sep 4, 2024 •

edited

Loading

shanicky left a comment •

edited

Loading

xxchan commented Sep 6, 2024

BugenZhao Sep 6, 2024

BugenZhao Sep 6, 2024

		# # Note: the parallelism depends on the risedev profile.
		# # So scale tests below are commented out.

feat: support scale SourceBackfill executor #16825

feat: support scale SourceBackfill executor #16825

Conversation

xxchan commented May 20, 2024 • edited Loading

What's changed and what's your intention?

Checklist

Documentation

Release note

xxchan commented Jun 10, 2024 • edited Loading

xxchan commented Sep 4, 2024 • edited Loading

shanicky left a comment • edited Loading

Choose a reason for hiding this comment

xxchan commented Sep 6, 2024

BugenZhao Sep 6, 2024

Choose a reason for hiding this comment

BugenZhao Sep 6, 2024

Choose a reason for hiding this comment

xxchan commented May 20, 2024 •

edited

Loading

xxchan commented Jun 10, 2024 •

edited

Loading

xxchan commented Sep 4, 2024 •

edited

Loading

shanicky left a comment •

edited

Loading