feat: new table stream graph #12240

shanicky · 2023-09-12T13:23:27Z

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

~~WIP, Some details require further discussion~~

with external source

no user defined primary key

dev=> explain create table t (v1 int)  with (
    connector = 'datagen',
    fields.v1.kind = 'sequence',
    fields.v1.start = '1',
    fields.v1.end = '1000',
    datagen.split.num = '5',
) FORMAT PLAIN ENCODE JSON ;
                                                         QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------
 StreamMaterialize { columns: [v1, _row_id(hidden)], stream_key: [_row_id], pk_columns: [_row_id], pk_conflict: Overwrite }
 └─StreamRowIdGen { row_id_index: 1 }
   └─StreamUnion { all: true }
     ├─StreamExchange [no_shuffle] { dist: SomeShard }
     │ └─StreamSource { source: t, columns: [v1, _row_id] }
     └─StreamExchange { dist: HashShard(_row_id) }
       └─StreamDml { columns: [v1, _row_id] }
         └─StreamSource
(8 rows)

append only

dev=> explain create table t (v1 int) append only  with (
    connector = 'datagen',
    fields.v1.kind = 'sequence',
    fields.v1.start = '1',
    fields.v1.end = '1000',
    datagen.split.num = '5',
) FORMAT PLAIN ENCODE JSON ;
                                                        QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------
 StreamMaterialize { columns: [v1, _row_id(hidden)], stream_key: [_row_id], pk_columns: [_row_id], pk_conflict: NoCheck }
 └─StreamRowIdGen { row_id_index: 1 }
   └─StreamUnion { all: true }
     ├─StreamExchange [no_shuffle] { dist: SomeShard }
     │ └─StreamSource { source: t, columns: [v1, _row_id] }
     └─StreamExchange [no_shuffle] { dist: SomeShard }
       └─StreamDml { columns: [v1, _row_id] }
         └─StreamSource
(8 rows)

user defined primary key

dev=> explain create table t (v1 int primary key) with (
    connector = 'datagen',
    fields.v1.kind = 'sequence',
    fields.v1.start = '1',
    fields.v1.end = '1000',
    datagen.split.num = '5',
) FORMAT PLAIN ENCODE JSON ;
                                           QUERY PLAN
-------------------------------------------------------------------------------------------------
 StreamMaterialize { columns: [v1], stream_key: [v1], pk_columns: [v1], pk_conflict: Overwrite }
 └─StreamUnion { all: true }
   ├─StreamExchange { dist: HashShard(v1) }
   │ └─StreamSource { source: t, columns: [v1] }
   └─StreamExchange { dist: HashShard(v1) }
     └─StreamDml { columns: [v1] }
       └─StreamSource
(7 rows)

without external source

The result of removing the source fragment (even though it might seem strange).

no user defined primary key

append only

user defined primary key

Checklist

I have written necessary rustdoc comments
I have added necessary unit tests and integration tests
All checks passed in ./risedev check (or alias, ./risedev c)

st1page · 2023-09-19T08:49:46Z

src/frontend/src/optimizer/mod.rs

+            dml_node = match kind {
+                PrimaryKeyKind::UserDefinedPrimaryKey | PrimaryKeyKind::RowIdAsPrimaryKey => {
+                    RequiredDist::hash_shard(pk_column_indices)
+                        .enforce_if_not_satisfies(dml_node, &Order::any())?
+                }
+                PrimaryKeyKind::AppendOnly => StreamExchange::new_no_shuffle(dml_node).into(),
+            };


The dist keys changes in planner test is because now we generate exchange here but previouesly we add it in materialize node. with

risingwave/src/frontend/src/optimizer/mod.rs

Line 482 in aa5e798

let table_required_dist = {

shanicky · 2023-09-19T09:52:39Z

I used the following code to test its compatibility with the current main branch, and it seems to have no problems.

on main

dev=> create table t_no_pk (v1 int)  with (
    connector = 'datagen',
    fields.v1.kind = 'sequence',
    fields.v1.start = '1',
    fields.v1.end = '1000',
    datagen.split.num = '5',
) FORMAT PLAIN ENCODE JSON ;
CREATE_TABLE
dev=> create table t_ap (v1 int) append only  with (
    connector = 'datagen',
    fields.v1.kind = 'sequence',
    fields.v1.start = '1',
    fields.v1.end = '1000',
    datagen.split.num = '5',
) FORMAT PLAIN ENCODE JSON ;
NOTICE:  APPEND ONLY TABLE is currently an experimental feature.
CREATE_TABLE
dev=> create table t_pk (v1 int primary key) with (
    connector = 'datagen',
    fields.v1.kind = 'sequence',
    fields.v1.start = '1',
    fields.v1.end = '1000',
    datagen.split.num = '5',
) FORMAT PLAIN ENCODE JSON ;
CREATE_TABLE
dev=> create table tt_no_pk (v1 int);
CREATE_TABLE
dev=> create table tt_ap (v1 int) append only;
NOTICE:  APPEND ONLY TABLE is currently an experimental feature.
CREATE_TABLE
dev=> create table tt_pk (v1 int primary key);
CREATE_TABLE
dev=> insert into tt_no_pk values (1);
INSERT 0 1
dev=> insert into tt_ap values (1);
INSERT 0 1
dev=> insert into tt_pk values (1);
INSERT 0 1
dev=>

restart to current branch

dev=> alter table tt_no_pk add column v2 int;
ALTER_TABLE
dev=> select * from tt_no_pk ;
 v1 | v2
----+----
  1 |
(1 row)

dev=> alter table tt_ap add column v2 int ;
ALTER_TABLE
dev=> select * from tt_ap ;
 v1 | v2
----+----
  1 |
(1 row)

dev=> alter table tt_pk add column v2 int ;
ALTER_TABLE
dev=> select * from tt_pk ;
 v1 | v2
----+----
  1 |
(1 row)

dev=> alter table t_no_pk add column v2 int;
ALTER_TABLE
dev=> select * from t_no_pk ;
dev=> alter table t_ap add column v2 int;
ALTER_TABLE
dev=> select * from t_no_pk ;
dev=> select * from t_ap;
dev=> alter table t_pk add column v2 int;
ALTER_TABLE

codecov · 2023-09-19T13:43:57Z

Codecov Report

Attention: 13 lines in your changes are missing coverage. Please review.

Comparison is base (cde91a0) 67.86% compared to head (9a89780) 67.87%.
Report is 14 commits behind head on main.

Files	Patch %	Lines
src/meta/service/src/ddl_service.rs	0.00%	5 Missing ⚠️
src/frontend/src/optimizer/mod.rs	97.45%	3 Missing ⚠️
...ntend/src/optimizer/plan_node/stream_row_id_gen.rs	66.66%	3 Missing ⚠️
src/common/src/hash/consistent_hash/vnode.rs	66.66%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #12240      +/-   ##
==========================================
+ Coverage   67.86%   67.87%   +0.01%     
==========================================
  Files        1526     1526              
  Lines      260047   260156     +109     
==========================================
+ Hits       176468   176588     +120     
+ Misses      83579    83568      -11

Flag	Coverage Δ
rust	`67.87% <91.44%> (+0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

BugenZhao · 2023-09-20T06:22:00Z

Please ping me once this PR is ready for review. :)

shanicky · 2023-09-20T06:27:54Z

Please ping me once this PR is ready for review. :)

This appears to be ready for review now 🥵

shanicky · 2023-09-20T06:29:08Z

src/frontend/src/optimizer/mod.rs

+
+            StreamUnion::new(Union {
+                all: true,
+                inputs: vec![external_source_node, dml_node],


I’m not sure which id needs to be specified as the source here. In the case of ‘with connector’, we have two sources.

BugenZhao

The plans look great!

BugenZhao · 2023-09-20T06:37:36Z

src/common/src/hash/consistent_hash/vnode.rs

@@ -119,9 +119,10 @@ impl VirtualNode {
        if let Ok(idx) = keys.iter().exactly_one()
            && let ArrayImpl::Serial(serial_array) = &**data_chunk.column_at(*idx)
        {
+            let fallback_row_id = Serial::from(rand::random::<i64>());


Why do we need this? And also for compute_row?

In the scenario where there is no user-defined primary key, the user’s insert operation, upon entering through the DML node, first undergoes a shuffle before entering the union, and only then is the row ID node. Therefore, at the time of the shuffle, the row id column is still empty. This was previously handled by using unwrap_or_default, but this would shuffle all the traffic to vnode 0, resulting in a severe traffic skew. Therefore, we’ve adopted the random method to randomize it a bit, specifically, one chunk randomizes one vnode.

~~Indeed, we need to add support for compute_row. I’m going to make some modifications.~~ It seems like it might not be necessary. 🤔

This looks too hack to me. 🥵 It could be surprising that this is not a pure function. Since the motivation here is to avoid data skew, I guess using the hash code of the row is way much better as it guarantees stability at least.

Can we add another RowIdGen here as well? We just need to ensure that there's no overlapping with the latter one, which can be done by allocating a flag of one bit for this information I guess.

Makes sense, let’s use a hash of the entire row as a fallback when there’s no row id. 👍

Can we add another RowIdGen here as well? We just need to ensure that there's no overlapping with the latter one, which can be done by allocating a flag of one bit for this information I guess.

A little strange to me because we can not know which data source can give more records

BugenZhao · 2023-09-20T06:39:21Z

src/frontend/src/optimizer/plan_node/stream_union.rs

@@ -37,7 +37,6 @@ impl StreamUnion {
    pub fn new(logical: generic::Union<PlanRef>) -> Self {
        let inputs = &logical.inputs;
        let dist = inputs[0].distribution().clone();
-        assert!(inputs.iter().all(|input| *input.distribution() == dist));


Any violation of this assertion? 👀

The distribution here may be different 🤔

I'm wondering what the distribution will be in this case. 🤔 SomeShard?

~~Yeah, it's okay here. I'll add assert back in.~~
It doesn't seem to work. When executing the following statement, one side is someshard and the other side is hashshard.

CREATE TABLE msrc (v INT) WITH ( connector = 'datagen', fields.v.kind = 'sequence', fields.v.start = '1', fields.v.end = '10', datagen.rows.per.second='15', datagen.split.num = '1' ) FORMAT PLAIN ENCODE JSON;

input StreamExchange [no_shuffle] { dist: SomeShard } -> [SomeShard] input StreamExchange { dist: HashShard(_row_id) } -> [1]

Some shard and hash shard should be compatible here with -> some shard. I’ve handled it, and an assert will still be performed for typical unions.

BugenZhao · 2023-09-20T06:50:43Z

I guess it's time to utilize the BarrierRecv executor! 😄

#8595

stdrc

Could you plz elaborate more on the background and motivation? Also may be better to have a more descriptive PR title.

src/frontend/planner_test/tests/testdata/output/generated_columns.yaml

st1page · 2023-11-13T11:30:51Z

src/common/src/hash/consistent_hash/vnode.rs

+        if project.is_empty() {
+            return project.row().hash(Crc32FastBuilder).into();
+        }


pass a full indices here is better 🤔

st1page · 2023-11-13T11:34:12Z

src/frontend/src/optimizer/plan_node/stream_row_id_gen.rs

 }

 impl StreamRowIdGen {
    pub fn new(input: PlanRef, row_id_index: usize) -> Self {
-        let distribution = if input.append_only() {
+        if input.append_only() {


I guess we can not check in the new function, instead, passing the flag is needed. DmlExecutor is non-append only.

… optimize data exchange between fragments.

…amSource, StreamDml, StreamCdcTableScan & StreamWatermarkFilter

src/frontend/src/optimizer/plan_node/stream_row_id_gen.rs

…distribution in `new_with_dist` function.

… for MySQL CDC sources

shanicky · 2023-11-14T11:21:40Z

For chunks where the row id is none, hashing each row individually can break up the chunk and create a time difference in the calculation of data within the same chunk, leading to unexpected results (like ‘distinct on’ without ‘order by’). For most of our end-to-end tests, this might produce random results. Some potential solutions we can consider are:

Creating a random vnode for chunks where the row id is none and transferring it.
Inserting a row id generator at the entry point of each union and then centrally regenerating it after the union.
Adding ‘order by’ and ‘row sort’ to all necessary end-to-end tests. 🥵

BugenZhao · 2023-11-15T05:27:22Z

For chunks where the row id is none, hashing each row individually can break up the chunk

That's true. 🥵 But why the random vnode approach does not encounter this problem? Did we use the same vnode for all rows in a single chunk?

BugenZhao

Generally LGTM!

BugenZhao · 2023-11-15T05:36:18Z

I guess it's time to utilize the BarrierRecv executor! 😄

#8595

BTW, I realize that BarrierRecv can be a backend-only executor. Every executor that does not have input should be attached with a BarrierRecv as its input. So it seems not to be a breaking change.

shanicky · 2023-11-15T07:14:17Z

For chunks where the row id is none, hashing each row individually can break up the chunk

That's true. 🥵 But why the random vnode approach does not encounter this problem? Did we use the same vnode for all rows in a single chunk?

Oh, I think the possible reason could be that many of our tests are run on initial tables, which haven’t been scaled or undergone vnode redistribution. So according to our current implementation, when the parallelism of the entire graph is the same, the vnode allocation is also sequential, like (1,2,3), (4,5,6), etc. In the previous implementation, after the DML comes in, each row id generator would add a row id according to its own allocated vnodes. Hence, the row ids assigned to a data chunk coming in are also (1,2,3,1,2,3,1,2,3), and after being passed through the serial type, they are also distributed to the following (1,2,3) operator. This keeps these rows together.

But if we change it to calculate the hash based on the entire row, it would lead to a very scattered distribution of hashes, so they would be randomly assigned to the subsequent operators.

The data in our e2e tests is quite limited, so we won’t encounter this problem before. 🥵

BugenZhao · 2023-11-15T07:22:28Z

The data in our e2e tests is quite limited, so we won’t encounter this problem before. 🥵

That's interesting. If I understand you correctly, records fewer than 256/#pu were kept in order. Such a coincidence!

Signed-off-by: Shanicky Chen <peng@risingwave-labs.com>

shanicky · 2023-11-15T08:06:01Z

retest this pr's compatibility with the current main branch

run in main branch

dev=> create table t_no_pk (v1 int)  with (
    connector = 'datagen',
    fields.v1.kind = 'sequence',
    fields.v1.start = '1',
    fields.v1.end = '1000',
    datagen.split.num = '5',
) FORMAT PLAIN ENCODE JSON ;
CREATE_TABLE
dev=> create table t_ap (v1 int) append only  with (
    connector = 'datagen',
    fields.v1.kind = 'sequence',
    fields.v1.start = '1',
    fields.v1.end = '1000',
    datagen.split.num = '5',
) FORMAT PLAIN ENCODE JSON ;
NOTICE:  APPEND ONLY TABLE is currently an experimental feature.
CREATE_TABLE
dev=> create table t_pk (v1 int primary key) with (
    connector = 'datagen',
    fields.v1.kind = 'sequence',
    fields.v1.start = '1',
    fields.v1.end = '1000',
    datagen.split.num = '5',
) FORMAT PLAIN ENCODE JSON ;
CREATE_TABLE
dev=> create table tt_no_pk (v1 int);
CREATE_TABLE
dev=> create table tt_ap (v1 int) append only;
NOTICE:  APPEND ONLY TABLE is currently an experimental feature.
CREATE_TABLE
dev=> create table tt_pk (v1 int primary key);
CREATE_TABLE
dev=> insert into tt_no_pk select * from generate_series(0, 9999, 1);
INSERT 0 10000
dev=> select count(*) from tt_no_pk;
 count
-------
 10000
(1 row)

dev=> insert into tt_ap select * from generate_series(0, 9999, 1);
INSERT 0 10000
dev=> insert into tt_pk select * from generate_series(0, 9999, 1);
INSERT 0 10000

restart to current branch

dev=> select count(*) from t_no_pk;
 count
-------
  1000
(1 row)

dev=> alter table t_no_pk add column v2 int;
ALTER_TABLE

dev=> select count(*) from t_no_pk;
 count
-------
  1000
(1 row)

dev=> select count(*) from t_ap;
 count
-------
  1000
(1 row)

dev=> alter table t_ap add column v2 int;
ALTER_TABLE
dev=> select count(*) from t_ap;
 count
-------
  1000
(1 row)

dev=> alter table t_pk add column v2 int;
ALTER_TABLE
dev=> select count(*) from t_pk;
 count
-------
  1000
(1 row)

dev=> alter table tt_pk add column v2 int;
ALTER_TABLE

dev=> select count(*) from tt_pk;
 count
-------
 10000
(1 row)

dev=> alter table tt_no_pk add column v2 int;
ALTER_TABLE
dev=> select count(*) from tt_no_pk;
 count
-------
 10000
(1 row)

dev=> alter table tt_ap add column v2 int;
ALTER_TABLE
dev=> select count(*) from tt_ap;
 count
-------
 10000
(1 row)

dev=> insert into t_ap values (100000000);
INSERT 0 1
dev=> select count(*) from t_ap;
 count
-------
  1001
(1 row)

BugenZhao · 2023-11-15T10:01:30Z

retest this pr's compatibility with the current main branch

Cool! Is it possible to add it to CI compatibility test? cc @kwannoel

kwannoel · 2023-11-16T08:33:13Z

retest this pr's compatibility with the current main branch

Cool! Is it possible to add it to CI compatibility test? cc @kwannoel

#13403 will require testing against 4 latest versions. That will be a prerequisite to test this PR.

What additional things need to be tested? I saw some ALTER etc... @shanicky you may add these to the backwards compat test.

If it's just normal DML, DQL, DDL, the backwards compat test already uses nexmark and tpch query for testing.

This reverts commit 48bf62e.

This reverts commit aaf41e6.

github-actions bot added the Invalid PR Title label Sep 12, 2023

shanicky changed the title ~~[wip] feat: new table dag~~ [wip] feat: new table stream graph Sep 12, 2023

shanicky force-pushed the peng/new-table-dag branch from 1d635a5 to dbbe9db Compare September 19, 2023 06:00

shanicky marked this pull request as ready for review September 19, 2023 08:09

st1page reviewed Sep 19, 2023

View reviewed changes

github-actions bot added the breaking-change label Sep 19, 2023

github-actions bot removed the breaking-change label Sep 19, 2023

shanicky changed the title ~~[wip] feat: new table stream graph~~ feat: new table stream graph Sep 19, 2023

github-actions bot added type/feature and removed Invalid PR Title labels Sep 19, 2023

shanicky commented Sep 20, 2023

View reviewed changes

BugenZhao reviewed Sep 20, 2023

View reviewed changes

shanicky force-pushed the peng/new-table-dag branch from a684fd3 to e66762b Compare September 21, 2023 08:46

shanicky force-pushed the peng/new-table-dag branch 2 times, most recently from 44e3ba4 to 692035c Compare October 23, 2023 06:56

shanicky force-pushed the peng/new-table-dag branch from 692035c to 3cbd350 Compare October 23, 2023 15:32

shanicky force-pushed the peng/new-table-dag branch from 3cbd350 to db7728e Compare October 30, 2023 16:25

shanicky mentioned this pull request Oct 31, 2023

feat: Experimentally introducing sink into table #13185

Merged

3 tasks

shanicky force-pushed the peng/new-table-dag branch 2 times, most recently from 2e994db to d965f44 Compare November 7, 2023 05:39

stdrc reviewed Nov 7, 2023

View reviewed changes

src/frontend/planner_test/tests/testdata/output/generated_columns.yaml Show resolved Hide resolved

shanicky force-pushed the peng/new-table-dag branch 3 times, most recently from 531c202 to 2911174 Compare November 13, 2023 07:04

st1page reviewed Nov 13, 2023

View reviewed changes

shanicky added 2 commits November 14, 2023 17:06

Added StreamExchange operator with HashShard distribution strategy to…

e12d2fa

… optimize data exchange between fragments.

Add StreamUnion & StreamExchange operators for shuffling, modify Stre…

0b87816

…amSource, StreamDml, StreamCdcTableScan & StreamWatermarkFilter

shanicky force-pushed the peng/new-table-dag branch from e5af73b to 0b87816 Compare November 14, 2023 09:09

st1page approved these changes Nov 14, 2023

View reviewed changes

src/frontend/src/optimizer/plan_node/stream_row_id_gen.rs Outdated Show resolved Hide resolved

st1page requested a review from BugenZhao November 14, 2023 09:14

shanicky added 2 commits November 14, 2023 17:32

Remove commented out portion, refactor code to clone and use input's …

9eb0db1

…distribution in `new_with_dist` function.

Skip empty sources, fill ID with connector, generate random server ID…

2e447d8

… for MySQL CDC sources

shanicky force-pushed the peng/new-table-dag branch from c6b774e to e1f9c2b Compare November 14, 2023 16:39

BugenZhao approved these changes Nov 15, 2023

View reviewed changes

BugenZhao mentioned this pull request Nov 15, 2023

streaming: leverage barrier recv executor #13432

Open

shanicky force-pushed the peng/new-table-dag branch from e1f9c2b to 5059920 Compare November 15, 2023 06:52

try update e2e

9a89780

Signed-off-by: Shanicky Chen <peng@risingwave-labs.com>

shanicky force-pushed the peng/new-table-dag branch from 5059920 to 9a89780 Compare November 15, 2023 07:27

shanicky added this pull request to the merge queue Nov 15, 2023

Merged via the queue into main with commit 48bf62e Nov 15, 2023
27 checks passed

shanicky deleted the peng/new-table-dag branch November 15, 2023 08:37

st1page mentioned this pull request Nov 16, 2023

bug(stream): group_top_n/main.slt unexpected results in end-to-end test (parallel) #13461

Closed

StrikeW added a commit that referenced this pull request Nov 20, 2023

Revert "feat: new table stream graph (#12240)"

aaf41e6

This reverts commit 48bf62e.

StrikeW added a commit that referenced this pull request Nov 20, 2023

Revert "Revert "feat: new table stream graph (#12240)""

e77b89b

This reverts commit aaf41e6.

xxchan mentioned this pull request Jul 30, 2024

disallow usage of jsonb/map type in group by / order by / primary key #7981

Open

st1page mentioned this pull request Sep 23, 2024

feat: allow append only table with pk #18634

Merged

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: new table stream graph #12240

feat: new table stream graph #12240

shanicky commented Sep 12, 2023 •

edited

Loading

st1page Sep 19, 2023 •

edited

Loading

shanicky commented Sep 19, 2023

codecov bot commented Sep 19, 2023 •

edited

Loading

BugenZhao commented Sep 20, 2023

shanicky commented Sep 20, 2023

shanicky Sep 20, 2023

BugenZhao left a comment

BugenZhao Sep 20, 2023

shanicky Sep 20, 2023 •

edited

Loading

BugenZhao Oct 17, 2023

shanicky Nov 7, 2023

st1page Nov 7, 2023

BugenZhao Sep 20, 2023

shanicky Sep 20, 2023

BugenZhao Oct 17, 2023

shanicky Nov 8, 2023 •

edited

Loading

shanicky Nov 13, 2023

BugenZhao commented Sep 20, 2023

stdrc left a comment

st1page Nov 13, 2023

st1page Nov 13, 2023

shanicky commented Nov 14, 2023

BugenZhao commented Nov 15, 2023

BugenZhao left a comment

BugenZhao commented Nov 15, 2023

shanicky commented Nov 15, 2023

BugenZhao commented Nov 15, 2023

shanicky commented Nov 15, 2023

BugenZhao commented Nov 15, 2023

kwannoel commented Nov 16, 2023

feat: new table stream graph #12240

feat: new table stream graph #12240

Conversation

shanicky commented Sep 12, 2023 • edited Loading

What's changed and what's your intention?

with external source

no user defined primary key

append only

user defined primary key

without external source

no user defined primary key

append only

user defined primary key

Checklist

st1page Sep 19, 2023 • edited Loading

Choose a reason for hiding this comment

shanicky commented Sep 19, 2023

codecov bot commented Sep 19, 2023 • edited Loading

Codecov Report

BugenZhao commented Sep 20, 2023

shanicky commented Sep 20, 2023

Choose a reason for hiding this comment

BugenZhao left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shanicky Sep 20, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shanicky Nov 8, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BugenZhao commented Sep 20, 2023

stdrc left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shanicky commented Nov 14, 2023

BugenZhao commented Nov 15, 2023

BugenZhao left a comment

Choose a reason for hiding this comment

BugenZhao commented Nov 15, 2023

shanicky commented Nov 15, 2023

BugenZhao commented Nov 15, 2023

shanicky commented Nov 15, 2023

BugenZhao commented Nov 15, 2023

kwannoel commented Nov 16, 2023

shanicky commented Sep 12, 2023 •

edited

Loading

st1page Sep 19, 2023 •

edited

Loading

codecov bot commented Sep 19, 2023 •

edited

Loading

shanicky Sep 20, 2023 •

edited

Loading

shanicky Nov 8, 2023 •

edited

Loading