[IR][refactor] Convert loop_var into LoopIndexStmt #953

xumingkuan · 2020-05-11T21:17:57Z

Related PR = #932 (comment)

There are more places than I thought that need to be refactored...

[Click here for the format server]

xumingkuan · 2020-05-11T21:20:28Z

The offload pass and all backends still need to be modified.

If we put this pass before demote_dense_struct_fors, it also needs to be modified.

I think ideally we should put this pass right after lower (lower_ast). WDYT?

xumingkuan · 2020-05-11T21:21:34Z

taichi/ir/statements.h

@@ -217,19 +217,20 @@ class OffloadedStmt : public Stmt {

 class LoopIndexStmt : public Stmt {
 public:
+  Stmt *loop;
  int index;
  bool is_struct_for;


I think the field is_struct_for would be unnecessary.

Yes, please feel free to remove that.

yuanming-hu · 2020-05-11T21:27:08Z

The offload pass and all backends still need to be modified.

If we put this pass before demote_dense_struct_fors, it also needs to be modified.

I think ideally we should put this pass right after lower (lower_ast). WDYT?

You mean moving offload to right after lower? I'm afraid that will lead to some issues in autodiff.

xumingkuan · 2020-05-12T00:42:01Z

The offload pass and all backends still need to be modified.
If we put this pass before demote_dense_struct_fors, it also needs to be modified.
I think ideally we should put this pass right after lower (lower_ast). WDYT?

You mean moving offload to right after lower? I'm afraid that will lead to some issues in autodiff.

Oh, I mean moving irpass::convert_into_loop_index(ir); to right after lower.

yuanming-hu · 2020-05-12T01:26:15Z

Oh, I see. That sounds good. Actually, let's make it part of lower.

taichi/taichi/transforms/lower_ast.cpp

Lines 446 to 449 in f78f387

    
           void lower(IRNode *root) { 
        
             auto offsets = LowerAST::run(root); 
        
             FixStructForOffsets::run(root, offsets); 
        
           }

# Conflicts: # taichi/ir/ir.h

xumingkuan · 2020-05-13T01:39:50Z

Why is this?

taichi/taichi/ir/ir.cpp

Lines 243 to 246 in e0ef399

    
           void Stmt::replace_with(Stmt *new_stmt) { 
        
             auto root = get_ir_root(); 
        
             irpass::replace_all_usages_with(root, this, new_stmt); 
        
             // Note: the current structure should have been destroyed now..

I think I need

taichi/taichi/transforms/demote_dense_struct_fors.cpp

Line 84 in e0ef399

irpass::replace_all_usages_with(body.get(), old_loop_vars[i], alloca);

to not throw exceptions. I'm changing the above line to irpass::replace_statements_with(...) and trying to add an argument for irpass::replace_statements_with so as to not throw exceptions, but I'm getting Windows fatal exception: access violation.

yuanming-hu · 2020-05-13T01:54:40Z

I have no idea on this. get_ir_root will follow the parent field of blocks, which might have been nullptr - could this be the cause?

xumingkuan · 2020-05-13T01:57:13Z

I have no idea on this. get_ir_root will follow the parent field of blocks, which might have been nullptr - could this be the cause?

Very probably...

# Conflicts: # taichi/transforms/statement_replace.cpp

xumingkuan · 2020-05-13T22:41:18Z

I wonder in which case this is false:

taichi/taichi/backends/metal/codegen_metal.cpp

Line 449 in e0ef399

if (loop_var->ret_type.data_type == DataType::i32) {

Since there are no loop_vars after this PR, I'm afraid that the information on data type will be lost.

yuanming-hu · 2020-05-13T23:12:51Z

Thanks for pointing that out. Shall we simply add a rule to enforce the data type of loop variables to be i32? In the future maybe i64 should also be allowed. Maybe the data type should be a member of the for statements so that LoopVarStmt can infer from it.

This reverts commit f284688.

xumingkuan · 2020-05-14T00:07:07Z

Thanks for pointing that out. Shall we simply add a rule to enforce the data type of loop variables to be i32? In the future maybe i64 should also be allowed. Maybe the data type should be a member of the for statements so that LoopVarStmt can infer from it.

I think in LLVM and CUDA backends, the loop variables are i32 by default. So I'm going to make them i32 in all backends by default for now.

yuanming-hu · 2020-05-14T00:08:14Z

Thanks for pointing that out. Shall we simply add a rule to enforce the data type of loop variables to be i32? In the future maybe i64 should also be allowed. Maybe the data type should be a member of the for statements so that LoopVarStmt can infer from it.

I think in LLVM and CUDA backends, the loop variables are i32 by default. So I'm going to make them i32 in all backends by default for now.

Sounds good. i32 should work for a while.

xumingkuan · 2020-05-14T00:11:55Z

taichi/transforms/demote_dense_struct_fors.cpp

-  TI_ASSERT(total_bits <= 31);
+  TI_ASSERT(total_bits <= 30);

  auto upper_bound = 1 << total_bits;


1 << 31 looks dangerous. So I suppose not supporting tensors of size 2^31 for now.

xumingkuan · 2020-05-14T00:18:09Z

TODO: Remove RangeForStmt::loop_var and StructForStmt::loop_vars. Maybe in the next PR.

yuanming-hu

Looks great! Thank you so much for refactoring this. The LLVM backend modifications look good to me. (I didn't look at the OpenGL and Metal backends.)

~~Btw, would it be possible to remove RangeForStmt::loop_var and StructForStmt::loop_vars in the future? How easy/hard is that?~~ Sorry I missed your last message. Great work!

xumingkuan · 2020-05-14T01:54:33Z

Looks great! Thank you so much for refactoring this. The LLVM backend modifications look good to me. (I didn't look at the OpenGL and Metal backends.)

Btw, would it be possible to remove RangeForStmt::loop_var and StructForStmt::loop_vars in the future? How easy/hard is that?

After lower_ast, it's easy -- I've already set them to nullptrs. I think only StructForStmt::loop_vars->size() contains some information now, but it should be the same as StructForStmt::snode->num_active_indices.

But during lower_ast, they are used temporarily, and maybe we need an std::unordered_map.

I wonder if it's feasible to create LoopIndexStmts directly in LowerAST::visit(FrontendForStmt *stmt). In this way, we can even obviate the need of eliminating identical LoopIndexStmts in simplify. (Because we don't eliminate them but we do eliminate identical local loads now, the performance is impaired.)

yuanming-hu · 2020-05-14T02:03:39Z

I wonder if it's feasible to create LoopIndexStmts directly in LowerAST::visit(FrontendForStmt *stmt). In this way, we can even obviate the need of eliminating identical LoopIndexStmts in simplify. (Because we don't eliminate them but we do eliminate identical local loads now, the performance is impaired.)

My feeling is that this is doable, but probably something in transformer.py needs to be changed. You can actually directly insert LoopIndexStmt in transformer.py and fix the loop pointers later in lower_ast.

xumingkuan · 2020-05-14T02:28:25Z

I wonder if it's feasible to create LoopIndexStmts directly in LowerAST::visit(FrontendForStmt *stmt). In this way, we can even obviate the need of eliminating identical LoopIndexStmts in simplify. (Because we don't eliminate them but we do eliminate identical local loads now, the performance is impaired.)

My feeling is that this is doable, but probably something in transformer.py needs to be changed. You can actually directly insert LoopIndexStmt in transformer.py and fix the loop pointers later in lower_ast.

I would prefer not touching transformer.py, as there are too many kinds of "frontend for"s... As the allocas are inserted in LowerAST::visit(FrontendForStmt *stmt), I think maybe I can replace them with LoopIndexStmts.

Currently, for this IR,

kernel {
  $0 : for @tmp1, @tmp2, @tmp3 where S2place_i32 active {
    $1 = alloca @tmp7
    @tmp7 = @tmp1
  }
}

after LowerAST::run, we have

kernel {
  <i32 x1> $99 = alloca
  <i32 x1> $100 = alloca
  <i32 x1> $101 = alloca
  $102 : for $99, $100, $101 where S1dense active, step 1 {
    $103 = alloca
    $104 = local load [ [$99[0]]]
    $105 : local store [$103 <- $104]
  }
}

Shall we modify

taichi/taichi/ir/frontend_ir.h

Lines 448 to 452 in e0ef399

    
           void flatten(FlattenContext *ctx) override { 
        
             ctx->push_back(std::make_unique<LocalLoadStmt>( 
        
                 LocalAddress(ctx->current_block->lookup_var(id), 0))); 
        
             stmt = ctx->back_stmt(); 
        
           }

and Stmt *Block::lookup_var(const Identifier &ident) to treat loop vars specifically?

yuanming-hu · 2020-05-14T03:35:57Z

Sounds good. Feel free to pick the most convenient solution for now (keep in mind the presence of #925)...

xumingkuan · 2020-05-14T03:48:35Z

Feel free to pick the most convenient solution for now

Shall I do it in this PR or the next?

keep in mind the presence of #925

I think even if lower_ast is in python, dealing with for loops in lower_ast will still be easier than in transformer.

yuanming-hu · 2020-05-14T04:33:54Z

Feel free to pick the most convenient solution for now

Shall I do it in this PR or the next?

Next PR sounds good.

keep in mind the presence of #925

I think even if lower_ast is in python, dealing with for loops in lower_ast will still be easier than in transformer.

That's true. Testing if a variable is a loop variable is easier in AST than in transformer.py.

archibate

OpenGL is OK, thank you!

archibate · 2020-05-14T05:07:07Z

codegen_kernel_statements                    194 ->   225   +16.0%
codegen_kernel_statements                     47 ->    55   +17.0%
codegen_kernel_statements                     53 ->    58    +9.4%
codegen_kernel_statements                     26 ->     0  -100.0%
codegen_kernel_statements                     57 ->   150  +163.2%
codegen_kernel_statements                     15 ->    16    +6.7%
codegen_kernel_statements                    135 ->     0  -100.0%
codegen_kernel_statements                     72 ->   146  +102.8%
codegen_kernel_statements                     16 ->    27   +68.8%
codegen_kernel_statements                     21 ->    40   +90.5%
codegen_kernel_statements                     17 ->    19   +11.8%
codegen_kernel_statements                     37 ->     0  -100.0%
codegen_kernel_statements                     37 ->    38    +2.7%
codegen_kernel_statements                      0 ->    72    +inf%
codegen_kernel_statements                     18 ->    20   +11.1%
codegen_kernel_statements                     24 ->    10   -58.3%
codegen_kernel_statements                     53 ->    28   -47.2%
codegen_kernel_statements                    135 ->   150   +11.1%
codegen_kernel_statements                     74 ->    62   -16.2%
codegen_kernel_statements                     40 ->    39    -2.5%
codegen_kernel_statements                     53 ->    17   -67.9%
codegen_kernel_statements                     24 ->    12   -50.0%
codegen_kernel_statements                     22 ->   136  +518.2%
codegen_kernel_statements                     17 ->    19   +11.8%
codegen_kernel_statements                     16 ->    12   -25.0%
codegen_kernel_statements                     50 ->    51    +2.0%
codegen_kernel_statements                     32 ->    16   -50.0%
codegen_kernel_statements                      0 ->    11    +inf%
codegen_kernel_statements                     16 ->    18   +12.5%
codegen_kernel_statements                     18 ->    53  +194.4%
codegen_kernel_statements                     20 ->    34   +70.0%
codegen_kernel_statements                     27 ->    18   -33.3%
codegen_kernel_statements                     34 ->    36    +5.9%
codegen_kernel_statements                     27 ->    29    +7.4%
codegen_kernel_statements                     12 ->    23   +91.7%
codegen_kernel_statements                     14 ->    85  +507.1%
codegen_kernel_statements                     37 ->    38    +2.7%
codegen_kernel_statements                    158 ->     0  -100.0%
codegen_kernel_statements                     37 ->    38    +2.7%
codegen_kernel_statements                     60 ->    66   +10.0%
codegen_kernel_statements                     15 ->     0  -100.0%
codegen_kernel_statements                     42 ->    44    +4.8%
codegen_kernel_statements                     16 ->    18   +12.5%
codegen_kernel_statements                      3 ->    52 +1633.3%
codegen_kernel_statements                     12 ->    65  +441.7%
codegen_kernel_statements                   2789 ->    11   -99.6%
codegen_kernel_statements                     50 ->    37   -26.0%
codegen_kernel_statements                   2085 ->  2084    -0.0%
codegen_kernel_statements                      0 ->    22    +inf%
codegen_kernel_statements                     69 ->     3   -95.7%
codegen_kernel_statements                     12 ->    51  +325.0%
codegen_kernel_statements                   2788 ->  2787    -0.0%
codegen_kernel_statements                     71 ->    83   +16.9%
codegen_kernel_statements                     38 ->    41    +7.9%
codegen_kernel_statements                     16 ->    18   +12.5%
codegen_kernel_statements                     28 ->   158  +464.3%
codegen_kernel_statements                     17 ->    18    +5.9%
codegen_kernel_statements                      0 ->  2788    +inf%
codegen_kernel_statements                     38 ->    53   +39.5%
codegen_kernel_statements                     52 ->     0  -100.0%
codegen_kernel_statements                     23 ->    24    +4.3%
codegen_kernel_statements                     76 ->    70    -7.9%
codegen_kernel_statements                     17 ->    18    +5.9%
codegen_kernel_statements                     10 ->    32  +220.0%
codegen_kernel_statements                     17 ->    24   +41.2%
codegen_kernel_statements                     48 ->    16   -66.7%
codegen_kernel_statements                     68 ->    37   -45.6%
codegen_kernel_statements                     38 ->    37    -2.6%
codegen_kernel_statements                      4 ->    50 +1150.0%
codegen_kernel_statements                    150 ->   148    -1.3%
codegen_kernel_statements                     20 ->    19    -5.0%
codegen_kernel_statements                   2727 ->  2726    -0.0%
codegen_kernel_statements                      4 ->    16  +300.0%
codegen_kernel_statements                     18 ->    17    -5.6%
codegen_kernel_statements                     64 ->    65    +1.6%
codegen_kernel_statements                     16 ->   101  +531.2%
codegen_kernel_statements                     17 ->    23   +35.3%
codegen_kernel_statements                     18 ->    24   +33.3%
codegen_kernel_statements                     34 ->    35    +2.9%
codegen_kernel_statements                     28 ->    22   -21.4%
codegen_kernel_statements                     20 ->    12   -40.0%
codegen_kernel_statements                     16 ->    52  +225.0%
codegen_kernel_statements                     10 ->    72  +620.0%
codegen_kernel_statements                     55 ->     0  -100.0%
codegen_kernel_statements                     20 ->    21    +5.0%
codegen_kernel_statements                     76 ->    69    -9.2%
codegen_kernel_statements                     28 ->    29    +3.6%
codegen_kernel_statements                     34 ->     0  -100.0%
codegen_kernel_statements                     18 ->    19    +5.6%
codegen_kernel_statements                     38 ->    10   -73.7%
codegen_kernel_statements                     70 ->    56   -20.0%
codegen_kernel_statements                     38 ->    36    -5.3%
codegen_kernel_statements                     34 ->    48   +41.2%
codegen_kernel_statements                     61 ->   158  +159.0%
codegen_kernel_statements                     16 ->    18   +12.5%
codegen_kernel_statements                     56 ->     4   -92.9%
codegen_kernel_statements                     28 ->    10   -64.3%
codegen_kernel_statements                      0 ->    52    +inf%
codegen_kernel_statements                     20 ->    34   +70.0%
codegen_kernel_statements                     29 ->    33   +13.8%
codegen_kernel_statements                      0 ->    12    +inf%
codegen_kernel_statements                     32 ->    56   +75.0%
codegen_kernel_statements                     45 ->    52   +15.6%
codegen_kernel_statements                    100 ->   117   +17.0%
codegen_kernel_statements                      4 ->    22  +450.0%
codegen_kernel_statements                    148 ->    38   -74.3%
codegen_kernel_statements                     12 ->    51  +325.0%
codegen_kernel_statements                     22 ->    28   +27.3%
codegen_kernel_statements                     20 ->    26   +30.0%
codegen_kernel_statements                     23 ->    27   +17.4%
codegen_kernel_statements                     24 ->     0  -100.0%
codegen_kernel_statements                     16 ->    17    +6.2%
codegen_kernel_statements                     41 ->    43    +4.9%
codegen_kernel_statements                     62 ->    27   -56.5%
codegen_kernel_statements                     26 ->     4   -84.6%
codegen_kernel_statements                      0 ->    53    +inf%
codegen_kernel_statements                     14 ->    24   +71.4%
codegen_kernel_statements                     26 ->    14   -46.2%
codegen_kernel_statements                     69 ->   116   +68.1%
codegen_kernel_statements                     19 ->    57  +200.0%
codegen_kernel_statements                     51 ->    62   +21.6%
codegen_kernel_statements                    112 ->    78   -30.4%
codegen_kernel_statements                     28 ->    48   +71.4%
codegen_kernel_statements                     36 ->     0  -100.0%
codegen_kernel_statements                     63 ->    64    +1.6%
codegen_kernel_statements                    159 ->   261   +64.2%
codegen_kernel_statements                     62 ->    18   -71.0%
codegen_kernel_statements                     62 ->    28   -54.8%
codegen_kernel_statements                     62 ->    61    -1.6%
codegen_kernel_statements                     24 ->    26    +8.3%
codegen_kernel_statements                     18 ->    19    +5.6%
codegen_kernel_statements                      0 ->    28    +inf%
codegen_kernel_statements                     18 ->    17    -5.6%
codegen_kernel_statements                     40 ->    45   +12.5%
codegen_kernel_statements                    186 ->   225   +21.0%
codegen_kernel_statements                    158 ->     0  -100.0%
codegen_kernel_statements                     52 ->    70   +34.6%
codegen_kernel_statements                     10 ->    34  +240.0%
codegen_kernel_statements                     18 ->    17    -5.6%
codegen_kernel_statements                     37 ->    38    +2.7%
codegen_kernel_statements                     41 ->    43    +4.9%
codegen_kernel_statements                     14 ->    13    -7.1%
codegen_kernel_statements                      0 ->    15    +inf%
codegen_kernel_statements                     56 ->    15   -73.2%
codegen_kernel_statements                     11 ->    27  +145.5%
codegen_kernel_statements                      0 ->   148    +inf%
codegen_kernel_statements                     16 ->    46  +187.5%
codegen_kernel_statements                    112 ->    78   -30.4%
codegen_kernel_statements                     11 ->    18   +63.6%
codegen_kernel_statements                     23 ->    65  +182.6%
codegen_kernel_statements                     90 ->    32   -64.4%
codegen_kernel_statements                     72 ->     0  -100.0%
codegen_kernel_statements                     16 ->    18   +12.5%
codegen_kernel_statements                     49 ->    29   -40.8%
codegen_kernel_statements                     52 ->     4   -92.3%
codegen_kernel_statements                     18 ->    17    -5.6%
codegen_kernel_statements                    136 ->    16   -88.2%
codegen_kernel_statements                      0 ->    26    +inf%
codegen_kernel_statements                     14 ->    16   +14.3%
codegen_kernel_statements                     21 ->    26   +23.8%
codegen_kernel_statements                     18 ->    20   +11.1%
codegen_kernel_statements                     53 ->    54    +1.9%
codegen_kernel_statements                    146 ->     0  -100.0%

k-ye

Great! I can confirm that the change works for Metal as well, thx!

k-ye · 2020-05-14T11:09:40Z

taichi/codegen/codegen_llvm.h

@@ -72,8 +72,7 @@ class CodeGenLLVM : public IRVisitor, public LLVMModuleBuilder {
  std::vector<OffloadedTask> offloaded_tasks;
  BasicBlock *func_body_bb;

-  std::unordered_map<OffloadedStmt *, std::vector<llvm::Value *>>
-      offloaded_loop_vars_llvm;
+  std::unordered_map<Stmt *, std::vector<llvm::Value *>> loop_vars_llvm;


nit: is it possible to use const Stmt* here? i don't think the key is modified?

xumingkuan · 2020-05-14T18:18:24Z

codegen_kernel_statements                    194 ->   225   +16.0%
codegen_kernel_statements                     47 ->    55   +17.0%
codegen_kernel_statements                     53 ->    58    +9.4%
codegen_kernel_statements                     26 ->     0  -100.0%
...
codegen_kernel_statements                      0 ->    26    +inf%
codegen_kernel_statements                     14 ->    16   +14.3%
codegen_kernel_statements                     21 ->    26   +23.8%
codegen_kernel_statements                     18 ->    20   +11.1%
codegen_kernel_statements                     53 ->    54    +1.9%
codegen_kernel_statements                    146 ->     0  -100.0%

I think maybe we're encountering this issue #656 (comment), that is, some tests are mismatched, so there are some -100.0% and +inf%.

[skip ci] Convert loop_var into LoopIndexStmt (stage 1)

b45424e

xumingkuan commented May 11, 2020

View reviewed changes

xumingkuan added 3 commits May 12, 2020 16:29

Merge branch 'master' into loop-index

2d75cd1

# Conflicts: # taichi/ir/ir.h

[skip ci] Fix irpass::offload

17dfe8d

Should work on CPU now

eaf692c

Move convert_into_loop_index to right after lower_ast

7263e02

xumingkuan mentioned this pull request May 13, 2020

[ir][refactor] Avoid throwing exception in replace_statements_with #968

Merged

xumingkuan added 5 commits May 13, 2020 15:43

Merge branch 'master' into loop-index

e29441a

# Conflicts: # taichi/transforms/statement_replace.cpp

Merge master

8c7898a

Fix tests

cf3b420

Fix CUDA compilation

e0da2cd

[skip ci] Remove comments

35ca973

xumingkuan and others added 5 commits May 13, 2020 19:53

Metal backend

e26a909

[skip ci] Metal backend (continued)

2f1e86b

opengl backend

3d7424b

[skip ci] enforce code format

f284688

Revert "[skip ci] enforce code format"

7b0fa75

This reverts commit f284688.

[skip ci] Remove LoopIndexStmt::is_struct_for

dbde2fb

xumingkuan commented May 14, 2020

View reviewed changes

Finally move the pass into lower

c1e91dd

xumingkuan marked this pull request as ready for review May 14, 2020 00:13

[skip ci] add comment

f0276bd

xumingkuan requested review from yuanming-hu, archibate and k-ye May 14, 2020 00:19

Merge branch 'master' into loop-index

d61030a

yuanming-hu approved these changes May 14, 2020

View reviewed changes

Fix opengl backend

3a73b77

xumingkuan changed the title ~~[IR][Refactor] Convert loop_var into LoopIndexStmt~~ [IR][refactor] Convert loop_var into LoopIndexStmt May 14, 2020

archibate reviewed May 14, 2020

View reviewed changes

k-ye approved these changes May 14, 2020

View reviewed changes

xumingkuan and others added 2 commits May 14, 2020 14:25

Use const Stmt*

ac03804

[skip ci] enforce code format

404b6e4

xumingkuan merged commit c6086e0 into taichi-dev:master May 14, 2020

xumingkuan mentioned this pull request May 21, 2020

[ir] [refactor] Deprecate loop_var #1032

Merged

xumingkuan deleted the loop-index branch May 27, 2020 20:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[IR][refactor] Convert loop_var into LoopIndexStmt #953

[IR][refactor] Convert loop_var into LoopIndexStmt #953

xumingkuan commented May 11, 2020

xumingkuan commented May 11, 2020

xumingkuan May 11, 2020

yuanming-hu May 11, 2020

yuanming-hu commented May 11, 2020

xumingkuan commented May 12, 2020

yuanming-hu commented May 12, 2020

xumingkuan commented May 13, 2020

yuanming-hu commented May 13, 2020

xumingkuan commented May 13, 2020

xumingkuan commented May 13, 2020

yuanming-hu commented May 13, 2020

xumingkuan commented May 14, 2020

yuanming-hu commented May 14, 2020

xumingkuan May 14, 2020

xumingkuan commented May 14, 2020

yuanming-hu left a comment •

edited

Loading

xumingkuan commented May 14, 2020

yuanming-hu commented May 14, 2020

xumingkuan commented May 14, 2020

yuanming-hu commented May 14, 2020

xumingkuan commented May 14, 2020

yuanming-hu commented May 14, 2020

archibate left a comment

archibate commented May 14, 2020

k-ye left a comment

k-ye May 14, 2020

xumingkuan commented May 14, 2020

[IR][refactor] Convert loop_var into LoopIndexStmt #953

[IR][refactor] Convert loop_var into LoopIndexStmt #953

Conversation

xumingkuan commented May 11, 2020

xumingkuan commented May 11, 2020

xumingkuan May 11, 2020

Choose a reason for hiding this comment

yuanming-hu May 11, 2020

Choose a reason for hiding this comment

yuanming-hu commented May 11, 2020

xumingkuan commented May 12, 2020

yuanming-hu commented May 12, 2020

xumingkuan commented May 13, 2020

yuanming-hu commented May 13, 2020

xumingkuan commented May 13, 2020

xumingkuan commented May 13, 2020

yuanming-hu commented May 13, 2020

xumingkuan commented May 14, 2020

yuanming-hu commented May 14, 2020

xumingkuan May 14, 2020

Choose a reason for hiding this comment

xumingkuan commented May 14, 2020

yuanming-hu left a comment • edited Loading

Choose a reason for hiding this comment

xumingkuan commented May 14, 2020

yuanming-hu commented May 14, 2020

xumingkuan commented May 14, 2020

yuanming-hu commented May 14, 2020

xumingkuan commented May 14, 2020

yuanming-hu commented May 14, 2020

archibate left a comment

Choose a reason for hiding this comment

archibate commented May 14, 2020

k-ye left a comment

Choose a reason for hiding this comment

k-ye May 14, 2020

Choose a reason for hiding this comment

xumingkuan commented May 14, 2020

yuanming-hu left a comment •

edited

Loading