GpuCast from ArrayType to StringType [databricks] #4221

HaoYang670 · 2021-11-26T08:10:17Z

Signed-off-by: HaoYang670 13716567376yh@gmail.com

add a new feature: support GpuCast from ArrayType to StringType

close #4028

Signed-off-by: Remzi Yang <13716567376@163.com>

…pids into branch-22.02

Signed-off-by: Remzi Yang <13716567376@163.com>

Signed-off-by: remzi <13716567376yh@gmail.com>

HaoYang670 · 2021-11-26T08:10:39Z

build

Signed-off-by: remzi <13716567376yh@gmail.com>

HaoYang670 · 2021-11-26T09:53:41Z

build

pxLi · 2021-11-26T09:58:29Z

build

Signed-off-by: remzi <13716567376yh@gmail.com>

HaoYang670 · 2021-11-29T02:23:40Z

build

sperlingxx

It looks good except several small issues.

sperlingxx · 2021-11-29T02:09:53Z

integration_tests/src/main/python/array_test.py

+
+
+@pytest.mark.parametrize('data_gen', all_gens_for_cast_to_string, ids=idfn)
+def test_legacy_cast_array_to_string(data_gen):


To me, it is better to merge this test with test_cast_array_to_string, since castComplexTypesToString config can be a pytest param of this test.

Absolutely agree with you.
However, the current style is more consistent with tests for casting struct to string.
Just as Bobby said, we should have a follow-on issue for moving up those tests to test_cast.py and cleaning up the code.

sperlingxx · 2021-11-29T02:10:47Z

integration_tests/src/main/python/array_test.py

+    lambda spark: unary_op_df(spark, data_gen).select(
+        f.col('a').cast("STRING")
+    )
+)


missing newline at end of file

sperlingxx · 2021-11-29T02:35:50Z

sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala

+              withResource(ColumnVector.fromScalar(space, child.getRowCount.toInt)) { spaceVec =>
+                withResource(
+                  ColumnVector.stringConcatenate(Array(spaceVec, strChild))
+                ) { AddSpace =>


nit: lower the first character.

sperlingxx · 2021-11-29T02:43:15Z

sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala

+       */
+      def removeFirstSpace(strVec: ColumnVector): ColumnVector = {
+        if (legacyCastToString){
+          withResource(ColumnVector.fromScalar(space, strVec.getRowCount.toInt)){spaceVec =>


It doesn't seem necessary to construct a spaceVec, since equalToworks between a column and a scalar as well.

Thanks for your advice, I will fix it

sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala

sperlingxx

~~Another issue is that the config LEGACY_COMPLEX_TYPES_TO_STRING only exists in 3.1.0+. However, we also need to support Spark 3.0.X in current.~~

Signed-off-by: remzi <13716567376yh@gmail.com>

HaoYang670 · 2021-11-29T05:55:11Z

build

revans2 · 2021-11-29T16:34:54Z

integration_tests/src/main/python/array_test.py

+# casting these types to string are not exact match, marked as xfail when testing
+not_matched_gens = [float_gen, double_gen, timestamp_gen, decimal_gen_neg_scale]
+# casting these types to string are not supported, marked as xfail when testing
+not_support_gens = decimal_128_gens


The names of these variables are too generic. Can we put _for_cast_to_string in them.

Thank you, I have revised the names

revans2 · 2021-11-29T16:37:01Z

integration_tests/src/main/python/array_test.py

@@ -249,3 +249,63 @@ def test_sql_array_scalars(query):
    assert_gpu_and_cpu_are_equal_collect(
            lambda spark : spark.sql('SELECT {}'.format(query)),
            conf={'spark.sql.legacy.allowNegativeScaleOfDecimal': 'true'})
+
+


nit: a lot of the other tests around casting are in cast_test.py. I can see arguments to have them here because it is specifically for arrays, but we already have tests there for casting arrays to arrays. As such I would rather see the tests moved there.

It is much more reasonable to move the tests to cast_test.py, I agree with you!

However, you may find that tests for casting struct to string are all in struct_test.py instead of cast_test.py:

spark-rapids/integration_tests/src/main/python/struct_test.py

Lines 75 to 166 in 877c1b2

# conf with legacy cast to string on

legacy_complex_types_to_string = {'spark.sql.legacy.castComplexTypesToString.enabled': 'true'}

@pytest.mark.parametrize('data_gen', [StructGen([["first", boolean_gen], ["second", byte_gen], ["third", short_gen], ["fourth", int_gen], ["fifth", long_gen], ["sixth", string_gen], ["seventh", date_gen]])], ids=idfn)

def test_legacy_cast_struct_to_string(data_gen):

assert_gpu_and_cpu_are_equal_collect(

lambda spark : unary_op_df(spark, data_gen).select(

f.col('a').cast("STRING")),

conf = legacy_complex_types_to_string)

# https://github.com/NVIDIA/spark-rapids/issues/2309

@pytest.mark.parametrize('cast_conf', ['LEGACY', 'SPARK311+'])

def test_one_nested_null_field_legacy_cast(cast_conf):

def was_broken_for_nested_null(spark):

data = [

(('foo',),),

((None,),),

(None,)

]

df = spark.createDataFrame(data)

return df.select(df._1.cast(StringType()))

assert_gpu_and_cpu_are_equal_collect(was_broken_for_nested_null, {

'spark.sql.legacy.castComplexTypesToString.enabled': cast_conf == 'LEGACY'

})

# https://github.com/NVIDIA/spark-rapids/issues/2315

@pytest.mark.parametrize('cast_conf', ['LEGACY', 'SPARK311+'])

def test_two_col_struct_legacy_cast(cast_conf):

def broken_df(spark):

key_data_gen = StructGen([

('a', IntegerGen(min_val=0, max_val=4)),

('b', IntegerGen(min_val=5, max_val=9)),

], nullable=False)

val_data_gen = IntegerGen()

df = two_col_df(spark, key_data_gen, val_data_gen)

return df.select(df.a.cast(StringType())).filter(df.b > 1)

assert_gpu_and_cpu_are_equal_collect(broken_df, {

'spark.sql.legacy.castComplexTypesToString.enabled': cast_conf == 'LEGACY'

})

@pytest.mark.parametrize('data_gen', [StructGen([["first", float_gen]])], ids=idfn)

@pytest.mark.xfail(reason='casting float to string is not an exact match')

def test_legacy_cast_struct_with_float_to_string(data_gen):

assert_gpu_and_cpu_are_equal_collect(

lambda spark : unary_op_df(spark, data_gen).select(

f.col('a').cast("STRING")),

conf = legacy_complex_types_to_string)

@pytest.mark.parametrize('data_gen', [StructGen([["first", double_gen]])], ids=idfn)

@pytest.mark.xfail(reason='casting double to string is not an exact match')

def test_legacy_cast_struct_with_double_to_string(data_gen):

assert_gpu_and_cpu_are_equal_collect(

lambda spark : unary_op_df(spark, data_gen).select(

f.col('a').cast("STRING")),

conf = legacy_complex_types_to_string)

@pytest.mark.parametrize('data_gen', [StructGen([["first", timestamp_gen]])], ids=idfn)

@pytest.mark.xfail(reason='https://github.com/NVIDIA/spark-rapids/issues/219')

def test_legacy_cast_struct_with_timestamp_to_string(data_gen):

assert_gpu_and_cpu_are_equal_collect(

lambda spark : unary_op_df(spark, data_gen).select(

f.col('a').cast("STRING")),

conf = legacy_complex_types_to_string)

@pytest.mark.parametrize('data_gen', [StructGen([["first", boolean_gen], ["second", byte_gen], ["third", short_gen], ["fourth", int_gen], ["fifth", long_gen], ["sixth", string_gen], ["seventh", date_gen]])], ids=idfn)

def test_cast_struct_to_string(data_gen):

assert_gpu_and_cpu_are_equal_collect(

lambda spark : unary_op_df(spark, data_gen).select(

f.col('a').cast("STRING")))

@pytest.mark.parametrize('data_gen', [StructGen([["first", float_gen]])], ids=idfn)

@pytest.mark.xfail(reason='casting float to string is not an exact match')

def test_cast_struct_with_float_to_string(data_gen):

assert_gpu_and_cpu_are_equal_collect(

lambda spark : unary_op_df(spark, data_gen).select(

f.col('a').cast("STRING")))

@pytest.mark.parametrize('data_gen', [StructGen([["first", double_gen]])], ids=idfn)

@pytest.mark.xfail(reason='casting double to string is not an exact match')

def test_cast_struct_with_double_to_string(data_gen):

assert_gpu_and_cpu_are_equal_collect(

lambda spark : unary_op_df(spark, data_gen).select(

f.col('a').cast("STRING")))

@pytest.mark.parametrize('data_gen', [StructGen([["first", timestamp_gen]])], ids=idfn)

@pytest.mark.xfail(reason='https://github.com/NVIDIA/spark-rapids/issues/219')

def test_cast_struct_with_timestamp_to_string(data_gen):

assert_gpu_and_cpu_are_equal_collect(

lambda spark : unary_op_df(spark, data_gen).select(

f.col('a').cast("STRING")))

Should we be consistent with it?

Yes we should, and the struct tests should probably move too at some point. If you want to file a follow on issue you can. But this is a nit so if you just want both to be a follow on issue that is fine too.

sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala

sql-plugin/src/main/scala/com/nvidia/spark/rapids/TypeChecks.scala

…ateListElements API Signed-off-by: remzi <13716567376yh@gmail.com>

HaoYang670 · 2021-11-30T06:37:59Z

build

revans2

The computation is at least documented. I would love it if we could have two follow on issues to find a way to make this cast better.

I would like to see if we can clean up the code. Even if we have to have a Spark specific kernel for doing some of the concat.
We should look a the performance impact of this vs a version that has spit paths for legacy vs not legacy. I am a little concerned that we are taking a performance hit when legacy is false because we wanted to reuse as much code as possible.

The only reason I am not approving this is because some of the comments by @sperlingxx have not been addressed yet.

revans2 · 2021-11-30T14:38:07Z

integration_tests/src/main/python/array_test.py

@@ -249,3 +249,63 @@ def test_sql_array_scalars(query):
    assert_gpu_and_cpu_are_equal_collect(
            lambda spark : spark.sql('SELECT {}'.format(query)),
            conf={'spark.sql.legacy.allowNegativeScaleOfDecimal': 'true'})
+
+


Yes we should, and the struct tests should probably move too at some point. If you want to file a follow on issue you can. But this is a nit so if you just want both to be a follow on issue that is fine too.

ttnghia · 2021-11-30T16:46:02Z

sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala

+  private  def castArrayToString(input:                    ColumnView,
+                                 elementType:                DataType,
+                                 ansiMode:                    Boolean,
+                                 legacyCastToString:          Boolean,
+                                 stringToDateAnsiModeEnabled: Boolean): ColumnVector = {


I'm suspect that our Scala style doesn't support such alignment 😃

This alignment is auto generated by IDEA when I press enter key. I find no way to change it. But it has passed the Scala style check.

sperlingxx

LGTM

HaoYang670 · 2021-12-07T03:35:10Z

This is a performance report of casting array to string.

Aims.

Compare the performance of CPU-version and GPU-version (end-to-end)
Compare the performance of spark-rapids when legacy enabled and legacy disabled (GpuProject transformation)

Hardware / Software environment:

CPU: Intel Core i7-10700K, 16 cores
Memory: 64 GB
Storage: 813 GB
GPU: Quadro RTX 8000, memory 48578 MB, driver 495.29.05, CUDA 11.5
Spark: spark-3.2.0-bin-hadoop-3.2
Spark-rapids: 22.02
Cudf: 22.02

test method:

spark mode: local
We use DataFrames with 1 column. Each row contains one array of bytes. The number of rows is from 1e8 to 1e9. Array length is from 1 to 50. All the DataFrames are stored as parquet files.
All spark configurations are defaults except spark.sql.legacy.castComplexTypesToString.enable.
In each test, we first load the dataframe, and then cast the column to string, and then call the aggregate max, and finally call collect. In Python, we can write the code as:
spark.read.parquet(data_path).withColumn("b", f.col("value").cast("STRING")).agg({"b": "max"}).collect().
For each DataFrame, we firstly run the test code 5 times for warm up. Then we run the test code 5 times and calculate the average consumed time.
When comparing CPU-version and GPU-version, we test the end-to-end performance (from read parquet to collect()). When compare GPU -version with enabled / disabled legacy, we test the performance of GpuProject transformation which only contains cast array to string.

test result:
use ETE as a shorthand of end-to-end, GP as a shorthand of GpuProject transformation.
Measure time in seconds.

Rows	Array length	CPU ETE	CPU-legacy ETE	GPU ETE	GPU-legacy ETE	GPU GP (total)	GPU-legacy GP (total)
1e8	1	6.14 secs	6.13 secs.	0.24 secs	0.28 secs	0.050 secs	0.098 secs
1e8	5	25.03 secs	23.12 secs	0.48 secs	0.62 secs	0.096 secs	0.253 secs
1e8	10	47.11 secs	47.16 secs	0.87 secs	1.20 secs	0.218 secs	0.563secs
1e8	50	228.55 secs	226.89 secs	20.67 secs	41.19 secs	16.00 secs	36.50 secs
1e9	1	60.23 secs	63.85 secs	1.08 secs	1.44 secs	0.388 secs	0.815 secs

conclusion:

For end-to-end performance, GPU has about 5 ~ 60 times acceleration. The acceleration increases as the number of rows increases, and the acceleration decreases as array length increases.
When legacy is enabled, spark-rapids will take twice as long as disabling legacy. The size of dataframe does not have an obvious impact on this slow-down.

discussion.
So far, cast array to string takes two different paths for legacy and no-legacy, which impacts the performance. We should clean up the code in this method to reduce the difference.

HaoYang670 · 2021-12-07T03:52:34Z

The testing code:
https://github.com/HaoYang670/spark-rapids/blob/743b870b84a303d0f71355633edf7ec82c5c95e6/integration_tests/src/main/python/array_test.py#L317-L379

HaoYang670 · 2021-12-07T09:14:51Z

So far, I test the performance on local mode. I will upload another report of standalone mode.

HaoYang670 · 2021-12-08T07:27:17Z

Similar results are gotten in standalone mode

HaoYang670 and others added 11 commits November 19, 2021 11:42

for issue 4028

dd0a548

Signed-off-by: Remzi Yang <13716567376@163.com>

Merge branch 'branch-22.02' of https://github.com/HaoYang670/spark-ra…

13dfe3b

…pids into branch-22.02

basic logic of casting array to string

4006035

Signed-off-by: Remzi Yang <13716567376@163.com>

some cast test

05d32bb

Signed-off-by: remzi <13716567376yh@gmail.com>

merge 22.02

d58a6b7

Signed-off-by: remzi <13716567376yh@gmail.com>

find cast timestamp to string bug

da6c053

Signed-off-by: remzi <13716567376yh@gmail.com>

Merge branch 'branch-22.02' into fix4028

6bac23f

add tests. test fails on legacy casting

c544677

Signed-off-by: remzi <13716567376yh@gmail.com>

pass legacy tests

f307b5a

Signed-off-by: remzi <13716567376yh@gmail.com>

Merge branch 'branch-22.02' into fix4028

ebced33

remove my test

2e0c084

Signed-off-by: remzi <13716567376yh@gmail.com>

HaoYang670 added the feature request New feature or request label Nov 26, 2021

verify scala style

232029d

Signed-off-by: remzi <13716567376yh@gmail.com>

pxLi changed the title ~~Support "GpuCast from non-nested ArrayType to StringType"~~ Support "GpuCast from non-nested ArrayType to StringType" [databricks] Nov 26, 2021

maven verify modified

ff20ca7

Signed-off-by: remzi <13716567376yh@gmail.com>

sperlingxx reviewed Nov 29, 2021

View reviewed changes

simplify some expressions

8e37820

Signed-off-by: remzi <13716567376yh@gmail.com>

revans2 changed the title ~~Support "GpuCast from non-nested ArrayType to StringType" [databricks]~~ GpuCast from ArrayType to StringType [databricks] Nov 29, 2021

revans2 requested changes Nov 29, 2021

View reviewed changes

add document, revise value name and switch to a simple stringConcaten…

b43dd0b

…ateListElements API Signed-off-by: remzi <13716567376yh@gmail.com>

HaoYang670 requested review from revans2 and sperlingxx November 30, 2021 09:40

revans2 reviewed Nov 30, 2021

View reviewed changes

ttnghia reviewed Nov 30, 2021

View reviewed changes

sperlingxx approved these changes Dec 1, 2021

View reviewed changes

HaoYang670 requested a review from revans2 December 1, 2021 02:39

revans2 approved these changes Dec 1, 2021

View reviewed changes

HaoYang670 merged commit f4ad835 into NVIDIA:branch-22.02 Dec 2, 2021

HaoYang670 deleted the fix4028 branch December 2, 2021 01:23

This was referenced Dec 2, 2021

[QST] Test the performance of casting array to string #4260

Closed

[QST]Move tests of array (struct) casting to the file cast_test.py #4262

Closed

HaoYang670 mentioned this pull request Dec 16, 2021

Speed up casting array to string when enabling legacy #4370

Open

revans2 mentioned this pull request Mar 7, 2022

[FEA] - Some unsupported operations on the GPU #4900

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GpuCast from ArrayType to StringType [databricks] #4221

GpuCast from ArrayType to StringType [databricks] #4221

HaoYang670 commented Nov 26, 2021 •

edited

Loading

HaoYang670 commented Nov 26, 2021

HaoYang670 commented Nov 26, 2021

pxLi commented Nov 26, 2021

HaoYang670 commented Nov 29, 2021

sperlingxx left a comment

sperlingxx Nov 29, 2021

HaoYang670 Dec 1, 2021

sperlingxx Nov 29, 2021

HaoYang670 Nov 29, 2021

sperlingxx Nov 29, 2021

HaoYang670 Nov 29, 2021

sperlingxx Nov 29, 2021

HaoYang670 Nov 29, 2021

sperlingxx left a comment •

edited

Loading

HaoYang670 commented Nov 29, 2021

revans2 Nov 29, 2021

HaoYang670 Nov 30, 2021

revans2 Nov 29, 2021

HaoYang670 Nov 30, 2021

revans2 Nov 30, 2021

HaoYang670 commented Nov 30, 2021

revans2 left a comment

revans2 Nov 30, 2021

ttnghia Nov 30, 2021

HaoYang670 Dec 1, 2021

sperlingxx left a comment

HaoYang670 commented Dec 7, 2021 •

edited

Loading

HaoYang670 commented Dec 7, 2021

HaoYang670 commented Dec 7, 2021

HaoYang670 commented Dec 8, 2021



		@pytest.mark.parametrize('data_gen', all_gens_for_cast_to_string, ids=idfn)
		def test_legacy_cast_array_to_string(data_gen):

	# conf with legacy cast to string on
	legacy_complex_types_to_string = {'spark.sql.legacy.castComplexTypesToString.enabled': 'true'}
	@pytest.mark.parametrize('data_gen', [StructGen([["first", boolean_gen], ["second", byte_gen], ["third", short_gen], ["fourth", int_gen], ["fifth", long_gen], ["sixth", string_gen], ["seventh", date_gen]])], ids=idfn)
	def test_legacy_cast_struct_to_string(data_gen):
	assert_gpu_and_cpu_are_equal_collect(
	lambda spark : unary_op_df(spark, data_gen).select(
	f.col('a').cast("STRING")),
	conf = legacy_complex_types_to_string)

	# https://github.com/NVIDIA/spark-rapids/issues/2309
	@pytest.mark.parametrize('cast_conf', ['LEGACY', 'SPARK311+'])
	def test_one_nested_null_field_legacy_cast(cast_conf):
	def was_broken_for_nested_null(spark):
	data = [
	(('foo',),),
	((None,),),
	(None,)
	]
	df = spark.createDataFrame(data)
	return df.select(df._1.cast(StringType()))

	assert_gpu_and_cpu_are_equal_collect(was_broken_for_nested_null, {
	'spark.sql.legacy.castComplexTypesToString.enabled': cast_conf == 'LEGACY'
	})


	# https://github.com/NVIDIA/spark-rapids/issues/2315
	@pytest.mark.parametrize('cast_conf', ['LEGACY', 'SPARK311+'])
	def test_two_col_struct_legacy_cast(cast_conf):
	def broken_df(spark):
	key_data_gen = StructGen([
	('a', IntegerGen(min_val=0, max_val=4)),
	('b', IntegerGen(min_val=5, max_val=9)),
	], nullable=False)
	val_data_gen = IntegerGen()
	df = two_col_df(spark, key_data_gen, val_data_gen)
	return df.select(df.a.cast(StringType())).filter(df.b > 1)

	assert_gpu_and_cpu_are_equal_collect(broken_df, {
	'spark.sql.legacy.castComplexTypesToString.enabled': cast_conf == 'LEGACY'
	})

	@pytest.mark.parametrize('data_gen', [StructGen([["first", float_gen]])], ids=idfn)
	@pytest.mark.xfail(reason='casting float to string is not an exact match')
	def test_legacy_cast_struct_with_float_to_string(data_gen):
	assert_gpu_and_cpu_are_equal_collect(
	lambda spark : unary_op_df(spark, data_gen).select(
	f.col('a').cast("STRING")),
	conf = legacy_complex_types_to_string)

	@pytest.mark.parametrize('data_gen', [StructGen([["first", double_gen]])], ids=idfn)
	@pytest.mark.xfail(reason='casting double to string is not an exact match')
	def test_legacy_cast_struct_with_double_to_string(data_gen):
	assert_gpu_and_cpu_are_equal_collect(
	lambda spark : unary_op_df(spark, data_gen).select(
	f.col('a').cast("STRING")),
	conf = legacy_complex_types_to_string)

	@pytest.mark.parametrize('data_gen', [StructGen([["first", timestamp_gen]])], ids=idfn)
	@pytest.mark.xfail(reason='https://github.com/NVIDIA/spark-rapids/issues/219')
	def test_legacy_cast_struct_with_timestamp_to_string(data_gen):
	assert_gpu_and_cpu_are_equal_collect(
	lambda spark : unary_op_df(spark, data_gen).select(
	f.col('a').cast("STRING")),
	conf = legacy_complex_types_to_string)

	@pytest.mark.parametrize('data_gen', [StructGen([["first", boolean_gen], ["second", byte_gen], ["third", short_gen], ["fourth", int_gen], ["fifth", long_gen], ["sixth", string_gen], ["seventh", date_gen]])], ids=idfn)
	def test_cast_struct_to_string(data_gen):
	assert_gpu_and_cpu_are_equal_collect(
	lambda spark : unary_op_df(spark, data_gen).select(
	f.col('a').cast("STRING")))

	@pytest.mark.parametrize('data_gen', [StructGen([["first", float_gen]])], ids=idfn)
	@pytest.mark.xfail(reason='casting float to string is not an exact match')
	def test_cast_struct_with_float_to_string(data_gen):
	assert_gpu_and_cpu_are_equal_collect(
	lambda spark : unary_op_df(spark, data_gen).select(
	f.col('a').cast("STRING")))

	@pytest.mark.parametrize('data_gen', [StructGen([["first", double_gen]])], ids=idfn)
	@pytest.mark.xfail(reason='casting double to string is not an exact match')
	def test_cast_struct_with_double_to_string(data_gen):
	assert_gpu_and_cpu_are_equal_collect(
	lambda spark : unary_op_df(spark, data_gen).select(
	f.col('a').cast("STRING")))

	@pytest.mark.parametrize('data_gen', [StructGen([["first", timestamp_gen]])], ids=idfn)
	@pytest.mark.xfail(reason='https://github.com/NVIDIA/spark-rapids/issues/219')
	def test_cast_struct_with_timestamp_to_string(data_gen):
	assert_gpu_and_cpu_are_equal_collect(
	lambda spark : unary_op_df(spark, data_gen).select(
	f.col('a').cast("STRING")))

GpuCast from ArrayType to StringType [databricks] #4221

GpuCast from ArrayType to StringType [databricks] #4221

Conversation

HaoYang670 commented Nov 26, 2021 • edited Loading

HaoYang670 commented Nov 26, 2021

HaoYang670 commented Nov 26, 2021

pxLi commented Nov 26, 2021

HaoYang670 commented Nov 29, 2021

sperlingxx left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sperlingxx left a comment • edited Loading

Choose a reason for hiding this comment

HaoYang670 commented Nov 29, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HaoYang670 commented Nov 30, 2021

revans2 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sperlingxx left a comment

Choose a reason for hiding this comment

HaoYang670 commented Dec 7, 2021 • edited Loading

HaoYang670 commented Dec 7, 2021

HaoYang670 commented Dec 7, 2021

HaoYang670 commented Dec 8, 2021

HaoYang670 commented Nov 26, 2021 •

edited

Loading

sperlingxx left a comment •

edited

Loading

HaoYang670 commented Dec 7, 2021 •

edited

Loading