Support coalescing reading for avro #5306

firestarman · 2022-04-25T11:35:51Z

This PR is to enable the coalescing reading for avro.

It has mainly

added the relevant configs.
created the classes for the multi-file reading factory and reader, along with some utils.
done some small refactor to reduce some duplicated code.
updated the avro tests for the coalescing reading case.

Performance on Local

CPU 12 cores, and one GPU (Titan V, with 12GB memory)
Non-partitioned 2000 avro files, 4.4GB in total in LOCAL storage

CPU PERFILE COALESCING

time(sec) 27.844 24.758 16.005

closes #5149

Signed-off-by: Firestarman firestarmanllc@gmail.com

Signed-off-by: Firestarman <firestarmanllc@gmail.com>

firestarman · 2022-04-25T11:36:36Z

build

Signed-off-by: Firestarman <firestarmanllc@gmail.com>

firestarman · 2022-04-26T00:56:05Z

build

firestarman · 2022-04-26T01:33:28Z

A follow-up issue #5312

integration_tests/src/main/python/avro_test.py

wbo4958 · 2022-04-27T02:15:57Z

tests/src/test/scala/com/nvidia/spark/rapids/GpuReaderTypeSuites.scala

+
+  private lazy val hasAvroJar = ExternalSource.hasSparkAvroJar
+
+  test("Use coalescing reading for local files") {


Looks like many duplicated test code for avro and MultiReaderTypeSuite ?

I think you can wrap the same code into MultiReaderTypeSuite and pass the "assume condition" to it

This implementation is clear and easy to understand, and these tests will be removed once multi-threaded reading is done.

wbo4958 · 2022-04-27T02:24:29Z

sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuAvroScan.scala

+      val singleFileInfo = try {
+        filterHandler.filterBlocks(file)
+      } catch {
+        case e: FileNotFoundException if ignoreMissingFiles =>


Could you add the related tests for this function?

Do you mean unit test or IT ?

I failed to figure out how to create such case ? Also there are no related tests in ORC and Parquet.
Do you have any suggestion for this ?

sorry for the wrong place. I mean we should add a related test for corrupt files, Please refer to test_parquet_read_with_corrupt_files

Added it here https://github.com/NVIDIA/spark-rapids/pull/5421/files#diff-77f524a6136cfcfe5884cbc67ecd7e5dca8f8faed9837b21cdb3140231451e5fR103

sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuAvroScan.scala

wbo4958 · 2022-04-27T02:27:52Z

sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuAvroScan.scala

+            val headerAndBlocks = BlockInfo(0, header.firstBlockStart, 0, 0) +: blocks
+            copyBlocksData(headerAndBlocks, in, out)
+            // check we didn't go over memory
+            if (out.getPos > estOutSize) {


Could we do re-allocate here instead of throwing an exception?

I just copied the code from ORC and Parquet. If this is a corner case, I will do it in a follow-up PR.

wbo4958 · 2022-04-27T02:30:40Z

sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuAvroScan.scala

+      out: OutputStream): Seq[BlockInfo] = {
+    val copyRanges = computeCopyRanges(blocks)
+    // copy cache: 8MB
+    val copyCache = new Array[Byte](8 * 1024 * 1024)


Would you consider to use the same way like parquet?

val copyBufferSize = conf.getInt("parquet.read.allocation.size", 8 * 1024 * 1024)

Yes, nice catch, I am doing it and it will come with the multi-threaded reading code.

Please file an followup, just in case you have forgot this

Added it here https://github.com/NVIDIA/spark-rapids/pull/5421/files#diff-c6a5c198f62669478a7f1b33703a4a2f1cc901190de4d75a56949174351db4b3R291

sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuAvroScan.scala

integration_tests/src/main/python/avro_test.py

sql-plugin/src/main/scala/com/nvidia/spark/rapids/AvroDataReader.scala

tests/pom.xml

sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuAvroScan.scala

sql-plugin/src/main/scala/com/nvidia/spark/rapids/AvroDataReader.scala

Signed-off-by: Firestarman <firestarmanllc@gmail.com>

firestarman · 2022-04-29T04:22:51Z

build

Signed-off-by: Firestarman <firestarmanllc@gmail.com>

firestarman · 2022-04-29T13:56:57Z

build

sql-plugin/src/main/scala/com/nvidia/spark/rapids/AvroDataReader.scala

pom.xml

sql-plugin/src/main/scala/com/nvidia/spark/rapids/AvroDataReader.scala

sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuAvroScan.scala

Signed-off-by: Firestarman <firestarmanllc@gmail.com>

firestarman · 2022-05-02T09:04:42Z

build

sql-plugin/src/main/scala/com/nvidia/spark/rapids/AvroDataFileReader.scala

Signed-off-by: Firestarman <firestarmanllc@gmail.com>

firestarman · 2022-05-03T01:45:23Z

build

Signed-off-by: Firestarman <firestarmanllc@gmail.com>

firestarman · 2022-05-03T01:58:29Z

build

Signed-off-by: Firestarman <firestarmanllc@gmail.com>

firestarman · 2022-05-03T03:13:45Z

build

Support coalescing reading for avro

61f7a2e

Signed-off-by: Firestarman <firestarmanllc@gmail.com>

firestarman requested review from jlowe, revans2, tgravescs, GaryShen2008 and NvTimLiu as code owners April 25, 2022 11:35

Remove unused code

9009e5e

Signed-off-by: Firestarman <firestarmanllc@gmail.com>

sameerz added the performance A performance related task/issue label Apr 25, 2022

sameerz added this to the Apr 18 - Apr 29 milestone Apr 25, 2022

firestarman mentioned this pull request Apr 26, 2022

[BUG] The coalesced AVRO file may contain different sync markers if the sync marker varies in the avro files being coalesced. #5312

Closed

wbo4958 reviewed Apr 27, 2022

View reviewed changes

firestarman mentioned this pull request Apr 28, 2022

[FEA]Sort the single block info by the header for AVRO coalescing reading. #5347

Open

firestarman requested a review from wbo4958 April 28, 2022 12:51

jlowe reviewed Apr 28, 2022

View reviewed changes

address the comments

f1554df

Signed-off-by: Firestarman <firestarmanllc@gmail.com>

firestarman requested a review from jlowe April 29, 2022 04:22

A small fix

9f7e893

Signed-off-by: Firestarman <firestarmanllc@gmail.com>

jlowe reviewed Apr 29, 2022

View reviewed changes

sameerz modified the milestones: Apr 18 - Apr 29, May 2 - May 20 Apr 29, 2022

Address the new comments

22a45d4

Signed-off-by: Firestarman <firestarmanllc@gmail.com>

firestarman requested a review from jlowe May 2, 2022 09:04

jlowe reviewed May 2, 2022

View reviewed changes

sql-plugin/src/main/scala/com/nvidia/spark/rapids/AvroDataFileReader.scala Outdated Show resolved Hide resolved

Merge branch 'branch-22.06' into coal-avro

f27db65

Add the latest comment

ce84a87

Signed-off-by: Firestarman <firestarmanllc@gmail.com>

firestarman requested a review from jlowe May 3, 2022 01:45

Remove the used config

6212b9f

Signed-off-by: Firestarman <firestarmanllc@gmail.com>

Update the doc

3ab0ac9

Signed-off-by: Firestarman <firestarmanllc@gmail.com>

jlowe approved these changes May 3, 2022

View reviewed changes

firestarman merged commit 4b13b04 into NVIDIA:branch-22.06 May 4, 2022

firestarman deleted the coal-avro branch May 4, 2022 10:07

firestarman mentioned this pull request May 6, 2022

Make the sync marker uniform for the Avro coalescing reader #5428

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support coalescing reading for avro #5306

Support coalescing reading for avro #5306

firestarman commented Apr 25, 2022

firestarman commented Apr 25, 2022

firestarman commented Apr 26, 2022

firestarman commented Apr 26, 2022

wbo4958 Apr 27, 2022

wbo4958 Apr 27, 2022

firestarman Apr 28, 2022 •

edited

Loading

wbo4958 Apr 27, 2022

firestarman Apr 28, 2022 •

edited

Loading

firestarman Apr 28, 2022

wbo4958 Apr 29, 2022

firestarman May 6, 2022

wbo4958 Apr 27, 2022

firestarman Apr 28, 2022

wbo4958 Apr 27, 2022

firestarman Apr 28, 2022 •

edited

Loading

wbo4958 Apr 29, 2022

firestarman May 6, 2022

firestarman commented Apr 29, 2022

firestarman commented Apr 29, 2022

firestarman commented May 2, 2022

firestarman commented May 3, 2022

firestarman commented May 3, 2022

firestarman commented May 3, 2022


		private lazy val hasAvroJar = ExternalSource.hasSparkAvroJar

		test("Use coalescing reading for local files") {

Support coalescing reading for avro #5306

Support coalescing reading for avro #5306

Conversation

firestarman commented Apr 25, 2022

Performance on Local

firestarman commented Apr 25, 2022

firestarman commented Apr 26, 2022

firestarman commented Apr 26, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

firestarman Apr 28, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

firestarman Apr 28, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

firestarman Apr 28, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

firestarman commented Apr 29, 2022

firestarman commented Apr 29, 2022

firestarman commented May 2, 2022

firestarman commented May 3, 2022

firestarman commented May 3, 2022

firestarman commented May 3, 2022

firestarman Apr 28, 2022 •

edited

Loading

firestarman Apr 28, 2022 •

edited

Loading

firestarman Apr 28, 2022 •

edited

Loading