[GLUTEN-7524][VL][UNIFFLE] Reset rss.row.based configuration of uniffle #7525
Conversation
| @@ -57,6 +57,8 @@ public <K, V> ShuffleWriter<K, V> getWriter( | |||
| } else { | |||
| writeMetrics = context.taskMetrics().shuffleWriteMetrics(); | |||
| } | |||
| sparkConf.set( | |||
| RssSparkConfig.SPARK_RSS_CONFIG_PREFIX + RssSparkConfig.RSS_ROW_BASED.key(), "false"); | |||
There was a problem hiding this comment.
It seems we have already set rss.row.based to false while initlizing UniflleShuffleManager, do we still need this line?
There was a problem hiding this comment.
Is it possible to create a fallbacked shuffle writer and then create a non-fallbacked shuffle writer?
There was a problem hiding this comment.
For example, there are two stages running on the same executor, one shuffle without fallback and one with fallback
There was a problem hiding this comment.
Good Catch. Just as @summaryzb said, this may cause some problem when there are two stages, with different rss.row.based, trying to create writer at the same time.
| @@ -57,6 +57,7 @@ public <K, V> ShuffleWriter<K, V> getWriter( | |||
| } else { | |||
| writeMetrics = context.taskMetrics().shuffleWriteMetrics(); | |||
| } | |||
| sparkConf.set(RssSparkConfig.SPARK_RSS_CONFIG_PREFIX + RssSparkConfig.RSS_ROW_BASED.key(), "false"); | |||
There was a problem hiding this comment.
This conf change may effect other columnar write task
There was a problem hiding this comment.
Currently it only affects shuffle writes, I have reset it when creating shuffle writer
There was a problem hiding this comment.
Oh, I see, how about we use a sparkconf.clone ?
There was a problem hiding this comment.
Sounds good to me. @summaryzb any better ideas?
What changes were proposed in this pull request?
closes #7524
How was this patch tested?