when we will support spark -gpu ,now we need spark -gpu feature to train big model

[mullerhai]

Hi, I know gluten is perfect tools for spark compute ,now we do some spark gpu work for train big model, need distribute train model with spark rdd/ dataframe data,so if we could support gpu ,maybe it is good

[weiting-chen]

What kind of the GPU work you are asking to support in Gluten?

1. To run Spark SQL on GPU
2. To run Spark SQL on CPU, then use Spark RDD/Dataframe to run Train model on GPU directly without writing to the disk.
3. Other scenarios

[mullerhai]

What kind of the GPU work you are asking to support in Gluten?

1. To run Spark SQL on GPU
2. To run Spark SQL on CPU, then use Spark RDD/Dataframe to run Train model on GPU directly without writing to the disk.
3. Other scenarios

just 2, as you know ,spark sql compute also complex and need a lot of time ,and train data also need more time ,these are need gpu support , I know your project want to add gpu support ,so ask the project owner

[weiting-chen]

Currently Gluten only focus on pure Spark SQL scenario, for case2, we are working with some customers to integrate ML/DL scenario with Gluten. There will be some works in 2nd half 2023, but no specific time guarantee.