No Spark Shim Provider found for 3.2.0

[puchengy]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

We run Spark-3.2 in house, when trying gluten + velox, we see issue "No Spark Shim Provider found for 3.2.0".

Describe the solution you'd like

Support Spark 3.2.0 version.

Describe alternatives you've considered

No alternatives I can think of right now.

Additional context

Nan.

[puchengy]

by any chance we can provide more flexibility over the spark minor version?

[CodingCat]

we need to run tests against a new version when adding a new minor one , because Spark seems not as compatible as years before over minor versions, <= might be a challenge given the current CI seems already slow

I had an internal patch to support minor versions and particularly I tested against 3.2.0 ....but may need to sync with intel folks before I submitted a PR

cc @FelixYBW

[FelixYBW]

@CodingCat Thank you. Sure, can you submit the PR? and mention it's not tracked by our CI tests in documents?

@puchengy, As @CodingCat mentioned exactly, Spark's API changed on minor versions, so each minor version needs some patches. But our CI already take more than 1 hour, it's not possible to track on minor versions. So we can add the minor version support but it won't be tracked by CI tests. So we decided only support dedicated versions now. We can accept the minor version support PR but will document clearly on this.

[puchengy]

@FelixYBW Thank you, do we have easy enough steps to run CI for the minor version (spark-3.2.0)?

If so, my team can run these CI internally whenever we plan on a new release. Thanks

[FelixYBW]

Yes, the CI script is in the repo. @zhztheplayer How @puchengy can start from?

[zhztheplayer]

Velox's GitHub Action CI job is basically capable for being ran from another environment:

https://github.com/oap-project/gluten/blob/3a8e7174f748aafb34cda56b289a65e9e41ae078/.github/workflows/unittests.yml#L58-L115

The job is docker-based so is portable. You can replay the steps in your machine after installing a docker service.