[FEA] Rework the shim layer to robustly handle ABI and API incompatibilities across Spark releases

[gerashegalov]

Is your feature request related to a problem? Please describe.

Apache Spark uses semantic versioning major.feature.patch, e.g. 3.1.2.

• Most compatibility promises are moot for the Plugin because our integration points are in the developer API and internal API land, not the Public APIs this language pertains to.
• Even if the compatibility guarantees were applicable we need to deal with breaking changes because we support multiple feature release lines 3.0.1+, 3.1.1+, and the future 3.2.x. In practice, incompatibilities at the API and ABI level may occur much more often in internal APIs.
• Per documentation and simple bytecode comparison, even if there is compatibility, there is often no ABI compatibility, just API (source) compatibility requiring recompilation. Thus our current implementation relying on the version-independent compilation using 3.0.1 as a dependency runs into runtime failures with the other Spark versions that is only detected with tests.

Describe the solution you'd like
A solution that catches 99% of the issues at build time and is robust at run time.

Describe alternatives you've considered
Rely on the existing solution that is based on test coverage to identify the issues at run time

Additional context
Prototype branches such as

• https://github.com/gerashegalov/spark-rapids/tree/reshim-source-deps-classifier-and-parallel-classloader-spark320-2
• https://github.com/gerashegalov/spark-rapids/tree/reshim-source-deps-classifier-and-parallel-classloader-spark320-refactor-on-the-fly

Tasks:

• Build one spark version at a time - [FEA] Add maven compile/package plugin executions, one for each supported Spark dependency version #3224
• Add shimming against incompatibility "inherited" via changes to catalyst TreeNode derivatives - Enable optional Spark 3.2.0 shim build #3323
• Classloader changes - Implement per-shim parallel world jar classloader #3381
• Build changes for combining classifier versions of Spark into single dist package - [FEA] Rework the shim layer to robustly handle ABI and API incompatibilities across Spark releases #3223 (comment)
• [DOC] Instructions to setup IDEs #3338
• Fix shim override config #3495
• ParquetCachedBatchSerializer move to common place for all spark 3.1+ builds - [FEA] Create non-shim specific version of ParquetCachedBatchSerializer #3314
• Fix any deploy targets and update jenkins builds
• deduplicate shim layer code
  • source code level dedupe of internal classes with the sole difference in the package name - Commonize more shim module files [databricks] #3577
  • binary class file level dedupe that consolidates identical classes even though compiled under different Spark versions in spark3xx-common Dynamic class file deduplication across shims in dist jar build  #3545
• remove unneeded build profiles after switching
• reduce pom sizes by refactoring Make new build default and combine into dist package #3411 (comment)
• remove 301emr shim - Remove 301emr shim #3443
• Test udf-examples native build
• test UCX with all shim flavors
• Kryo registartion: [BUG] GpuKryoRegistrator ClassNotFoundException  #3468
• All 3.2 integration tests pass [FEA] Enable integration tests in CI for Spark 3.2.0 without failing our regular builds  #3483

These are optional or nice to have tasks:

• Integration tests should only pick up one jar, also would need update builds for individual spark jars
• Build scripts to build all easily
• external facing api's into its own module (optional)

[tgravescs]

reopen because has other subtasks

[abellina]

Should we add a subtask about testing with UCX for each Spark distro?

[gerashegalov]

@abellina sure, go ahead re: testing with UCX for each Spark distro. Also we have the optimization / fix you suggested offline about triggering the shuffle manager init on the driver earlier that you can add

[tgravescs]

udf examples native build is passing so marking that off

[tgravescs]

all required tasks are complete so closing this.