Remove shims module [databricks]

[razajafri]

This is the first commit in a series of commit. The goal is to get rid of v1 shims usage and replace it with v2 usage

• Source files from shims module were moved to sql-plugin module
• Test files were moved to test folder inside sql-plugin module
• Added the ability to select test files based on profile for testing shims
• Moved README.md from shims module to docs/dev/ and renamed it to shims.md.
• Modified the relative links in the doc to match the new location

[razajafri]

build

[razajafri]

build

[razajafri]

build

[razajafri]

build

[razajafri]

CI failed due to [ERROR] Failed to execute goal on project rapids-4-spark-udf_2.12: Could not resolve dependencies for project com.nvidia:rapids-4-spark-udf_2.12:jar:22.04.0-SNAPSHOT: Could not find artifact com.nvidia:rapids-4-spark-shims-spark312db_2.12:jar:22.04.0-SNAPSHOT in snapshots-repo (https://oss.sonatype.org/content/repositories/snapshots) -> [Help 1]

Are we missing some jars for 22.04 for db?

[pxLi]

CI failed due to [ERROR] Failed to execute goal on project rapids-4-spark-udf_2.12: Could not resolve dependencies for project com.nvidia:rapids-4-spark-udf_2.12:jar:22.04.0-SNAPSHOT: Could not find artifact com.nvidia:rapids-4-spark-shims-spark312db_2.12:jar:22.04.0-SNAPSHOT in snapshots-repo (https://oss.sonatype.org/content/repositories/snapshots) -> [Help 1]

Are we missing some jars for 22.04 for db?

nope, the rapids-4-spark-shims-spark301db&rapids-4-spark-shims-spark312db_2.12 was supposed be built with in this CI (but this PR remove the 301db and 312db shims), and we never deploy nightly shims to sonatype snapshot repo.

failed at Building RAPIDS Accelerator for Apache Spark Scala UDF Plugin 22.04.0-SNAPSHOT [4/9]. This may require some dependency update. And there are still a few other modules have dep of
rapids-4-spark-shims-${spark.version.classifier}_${scala.binary.version}

FYI for pre-merge, since we have old nightly snapshot of shims on URM, so test in blossom could pass the build (which is not correct actually). And on DB runtime, it correctly failed as no those deps available.

[jlowe]

Ah, right, I totally missed in the review that a number of other poms (aggregator, tests, etc.) need to be updated, as they explicitly depend on the shims artifact being removed.

@razajafri you can build in a clean repo, either by cleaning out your local ~/.m2 or using something like -Dmaven.repo.local=/tmp/m2temprepo to reproduce these types of errors locally.

[razajafri]

Ah, right, I totally missed in the review that a number of other poms (aggregator, tests, etc.) need to be updated, as they explicitly depend on the shims artifact being removed.

@razajafri you can build in a clean repo, either by cleaning out your local ~/.m2 or using something like -Dmaven.repo.local=/tmp/m2temprepo to reproduce these types of errors locally.

aah! of course I should've removed the dependency from other modules but I don't get how the CI was passing until I built on databricks. Does the CI reuse previously used maven repo? Nevermind, read @pxLi comment above. This is something we should probably fix

[razajafri]

build

[razajafri]

build

[tgravescs]

build