Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add maven compile/package plugin executions for Spark302 and Spark301 #3257

Merged
merged 45 commits into from
Aug 23, 2021
Merged

Add maven compile/package plugin executions for Spark302 and Spark301 #3257

Show file tree
Hide file tree
Changes from 37 commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
a02f789
initial pom files
tgravescs Aug 16, 2021
02aae7b
Copy code from 3.0.1 shim to 3.1.1
tgravescs Aug 16, 2021
b99369d
fixes
tgravescs Aug 16, 2021
d15a5b5
Add Quatenary
tgravescs Aug 16, 2021
0a5ef7f
Merge branch 'build301' of github.com:tgravescs/spark-rapids into bui…
tgravescs Aug 16, 2021
47befb9
checkpoint
tgravescs Aug 16, 2021
7946e70
checkpoint
tgravescs Aug 16, 2021
e7f49aa
checkpoint
tgravescs Aug 16, 2021
158f41e
working build both
tgravescs Aug 17, 2021
c3a4d82
working include in jars
tgravescs Aug 17, 2021
e9a18fb
java trying to work
tgravescs Aug 17, 2021
f304417
checkpoint
tgravescs Aug 17, 2021
ec591cb
checkpoint
tgravescs Aug 17, 2021
9af5f62
revert poms to oroginal and add classifiers
tgravescs Aug 18, 2021
5f3e16e
Working build
tgravescs Aug 18, 2021
24b1e6c
cleanup
tgravescs Aug 18, 2021
d073913
Merge remote-tracking branch 'origin/branch-21.10' into buildperspark
tgravescs Aug 18, 2021
de3bb0c
upmerge
tgravescs Aug 18, 2021
5e0b745
cleanup
tgravescs Aug 18, 2021
4f44dc0
upmerge and cleanup
tgravescs Aug 18, 2021
6e9f929
move thigns to be inside profiles
tgravescs Aug 18, 2021
b29f19b
fix dependencies default
tgravescs Aug 18, 2021
ba214ea
clean
tgravescs Aug 18, 2021
2bcc995
work around dependency missing
tgravescs Aug 18, 2021
f8c625e
move dependencies inside profile to package properly
tgravescs Aug 18, 2021
8849b6a
revert 311 changes
tgravescs Aug 18, 2021
1e8da74
remove release 311 profiles
tgravescs Aug 18, 2021
4a9cdd5
fix shim dependencies
tgravescs Aug 18, 2021
5129b25
revert change for 3.1.1
tgravescs Aug 18, 2021
7f8ba94
Merge remote-tracking branch 'origin/branch-21.10' into buildperspark
tgravescs Aug 19, 2021
3641c45
change name of integration tests jar
tgravescs Aug 19, 2021
84b6945
change to use with-classifier profile
tgravescs Aug 19, 2021
ced230e
move dependencies back to wwhere they were
tgravescs Aug 19, 2021
7d251d9
fix copyright
tgravescs Aug 19, 2021
a16a567
fix spacing
tgravescs Aug 19, 2021
f7ab378
Fix dependencies on dist pre-merge profile
tgravescs Aug 20, 2021
19baa0f
Fix integration test depdnencies
tgravescs Aug 20, 2021
6c48775
Add property to set 2 profiles, address review comments
tgravescs Aug 20, 2021
3dca739
debug premerge failures print active profiles
tgravescs Aug 20, 2021
c37f29a
disable mvn_verify
tgravescs Aug 20, 2021
854e093
enable mvn debug
tgravescs Aug 20, 2021
9f99ae8
revert premerge build script changes
tgravescs Aug 20, 2021
d8ec7c1
Merge remote-tracking branch 'origin/branch-21.10' into buildperspark
tgravescs Aug 20, 2021
7c786ba
Update 302 to match changes to 301 shim
tgravescs Aug 20, 2021
3b9e4f1
Update missed 301 dep
tgravescs Aug 20, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
208 changes: 144 additions & 64 deletions dist/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,40 +29,150 @@
<description>Creates the distribution package of the RAPIDS plugin for Apache Spark</description>
<version>21.10.0-SNAPSHOT</version>

<dependencies>
<dependency>
<groupId>com.nvidia</groupId>
<artifactId>rapids-4-spark-sql_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>com.nvidia</groupId>
<artifactId>rapids-4-spark-shuffle_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>com.nvidia</groupId>
<artifactId>rapids-4-spark-shims-aggregator_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>com.nvidia</groupId>
<artifactId>rapids-4-spark-udf_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<!-- required for conf generation script -->
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.binary.version}</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<!-- required for conf generation script -->
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_${scala.binary.version}</artifactId>
<scope>provided</scope>
</dependency>
</dependencies>
<profiles>
<profile>
<id>default</id>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
<dependencies>
<dependency>
<groupId>com.nvidia</groupId>
<artifactId>rapids-4-spark-sql_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>com.nvidia</groupId>
<artifactId>rapids-4-spark-shuffle_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>com.nvidia</groupId>
<artifactId>rapids-4-spark-udf_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>com.nvidia</groupId>
<artifactId>rapids-4-spark-shims-aggregator_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<!-- required for conf generation script -->
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.binary.version}</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<!-- required for conf generation script -->
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_${scala.binary.version}</artifactId>
<scope>provided</scope>
</dependency>
jlowe marked this conversation as resolved.
Show resolved Hide resolved
</dependencies>
</profile>
<profile>
<id>with-classifier</id>
jlowe marked this conversation as resolved.
Show resolved Hide resolved
<dependencies>
<dependency>
<groupId>com.nvidia</groupId>
<artifactId>rapids-4-spark-sql_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<classifier>${spark.version.classifier}</classifier>
</dependency>
<dependency>
<groupId>com.nvidia</groupId>
<artifactId>rapids-4-spark-shuffle_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<classifier>${spark.version.classifier}</classifier>
</dependency>
<dependency>
<groupId>com.nvidia</groupId>
<artifactId>rapids-4-spark-udf_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<classifier>${spark.version.classifier}</classifier>
</dependency>
<dependency>
<groupId>com.nvidia</groupId>
<artifactId>rapids-4-spark-shims-aggregator_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<!-- required for conf generation script -->
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.binary.version}</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<!-- required for conf generation script -->
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_${scala.binary.version}</artifactId>
<scope>provided</scope>
</dependency>
</dependencies>
</profile>
<profile>
<id>pre-merge</id>
<dependencies>
<dependency>
<groupId>com.nvidia</groupId>
<artifactId>rapids-4-spark-sql_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>com.nvidia</groupId>
<artifactId>rapids-4-spark-shuffle_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>com.nvidia</groupId>
<artifactId>rapids-4-spark-udf_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>com.nvidia</groupId>
<artifactId>rapids-4-spark-shims-aggregator_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<!-- required for conf generation script -->
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.binary.version}</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<!-- required for conf generation script -->
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_${scala.binary.version}</artifactId>
<scope>provided</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<executions>
<execution>
<id>if_modified_files</id>
<phase>verify</phase>
<goals>
<goal>exec</goal>
</goals>
<configuration>
<executable>bash</executable>
<commandlineArgs>-c 'export MODIFIED=$(git status --porcelain | grep "^ M"); [[ -z $MODIFIED ]] &amp;&amp; exit 0 || { echo -e "found modified files during mvn verify:\n$MODIFIED"; exit 1;}'</commandlineArgs>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
<activation>
<activeByDefault>false</activeByDefault>
jlowe marked this conversation as resolved.
Show resolved Hide resolved
</activation>
</profile>
</profiles>


<build>
<plugins>
Expand Down Expand Up @@ -206,34 +316,4 @@
</plugins>
</build>

<profiles>
<profile>
<id>pre-merge</id>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<executions>
<execution>
<id>if_modified_files</id>
<phase>verify</phase>
<goals>
<goal>exec</goal>
</goals>
<configuration>
<executable>bash</executable>
<commandlineArgs>-c 'export MODIFIED=$(git status --porcelain | grep "^ M"); [[ -z $MODIFIED ]] &amp;&amp; exit 0 || { echo -e "found modified files during mvn verify:\n$MODIFIED"; exit 1;}'</commandlineArgs>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
<activation>
<activeByDefault>false</activeByDefault>
</activation>
</profile>
</profiles>

</project>
74 changes: 61 additions & 13 deletions integration_tests/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,21 +67,69 @@
<version>${project.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.nvidia</groupId>
<artifactId>rapids-4-spark-tests_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<type>test-jar</type>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.nvidia</groupId>
<artifactId>rapids-4-spark-udf-examples_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<scope>test</scope>
</dependency>
</dependencies>

<profiles>
<profile>
<id>default</id>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
<dependencies>
<dependency>
<groupId>com.nvidia</groupId>
<artifactId>rapids-4-spark-tests_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<type>test-jar</type>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.nvidia</groupId>
<artifactId>rapids-4-spark-udf-examples_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<scope>test</scope>
</dependency>
</dependencies>
</profile>
<profile>
<id>with-classifier</id>
<dependencies>
<dependency>
<groupId>com.nvidia</groupId>
<artifactId>rapids-4-spark-tests_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<classifier>${spark.version.classifier}tests</classifier>
<type>test-jar</type>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.nvidia</groupId>
<artifactId>rapids-4-spark-udf-examples_${scala.binary.version}</artifactId>
jlowe marked this conversation as resolved.
Show resolved Hide resolved
<version>${project.version}</version>
<classifier>${spark.version.classifier}</classifier>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<configuration>
<classifier>${spark.version.classifier}</classifier>
</configuration>
</plugin>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<finalName>rapids-4-spark-integration-tests_${scala.binary.version}-${project.version}-${spark.version.classifier}</finalName>
</configuration>
</plugin>
</plugins>
</build>
</profile>
</profiles>

<build>
<plugins>
<plugin>
Expand Down
21 changes: 21 additions & 0 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,22 @@
</modules>

<profiles>
<profile>
<id>release301</id>
<properties>
<spark.version.classifier>spark301</spark.version.classifier>
<spark.version>${spark301.version}</spark.version>
<spark.test.version>${spark301.version}</spark.test.version>
</properties>
</profile>
<profile>
<id>release302</id>
<properties>
<spark.version.classifier>spark302</spark.version.classifier>
<spark.version>${spark302.version}</spark.version>
<spark.test.version>${spark302.version}</spark.test.version>
</properties>
</profile>
<profile>
<id>udf-compiler</id>
<modules>
Expand Down Expand Up @@ -232,6 +248,7 @@
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<spark.version>${spark301.version}</spark.version>
<spark.version.classifier></spark.version.classifier>
<spark.test.version>${spark301.version}</spark.test.version>
<cuda.version>cuda11</cuda.version>
<cudf.version>21.10.0-SNAPSHOT</cudf.version>
Expand Down Expand Up @@ -592,9 +609,13 @@
<version>${maven.jar.plugin.version}</version>
<executions>
<execution>
<id>default-test-jar</id>
<goals>
<goal>test-jar</goal>
</goals>
<configuration>
<classifier>${spark.version.classifier}tests</classifier>
</configuration>
</execution>
</executions>
</plugin>
Expand Down
Loading