[FEA] Use conventional jar layout in dist jar if there is only one input shim #3682
Comments
gerashegalov
added
feature request
|
I personally disagree with this for exactly the same reasons pointed out here.
If what we ship and expect most people to use is the uber/parallel world jar, then I want everyone to be testing that situation. I personally build and test regularly for a single version because I would rather wait 3 mins to build a single version than 10 mins for buildall to finish. But if I am testing things in a very different way I don't want that. If someone has a targeted cloud environment where they are building, testing, and deploying for a single version of Spark, then they should make any changes needed to the build scripts. The common code should stay as is. |
|
I think the intent of my issue and the wording diverged in that it sounds generally applicable to I too advocate that we should definitely do most of our testing with the muti-shim jar, preferably minimumFeatureVersionMix profile. What I argue for vendor environments has little to do with the multi-shim build overhead, What we have there is an environment we have no control over, whereas we are pushing a jar that provides a solution that this type of environment most often doesn't have. E.g., if we deploy to Databricks for a specific runtime version known apriori there is absolutely no need for a complex mult-shim paralllel-world layout. Therefore, I think having an option to turn off non-standard jar layout creation at build time when needed is reasonable. Again, I agree that it should not be the default behavior. |
|
Can we wait on this until we have an actual customer that asks us for this? I don't really want to have to maintain/test a feature that someone somewhere might use. |
|
yes we can wait, i didn't put any release label on it. But if you will I am the "customer" asking for it to reduce our issues overhead with the vendors when possible. |
This PR closes #4649, #3682. - add a doc for JDK9+ property with potential cross-compilation to lower Java major versions - `allowConventionalDistJar` for conventional jar creation when only one shim is included - automatically detect conventional jar and suppress warnings about classloader issues - remove manual configuration of javac args for scala-maven-plugin. This plugin processes `maven .compiler.*` and adds the right set of javac opts automatically. 3.4.x we use for db doesn't handle maven.compiler.release, however, we don't need it at the moment and it's currently not used. - disambiguate calls to buf.position() generating errors on JDK11 Note this PR only builds skipping running tests `-DskipTests` because we still need to address #3851 (comment) Signed-off-by: Gera Shegalov <gera@apache.org>
|
This option was implemented in #5528 |
Is your feature request related to a problem? Please describe.
In a targeted cloud / vendor environment there is typically no need to have multi-spark-version jar artifact allowing to switch between Spark versions back and forth.
We expect that the dist jar for such a targeted environment is likely to be built using the
individualprofile. In this case it's unnecessary to use non-standard Parallel World jar layoutspark-rapids/dist/pom.xml
Line 1674 in a5963e5
Describe the solution you'd like
We need an option for dist jar based on a single
individualbuildver classified output to use the standard jar layout without/spark3*directories at the root.Describe alternatives you've considered
keep using PW layout even for single shim builds which elevates the risk of classloading issues
Additional context
#3381
The text was updated successfully, but these errors were encountered: