[BUG] UDF compiler can cause crashes with unexpected class input

[revans2]

Describe the bug
We recently saw with a customer the UDF compiler running into issues when the UDFs appeared to not match the format that it expected. This resulted in errors like the following that crashed the application.

java.lang.RuntimeException: java.io.IOException: invalid constant type: 16 at 98
        at javassist.CtClassType.getClassFile2(CtClassType.java:204)
        at javassist.CtClassType.makeFieldCache(CtClassType.java:835)
        at javassist.CtClassType.getMembers(CtClassType.java:826)
        at javassist.CtClassType.getDeclaredMethod(CtClassType.java:1188)
        at com.nvidia.spark.udf.LambdaReflection$.apply(LambdaReflection.scala:139)
        at com.nvidia.spark.udf.CatalystExpressionBuilder.(CatalystExpressionBuilder.scala:46)
        at com.nvidia.spark.udf.GpuScalaUDFLogical.compile(GpuScalaUDF.scala:45)
        at com.nvidia.spark.udf.LogicalPlanRules.attemptToReplaceExpression(Plugin.scala:53)

Generally we should catch all exceptions in UDF compiler, log them, and fall back to not replacing the UDFs.

[Niharikadutta]

Can we please prioritize this? Having a stable version of UDF compiler is very important for our customers.

[sameerz]

@jlowe can this be closed based on PR #3404?

[jlowe]

can this be closed based on PR #3404?

Depends upon whether this issue is tracking the fact that the UDF compiler threw the invalid constant type or whether it's tracking the fact that the entire query failed as a result. If it's the latter then yes, but we may also want to understand why the UDF compiler threw the invalid constant exception.

[jlowe]

#3404 is the issue tracking the desire to not fail the query for a UDF compiler issue on a UDF even if that means leaving the UDF on the CPU. That was fixed by #3454. We'll leave this issue track the specific problem of invalid constant type: 16 at 98

[viadea]

As per the @Niharikadutta , using the aggregator jar of 21.10 works fine for this issue.
So should we assume this issue is fixed @Niharikadutta after you built the rapids-4-spark jar based on your spark dependency?

[Niharikadutta]

Yes this issue is fixed with the latest v21.10 jar. We can close this issue. Thanks!