Skip to content

Commit

Permalink
Improve the diagnostics in udf compiler for try-and-catch. (#3668)
Browse files Browse the repository at this point in the history
Signed-off-by: Sean Lee <selee@nvidia.com>
  • Loading branch information
seanprime7 authored Sep 27, 2021
1 parent 4ee8ed1 commit 034ba9b
Show file tree
Hide file tree
Showing 3 changed files with 24 additions and 0 deletions.
2 changes: 2 additions & 0 deletions docs/additional-functionality/udf-to-catalyst-expressions.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,3 +110,5 @@ When translating UDFs to Catalyst expressions, the supported UDF functions are l
| | lhs :+ rhs |
| Method call | Only if the method being called 1. Consists of operations supported by the UDF compiler, and 2. is one of the folllowing: a final method, a method in a final class, or a method in a final object |

All other expressions, including but not limited to `try` and `catch`, are unsupported and UDFs
with such expressions cannot be compiled.
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,10 @@ class LambdaReflection private(private val classPool: ClassPool,

private val codeAttribute = methodInfo.getCodeAttribute

if (codeAttribute.getExceptionTable.size > 0) {
throw new SparkException("exception handling (try-and-catch) isn't supported")
}

lazy val codeIterator: CodeIterator = codeAttribute.iterator

lazy val parameters: Array[CtClass] = ctMethod.getParameterTypes
Expand Down
18 changes: 18 additions & 0 deletions udf-compiler/src/test/scala/com/nvidia/spark/OpcodeSuite.scala
Original file line number Diff line number Diff line change
Expand Up @@ -890,6 +890,24 @@ class OpcodeSuite extends FunSuite {
checkEquivNotCompiled(result, ref)
}

test("FALLBACK TO CPU: exception handling - unsupported") {
val myudf1: Int => Int = { a =>
var x:Int = 0
try {
x = a
} catch {
case ex: Exception => { }
}
x
}

val u1 = makeUdf(myudf1)
val dataset = List(1, 5, 3, 7).toDS()
val result = dataset.withColumn("new", u1('value))
val ref = dataset.withColumn("new", col("value"))
checkEquivNotCompiled(result, ref)
}

test("conditional doubles test2") {
val myudf: Double => Double = { x =>
val t =
Expand Down

0 comments on commit 034ba9b

Please sign in to comment.