Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix GpuCreateNamedStruct not serializable issue #2442

Merged
merged 6 commits into from
May 20, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,18 @@ object GpuExpressionsUtils extends Arm {
*/
def columnarEvalToColumn(expr: Expression, batch: ColumnarBatch): GpuColumnVector =
resolveColumnVector(expr.columnarEval(batch), batch.numRows, expr.dataType)

/**
* Extract the GpuLiteral
* @param exp the input expression to be extracted
* @return an optional GpuLiteral
*/
@scala.annotation.tailrec
def extractGpuLit(exp: Expression): Option[GpuLiteral] = exp match {
revans2 marked this conversation as resolved.
Show resolved Hide resolved
case gl: GpuLiteral => Some(gl)
case ga: GpuAlias => extractGpuLit(ga.child)
case _ => None
}
}

/**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -87,8 +87,16 @@ case class GpuCreateNamedStruct(children: Seq[Expression]) extends GpuExpression
case Seq(name, value) => (name, value)
}.toList.unzip

// Names will be serialized before Spark scheduling, and the returned type GpuScalar
// from GpuLiteral.columnarEval(null) can't be serializable, which causes
// `org.apache.spark.SparkException: Task not serializable` issue.
//
// And on the other hand, the calling for columnarEval(null) in the driver side is
// dangerous for GpuExpressions, we'll have to pull it apart manually.
private lazy val names = nameExprs.map {
case g: GpuExpression => g.columnarEval(null)
case ge: GpuExpression =>
GpuExpressionsUtils.extractGpuLit(ge).map(_.value)
.getOrElse(throw new IllegalStateException(s"Unexpected GPU expression $ge"))
case e => e.eval(EmptyRow)
}

Expand Down