[FEA]Audit - [SPARK-34605][SQL] Support java.time.Duration as an external type of the day-time interval type

[tgravescs]

Is your feature request related to a problem? Please describe.
spark made a change to support java.time.Duration for SPARK-27790 Support ANSI SQL INTERVAL types

apache/spark@17601e0

It looks like we copied the getConverterForType function in GpuRowToColumnarExec so we should look closer at pulling that in.

https://issues.apache.org/jira/browse/SPARK-34605

[sameerz]

@tgravescs will review with @revans2

[tgravescs]

getConverterFortype is used in GpuRowToColumnarExec:
https://github.com/NVIDIA/spark-rapids/blob/branch-0.5/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuRowToColumnarExec.scala#L94

I don't know if there is already another lira to add support for DayTimeIntervalType. I assume we need to add it to our isSupportedType checks and so forth as well. Although we may just let it fall though, do we have a standard for this @revans2 ?

[sameerz]

P1 for whether we support time.Duration for the cached batch serializer. P3 for whether we are going to support time.Duration.

[revans2]

There are a number of new types that have been added into spark and if we run into any of them we fall back to the CPU for it. Eventually we want to add in more types, but that should not block any of the existing code/queries from working.

[res-life]

Duration is an external type, Plugin do not need to handle it.
The Duration is used when creating a df, see below:

schema = StructType([
    StructField("c1", DayTimeIntervalType(
                DayTimeIntervalType.DAY, DayTimeIntervalType.DAY
            ), False),
])
data = [(timedelta( days=50, seconds=27), ),]
df = spark.createDataFrame(
        SparkContext.getOrCreate().parallelize(data, numSlices=2),
        schema)

day-time interval type is stored as int64 internally in Spark,
and Spark also stores it as int64 to parquet file along with an metada which indicating the mapping between day-time interval and int64

This issue supports ANSI intervals to/from Parquet:
#4145

Can we close it? @revans2

[revans2]

I think we can close this because we can do columnar to row and row to columnar on these types.