[BUG] Spark UT framework: Propagate Hadoop configs from parquet options to underlying file system

[Feng-Jiang28]

In Spark UT framework, there is a test case ensuring the Hadoop configurations are correctly propagated from the data source options to the underlying file system. '

val conf = Map("ds_option" -> "value", "mergeSchema" -> mergeSchema.toString)
inputDataset
   .write
   .options(conf)
   .format(dataSourceFormat)
   .save(path)

The test code then checks if the underlying FileSystem' conf has been set correctly.
However, the ds_option seems be to missed in FileSystem.

If I modified the test code to set spark.sparkContext.hadoopConfiguration.set("ds_option", "value"), then the test case will passs.

So the gap between Rapids and Spark is that options(conf) method on the DataFrameWriter of plugin cannot modify the Hadoop configuration.