Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes a minor deprecation warning #4223

Merged
merged 1 commit into from
Nov 29, 2021

Conversation

res-life
Copy link
Collaborator

This fixes #3374

Signed-off-by: Chong Gao res_life@163.com

Signed-off-by: Chong Gao <res_life@163.com>
@res-life res-life linked an issue Nov 26, 2021 that may be closed by this pull request
@res-life
Copy link
Collaborator Author

1. getAllStatistics deprecated

[WARNING] [Warn] GpuMultiFileReader.scala:87: 
method getAllStatistics in class FileSystem is deprecated: 
see corresponding Javadoc for more information.

This is because of getAllStatistics is deprecated in hadoop-client-api-3.3.1.jar which is depended by spark 3.2.0.
Recommends to use
1
see:

https://hadoop.apache.org/docs/r2.8.2/api/org/apache/hadoop/fs/FileSystem.html

But getGlobalStorageStatistics does not provide the capacity to fetch thread local statistics like previous version:

FileSystem.getAllStatistics.asScala.map(_.getThreadStatistics.getBytesRead).sum

So, just leave it as it is. BTW, one issue is filed: #4219
2. getDecimalMetadata deprecated

[WARNING] [Warn] sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuParquetScan.scala:604: 
method getDecimalMetadata in class PrimitiveType is deprecated: 
see corresponding Javadoc for more information.

spark 3.0.1 uses parquet-column-1.10.1.jar and spark 3.2.0 uses parquet-column-1.12.1.jar.
parquet-column 1.12.1 deprecated the getDecimalMetadata method and recommends to use "getLogicalTypeAnnotation()".
parquet-column-1.10.1 does not have getLogicalTypeAnnotation.
So if want to remove the warning we must write shim codes separately for the 2 versions.
It's a little heavy, let's keep it.

3. isDirectory deprecated

sql-plugin/src/main/scala/org/apache/spark/sql/execution/datasources/rapids/GpuPartitioningUtils.scala:121:
 method isDirectory in class FileSystem is deprecated: 
see corresponding Javadoc for more information.

Fixed.

@res-life
Copy link
Collaborator Author

build

@res-life res-life merged commit cd725b8 into NVIDIA:branch-22.02 Nov 29, 2021
@pxLi pxLi added the bug Something isn't working label Nov 29, 2021
@res-life res-life deleted the minor-warnings branch March 13, 2022 05:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] minor deprecation warnings in a 3.2 shim build
4 participants