Add support of HDFS as remote object store

[yahoNanJing]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Currently, we can only read parquet files from local file system. It would be nice to add support to read parquet files that reside on HDFS.

Describe the solution you'd like

Describe alternatives you've considered

Additional context
Add any other context or screenshots about the feature request here.

[yahoNanJing]

@yjshen could you help review the related PR and give some comments?

[yjshen]

@yahoNanJing, Great to see this! Previously, we have some discussions at #907. Please also check the discussions there for more backgrounds.

[yjshen]

Besides, I have a partially implemented repo for native HDFS Rust binding, based on libhdfs3 originates from HAWQ. It will be great we can cooperate to make this happen.

https://github.com/yjshen/hdfs-native
The repo provides a wrapper of libhdfs3, which also originates from hdfs-rs as your fs-hdfs. I also have a HDFS object-store impl in src/hdfs_store.rs for fast development iteration.

[yjshen]

Per discussions in #907, a more preferable way for the community may be to put connectors such as S3, HDFS in their own repositories for fast development iterations, reduce unnecessarily dependencies and keep compilation time reasonable.

[yahoNanJing]

Agree to have a separate crate from each remote storage. Later will refine this PR and make it for the HDFS related crate.

[zuston]

Besides, I have a partially implemented repo for native HDFS Rust binding, based on libhdfs3 originates from HAWQ. It will be great we can cooperate to make this happen.

https://github.com/yjshen/hdfs-native The repo provides a wrapper of libhdfs3, which also originates from hdfs-rs as your fs-hdfs. I also have a HDFS object-store impl in src/hdfs_store.rs for fast development iteration.

Great work for https://github.com/yjshen/hdfs-native. Is it available for production env?

[drauschenbach]

Obsoleted by #1062 where consensus led to this feature landing in https://github.com/datafusion-contrib/hdfs-native-object-store.

[andygrove]

Closing this. Thanks @drauschenbach.