A native Rust implementation of the Apache ORC file format, providing API's to read data into Apache Arrow in-memory arrays.
See the documentation for examples on how to use this crate.
This crate currently only supports reading ORC files into Arrow arrays. Write support is planned (see Roadmap). The below features listed relate only to reading ORC files. At this time, we aim to support the ORCv1 specification only.
- Read synchronously & asynchronously (using Tokio)
- All compression types (Zlib, Snappy, Lzo, Lz4, Zstd)
- All ORC data types
- All encodings
- Rudimentary support for retrieving statistics
- Retrieving user metadata into Arrow schema metadata
The long term vision for this crate is to be feature complete enough to be donated to the arrow-rs project.
The following lists the rough roadmap for features to be implemented, from highest to lowest priority.
- Performance enhancements
- DataFusion integration
- Predicate pushdown
- Row indices
- Bloom filters
- Write from Arrow arrays
- Encryption
A non-Arrow API interface is not planned at the moment. Feel free to raise an issue if there is such a use case.
No guarantees are provided about stability across versions. We will endeavour to keep the top level API's
(ArrowReader and ArrowStreamReader) as stable as we can, but other API's provided may change as we
explore the interface we want the library to expose.
Versions will be released on an ad-hoc basis (with no fixed schedule).
The following table lists how ORC data types are read into Arrow data types:
| ORC Data Type | Arrow Data Type | Notes |
|---|---|---|
| Boolean | Boolean | |
| TinyInt | Int8 | |
| SmallInt | Int16 | |
| Int | Int32 | |
| BigInt | Int64 | |
| Float | Float32 | |
| Double | Float64 | |
| String | Utf8 | |
| Char | Utf8 | |
| VarChar | Utf8 | |
| Binary | Binary | |
| Decimal | Decimal128 | |
| Date | Date32 | |
| Timestamp | Timestamp(Nanosecond, None) | ¹ |
| Timestamp instant | Timestamp(Nanosecond, UTC) | ¹ |
| Struct | Struct | |
| List | List | |
| Map | Map | |
| Union | Union(_, Sparse) | ² |
¹: ArrowReaderBuilder::with_schema allows configuring different time units or decoding to
Decimal128(38, 9) (i128 of non-leap nanoseconds since UNIX epoch).
Overflows may happen while decoding to a non-Seconds time unit, and results in OrcError.
Loss of precision may happen while decoding to a non-Nanosecond time unit, and results in OrcError.
Decimal128(38, 9) avoids both overflows and loss of precision.
²: Currently only supports a maximum of 127 variants
All contributions are welcome! Feel free to raise an issue if you have a feature request, bug report, or a question. Feel free to raise a Pull Request without raising an issue first, as long as the Pull Request is descriptive enough.
Some tools we use in addition to the standard cargo that require installation are:
cargo install typos-cli
cargo install taplo-cli# Building the crate
cargo build
# Running the test suite
cargo test
# Simple benchmarks
cargo bench
# Formatting TOML files
taplo format
# Detect any typos in the codebase
typosTo regenerate/update the proto.rs file, execute the regen.sh script.
./regen.sh