Page indexes in decode_metadata and encode_metadata

[adriangb]

Hi folks,

I'm trying to work with parquet metadata directly. In particular, I'd like to be able to serialize and deserialize it so I can store it in a cache outside of parquet files and avoid slow object store requests. I see that there is a decode_metadata function but it doesn't handle some of the newer features like page indexes. I also don't see an inverse method, the pieces necessary seem to only exist in readers, etc. Short of re-implementing all of this logic myself, is there an easy way to serialize and deserialize entire parquet metadata?

Thanks!

[alamb]

Confusingly, the Parquet "Page Index" https://github.com/apache/parquet-format/blob/master/PageIndex.md

Refers to two separate structures in the metadata, both of which are available in the ParquetMetadata:

• ColumnIndex: https://docs.rs/parquet/latest/parquet/file/metadata/struct.ParquetMetaData.html#method.column_index
• OffsetIndex: https://docs.rs/parquet/latest/parquet/file/metadata/struct.ParquetMetaData.html#method.offset_index

If you mean how can you get those fields populated with just decode metadata, I am not sure it is possible outside the readers uet

One thing you could do is use https://docs.rs/parquet/latest/parquet/arrow/arrow_reader/struct.ArrowReaderBuilder.html#method.new_with_options to say fetch the page/offset index and and then use that to read the metadata with the page/offset index.

That is the approach taken in the datafusion example to read the metadata (without actually reading the data): https://github.com/apache/datafusion/blob/58f79e143e1a90e5caa59eecc9b36dbdd082a7eb/datafusion-examples/examples/advanced_parquet_index.rs#L408-L415

It would be nice to have a nicer API I think

[adriangb]

Thanks for explaining.

If you mean how can you get those fields populated with just decode metadata, I am not sure it is possible outside the readers uet

Yep this is what I was asking for confirmation on.

To clarify, currently you can do something like this as you point out above:

let mut buffer = Vec::new();
let mut writer = ArrowWriter::try_new(&mut buffer, schema, None)?;
// write the batches
writer.finish()?;

let b = Bytes::from(buf);
let read_options = ArrowReaderOptions::new().with_page_index(true);
let reader = ParquetRecordBatchReaderBuilder::try_new_with_options(b.clone(), read_options).unwrap();
let metadata = reader.metadata().clone();  // A `ParquetMetadata` with `ColumnIndex` and `OffsetIndex` set

What I then want to do is:

let mut buffer = Vec::new();
encode_metadata(metadata, &mut buffer);
// store the metadata somewhere
let metadata: ParquetMetadata = decode_metadata(metadata, &buffer);  // should include `ColumnIndex` and `OffsetIndex`

I think the things missing are:

• encode_metadata does not exist at all, although the logic does obviously exist in ArrowWriter
• decode_metadata does not decode ColumnIndex and OffsetIndex

[alamb]

Makes sense

If we want more control over how the encoding/decoding works, it is probably worth creating a new API

Maybe something like

let encoder = ParquetMetadataEncoder::new()
   .with_some_options(foo);
let mut buffer = vec![]
encoder.encode(metadata, &mut buffer)

Similarly for decoding

let decoder = ParquetMetadataDecoder::new()
   .with_offset_index(true)
   .with_column_index(true);

let result = decoder.decode(&buffer);
// decoder need to have some way to ask for more data
match result {
  FullDecode(metadata) => return metadata,
  NeedMoreData(range) => {
    // fetch additional data and pass to deocder?
    todo!()
  } 
  ..
};

🤔

[adriangb]

@alamb I opened #6000 to at least add encode_metadata, I had pushed before your comment.

I do agree a better API like you propose would be good. Do you envision this being integrated into the rest of the crate, e.g. this replaces some of SerializedFileWriter?

The fetching of more data part is what seems a bit complicated to me. For my use case I'd essentially always be passing in all of the data so it'd be simpler to always assume the data is complete and return an error if it's not (mainly because I can just forward any errors instead of determining if it's an EOF / missing data error or something else). But maybe it's not that bad and it's worth generalizing.

But that's for the read side, I think the write side should be more straightforward, I see no reason why we can't have the ParquetMetadataEncoder you propose. Maybe I can transform #6000 into that and incorporate it into SerializedFileWriter

[alamb]

I do agree a better API like you propose would be good. Do you envision this being integrated into the rest of the crate, e.g. this replaces some of SerializedFileWriter?

Yes, my thinking is that as the parquet crate evolves, the need for more control over metadata decoding shows up more and more. In addition to control over page index / offset index, I can imagine a use for providing filters (e.g only decode metadata for certain row groups) as well as for encoding/decoding the

the data so it'd be simpler to always assume the data is complete and return an error

This makes sense. The reason I was thinking it would be nice to return additional information about what was missing was to help implement serielized reader -- specifically so that it would know how much extra data to request if the requested information was not available

I will file a ticket with a more concrete proposal

[alamb]

I filed #6002 for discussion

[alamb]

The conversation is on #6002 so let's continue the discussion there