bug: lance error to peek lance schema

[eddyxu]

Problem

lance/rust/lance-datafusion/src/utils.rs

Line 41 in ef0953d

return Err(Error::from(batch.as_ref().unwrap_err()));

Database { table: Some("..."), cause: "lance error: LanceError(Arrow): Io error: failed to fill whole buffer, /src/lance/rust/lance-datafusion/src/utils.rs:41:28" }

It is not clear how to reproduce it yet.

[LuQQiu]

• The error message "failed to fill the whole buffer" originates from Rust's standard library (std::io), and occurs when "read_exact" or similar methods fail to read the expected number of bytes
• In our case, failed to read from the input data RecordBatchReader when writing/appending data to the dataset/fragments

@eddyxu do you have any information about what types of data sources are used? e.g. what's the python data input type?
Could we have more information on their workflow, what did they read? what libraries are used for intermediate data processing, and what do they pass to Lance