Use fixed arrow version rather than flexible arrow version.

[liurenjie1024]

Currently in icelake, we use following format to specify arrow version:

arrow = { version = ">= 46"}

This may cause some problems since arrow doesn't guarantee compatibility in upgrading major version. For example, it will make icelake unable to compile when user specifies a higher version such as 48, which makes this range version meaningless.

cc @Xuanwo @ZENOTME

[Xuanwo]

The main issue we're facing is supporting users with varying arrow versions. This becomes more complicated for icelake, as it uses arrow for both input and output types, preventing us from simply allowing arrow 46 & 47 to coexist simultaneously. Users' additional components may need the use of arrow 46 types.

To further complicate matters, all arrow components released under the same major version. For instance:

arrow = { version = "47.0.0", features = ["pyarrow"] }
arrow-select = { version = "47.0.0" }
arrow-array = { version = "47.0.0" }
arrow-buffer = { version = "47.0.0" }
arrow-flight = { version = "47.0.0", features = ["flight-sql-experimental", "tls"] }
arrow-ipc = { version = "47.0.0" }
arrow-schema = { version = "47.0.0", features = ["serde"] }
arrow-ord = { version = "47.0.0" }
arrow-cast = { version = "47.0.0", features = ["prettyprint"] }
parquet = { version = "47.0.0", features = ["async"] }
parquet_rs = { package = "parquet", version = "47.0.0" }
arrow-format = { version = "0.8.1", features = ["flight-data", "flight-service", "ipc"] }

A potential solution could be to implement feature gates that support the latest three versions of arrow, but this might significantly increase our workload. And I'm not sure if this works.

[liurenjie1024]

I think in apache/iceberg-rust, we should be careful with exposing arrow api.

[Xuanwo]

How about maintain a MSAV (minimum supported arrow version) for project? We will incorporate CIs to ensure versions exceeding MSAV compile successfully, and increment MSAV if it fails.

[liurenjie1024]

This idea sounds reasonable. Is it possible to make it run automatically?

[Xuanwo]

Should be possible via github action matrix. Let me give it a try.

[ZENOTME]

How about maintain a MSAV (minimum supported arrow version) for project? We will incorporate CIs to ensure versions exceeding MSAV compile successfully, and increment MSAV if it fails.

This idea sounds good!

[xxhZs]

Hi, I believe using >=xxx might not be a good approach because Icelake uses Arrow for input and output. This means I must introduce the same dependencies as Icelake in their own code, and >=xxx would cause Icelake to fetch the latest Arrow version. So, even if a user is using a version that satisfies >=xxx. For example, I'm using version 48, I still cannot use Icelake because it has already declared a dependency on version 49. Alternatively, there might be a method to specify that Icelake should use version 48?

[Xuanwo]

Alternatively, there might be a method to specify that Icelake should use version 48?

Pin arrow to 48 in Cargo.lock

[liurenjie1024]

This sound reasonable.

[xxhZs]

Thanks for help