Extend ArrowNativeType to support more types convertion

[edgarriba]

hi, we are developing a Tensor struct out of ScalarBuffer in here: https://github.com/kornia/kornia-rs/blob/main/crates/kornia-core/src/storage.rs#L52 for cv and dl purposes.

For some use cases we have, we need to be able to extend the supported types by arrow_buffer::ArrowNativeType to things like ScalarBuffer<[u8; 3]> (and other similar) in order to speed up transform operations with special types of images.

Is there any way to achieve that ? We understand the point around arrow-rs and safety but as users we might need some more flexibility. We just switched from ndarray to use our arrow based tensor as we have plans also to support different backends (cuda, wgpu, etc) but we are facing this unfortunate constraints. Any help would be appreciated as we really like the project and would like to stick to it as much as possible.

[tustvold]

It's probably out of scope for ArrowNativeType, given no arrow arrays will use that type, but nothing prevents you from using the lower-level Buffer type.

That all being said, given arrow arrays don't have such types, I'm not sure what benefit there is of using arrow's types, as opposed to using Vec or bytes::Bytes?