Split optional dependencies in pyproject.toml

[TomNicholas]

Takes the splitting of optional dependencies idea started in #87 to its logical conclusion.

• Changes are documented in docs/releases.rst
• New functionality has documentation

[TomNicholas]

Right now I don't think this can possibly work, because icechunk requires zarr-python v3 but kerchunk requires <3.

[maxrjones]

not without truly living on the bleeding edge 😅

https://github.com/developmentseed/virtualize-nex-gddp-cmip6/blob/f331421ee87adcfb42cb4a1904c05ec1214eb932/pyproject.toml#L24

[TomNicholas]

Not sure we should change the public pip install to use some random branch...

Aside: Is there a reason you are using virtualizarr main there? Would it be helpful to issue another release?

[maxrjones]

Not sure we should change the public pip install to use some random branch...

sorry, I wasn't suggesting you should use that approach, but just was sharing for information about what is/isn't possible. I should've been more clear.

Aside: Is there a reason you are using virtualizarr main there? Would it be helpful to issue another release?

I wanted strict type hinting (#306)

[TomNicholas]

None of this is currently tested because the CI uses conda envs rather than pip to install deps. We could change that though (FYI @maxrjones).

However we don't really want one env and corresponding CI job for every single reader, it's fine to test them all at once, but that means we still wouldn't really be testing the ability to do pip install virtualizarr[netcdf3] specifically for example.

Also @norlandrhagen fastparquet is in the CI but doesn't seem to be used anywhere?

[maxrjones]

I could change the CI to use pip. I have been using uv a bit recently and it's been super promising for managing deps, but would be a bigger change.

[cisaacstern]

without much context, obligatory pixi endorsement comment

[TomNicholas]

I think the CI failure is because the mypy CI job is apparently the only one that uses pip to install currently

[norlandrhagen]

Also @norlandrhagen fastparquet is in the CI but doesn't seem to be used anywhere?

I think it's a hidden requirement of using the Kerchunk parquet format.

[TomNicholas]

I think it's a hidden requirement of using the Kerchunk parquet format.

So does that mean we should have another writer split out in these dependencies?

[maxrjones]

without much context, obligatory pixi endorsement comment

emerging from a rabbit hole (thanks @cisaacstern 😂) I'm floating around this decision framework for feedback:

• If you're starting a new Python package, use uv
• If you need to manage multiple Python virtual environments at a time, also use hatch
• If you need Conda-Forge in addition to PyPI distributions, also use pixi

More on topic, feel welcome to ping me if you'd like to assign any CI / packaging updates.

[norlandrhagen]

So does that mean we should have another writer split out in these dependencies?

Probably! Also, maybe for reading existing Kerchunk parquet refs.

[TomNicholas]

Probably! Also, maybe for reading existing Kerchunk parquet refs.

Does reading existing Kerchunk parquet refs require the same additional dependencies? If so we could have kerchunk_format = ["fastparquet"] as another group?

[norlandrhagen]

Does reading existing Kerchunk parquet refs require the same additional dependencies? If so we could have kerchunk_format = ["fastparquet"] as another group?

I just checked, both reading the writing of Kerchunk Parquet refs require fastparquet.

[TomNicholas]

In a633a0c I embedded the pyproject.toml into the installation docs page so that the list of pip-installable optional dependencies always stays up-to-date. See

https://virtualizarr--309.org.readthedocs.build/en/309/installation.html

[TomNicholas]

note to self: add various parallel options to the splitting here once #349 is in, because that adds optional dependencies on lithops and dask