-
-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make chunk manager an option in set_options
#9362
Conversation
Good idea! I think this should be documented and tested though. It can be documented in the paragraph you linked to, and tested using the options context manager in a simple new test. |
Please add a line in the docstring of |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry I should have mentioned to add a whatsnew entry - then happy to merge!
Done - thanks @TomNicholas |
* main: Improve error message for missing coordinate index (pydata#9370) Add flaky to TestNetCDF4ViaDaskData (pydata#9373) Make chunk manager an option in `set_options` (pydata#9362) Revise (pydata#9371) Remove duplicate word from docs (pydata#9367) Adding open_groups to BackendEntryPointEngine, NetCDF4BackendEntrypoint, and H5netcdfBackendEntrypoint (pydata#9243)
* main: (214 commits) Adds copy parameter to __array__ for numpy 2.0 (pydata#9393) `numpy 2` compatibility in the `pydap` backend (pydata#9391) pyarrow dependency added to doc environment (pydata#9394) Extend padding functionalities (pydata#9353) refactor GroupBy internals (pydata#9389) Combine `UnsignedIntegerCoder` and `CFMaskCoder` (pydata#9274) passing missing parameters to ZarrStore.open_store when opening a datatree (pydata#9377) Fix tests on big-endian systems (pydata#9380) Improve error message on `ds['x', 'y']` (pydata#9375) Improve error message for missing coordinate index (pydata#9370) Add flaky to TestNetCDF4ViaDaskData (pydata#9373) Make chunk manager an option in `set_options` (pydata#9362) Revise (pydata#9371) Remove duplicate word from docs (pydata#9367) Adding open_groups to BackendEntryPointEngine, NetCDF4BackendEntrypoint, and H5netcdfBackendEntrypoint (pydata#9243) Revise (pydata#9366) Fix rechunking to a frequency with empty bins. (pydata#9364) whats-new entry for dropping python 3.9 (pydata#9359) drop support for `python=3.9` (pydata#8937) Revise (pydata#9357) ...
It would be useful to have a way to choose a chunk manager if more than one is found (see https://docs.xarray.dev/en/latest/internals/chunked-arrays.html#user-interface for how it works). The use case I have is a codebase that uses Xarray and Dask (and has lots of Dask imports), and which I'd like to be able to run on Cubed too (as well as Dask). Rather than converting it all in one go, it would be easier to convert it to use Xarray-only APIs gradually. The changes involve moving
da.map_blocks
toxr.apply_ufunc
, for example.Using the the current PR I can have both Dask and Cubed packages installed and switch to use Cubed with
xr.set_option(chunk_manager="cubed")
. This allows me to run a subset of the tests on Cubed while not having to worry about the rest of the Dask imports in the codebase.Does this look like a good way to achieve this @TomNicholas?
whats-new.rst
api.rst