Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make chunk manager an option in set_options #9362

Merged
merged 6 commits into from
Aug 16, 2024

Conversation

tomwhite
Copy link
Contributor

@tomwhite tomwhite commented Aug 14, 2024

It would be useful to have a way to choose a chunk manager if more than one is found (see https://docs.xarray.dev/en/latest/internals/chunked-arrays.html#user-interface for how it works). The use case I have is a codebase that uses Xarray and Dask (and has lots of Dask imports), and which I'd like to be able to run on Cubed too (as well as Dask). Rather than converting it all in one go, it would be easier to convert it to use Xarray-only APIs gradually. The changes involve moving da.map_blocks to xr.apply_ufunc, for example.

Using the the current PR I can have both Dask and Cubed packages installed and switch to use Cubed with xr.set_option(chunk_manager="cubed"). This allows me to run a subset of the tests on Cubed while not having to worry about the rest of the Dask imports in the codebase.

Does this look like a good way to achieve this @TomNicholas?

  • Closes #xxxx
  • Tests added
  • User visible changes (including notable bug fixes) are documented in whats-new.rst
  • New functions/methods are listed in api.rst

@TomNicholas TomNicholas added the topic-chunked-arrays Managing different chunked backends, e.g. dask label Aug 14, 2024
@TomNicholas
Copy link
Member

Good idea! I think this should be documented and tested though. It can be documented in the paragraph you linked to, and tested using the options context manager in a simple new test.

@Illviljan
Copy link
Contributor

Please add a line in the docstring of xr.set_option as well.

Copy link
Member

@TomNicholas TomNicholas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry I should have mentioned to add a whatsnew entry - then happy to merge!

@tomwhite
Copy link
Contributor Author

Sorry I should have mentioned to add a whatsnew entry - then happy to merge!

Done - thanks @TomNicholas

@TomNicholas TomNicholas merged commit 6166c87 into pydata:main Aug 16, 2024
28 checks passed
dcherian added a commit to dcherian/xarray that referenced this pull request Aug 17, 2024
* main:
  Improve error message for missing coordinate index (pydata#9370)
  Add flaky to TestNetCDF4ViaDaskData (pydata#9373)
  Make chunk manager an option in `set_options` (pydata#9362)
  Revise (pydata#9371)
  Remove duplicate word from docs (pydata#9367)
  Adding open_groups to BackendEntryPointEngine, NetCDF4BackendEntrypoint, and H5netcdfBackendEntrypoint (pydata#9243)
dcherian added a commit to TomNicholas/xarray that referenced this pull request Aug 22, 2024
* main: (214 commits)
  Adds copy parameter to __array__ for numpy 2.0 (pydata#9393)
  `numpy 2` compatibility in the `pydap` backend (pydata#9391)
  pyarrow dependency added to doc environment (pydata#9394)
  Extend padding functionalities (pydata#9353)
  refactor GroupBy internals (pydata#9389)
  Combine `UnsignedIntegerCoder` and `CFMaskCoder` (pydata#9274)
  passing missing parameters to ZarrStore.open_store when opening a datatree (pydata#9377)
  Fix tests on big-endian systems (pydata#9380)
  Improve error message on `ds['x', 'y']` (pydata#9375)
  Improve error message for missing coordinate index (pydata#9370)
  Add flaky to TestNetCDF4ViaDaskData (pydata#9373)
  Make chunk manager an option in `set_options` (pydata#9362)
  Revise (pydata#9371)
  Remove duplicate word from docs (pydata#9367)
  Adding open_groups to BackendEntryPointEngine, NetCDF4BackendEntrypoint, and H5netcdfBackendEntrypoint (pydata#9243)
  Revise (pydata#9366)
  Fix rechunking to a frequency with empty bins. (pydata#9364)
  whats-new entry for dropping python 3.9 (pydata#9359)
  drop support for `python=3.9` (pydata#8937)
  Revise (pydata#9357)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
topic-chunked-arrays Managing different chunked backends, e.g. dask
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants