You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Arrays whose values are a function of the index do not need to be materialized - either in memory or on disk. These include arrays where every entry is the same (created with full), and arrays of offsets (used in map_blocks to provide block IDs to the map function).
Although the array is small since its size is the number of chunks, there is a scalability issue due to the fact that every chunk is stored in a separate file in the Zarr store.
These problems could be solved by writing an array implementation that implements indexing on the fly: either by returning a subarray of fill values for the case of full, or np.ravel_multi_index to turn an index into an offset for an array of offsets.
The text was updated successfully, but these errors were encountered:
Arrays whose values are a function of the index do not need to be materialized - either in memory or on disk. These include arrays where every entry is the same (created with
full
), and arrays of offsets (used inmap_blocks
to provide block IDs to the map function).Zarr arrays created with
full
are not expensive to create since they specifywrite_empty_chunks=False
so no chunks are written - just the metadata. However, offsets are materialized to disk: https://github.com/tomwhite/cubed/blob/e21786591d9832a85f2e492641ad2061bdb8c14a/cubed/core/ops.py#L431-L432Although the array is small since its size is the number of chunks, there is a scalability issue due to the fact that every chunk is stored in a separate file in the Zarr store.
These problems could be solved by writing an array implementation that implements indexing on the fly: either by returning a subarray of fill values for the case of
full
, ornp.ravel_multi_index
to turn an index into an offset for an array of offsets.The text was updated successfully, but these errors were encountered: