Add scan.

[dcherian]

Closes #277

[dcherian]

need input here on choosing a new intermediate chunksize to rechunk to based on memory info.

[tomwhite]

There are a couple of things to consider here: the number of chunks to combine at each stage, and the memory limits.

The first is like split_every in reduction, where the default is 4, although 6 or 8 may be better for larger workloads.

For the second, we should make sure the new chunksize is no larger than (x.spec.allowed_mem - x.spec.reserved_mem) // 4, where the factor of 4 is comes about because of the {compressed,uncompressed} * {input,output} copies.

There is an error case where this memory constraint means the new chunksize is no larger than the existing one, so the computation can't proceed. The user can fix this either by reducing the chunksize or by increasing the memory. This is similar to this case:

cubed/cubed/core/ops.py

Lines 985 to 991 in 88c5dc4

	# single axis: see how many result chunks fit in max_mem
	# factor of 4 is memory for {compressed, uncompressed} x {input, output}
	target_chunk_size = (max_mem - chunk_mem) // (chunk_mem * 4)
	if target_chunk_size <= 1:
	raise ValueError(
	f"Not enough memory for reduction. Increase allowed_mem ({allowed_mem}) or decrease chunk size"
	)

[dcherian]

need input here too.

[tomwhite]

This should be the memory allocated to read from the side inputs (scanned and increment here). We double the chunk memory to account for reading the compressed Zarr chunk, so the result would be

extra_projected_mem=scanned.chunkmem * 2 + increment.chunkmem * 2

(There's an open issue #288 to make this a bit more transparent.)

[tomwhite]

Are you going to add a user-facing cumulative_sum function from the Array API? This would be a good function for the unit tests to test.

[tomwhite]

This should be the memory allocated to read from the side inputs (scanned and increment here). We double the chunk memory to account for reading the compressed Zarr chunk, so the result would be

extra_projected_mem=scanned.chunkmem * 2 + increment.chunkmem * 2

(There's an open issue #288 to make this a bit more transparent.)

[tomwhite]

There are a couple of things to consider here: the number of chunks to combine at each stage, and the memory limits.

The first is like split_every in reduction, where the default is 4, although 6 or 8 may be better for larger workloads.

For the second, we should make sure the new chunksize is no larger than (x.spec.allowed_mem - x.spec.reserved_mem) // 4, where the factor of 4 is comes about because of the {compressed,uncompressed} * {input,output} copies.

There is an error case where this memory constraint means the new chunksize is no larger than the existing one, so the computation can't proceed. The user can fix this either by reducing the chunksize or by increasing the memory. This is similar to this case:

cubed/cubed/core/ops.py

Lines 985 to 991 in 88c5dc4

	# single axis: see how many result chunks fit in max_mem
	# factor of 4 is memory for {compressed, uncompressed} x {input, output}
	target_chunk_size = (max_mem - chunk_mem) // (chunk_mem * 4)
	if target_chunk_size <= 1:
	raise ValueError(
	f"Not enough memory for reduction. Increase allowed_mem ({allowed_mem}) or decrease chunk size"
	)

[tomwhite]

Using map_blocks would be simpler and avoid the 2D assumption

[tomwhite]

Maybe call something like _scan_binop to link it to the scan implementation? I've been using a naming convention like that elsewhere in the file.