diff --git a/_sources/operations.md b/_sources/operations.md index 751dc97d..0f134b37 100644 --- a/_sources/operations.md +++ b/_sources/operations.md @@ -10,6 +10,7 @@ The following diagram shows the dependencies between operations. Array API funct Note how fundamental `blockwise` is - all array API functions depend on it. +(elemwise-operation)= ## `elemwise` The simplest core operation is `elemwise`, which maps input array elements to output array elements, a block at a time. @@ -62,6 +63,7 @@ This example shows how `outer` is implemented using `blockwise`. Each block from ![The blockwise primitive operation](images/blockwise.svg) +(rechunk-operation)= ## `rechunk` The `rechunk` operation is a primitive operation for changing the chunking of an array, without changing its shape or dtype. diff --git a/_sources/user-guide/scaling.md b/_sources/user-guide/scaling.md index 5d61d739..7d7db6a1 100644 --- a/_sources/user-guide/scaling.md +++ b/_sources/user-guide/scaling.md @@ -23,7 +23,7 @@ To understand factors affecting the scaling and performance of Cubed, we will st ### Single-step Calculation The simplest non-trivial operation in Cubed would map one block from the input array to one block in the output array, with no intermediate persistent stores. -Changing the sign of every element in an array would be an example of this type of operation, known as an [elementwise](Operations/elemwise) operation. +Changing the sign of every element in an array would be an example of this type of operation, known as an [elementwise](#elemwise-operation) operation. In an ideal environment where the serverless service provides infinite workers, the limiting factor for scaling would be concurrent writes to Zarr. In such a case weak scaling should be linear, i.e. an array with more blocks could be processed in the same amount of time given proportionally more workers to process those blocks. @@ -41,7 +41,7 @@ Worker start-up time is another practical speed consideration, though it would d ### Multi-step Calculation A multi-step calculation requires writing one or more intermediate arrays to persistent storage. -One important example in Cubed is the [rechunk](Operations/rechunk) operation, which guarantees bounded memory usage by writing and reading from one intermediate persistent Zarr store. +One important example in Cubed is the [rechunk](#rechunk-operation) operation, which guarantees bounded memory usage by writing and reading from one intermediate persistent Zarr store. In multi-step calculations, the number of steps in the plan sets the minimum total execution time. Hence, reducing the number of steps in the plan can lead to significant performance improvements. diff --git a/_static/documentation_options.js b/_static/documentation_options.js index 2119e76d..15fd8555 100644 --- a/_static/documentation_options.js +++ b/_static/documentation_options.js @@ -1,6 +1,6 @@ var DOCUMENTATION_OPTIONS = { URL_ROOT: document.getElementById("documentation_options").getAttribute('data-url_root'), - VERSION: '0.9.0-2-g819bc6b', + VERSION: '0.9.0-3-g23e4fb8', LANGUAGE: 'en', COLLAPSE_INDEX: false, BUILDER: 'html', diff --git a/objects.inv b/objects.inv index 8ba8c2db..3b32b224 100644 Binary files a/objects.inv and b/objects.inv differ diff --git a/operations.html b/operations.html index 6587b0a6..51a1631d 100644 --- a/operations.html +++ b/operations.html @@ -345,7 +345,7 @@

Dependency Treeblockwise is - all array API functions depend on it.

-

elemwise#

+

elemwise#

The simplest core operation is elemwise, which maps input array elements to output array elements, a block at a time.

-

rechunk#

+

rechunk#

The rechunk operation is a primitive operation for changing the chunking of an array, without changing its shape or dtype.

  • Preserves: shape, dtype

    • diff --git a/user-guide/scaling.html b/user-guide/scaling.html index 48c09b5f..8173127e 100644 --- a/user-guide/scaling.html +++ b/user-guide/scaling.html @@ -366,7 +366,7 @@

    Theoretical vs Practical Scaling of Cubed

    Single-step Calculation#

    The simplest non-trivial operation in Cubed would map one block from the input array to one block in the output array, with no intermediate persistent stores. -Changing the sign of every element in an array would be an example of this type of operation, known as an elementwise operation.

    +Changing the sign of every element in an array would be an example of this type of operation, known as an elementwise operation.

    In an ideal environment where the serverless service provides infinite workers, the limiting factor for scaling would be concurrent writes to Zarr. In such a case weak scaling should be linear, i.e. an array with more blocks could be processed in the same amount of time given proportionally more workers to process those blocks.

    In practice, this ideal scenario may not be achieved, for a number of reasons. @@ -381,7 +381,7 @@

    Single-step Calculation

    Multi-step Calculation#

    A multi-step calculation requires writing one or more intermediate arrays to persistent storage. -One important example in Cubed is the rechunk operation, which guarantees bounded memory usage by writing and reading from one intermediate persistent Zarr store.

    +One important example in Cubed is the rechunk operation, which guarantees bounded memory usage by writing and reading from one intermediate persistent Zarr store.

    In multi-step calculations, the number of steps in the plan sets the minimum total execution time. Hence, reducing the number of steps in the plan can lead to significant performance improvements. Reductions can be carried out in fewer iterative steps if allowed_mem is larger.