Skip to content

Commit

Permalink
Add changes for 23e4fb8
Browse files Browse the repository at this point in the history
  • Loading branch information
actions-user committed Jul 10, 2023
1 parent 2b663de commit 95749d6
Show file tree
Hide file tree
Showing 6 changed files with 9 additions and 7 deletions.
2 changes: 2 additions & 0 deletions _sources/operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ The following diagram shows the dependencies between operations. Array API funct

Note how fundamental `blockwise` is - all array API functions depend on it.

(elemwise-operation)=
## `elemwise`

The simplest core operation is `elemwise`, which maps input array elements to output array elements, a block at a time.
Expand Down Expand Up @@ -62,6 +63,7 @@ This example shows how `outer` is implemented using `blockwise`. Each block from

![The blockwise primitive operation](images/blockwise.svg)

(rechunk-operation)=
## `rechunk`

The `rechunk` operation is a primitive operation for changing the chunking of an array, without changing its shape or dtype.
Expand Down
4 changes: 2 additions & 2 deletions _sources/user-guide/scaling.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ To understand factors affecting the scaling and performance of Cubed, we will st
### Single-step Calculation

The simplest non-trivial operation in Cubed would map one block from the input array to one block in the output array, with no intermediate persistent stores.
Changing the sign of every element in an array would be an example of this type of operation, known as an [elementwise](Operations/elemwise) operation.
Changing the sign of every element in an array would be an example of this type of operation, known as an [elementwise](#elemwise-operation) operation.

In an ideal environment where the serverless service provides infinite workers, the limiting factor for scaling would be concurrent writes to Zarr.
In such a case weak scaling should be linear, i.e. an array with more blocks could be processed in the same amount of time given proportionally more workers to process those blocks.
Expand All @@ -41,7 +41,7 @@ Worker start-up time is another practical speed consideration, though it would d
### Multi-step Calculation

A multi-step calculation requires writing one or more intermediate arrays to persistent storage.
One important example in Cubed is the [rechunk](Operations/rechunk) operation, which guarantees bounded memory usage by writing and reading from one intermediate persistent Zarr store.
One important example in Cubed is the [rechunk](#rechunk-operation) operation, which guarantees bounded memory usage by writing and reading from one intermediate persistent Zarr store.

In multi-step calculations, the number of steps in the plan sets the minimum total execution time.
Hence, reducing the number of steps in the plan can lead to significant performance improvements.
Expand Down
2 changes: 1 addition & 1 deletion _static/documentation_options.js
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
var DOCUMENTATION_OPTIONS = {
URL_ROOT: document.getElementById("documentation_options").getAttribute('data-url_root'),
VERSION: '0.9.0-2-g819bc6b',
VERSION: '0.9.0-3-g23e4fb8',
LANGUAGE: 'en',
COLLAPSE_INDEX: false,
BUILDER: 'html',
Expand Down
Binary file modified objects.inv
Binary file not shown.
4 changes: 2 additions & 2 deletions operations.html
Original file line number Diff line number Diff line change
Expand Up @@ -345,7 +345,7 @@ <h2>Dependency Tree<a class="headerlink" href="#dependency-tree" title="Permalin
<p>Note how fundamental <code class="docutils literal notranslate"><span class="pre">blockwise</span></code> is - all array API functions depend on it.</p>
</section>
<section id="elemwise">
<h2><code class="docutils literal notranslate"><span class="pre">elemwise</span></code><a class="headerlink" href="#elemwise" title="Permalink to this heading">#</a></h2>
<span id="elemwise-operation"></span><h2><code class="docutils literal notranslate"><span class="pre">elemwise</span></code><a class="headerlink" href="#elemwise" title="Permalink to this heading">#</a></h2>
<p>The simplest core operation is <code class="docutils literal notranslate"><span class="pre">elemwise</span></code>, which maps input array elements to output array elements, a block at a time.</p>
<ul class="simple">
<li><p>Preserves: <code class="docutils literal notranslate"><span class="pre">shape</span></code>, <code class="docutils literal notranslate"><span class="pre">chunks</span></code>, <code class="docutils literal notranslate"><span class="pre">numblocks</span></code></p></li>
Expand Down Expand Up @@ -389,7 +389,7 @@ <h2><code class="docutils literal notranslate"><span class="pre">blockwise</span
<p><img alt="The blockwise primitive operation" src="_images/blockwise.svg" /></p>
</section>
<section id="rechunk">
<h2><code class="docutils literal notranslate"><span class="pre">rechunk</span></code><a class="headerlink" href="#rechunk" title="Permalink to this heading">#</a></h2>
<span id="rechunk-operation"></span><h2><code class="docutils literal notranslate"><span class="pre">rechunk</span></code><a class="headerlink" href="#rechunk" title="Permalink to this heading">#</a></h2>
<p>The <code class="docutils literal notranslate"><span class="pre">rechunk</span></code> operation is a primitive operation for changing the chunking of an array, without changing its shape or dtype.</p>
<ul class="simple">
<li><p>Preserves: <code class="docutils literal notranslate"><span class="pre">shape</span></code>, <code class="docutils literal notranslate"><span class="pre">dtype</span></code></p></li>
Expand Down
4 changes: 2 additions & 2 deletions user-guide/scaling.html
Original file line number Diff line number Diff line change
Expand Up @@ -366,7 +366,7 @@ <h2>Theoretical vs Practical Scaling of Cubed<a class="headerlink" href="#theore
<section id="single-step-calculation">
<h3>Single-step Calculation<a class="headerlink" href="#single-step-calculation" title="Permalink to this heading">#</a></h3>
<p>The simplest non-trivial operation in Cubed would map one block from the input array to one block in the output array, with no intermediate persistent stores.
Changing the sign of every element in an array would be an example of this type of operation, known as an <a class="reference internal" href="#Operations/elemwise"><span class="xref myst">elementwise</span></a> operation.</p>
Changing the sign of every element in an array would be an example of this type of operation, known as an <a class="reference internal" href="../operations.html#elemwise-operation"><span class="std std-ref">elementwise</span></a> operation.</p>
<p>In an ideal environment where the serverless service provides infinite workers, the limiting factor for scaling would be concurrent writes to Zarr.
In such a case weak scaling should be linear, i.e. an array with more blocks could be processed in the same amount of time given proportionally more workers to process those blocks.</p>
<p>In practice, this ideal scenario may not be achieved, for a number of reasons.
Expand All @@ -381,7 +381,7 @@ <h3>Single-step Calculation<a class="headerlink" href="#single-step-calculation"
<section id="multi-step-calculation">
<h3>Multi-step Calculation<a class="headerlink" href="#multi-step-calculation" title="Permalink to this heading">#</a></h3>
<p>A multi-step calculation requires writing one or more intermediate arrays to persistent storage.
One important example in Cubed is the <a class="reference internal" href="#Operations/rechunk"><span class="xref myst">rechunk</span></a> operation, which guarantees bounded memory usage by writing and reading from one intermediate persistent Zarr store.</p>
One important example in Cubed is the <a class="reference internal" href="../operations.html#rechunk-operation"><span class="std std-ref">rechunk</span></a> operation, which guarantees bounded memory usage by writing and reading from one intermediate persistent Zarr store.</p>
<p>In multi-step calculations, the number of steps in the plan sets the minimum total execution time.
Hence, reducing the number of steps in the plan can lead to significant performance improvements.
Reductions can be carried out in fewer iterative steps if <code class="docutils literal notranslate"><span class="pre">allowed_mem</span></code> is larger.
Expand Down

0 comments on commit 95749d6

Please sign in to comment.