From 95749d6d4159458b0d6ca056f9138ff21b97b10d Mon Sep 17 00:00:00 2001 From: GitHub Action Date: Mon, 10 Jul 2023 14:04:52 +0000 Subject: [PATCH] Add changes for 23e4fb8b46d31c276f1a8fe0faaf80d2d03e4a37 --- _sources/operations.md | 2 ++ _sources/user-guide/scaling.md | 4 ++-- _static/documentation_options.js | 2 +- objects.inv | Bin 1206 -> 1229 bytes operations.html | 4 ++-- user-guide/scaling.html | 4 ++-- 6 files changed, 9 insertions(+), 7 deletions(-) diff --git a/_sources/operations.md b/_sources/operations.md index 751dc97d..0f134b37 100644 --- a/_sources/operations.md +++ b/_sources/operations.md @@ -10,6 +10,7 @@ The following diagram shows the dependencies between operations. Array API funct Note how fundamental `blockwise` is - all array API functions depend on it. +(elemwise-operation)= ## `elemwise` The simplest core operation is `elemwise`, which maps input array elements to output array elements, a block at a time. @@ -62,6 +63,7 @@ This example shows how `outer` is implemented using `blockwise`. Each block from ![The blockwise primitive operation](images/blockwise.svg) +(rechunk-operation)= ## `rechunk` The `rechunk` operation is a primitive operation for changing the chunking of an array, without changing its shape or dtype. diff --git a/_sources/user-guide/scaling.md b/_sources/user-guide/scaling.md index 5d61d739..7d7db6a1 100644 --- a/_sources/user-guide/scaling.md +++ b/_sources/user-guide/scaling.md @@ -23,7 +23,7 @@ To understand factors affecting the scaling and performance of Cubed, we will st ### Single-step Calculation The simplest non-trivial operation in Cubed would map one block from the input array to one block in the output array, with no intermediate persistent stores. -Changing the sign of every element in an array would be an example of this type of operation, known as an [elementwise](Operations/elemwise) operation. +Changing the sign of every element in an array would be an example of this type of operation, known as an [elementwise](#elemwise-operation) operation. In an ideal environment where the serverless service provides infinite workers, the limiting factor for scaling would be concurrent writes to Zarr. In such a case weak scaling should be linear, i.e. an array with more blocks could be processed in the same amount of time given proportionally more workers to process those blocks. @@ -41,7 +41,7 @@ Worker start-up time is another practical speed consideration, though it would d ### Multi-step Calculation A multi-step calculation requires writing one or more intermediate arrays to persistent storage. -One important example in Cubed is the [rechunk](Operations/rechunk) operation, which guarantees bounded memory usage by writing and reading from one intermediate persistent Zarr store. +One important example in Cubed is the [rechunk](#rechunk-operation) operation, which guarantees bounded memory usage by writing and reading from one intermediate persistent Zarr store. In multi-step calculations, the number of steps in the plan sets the minimum total execution time. Hence, reducing the number of steps in the plan can lead to significant performance improvements. diff --git a/_static/documentation_options.js b/_static/documentation_options.js index 2119e76d..15fd8555 100644 --- a/_static/documentation_options.js +++ b/_static/documentation_options.js @@ -1,6 +1,6 @@ var DOCUMENTATION_OPTIONS = { URL_ROOT: document.getElementById("documentation_options").getAttribute('data-url_root'), - VERSION: '0.9.0-2-g819bc6b', + VERSION: '0.9.0-3-g23e4fb8', LANGUAGE: 'en', COLLAPSE_INDEX: false, BUILDER: 'html', diff --git a/objects.inv b/objects.inv index 8ba8c2db45ef0d952437f12edca2c97993fadfe7..3b32b2245581bf2581c54a1a4e823ccfebfbac12 100644 GIT binary patch delta 1125 zcmV-r1e*J{3C#(RKLIn5KsK+PTuBxz1&k%|xS-cC^uCS9?TmsC2V! zFtSZ%$gf`;^I?NA;N%jZs@|)jXuF=1HD}0Acbte1Qr$k&u7PrhwB` zmh`s|I6<~CO^d3KX{zBR$Scc_;WJj&jf5TWKGILaA-gu!Z3j&s>8F2rDMUkP@w%s; zF?LRMPbfW;>=x?i5VMBe*Vi?cry8 zE66~0te`VzB+rPDsbqi0(NNa1jfADNxJ7gj+6d5dLuni;vPxQ1q9$1cl2st$P?4p+ z0%sx;b||H_y9d_a%FdoK4ko*2OHHhWi|1CfN2LZ6 ztf^YO?ni$?DI3bR)V$c^KygYkn0UVy#YYh6Ekf^D!5e}RAIG70kYC5%VV|1A(Qp*6 zo{qgpP=n#4DCCu8p^7HmhAjhH(P$wiTgG9>gjOY6h|fD!oq1YRIs9_v&R{i|Z~9h6 zO0B07Aqm&?Ih4#zyz~zmt7HrKlywDf74X%#@RNUWdAQUar3*P3iW^>SvMgwSUfZPW zSBP^AGKBwLTYRq{e@BZqwBD!-L2CVYnqRlc;4r~zRW%g)&oC)bn|h{C@n7RJvgkJQ z^eWv+*Q>kzx_FR-9H{;kCN*dJf7(v9bDv57!&vcSZ$$zK&#rDlV>5s!<9{R8O rh&8+wF}7_m*i0f*3*0>@Exmmo&4Yth4N+}fLmO|8HV^23gxpIOD>5?J delta 1102 zcmV-U1hMb)w8cKJDZn?Z59m0c3;NW>xu00HW+>)RoO`1`bU&RK)9JrzYB3ce92LXTl1&XwSYKzPPS<#!1lI}R#ptHweWp9+@u`_!~^;T}2lwTbZyaBLV*@KCzVXKEbmaG>C6b(`Nj)?$B$18p2)w=>v3BkXh`pC${@ zTe=O~HssP|A+noh!}Bu=>97#lO0VIy9feMlA@bF!6R+*aBb7tEj$fMZH)cr80O3Mk z;-wXN<>V&oZLHyi5%~pXh;$)C@zRR+NX=jZpESjOKMLy6P`0&j#SRCGlY_y;`voOF zf~=T+re`_ydJkiTxv$e`QEGfHhM9S7R~n39IG{{PEX z(xp$c{LohXI9iba!t;xrTiZ0?N&6qk8BnWswdCBE1$?|MQ!}N@uUV$GD@=~nCs_J4 z_8&rJ#b0`>306BRDependency Treeblockwise is - all array API functions depend on it.

-

elemwise#

+

elemwise#

The simplest core operation is elemwise, which maps input array elements to output array elements, a block at a time.

  • Preserves: shape, chunks, numblocks

    • @@ -389,7 +389,7 @@

    blockwiseThe blockwise primitive operation

-

rechunk#

+

rechunk#

The rechunk operation is a primitive operation for changing the chunking of an array, without changing its shape or dtype.

  • Preserves: shape, dtype

    • diff --git a/user-guide/scaling.html b/user-guide/scaling.html index 48c09b5f..8173127e 100644 --- a/user-guide/scaling.html +++ b/user-guide/scaling.html @@ -366,7 +366,7 @@

    Theoretical vs Practical Scaling of Cubed

    Single-step Calculation#

    The simplest non-trivial operation in Cubed would map one block from the input array to one block in the output array, with no intermediate persistent stores. -Changing the sign of every element in an array would be an example of this type of operation, known as an elementwise operation.

    +Changing the sign of every element in an array would be an example of this type of operation, known as an elementwise operation.

    In an ideal environment where the serverless service provides infinite workers, the limiting factor for scaling would be concurrent writes to Zarr. In such a case weak scaling should be linear, i.e. an array with more blocks could be processed in the same amount of time given proportionally more workers to process those blocks.

    In practice, this ideal scenario may not be achieved, for a number of reasons. @@ -381,7 +381,7 @@

    Single-step Calculation

    Multi-step Calculation#

    A multi-step calculation requires writing one or more intermediate arrays to persistent storage. -One important example in Cubed is the rechunk operation, which guarantees bounded memory usage by writing and reading from one intermediate persistent Zarr store.

    +One important example in Cubed is the rechunk operation, which guarantees bounded memory usage by writing and reading from one intermediate persistent Zarr store.

    In multi-step calculations, the number of steps in the plan sets the minimum total execution time. Hence, reducing the number of steps in the plan can lead to significant performance improvements. Reductions can be carried out in fewer iterative steps if allowed_mem is larger.