Skip to content

Commit

Permalink
DOC: Add references to connect examples
Browse files Browse the repository at this point in the history
  • Loading branch information
snowman2 committed Feb 1, 2021
1 parent d6e17db commit b5d39e8
Show file tree
Hide file tree
Showing 2 changed files with 8 additions and 11 deletions.
9 changes: 2 additions & 7 deletions docs/examples/dask_read_write.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@
"\n",
"See docs for:\n",
"\n",
"- [Reading COGs in Parallel](read-locks.ipynb)\n",
"- [rioxarray.open_rasterio](../rioxarray.rst#rioxarray-open-rasterio)\n",
"- DataArray: [rio.to_raster()](../rioxarray.rst#rioxarray.raster_array.RasterArray.to_raster)\n",
"- Dataset: [rio.to_raster()](../rioxarray.rst#rioxarray.raster_dataset.RasterDataset.to_raster)"
Expand Down Expand Up @@ -96,13 +97,7 @@
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": []
}
],
"outputs": [],
"source": [
"with LocalCluster() as cluster, Client(cluster) as client:\n",
" xds = rioxarray.open_rasterio(\n",
Expand Down
10 changes: 6 additions & 4 deletions docs/examples/read-locks.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -4,17 +4,19 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# Reading COGs in Parallel\n",
"# Example - Reading COGs in Parallel\n",
"\n",
"COGs can be internally chunked, which makes it possible to read them in parallel from multiple threads. However, the libraries `rioxarray` builds on, `rasterio` and `GDAL`, require some care to be used safely from multiple threads within a single process. By default, `rio.open_rasterio` will acquire a per-process lock when reading a chunk of a COG.\n",
"Cloud Optimized Geotiffs (COGs) can be internally chunked, which makes it possible to read them in parallel from multiple threads. However, the libraries `rioxarray` builds on, `rasterio` and `GDAL`, require some care to be used safely from multiple threads within a single process. By default, [rioxarray.open_rasterio](../rioxarray.rst#rioxarray-open-rasterio) will acquire a per-process lock when reading a chunk of a COG.\n",
"\n",
"If you're using `rioxarray` with [Dask](http://docs.dask.org/) through the `chunks` keyword, you can also specify the `lock=False` keyword to ensure that reading *and* operating on your data happen in parallel.\n",
"\n",
"## Scheduler Choice\n",
"\n",
"Dask has [several schedulers](https://docs.dask.org/en/latest/scheduling.html) which run computations in parallel. Which scheduler is best depends on a variety of factors, including whether your computation holds Python's Global Interpreter Lock, whether how much data needs to be moved around, and whether you need more than one machine's computational power. This section about read-locks only applies if you have more than one thread in a process. This will happen with Dask's [local threaded scheduler](https://docs.dask.org/en/latest/scheduling.html#local-threads) and its [distributed scheduler](https://distributed.dask.org/en/latest/) when configured to use more than one thread per worker.\n",
"\n",
"By default, `xarray` objects will use the local `threaded` scheduler."
"By default, `xarray` objects will use the local `threaded` scheduler.\n",
"\n",
"Note: Also see [Reading and Writing with Dask](dask_read_write.ipynb)"
]
},
{
Expand Down Expand Up @@ -219,7 +221,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.6"
"version": "3.9.1"
}
},
"nbformat": 4,
Expand Down

0 comments on commit b5d39e8

Please sign in to comment.