Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dask improvements #217

Merged
merged 46 commits into from
Nov 13, 2023
Merged

Dask improvements #217

merged 46 commits into from
Nov 13, 2023

Conversation

rosepearson
Copy link
Owner

@rosepearson rosepearson commented Nov 2, 2023 •
edited
Loading

  • Try improve coarse DEM inclusion - leave to Dask to decide
  • Identify and fix hot-spots causing 'Large Graph Errors"

DESCRIPTION OF PR:

  • Make code change
  • Update tests
    • Update / create new tests
    • Ensure these tests have the expected behavour
    • Test locally and ensure tests are passing
  • Ensure tests passing on GH Actions
  • Update documentation
    • Doc strings
    • Wiki
  • Update package (version.py & pyproject.toml) version (if needed)

@rosepearson rosepearson linked an issue Nov 2, 2023 that may be closed by this pull request
Dask improvements #208
Closed
github-actions and others added 28 commits November 2, 2023 20:17
fixup: Format Python code with Black
240f5f0
@rosepearson
Typo correction
dfe1bf0
@rosepearson
Add missing removal of bland to buffered foreshore
de1fa1b
@rosepearson
Make robust to adding coarse DEM with no foreshore (e.g. landlocked)
5459b95
fixup: Format Python code with Black
bf67d73
@rosepearson
Updated benchmarks with minor changes from shifting the interpolation…
a9c791d 
… to xarray/rioxarray
@rosepearson
Tidy-up to remove the now unused coarse DEM chunking code
4037956
@rosepearson
Specify chunking for interpolation
df17edc
fixup: Format Python code with Black
a532d33
@rosepearson
fix change to chunks
05a0f86
@rosepearson
Try force save after LiDAR added
9450558
fixup: Format Python code with Black
dd64070
@rosepearson
load coarse DEM no chunks
06751b4
@rosepearson
Merge branch '208-dask-improvements' of github.com:rosepearson/GeoFab…
89e13be 
…rics into 208-dask-improvements
@rosepearson
create save functions to simplify code when attempting caching files
f36d6df
fixup: Format Python code with Black
f9adf35
@rosepearson
minor fixes after last change
7352d0c
fixup: Format Python code with Black
36df7e8
@rosepearson
try chunking the coarse DEM
ec45e8a
@rosepearson
typo fix
483eb6d
fixup: Format Python code with Black
95cb6df
@rosepearson
Write out temp nc files during RawDem class execution
bd7a1b8
fixup: Format Python code with Black
2a42f46
@rosepearson
only delete cache if it exists
bdba882
@rosepearson
minor fixes
ae30061
@rosepearson @jennan
try ensure efficient and robust use of compute resources available
3191149 
Code adapted from some provided by @jennan. Aims to better control parallelisation.

Co-Authored-By: Maxime Rio <2003548+jennan@users.noreply.github.com>
fixup: Format Python code with Black
16c2f55
@rosepearson
Fixes associated with the explicit dask interpolation
18cbc48
@rosepearson rosepearson removed a link to an issue Nov 12, 2023
Dask improvements #208
Closed
@rosepearson rosepearson linked an issue Nov 12, 2023 that may be closed by this pull request
Dask Improvements part 1 #219
Closed
rosepearson and others added 13 commits November 13, 2023 13:04
@rosepearson rosepearson merged commit 7783462 into main Nov 13, 2023
@rosepearson rosepearson deleted the 208-dask-improvements branch November 13, 2023 03:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Dask Improvements part 1
1 participant
@rosepearson