Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lazy netcdf saves #5191

Merged
merged 92 commits into from
Apr 21, 2023
Merged

Lazy netcdf saves #5191

merged 92 commits into from
Apr 21, 2023

Commits on Mar 10, 2023

  1. Basic functional lazy saving.

    pp-mo committed Mar 10, 2023
    Configuration menu
    Copy the full SHA
    cd7fa42 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    1f32800 View commit details
    Browse the repository at this point in the history
  3. Non-lazy saves return nothing.

    pp-mo committed Mar 10, 2023
    Configuration menu
    Copy the full SHA
    e0f980f View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    67b96cf View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    8cdbc9b View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    8723f24 View commit details
    Browse the repository at this point in the history
  7. Fixes for non-lazy save.

    pp-mo committed Mar 10, 2023
    Configuration menu
    Copy the full SHA
    d19a87f View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    45e0e60 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    6a83200 View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    78a9346 View commit details
    Browse the repository at this point in the history
  11. Document dask-specific locking; implement differently for threads or …

    …distributed schedulers.
    pp-mo committed Mar 10, 2023
    Configuration menu
    Copy the full SHA
    31b12f7 View commit details
    Browse the repository at this point in the history
  12. Minor fix for unit-tests.

    pp-mo committed Mar 10, 2023
    Configuration menu
    Copy the full SHA
    99b4f41 View commit details
    Browse the repository at this point in the history
  13. Configuration menu
    Copy the full SHA
    5d0a707 View commit details
    Browse the repository at this point in the history
  14. Minor test fix.

    pp-mo committed Mar 10, 2023
    Configuration menu
    Copy the full SHA
    431036f View commit details
    Browse the repository at this point in the history

Commits on Mar 13, 2023

  1. Move DeferredSaveWrapper into _thread_safe_nc; replicate the NetCDFDa…

    …taProxy fix; use one lock per Saver; add extra up-scaled test
    pp-mo committed Mar 13, 2023
    Configuration menu
    Copy the full SHA
    dc368d9 View commit details
    Browse the repository at this point in the history

Commits on Mar 16, 2023

  1. Update lib/iris/fileformats/netcdf/saver.py

    Co-authored-by: Bouwe Andela <bouweandela@gmail.com>
    pp-mo and bouweandela authored Mar 16, 2023
    Configuration menu
    Copy the full SHA
    6756a46 View commit details
    Browse the repository at this point in the history
  2. Update lib/iris/fileformats/netcdf/_dask_locks.py

    Co-authored-by: Bouwe Andela <bouweandela@gmail.com>
    pp-mo and bouweandela authored Mar 16, 2023
    Configuration menu
    Copy the full SHA
    80b4b6c View commit details
    Browse the repository at this point in the history
  3. Update lib/iris/fileformats/netcdf/saver.py

    Co-authored-by: Bouwe Andela <bouweandela@gmail.com>
    pp-mo and bouweandela authored Mar 16, 2023
    Configuration menu
    Copy the full SHA
    78a8716 View commit details
    Browse the repository at this point in the history

Commits on Mar 17, 2023

  1. Small rename + reformat.

    pp-mo committed Mar 17, 2023
    Configuration menu
    Copy the full SHA
    47bb08b View commit details
    Browse the repository at this point in the history

Commits on Mar 18, 2023

  1. Remove Saver lazy option; all lazy saves are delayed; factor out fill…

    …value checks and make them delayable.
    pp-mo committed Mar 18, 2023
    Configuration menu
    Copy the full SHA
    0ece09a View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    940f544 View commit details
    Browse the repository at this point in the history

Commits on Mar 20, 2023

  1. Configuration menu
    Copy the full SHA
    eb97130 View commit details
    Browse the repository at this point in the history
  2. Repurposed 'test__FillValueMaskCheckAndStoreTarget' to 'test__data_fi…

    …llvalue_check', since old class is gone.
    pp-mo committed Mar 20, 2023
    Configuration menu
    Copy the full SHA
    4596081 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    ad49fbe View commit details
    Browse the repository at this point in the history
  4. Fix test problems; Saver automatically completes to preserve existing…

    … direct usage (which is public API).
    pp-mo committed Mar 20, 2023
    Configuration menu
    Copy the full SHA
    b29c927 View commit details
    Browse the repository at this point in the history
  5. Fix docstring error.

    pp-mo committed Mar 20, 2023
    Configuration menu
    Copy the full SHA
    8f10281 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    6a564d9 View commit details
    Browse the repository at this point in the history
  7. Fix Saver docstring.

    pp-mo committed Mar 20, 2023
    Configuration menu
    Copy the full SHA
    2fb4d6c View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    c84bfdc View commit details
    Browse the repository at this point in the history

Commits on Mar 21, 2023

  1. Configuration menu
    Copy the full SHA
    5b78085 View commit details
    Browse the repository at this point in the history

Commits on Mar 23, 2023

  1. Configuration menu
    Copy the full SHA
    478332e View commit details
    Browse the repository at this point in the history

Commits on Mar 27, 2023

  1. Moved netcdf cell-method handling into nc_load_rules.helpers, and var…

    …ious tests into more specific test folders.
    pp-mo committed Mar 27, 2023
    Configuration menu
    Copy the full SHA
    34f154c View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    d3744ba View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    9673ea0 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    bcbcbc8 View commit details
    Browse the repository at this point in the history
  5. Remove debug-only code.

    pp-mo committed Mar 27, 2023
    Configuration menu
    Copy the full SHA
    05c04a1 View commit details
    Browse the repository at this point in the history

Commits on Mar 28, 2023

  1. Configuration menu
    Copy the full SHA
    679ea47 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    70ec9dd View commit details
    Browse the repository at this point in the history

Commits on Mar 29, 2023

  1. Configuration menu
    Copy the full SHA
    28a4674 View commit details
    Browse the repository at this point in the history
  2. Tidy test docstrings.

    pp-mo committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    67f4b2b View commit details
    Browse the repository at this point in the history
  3. Correct test import.

    pp-mo committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    ebec72f View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    1f5b904 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    5045c9f View commit details
    Browse the repository at this point in the history

Commits on Mar 30, 2023

  1. Configuration menu
    Copy the full SHA
    393407a View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    518360b View commit details
    Browse the repository at this point in the history

Commits on Mar 31, 2023

  1. Docstring fixes.

    pp-mo committed Mar 31, 2023
    Configuration menu
    Copy the full SHA
    5c9931f View commit details
    Browse the repository at this point in the history

Commits on Apr 4, 2023

  1. Configuration menu
    Copy the full SHA
    7daee68 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    97474f9 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    64c7251 View commit details
    Browse the repository at this point in the history

Commits on Apr 11, 2023

  1. Configuration menu
    Copy the full SHA
    75043f9 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    445fbe2 View commit details
    Browse the repository at this point in the history
  3. Review changes.

    pp-mo committed Apr 11, 2023
    Configuration menu
    Copy the full SHA
    09cb22e View commit details
    Browse the repository at this point in the history

Commits on Apr 12, 2023

  1. Review changes.

    pp-mo committed Apr 12, 2023
    Configuration menu
    Copy the full SHA
    3445f58 View commit details
    Browse the repository at this point in the history

Commits on Apr 13, 2023

  1. Enhance tests + debug.

    pp-mo committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    cb1e1f7 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    1c81cee View commit details
    Browse the repository at this point in the history
  3. Improve test.

    pp-mo committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    370837b View commit details
    Browse the repository at this point in the history
  4. Adding a whatsnew entry for 5224 (SciTools#5234)

    * Adding a whatsnew entry explaining 5224
    
    * Fixing link and format error
    HGWright authored and pp-mo committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    2f5f3c2 View commit details
    Browse the repository at this point in the history
  5. Replacing numpy legacy printing with array2string and remaking result…

    …s for dependent tests
    HGWright authored and pp-mo committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    a55c6f2 View commit details
    Browse the repository at this point in the history
  6. adding a whatsnew entry

    HGWright authored and pp-mo committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    4914e99 View commit details
    Browse the repository at this point in the history
  7. configure codecov

    HGWright authored and pp-mo committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    bd642cd View commit details
    Browse the repository at this point in the history
  8. remove results creation commit from blame

    HGWright authored and pp-mo committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    bc5bdd1 View commit details
    Browse the repository at this point in the history
  9. fixing whatsnew entry

    HGWright authored and pp-mo committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    301e59e View commit details
    Browse the repository at this point in the history
  10. Bump scitools/workflows from 2023.04.1 to 2023.04.2 (SciTools#5236)

    Bumps [scitools/workflows](https://github.com/scitools/workflows) from 2023.04.1 to 2023.04.2.
    - [Release notes](https://github.com/scitools/workflows/releases)
    - [Commits](SciTools/workflows@2023.04.1...2023.04.2)
    
    ---
    updated-dependencies:
    - dependency-name: scitools/workflows
      dependency-type: direct:production
    ...
    
    Signed-off-by: dependabot[bot] <support@github.com>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    dependabot[bot] authored and pp-mo committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    7b3044d View commit details
    Browse the repository at this point in the history
  11. Use real array for data of of small netCDF variables. (SciTools#5229)

    * Small netCDF variable data is real.
    
    * Various test fixes.
    
    * More test fixing.
    
    * Fix printout in Mesh documentation.
    
    * Whatsnew + doctests fix.
    
    * Tweak whatsnew.
    pp-mo committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    02f2b66 View commit details
    Browse the repository at this point in the history
  12. Handle derived coordinates correctly in concatenate (SciTools#5096)

    * First working prototype of concatenate that handels derived coordinates correctly
    
    * Added checks for derived coord metadata during concatenation
    
    * Added tests
    
    * Fixed defaults
    
    * Added what's new entry
    
    * Optimized test coverage
    schlunma authored and pp-mo committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    a7e0689 View commit details
    Browse the repository at this point in the history
  13. Configuration menu
    Copy the full SHA
    c4e8bbb View commit details
    Browse the repository at this point in the history
  14. Modernize and simplify iris.analysis._Groupby (SciTools#5015)

    * Modernize and simplify _Groupby
    
    * Rename variable to improve readability
    
    Co-authored-by: Martin Yeo <40734014+trexfeathers@users.noreply.github.com>
    
    * Add a whatsnew entry
    
    * Add a type hint to _add_shared_coord
    
    * Add a test for iris.analysis._Groupby.__repr__
    
    ---------
    
    Co-authored-by: Martin Yeo <40734014+trexfeathers@users.noreply.github.com>
    2 people authored and pp-mo committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    e6661b8 View commit details
    Browse the repository at this point in the history
  15. Finalises Lazy Data documentation (SciTools#5137)

    * cube and io lazy data notes added
    
    * [pre-commit.ci] auto fixes from pre-commit.com hooks
    
    for more information, see https://pre-commit.ci
    
    * Added comments within analysis, as well as palette and iterate, and what's new
    
    * fixed docstrings as requested in @trexfeathers review
    
    * reverted cube.py for time being
    
    * fixed flake8 issue
    
    * Lazy data second batch
    
    * [pre-commit.ci] auto fixes from pre-commit.com hooks
    
    for more information, see https://pre-commit.ci
    
    * updated lastest what'snew
    
    * I almost hope this wasn't the fix, I'm such a moron
    
    * adressed review changes
    
    ---------
    
    Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
    Co-authored-by: Bill Little <bill.james.little@gmail.com>
    3 people authored and pp-mo committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    afbdbbd View commit details
    Browse the repository at this point in the history
  16. Configuration menu
    Copy the full SHA
    b8bb753 View commit details
    Browse the repository at this point in the history
  17. update ci locks location (SciTools#5228)

    bjlittle authored and pp-mo committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    97cc149 View commit details
    Browse the repository at this point in the history
  18. Updated environment lockfiles (SciTools#5211)

    Co-authored-by: Lockfile bot <noreply@github.com>
    2 people authored and pp-mo committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    f14a321 View commit details
    Browse the repository at this point in the history
  19. Increase retries.

    pp-mo committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    f7a0b87 View commit details
    Browse the repository at this point in the history
  20. Configuration menu
    Copy the full SHA
    69ddd9d View commit details
    Browse the repository at this point in the history
  21. update cf standard units (SciTools#5244)

    * update cf standard units
    
    * added whatsnew entry
    
    * Correct pull number
    
    Co-authored-by: Martin Yeo <40734014+trexfeathers@users.noreply.github.com>
    
    ---------
    
    Co-authored-by: Martin Yeo <40734014+trexfeathers@users.noreply.github.com>
    2 people authored and pp-mo committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    8235d60 View commit details
    Browse the repository at this point in the history
  22. libnetcdf <4.9 pin (SciTools#5242)

    * Pin libnetcdf<4.9 and update lock files.
    
    * What's New entry.
    
    * libnetcdf not available on PyPI.
    
    * Fix for Pandas v2.0.
    
    * Fix for Pandas v2.0.
    trexfeathers authored and pp-mo committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    724c6d2 View commit details
    Browse the repository at this point in the history
  23. Configuration menu
    Copy the full SHA
    4f50dc7 View commit details
    Browse the repository at this point in the history
  24. Configuration menu
    Copy the full SHA
    0da68cf View commit details
    Browse the repository at this point in the history
  25. Configuration menu
    Copy the full SHA
    e8b7bfd View commit details
    Browse the repository at this point in the history
  26. Configuration menu
    Copy the full SHA
    ad48caf View commit details
    Browse the repository at this point in the history
  27. Revert some more debug changes.

    pp-mo committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    291b587 View commit details
    Browse the repository at this point in the history
  28. Configuration menu
    Copy the full SHA
    b2260ef View commit details
    Browse the repository at this point in the history

Commits on Apr 14, 2023

  1. Configuration menu
    Copy the full SHA
    33a7d86 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    db6932d View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    631e001 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    2869f97 View commit details
    Browse the repository at this point in the history

Commits on Apr 21, 2023

  1. Configuration menu
    Copy the full SHA
    419727b View commit details
    Browse the repository at this point in the history
  2. Fix attribution names.

    pp-mo committed Apr 21, 2023
    Configuration menu
    Copy the full SHA
    2f4458b View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    88b7a2a View commit details
    Browse the repository at this point in the history
  4. Fixed new py311 lockfile.

    pp-mo committed Apr 21, 2023
    Configuration menu
    Copy the full SHA
    98a20e7 View commit details
    Browse the repository at this point in the history
  5. Fix typos spotted by codespell.

    pp-mo committed Apr 21, 2023
    Configuration menu
    Copy the full SHA
    bbc1167 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    ed38e43 View commit details
    Browse the repository at this point in the history
  7. Fix lockfile for python 3.11

    pp-mo committed Apr 21, 2023
    Configuration menu
    Copy the full SHA
    54ec0f8 View commit details
    Browse the repository at this point in the history