Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[pull] main from xarray-contrib:main #48

Merged
merged 50 commits into from
Oct 18, 2024
Merged

[pull] main from xarray-contrib:main #48

merged 50 commits into from
Oct 18, 2024

Conversation

pull[bot]
Copy link

@pull pull bot commented Mar 17, 2024 •
edited
Loading

See Commits and Changes for more details.

Created by pull[bot]

Can you help keep this open source service alive? 💖 Please sponsor : )

dcherian and others added 4 commits March 13, 2024 14:33
@dcherian
Fix direct reductions of Xarray objects (#339)
b0cabf3 
* Fix direct reductions of Xarray objects

Closes pydata/xarray#8819

* Fix doctest
@dependabot
Bump codecov/codecov-action from 4.0.0 to 4.1.0 (#338)
f4d5baf 
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 4.0.0 to 4.1.0.
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md)
- [Commits](codecov/codecov-action@v4.0.0...v4.1.0)

---
updated-dependencies:
- dependency-name: codecov/codecov-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@dcherian
Fix upstream-dev CI, silence warnings (#341)
382dc14 
* Fix upstream-dev CI

Closes #337

* silence warnings

* fix mypy

* Trigger upstream workflow
@dcherian
Fix direct quantile reduction (#343)
20be463
@pull pull bot added the ⤵️ pull label Mar 17, 2024
dcherian and others added 23 commits March 19, 2024 15:37
@dcherian
Fix nanlen with strings (#344)
307899a 
* Fix nanlen with strings

Closes pydata/xarray#8853

* fix windows

* Silence warnings
@dcherian
method heuristics: Avoid dot product as much as possible (#347)
4952fe9 
* Another `method` detection optimization

* fix

* silence warnings

* silence one more warning

* Even better shortcut

* Update docs
@dependabot
Bump codecov/codecov-action from 4.1.0 to 4.1.1 (#349)
603ad2c 
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 4.1.0 to 4.1.1.
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md)
- [Commits](codecov/codecov-action@v4.1.0...v4.1.1)

---
updated-dependencies:
- dependency-name: codecov/codecov-action
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@tomwhite @pre-commit-ci @dcherian
Initial minimal working Cubed example for "map-reduce" (#352)
90393df 
* Initial minimal working Cubed example for "map-reduce"

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix misspelled `aggegrate_func`

* Update flox/core.py

Co-authored-by: Deepak Cherian <dcherian@users.noreply.github.com>

* Expand to ALL_FUNCS

* Use `_finalize_results` directly

* Add test for nan values

* Removed unused dtype from test

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Move example notebook to a gist https://gist.github.com/tomwhite/2d637d2581b44468da5b7e29c30c0c49

* Add CubedArray type

* Add Cubed to CI

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Make mypy happy

* Make mypy happy (again)

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Deepak Cherian <dcherian@users.noreply.github.com>
@pre-commit-ci
[pre-commit.ci] pre-commit autoupdate (#350)
a7e36f3 
* [pre-commit.ci] pre-commit autoupdate

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.1.9 → v0.3.5](astral-sh/ruff-pre-commit@v0.1.9...v0.3.5)
- [github.com/psf/black-pre-commit-mirror: 23.12.1 → 24.3.0](psf/black-pre-commit-mirror@23.12.1...24.3.0)
- [github.com/nbQA-dev/nbQA: 1.7.1 → 1.8.5](nbQA-dev/nbQA@1.7.1...1.8.5)
- [github.com/kynan/nbstripout: 0.6.1 → 0.7.1](kynan/nbstripout@0.6.1...0.7.1)
- [github.com/abravalheri/validate-pyproject: v0.15 → v0.16](abravalheri/validate-pyproject@v0.15...v0.16)
- [github.com/rhysd/actionlint: v1.6.26 → v1.6.27](rhysd/actionlint@v1.6.26...v1.6.27)

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
@dcherian
Trim CI (#355)
114d724
@dcherian
Fix benchmarks (#358)
497e7bc 
airspeed-velocity/asv#1389 (comment)
@dcherian
Edits to climatology doc (#361)
13cb229
@dcherian
Optimize bitmask finding for chunk size 1 and single chunk cases (#360)
627bf2b 
* Optimize bitmask finding for chunk size 1.

* Fix benchmark.

* bugfix

* Add single chunk benchmark

* Optimize single chunk case.

* Add test
@tomwhite
Add cubed notebook for hourly climatology example using "map-reduce" …
efd88e1 
…method (#356)

* Add cubed notebook for hourly climatology example using "map-reduce" method

* Add cubed dependencies to docs build

* Use Cubed version with HTML repr fix
@dependabot
Bump codecov/codecov-action from 4.1.1 to 4.3.1 (#362)
2439c5c 
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 4.1.1 to 4.3.1.
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md)
- [Commits](codecov/codecov-action@v4.1.1...v4.3.1)

---
updated-dependencies:
- dependency-name: codecov/codecov-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@dcherian
Manually fuse reindexing intermediates with blockwise reduction for c…
eb3c0ef 
…ohorts. (#300)

* Manually fuse reindexing intermediates with blockwise reduction for cohorts.

```
| Change   | Before [627bf2b] <main>   | After [9d710529] <optimize-cohorts-graph>   |   Ratio | Benchmark (Parameter)                           |
|----------|----------------------------|---------------------------------------------|---------|-------------------------------------------------|
| -        | 3.39±0.02ms                | 2.98±0.01ms                                 |    0.88 | cohorts.PerfectMonthly.time_graph_construct     |
| -        | 20                         | 17                                          |    0.85 | cohorts.PerfectMonthly.track_num_layers         |
| -        | 23.0±0.07ms                | 19.0±0.1ms                                  |    0.83 | cohorts.ERA5Google.time_graph_construct         |
| -        | 4878                       | 3978                                        |    0.82 | cohorts.ERA5Google.track_num_tasks              |
| -        | 179±0.8ms                  | 147±0.5ms                                   |    0.82 | cohorts.OISST.time_graph_construct              |
| -        | 159                        | 128                                         |    0.81 | cohorts.ERA5Google.track_num_layers             |
| -        | 936                        | 762                                         |    0.81 | cohorts.PerfectMonthly.track_num_tasks          |
| -        | 1221                       | 978                                         |    0.8  | cohorts.OISST.track_num_layers                  |
| -        | 4929                       | 3834                                        |    0.78 | cohorts.ERA5DayOfYear.track_num_tasks           |
| -        | 351                        | 274                                         |    0.78 | cohorts.NWMMidwest.track_num_layers             |
| -        | 4562                       | 3468                                        |    0.76 | cohorts.ERA5DayOfYear.track_num_tasks_optimized |
| -        | 164±1ms                    | 118±0.4ms                                   |    0.72 | cohorts.ERA5DayOfYear.time_graph_construct      |
| -        | 1100                       | 735                                         |    0.67 | cohorts.ERA5DayOfYear.track_num_layers          |
| -        | 3930                       | 2605                                        |    0.66 | cohorts.NWMMidwest.track_num_tasks              |
| -        | 3715                       | 2409                                        |    0.65 | cohorts.NWMMidwest.track_num_tasks_optimized    |
| -        | 28952                      | 18798                                       |    0.65 | cohorts.OISST.track_num_tasks                   |
| -        | 27010                      | 16858                                       |    0.62 | cohorts.OISST.track_num_tasks_optimized         |
```

* fix typing
@dcherian
Use threadpool for finding labels in chunk (#327)
c398f4e 
* Use threadpool for finding labels in chunk

Great when we have lots of decent size chunks, particularly the NWM
county groupby: 600ms -> 400ms.

```
| Before [0cccb90] <optimize-again>   | After [38fe8a6c] <threadpool>   |   Ratio | Benchmark (Parameter)                       |
|--------------------------------------|---------------------------------|---------|---------------------------------------------|
| 3.50±0.2ms                           | 2.93±0.07ms                     |    0.84 | cohorts.PerfectMonthly.time_graph_construct |
| 20.0±1ms                             | 9.66±1ms                        |    0.48 | cohorts.NWMMidwest.time_find_group_cohorts  |
```

* Add threshold

* Fix + comment

* Fix benchmark.

* Tweak threshold

* Small cleanup

* Comment

* Try single allocation

* Revert "Try single allocation"

This reverts commit c6b93367e2024e60d77af24a69d177670a040dfc.

* cleanup
@dcherian
Optimize min_count when expected_groups is not provided. (#236)
0083ab2 
* Optimize min_count for all numpy

For pure numpy arrays, min_count=1 (xarray default) is the same
as min_count=None, with the right fill_value. This avoids
one useless pass over the data, and one useless copy.

We need to always accumulate count with dask, to make sure we
get the right values at the end.

* Better?
@keewis
import normalize_axis_index from numpy.lib on numpy>=2 (#364)
fb180e8 
* import `normalize_axis_index` from `numpy.lib` on `numpy>=2`

* import the right thing
@dcherian
Remove errant print statement
d68d837
@tomwhite
Cubed blockwise (#357)
7166db3 
* Initial minimal working Cubed example for "blockwise"

* Update minimum cubed version that includes cubed-dev/cubed#448

* Fix mypy errors

* Update documentation with a 'blockwise' example for Cubed
@dependabot
Bump codecov/codecov-action from 4.3.1 to 4.4.1 (#366)
07ad826 
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 4.3.1 to 4.4.1.
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md)
- [Commits](codecov/codecov-action@v4.3.1...v4.4.1)

---
updated-dependencies:
- dependency-name: codecov/codecov-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@dcherian
Property tests with hypothesis (#348)
7f98f45 
* Property tests with hypothesis

* skip on minimal env

* fix typing

* fix test

* fix mypy

* remove docstring

* try again

* fix again

* more fix

* fix tests

* Try fix

* some debug logging instead of info

* try `int8`

* Update casting behaviour

* More dtypes

* Complex fixes

* Revert "try `int8`"

This reverts commit a9097c2.

* fix dtype

* skip complex var, std

* Start fixing timedelta64

* fix casting

* exclude timedelta64, datetime64

* tweak

* filter out too_slow

* update hypothesis cache

* fix

* fix more.

* update caching strategy

* WIP

* Skip float16

* Attempt to increase numerical stablity of var, std

* update tolerances

* fix

* update action

* fixes

* Trim CI

* Cast to int64 instead of intp

* revert?

* [revert]

* try again

* debug logging

* Revert "try again"

This reverts commit a02d947.

* adapt

* Revert "Revert "try again""

This reverts commit 35ff742.

* Fix cast

* remove prints

* Revert "[revert]"

This reverts commit d143a98.

* info -> debug

* Fix quantiles

* bring back notes

* Small opt

* Just skip var, std

* Fix mypy

* no-redef

* try again
@pre-commit-ci
[pre-commit.ci] pre-commit autoupdate (#372)
22257d2 
updates:
- [github.com/astral-sh/ruff-pre-commit: v0.3.5 → v0.5.0](astral-sh/ruff-pre-commit@v0.3.5...v0.5.0)
- [github.com/pre-commit/pre-commit-hooks: v4.5.0 → v4.6.0](pre-commit/pre-commit-hooks@v4.5.0...v4.6.0)
- [github.com/psf/black-pre-commit-mirror: 24.3.0 → 24.4.2](psf/black-pre-commit-mirror@24.3.0...24.4.2)
- [github.com/codespell-project/codespell: v2.2.6 → v2.3.0](codespell-project/codespell@v2.2.6...v2.3.0)
- [github.com/abravalheri/validate-pyproject: v0.16 → v0.18](abravalheri/validate-pyproject@v0.16...v0.18)
- [github.com/rhysd/actionlint: v1.6.27 → v1.7.1](rhysd/actionlint@v1.6.27...v1.7.1)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
@dcherian
Add release.yml
29b0da1
@dependabot @dcherian
Bump codecov/codecov-action from 4.4.1 to 4.5.0 (#371)
04338d4 
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 4.4.1 to 4.5.0.
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md)
- [Commits](codecov/codecov-action@v4.4.1...v4.5.0)

---
updated-dependencies:
- dependency-name: codecov/codecov-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Deepak Cherian <dcherian@users.noreply.github.com>
@dcherian
Add scans (#370)
1c10b74 
* Add scans

* grouped reduce

* Some fixes.

* Updates for ffill

* Better ffill

* Support numpy

* cleanup

* more tests

* Fix ffill

* [WIP] expand tests

* Fixes. we need two versions of binary_op

* Fix ffill again

* Disable cumsum for now.

* Fixes.

* Fix tests: Remove overflowing test cases, proper fill_value

* typing

* Fix tests

* Try and avoid some roundoff error

* Skip float32 for cumsum

* fix min deps test

* Another fix

* Silence warnings

* Cleanup

* Add docs

* fix

* bfill

* Fix test

* hypothesis: Better CI profile

* Small change.

* Add hypothesis to all envs

* Generate chunking along all dimensions

* lint

* more guards

* more guards

* fix

* Fix typing

* cleanup

* fix mypy

* Add comments
dcherian and others added 22 commits July 28, 2024 14:15
@dcherian
Add more array types for property tests
c413fa8
@dcherian
Refactor hypothesis strategies
d3cce58
@dcherian
Type strategies
40d50d5
@dcherian
Fix overflowing skipping to handle NaNs
38af8d7 
Remove from ffill, bfill

Fix fill_value for datetime64

isnan to isnull
@dcherian
Fix ffill, bfill bugs
573dd75
@dcherian
Testing improvements.
e6159a6 
1. Record duration
2. Speed up some tests
3. Silence logging
@dcherian
Update containment image with gridlines, higher dpi
8e522b5
@dcherian
Better gridlines
8d54c9b
@dcherian
Stricter tolerance in property tests (#377)
4d03d70 
* Stricter tolerance in property tests

now that ml31415/numpy-groupies#90 was merged

* skip float32 with cumsum
@dcherian
Add cohorts snapshot tests with syrupy (#379)
cb3fc1f 
* Add cohorts snapshot tests with syrupy

* Fix.

* fix again

* Rework CI

* [revery]

* improve

* fix mypy?

* Revert "[revery]"

This reverts commit 7664e5e.

* Try again

* fix mypy
@dcherian
Optimize for-loop merging of cohorts. (#378)
f8f34b9 
* Optimize for-loop merging of cohorts.

Do this by skipping perfect cohorts that we already know about.

* Add new benchmark

* Fix

* Cleanup print statements

* minimize diff

* cleanup

* Update snapshot
@dcherian
Fix docs build (#382)
b05586c 
* [skip-ci] Bump furo

* [skip-ci] bump rtd config

* fix docs
@dcherian
Fix first, last again (#381)
f0ce343 
* Fix first, last again

Add more first, last tests

* Fix

* fix type ignores

* Add one more property test

* Support cohorts and grouped_combine

* fix docs

* fix profile
@dcherian
Avoid explicit np.nan, np.inf (#383)
4dbadae 
* Handle dtypes.NA properly for datetime/timedelta

* Add Aggregation.preserves_dtype

* Fix ffill, bfill
@dcherian
Fix bug with NaNs in by and method='blockwise' (#384)
a9f51a1 
xref pydata/xarray#9320
@dcherian
Expand groupby_reduce property tests (#385)
edc1344 
* Expand groupby_reduce property tests

* Add back var, std

* cast quantile result

* Revert "Add back var, std"

This reverts commit 805b8d3.

* pin numpy in benchmark env

* Add benchmarks as test
@dcherian
silence warning (#390)
51310a3
@dcherian
Drop python 3.9, use ruff (#392)
0438a7e 
* Drop python 3.9, use ruff

* switch to Ruff

* fix mypy

* remove toctrees

* fix
@dcherian
Preserve dtype better when specified. (#389)
7421cb1 
* Preserve dtype better when specified.

* Add one more test

* tweak test

* more test

* [revert] test with Xarray PR branch

* tweak

* show versions

* Drop python 3.9, use ruff

* switch to Ruff

* fix mypy

* remove toctrees

* fix

* one more
@dcherian
Avoid rechunking when preferred_method="blockwise" (#394)
d65181c 
* Avoid rechunking when preferred_method="blockwise"

* Add test

* fix
@dcherian
Fix default int on windows, numpy<2 (#395)
8556811 
* Avoid rechunking when preferred_method="blockwise"

* Add new numpy1 environment

* try int_ instead of intp

* Use uintp instead

* Use np.uint instead

* more fixes

* Add test

* fix

* fix again
@dcherian
Faster subsetting for cohorts (#397)
07a15c4 
* Faster subsetting for cohorts

Closes #396

* tpying
@pull pull bot merged commit 07a15c4 into Illviljan:main Oct 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
⤵️ pull
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@dcherian @tomwhite @keewis