-
-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Efficient rolling 'trick' #1978
Comments
I'll let @fujiisoup / @shoyer confirm but this looks eerily similar to #1837. |
This is exactly what @fujiisoup just implemented in #1837. It's a nice trick. If the window is large, it's not quite as fast using a clever algorithm for rolling sums like bottleneck, but it still gives something like a 100x performance boost over the naive loop we used to use. |
Yes. In [7]: da.rolling(date=3).construct('rolling_date')
Out[7]:
<xarray.DataArray (item: 2, date: 6, rolling_date: 3)>
array([[[nan, nan, 0.],
[nan, 0., 1.],
[ 0., 1., 2.],
[ 1., 2., 3.],
[ 2., 3., 4.],
[ 3., 4., 5.]],
[[nan, nan, 6.],
[nan, 6., 7.],
[ 6., 7., 8.],
[ 7., 8., 9.],
[ 8., 9., 10.],
[ 9., 10., 11.]]])
Dimensions without coordinates: item, date, rolling_date does the similar thing ( FYI, using |
🤦♂️ I think my memory is getting worse every day! I glanced at this myself back in Jan. Closing! |
Based off http://www.rigtorp.se/2011/01/01/rolling-statistics-numpy.html, we wrote up a function that 'tricks' numpy into presenting an array that looks rolling, but without the O^2 memory requirements
Would people be interested in this going into xarray?
It seems to work really well on a few use-cases, but I imagine it's enough trickery that we might not want to support it in xarray.
And, to be clear, it's strictly worse where we have rolling algos. But where we don't, you get a rolling
apply
without the python loops.The text was updated successfully, but these errors were encountered: