selecting a point from an mfdataset #1396
Comments
|
Can you try using This might be an issue with xarray verifying aligned coordinates, which we should have an option to disable. (This came up somewhere else recently, but I couldn't find the issue with a quick search.) |
|
The relevant part of the stack trace is I think the issue you are referring to is also mine (#1385). |
|
OK, so that isn't terribly useful -- the slow-down is somewhere in dask-land. If it was an issue with alignment, that would come up when building the dask graph, not computing it. |
|
One thing worth trying is specifying |
This is definitely what I suspect is happening. The problem with adding more chunks is that I quickly hit my system ulimit (see #1394), since, for some reason, all the 1754 files are opened as soon as I call |
|
I have created a self-contained, reproducible example of this serious performance problem. This issue is becoming a big problem for me. I imagine other people must be experiencing it too. I am happy to try to dig in and fix it, but I think some of @shoyer's backend insight would be valuable first. |
|
This dask bug also explains why it is so slow to generate the |
|
We had similar performance issues with xarray+dask, which we solved by using a chunking heuristic when opening a dataset. You can read about it in #1440. Now, in our case the data really wouldn't fit in memory, which is clearly not the case in your gist. Anyway, I thought I'd play around with your gist and see if chunking can make a difference. I couldn't use your example directly, as the data it generates in memory is too large for the dev VM I'm on with this. So I changed the generated file size to (12, 1000, 2000), the essence of your gist remained though, it would take ~25 seconds to do the time series extraction, whereas ~800 ms using So, I thought I'd try our 'chunking heuristic' on the generated test datasets. Simply split the dataset in 2x2 chunks along spatial dimensions. So: ds = xr.open_mfdataset(all_files, decode_cf=False, chunks={'time':12, 'x':1000, 'y':500})To my surprise: # time extracting a timeseries of a single point
y, x = 200, 300
with ProgressBar():
%time ts = ds.data[:, y, x].load()results in I'm not entirely sure what's happening, as the file obviously fits in memory just fine because the looping thing works well. Maybe it's fine when you loop through them one by one, but the single file chunk turns out to be too large when dask wants to parallelize the whole thing. I really have no idea. I'd be very intrigued to see if you can get a similar result by doing a simple 2x2xtime chunking. By the way, EDIT: print(xr.__version__)
print(dask.__version__) |
|
Hi @JanisGailis. Thanks for looking into this issue! I will give your solution a try as soon as I get some free time. However, I would like to point out that the issue is completely resolved by dask/dask#2364. So this can probably be closed after the next dask release. |
|
That's great to know! I think there's no need to try my 'solution' then, maybe only out of pure interest. It would of course be interesting to know why a 'custom' chunked dataset was apparently not affected by the bug. And if it was indeed the case. EDIT: I read the discussion on dask github and the xarray mailinglist. It's probably because when explicit chunking is used, the chunks are not aliased and fusing works as expected. |
This comment has been minimized.
This comment has been minimized.
|
closed via dask/dask#2364 (a long time ago) |
Sorry to be opening so many vague performance issues. I am really having a hard time with my current dataset, which is exposing certain limitations of xarray and dask in a way none of my previous work has done.
I have a directory full of netCDF4 files. There are 1754 files, each 8.1GB in size, each representing a single model timestep. So there is ~14 TB of data total. (In addition to the time-dependent output, there is a single file with information about the grid.)
Imagine I want to extract a timeseries from a single point (indexed by
k, j, i) in this simulation. Without xarray, I would do something like this:Which goes reasonably quick: tqdm gives
[02:38<00:00, 11.56it/s].I could do the same sort of loop using xarray:
Which has a <50% performance overhead:
[03:29<00:00, 8.74it/s]. Totally acceptable.Of course, what I really want is to avoid a loop and deal with the whole dataset as a single self-contained object.
This alone takes between 4-5 minutes to run (see #1385). If I want to print the repr, it takes another 3 minutes or so to
print(ds). The full dataset looks like this:Now, to extract the same timeseries, I would like to say
I monitor what is happening under the hood using when I call this by using netdata and the dask.distributed dashboard, using only a single process and thread. First, all the files are opened (see #1394). Then they start getting read. Each read takes between 10 and 30 seconds, and the memory usage starts increasing steadily. My impression is that the entire dataset is being read into memory for concatenation. (I have dumped out the dask graph in case anyone can make sense of it.) I have never let this calculation complete, as it looks like it would eat up all the memory on my system...plus it's extremely slow.
To me, this seems like a failure of lazy indexing. I naively expected that the underlying file access would work similar to my loop, perhaps even in parallel.
Can anyone shed some light on what might be going wrong?
The text was updated successfully, but these errors were encountered: