Obtaining fresh data from the disk when reopening a NetCDF file a second time

[cjauvin]

I have a program where I open a .nc file, do something with it, and want to reopen it later, after an external program has been modifying it, and my issue is that the caching mechanism will give me the already opened version of the file, and not the refreshed version on the disk. To demonstrate this behavior, let's say you have two files: bla.nc and bla_mod.nc, with different content:

import shutil
import xarray as xr

a = xr.open_dataset("bla.nc")

# Simulate external process modifying bla.nc while this script is running
shutil.copy("bla_mod.nc", "bla.nc")

# a.close()  # this is the only thing that WOULD make it work!

b = xr.open_dataset("bla.nc") 

# Here I would expect b to be different than a, but it is not

I understand that the file SHOULD be closed (or that I should use a context manager) in an ideal world, and that if so it would work but let's say it is not (perhaps we forgot, or we're simply being lazy).

At first I thought that I could use the cache parameter to open_dataset for that purpose, but after studying the code, I discovered that it is connected to a different caching mechanism than the one that is at play here.

After some experiments to better understand the code, I came to the conclusion that the only way my particular use case could be supported (that is, without using an explicit close or a context manager, which is, in itself, debatable, I admit) is that if the underlying netCDF4._netCDF4.Dataset file object is explicitly closed, like it is when flushed out of the cache:

xarray/xarray/backends/file_manager.py

Line 222 in 5735e16

file.close()

Given that I cannot really see how, in the particular case where the user calls open_dataset for a second time, she wouldn't want the fresh version on disk, it made me think that a fix for that behavior would be to simply explicitly flush the cache immediately after the CachingFileManager for a particular dataset has been created, as I do here:

master...cjauvin:netcdf-caching-bug

Because I admit that this looks weird at first sight (why close an object immediately after having created it?), I imagine that a better option would probably be to add a boolean option to the CachingFileManager, in order to make it optional (something like flush_and_close_file_if_already_present).

I think this subtle change would result in a more coherent experience with the exact use case that I present, but admittedly, I didn't study the overall code deeply enough to be certain that it couldn't result in unwanted side effects for some other backends.

[shoyer]

I think this is basically the same issue as #4240

[shoyer]

Thanks for raising this issue, especially the in-depth explanation!

I have a tentative fix in #4879. Rather than closing existing files, it simply separately caches files for each time open_dataset() is called.