You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently gcsfs implements a cache_timeout and s3fs might want one too (fsspec/s3fs#271, fsspec/s3fs#253). So maybe we should just add it to AbstractFileSystem.
Proposal
Adds a cache_timeout to AbstractFileSystem.__init__. Taken from the gcsfs docs
cache_timeout: float, optional
Cache expiration time in seconds for object metadata cache.
Set cache_timeout <= 0 for no caching, None for no cache expiration.
As for the default, I think None makes the most sense for now since that's what gcsfs does. We can explore deprecating None in favor of something else (I think 0 for no caching) down the road.
One question: is this timeout per-key, per-directory, or global to the filesystem instance? In the gcsfs docs, we state
GCSFileSystem maintains a per-implied-directory cache of object listings and
fulfills all object information and listing requests from cache.
But from what I can tell, the cache is actually per object, not per-directory.
Currently gcsfs implements a
cache_timeout
and s3fs might want one too (fsspec/s3fs#271, fsspec/s3fs#253). So maybe we should just add it to AbstractFileSystem.Proposal
Adds a
cache_timeout
toAbstractFileSystem.__init__
. Taken from the gcsfs docsAs for the default, I think
None
makes the most sense for now since that's what gcsfs does. We can explore deprecatingNone
in favor of something else (I think0
for no caching) down the road.One question: is this timeout per-key, per-directory, or global to the filesystem instance? In the gcsfs docs, we state
But from what I can tell, the cache is actually per object, not per-directory.
and
The text was updated successfully, but these errors were encountered: