Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

document storage classes and some developer apis #2279

Merged
merged 37 commits into from
Oct 13, 2024
Merged
Show file tree
Hide file tree
Changes from 29 commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
fe49f5f
fix: zarr v2 compatability fixes
jhamman Sep 14, 2024
9a1580b
move zarr.store to zarr.storage
jhamman Sep 16, 2024
0d89912
make chunks a tuple
jhamman Sep 17, 2024
d78e384
Merge branch 'v3' of https://github.com/zarr-developers/zarr-python i…
jhamman Sep 17, 2024
e534279
Apply suggestions from code review
jhamman Sep 17, 2024
dea4a3d
Merge branch 'v3' of https://github.com/zarr-developers/zarr-python i…
jhamman Sep 18, 2024
7800f38
Merge branch 'v3' of https://github.com/zarr-developers/zarr-python i…
jhamman Sep 22, 2024
93b61fc
more merge conflict resolution
jhamman Sep 23, 2024
88afe52
Merge branch 'v3' of https://github.com/zarr-developers/zarr-python i…
jhamman Sep 29, 2024
fb6752d
fixups
jhamman Sep 29, 2024
0b1dedc
fixup zipstore
jhamman Sep 29, 2024
322918a
Apply suggestions from code review
jhamman Sep 29, 2024
a95d54a
Apply suggestions from code review
jhamman Sep 30, 2024
128eb53
add test
jhamman Sep 30, 2024
3c170ef
Merge branch 'v3' of https://github.com/zarr-developers/zarr-python i…
jhamman Sep 30, 2024
54ab9ef
extend test
jhamman Sep 30, 2024
77f2938
clean up parents
jhamman Sep 30, 2024
2295d76
debug race condition
jhamman Sep 30, 2024
5879d67
more debug
jhamman Sep 30, 2024
f54f6f2
document storage classes and some developer apis
jhamman Oct 1, 2024
3940d22
Update src/zarr/core/array.py
jhamman Oct 1, 2024
c9d1c50
Merge branch 'fix/dask-compat' into doc/storage
jhamman Oct 2, 2024
83fa3ac
Merge branch 'v3' of https://github.com/zarr-developers/zarr-python i…
jhamman Oct 7, 2024
f2137fb
Merge branch 'doc/storage' of github.com:jhamman/zarr-python into doc…
jhamman Oct 7, 2024
f44601b
Merge branch 'v3' into doc/storage
jhamman Oct 9, 2024
731c22a
Merge branch 'v3' of https://github.com/zarr-developers/zarr-python i…
jhamman Oct 10, 2024
4e93aa0
inherit docstrings from baseclass
jhamman Oct 10, 2024
dad94e2
Merge branch 'doc/storage' of github.com:jhamman/zarr-python into doc…
jhamman Oct 10, 2024
5477d32
fix sphinx warning
jhamman Oct 11, 2024
728d4f7
# docstring inherited
jhamman Oct 11, 2024
e293b47
Merge branch 'v3' of https://github.com/zarr-developers/zarr-python i…
jhamman Oct 11, 2024
cee730c
Merge branch 'v3' of https://github.com/zarr-developers/zarr-python i…
jhamman Oct 11, 2024
3f7290f
add storage guide
jhamman Oct 11, 2024
35a6428
add missing file
jhamman Oct 11, 2024
a78461e
update links
jhamman Oct 11, 2024
eff01ce
Merge branch 'v3' into doc/storage
jhamman Oct 11, 2024
c1f0923
Merge branch 'v3' into doc/storage
dstansby Oct 12, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
87 changes: 84 additions & 3 deletions src/zarr/abc/store.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@


class AccessMode(NamedTuple):
"""Access mode flags."""

str: AccessModeLiteral
readonly: bool
overwrite: bool
Expand All @@ -28,6 +30,24 @@ class AccessMode(NamedTuple):

@classmethod
def from_literal(cls, mode: AccessModeLiteral) -> Self:
"""
Create an AccessMode instance from a literal.

Parameters
----------
mode : AccessModeLiteral
One of 'r', 'r+', 'w', 'w-', 'a'.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we define these as a string somewhere, and then re-use it here and lower down by making the docstring a format string? I worry about lists like this getting out of sync if they're duplicated across docstrings.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I decided not to do that here because its just this one method. If we have a docstring template tool in the future, I think it would be great to bring that to bear here (and even more so on the Group/Array classes).


Returns
-------
AccessMode
The created instance.

Raises
------
ValueError
If mode is not one of 'r', 'r+', 'w', 'w-', 'a'.
"""
if mode in ("r", "r+", "a", "w", "w-"):
return cls(
str=mode,
Expand All @@ -40,6 +60,10 @@ def from_literal(cls, mode: AccessModeLiteral) -> Self:


class Store(ABC):
"""
Abstract base class for Zarr stores.
"""

_mode: AccessMode
_is_open: bool

Expand All @@ -49,6 +73,21 @@ def __init__(self, *args: Any, mode: AccessModeLiteral = "r", **kwargs: Any) ->

@classmethod
async def open(cls, *args: Any, **kwargs: Any) -> Self:
"""
Create and open the store.

Parameters
----------
*args : Any
Positional arguments to pass to the store constructor.
**kwargs : Any
Keyword arguments to pass to the store constructor.

Returns
-------
Store
The opened store instance.
"""
store = cls(*args, **kwargs)
await store._open()
return store
Expand All @@ -67,6 +106,20 @@ def __exit__(
self.close()

async def _open(self) -> None:
"""
Open the store.

Raises
------
ValueError
If the store is already open.
FileExistsError
If ``mode='w-'`` and the store already exists.

Notes
-----
* When ``mode='w'`` and the store already exists, it will be cleared.
"""
if self._is_open:
raise ValueError("store is already open")
if self.mode.str == "w":
Expand All @@ -76,14 +129,30 @@ async def _open(self) -> None:
self._is_open = True

async def _ensure_open(self) -> None:
"""Open the store if it is not already open."""
if not self._is_open:
await self._open()

@abstractmethod
async def empty(self) -> bool: ...
async def empty(self) -> bool:
"""
Check if the store is empty.

Returns
-------
bool
True if the store is empty, False otherwise.
"""
...

@abstractmethod
async def clear(self) -> None: ...
async def clear(self) -> None:
"""
Clear the store.

Remove all keys and values from the store.
"""
...

@abstractmethod
def with_mode(self, mode: AccessModeLiteral) -> Self:
Expand Down Expand Up @@ -116,6 +185,7 @@ def mode(self) -> AccessMode:
return self._mode

def _check_writable(self) -> None:
"""Raise an exception if the store is not writable."""
if self.mode.readonly:
raise ValueError("store mode does not support writing")

Expand Down Expand Up @@ -199,7 +269,7 @@ async def set_if_not_exists(self, key: str, value: Buffer) -> None:
Store a key to ``value`` if the key is not already present.

Parameters
-----------
----------
key : str
value : Buffer
"""
Expand Down Expand Up @@ -339,6 +409,17 @@ async def set_if_not_exists(self, default: Buffer) -> None: ...


async def set_or_delete(byte_setter: ByteSetter, value: Buffer | None) -> None:
"""Set or delete a value in a byte setter

Parameters
----------
byte_setter : ByteSetter
value : Buffer | None

Notes
-----
If value is None, the key will be deleted.
"""
if value is None:
await byte_setter.delete()
else:
Expand Down
30 changes: 30 additions & 0 deletions src/zarr/core/common.py
Original file line number Diff line number Diff line change
Expand Up @@ -151,3 +151,33 @@ def parse_order(data: Any) -> Literal["C", "F"]:
if data in ("C", "F"):
return cast(Literal["C", "F"], data)
raise ValueError(f"Expected one of ('C', 'F'), got {data} instead.")


def _inherit_docstrings(cls: type[Any]) -> type[Any]:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Docstrings are already automatically inherited, e.g. see Store and LocalStore. So I think this isn't needed and can be gotten rid of.

"""
Inherit docstrings from base class

Iterate over the methods of the class and if a method is missing a docstring,
try to inherit one from a base class (ABC).

Parameters
----------
cls : object
the class to inherit docstrings from

Returns
-------
cls
the class with updated docstrings
"""
# Iterate over the methods of the class
for name, method in cls.__dict__.items():
# Skip if it's not a callable (method or function)
if callable(method):
# Get the corresponding method from the base class (ABC)
for base in cls.__bases__:
base_method = getattr(base, name, None)
if base_method and not method.__doc__:
method.__doc__ = base_method.__doc__
break
return cls
2 changes: 2 additions & 0 deletions src/zarr/storage/__init__.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,13 @@
from zarr.storage.common import StoreLike, StorePath, make_store_path
from zarr.storage.local import LocalStore
from zarr.storage.logging import LoggingStore
from zarr.storage.memory import MemoryStore
from zarr.storage.remote import RemoteStore
from zarr.storage.zip import ZipStore

__all__ = [
"LocalStore",
"LoggingStore",
"MemoryStore",
"RemoteStore",
"StoreLike",
Expand Down
123 changes: 123 additions & 0 deletions src/zarr/storage/common.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,17 @@ def _dereference_path(root: str, path: str) -> str:


class StorePath:
"""
Path-like interface for a Store.

Parameters
----------
store : Store
The store to use.
path : str
The path within the store.
"""

store: Store
path: str

Expand All @@ -39,25 +50,80 @@ async def get(
prototype: BufferPrototype | None = None,
byte_range: ByteRangeRequest | None = None,
) -> Buffer | None:
"""
Read bytes from the store.

Parameters
----------
prototype : BufferPrototype, optional
The buffer prototype to use when reading the bytes.
byte_range : ByteRangeRequest, optional
The range of bytes to read.

Returns
-------
buffer : Buffer or None
The read bytes, or None if the key does not exist.
"""
if prototype is None:
prototype = default_buffer_prototype()
return await self.store.get(self.path, prototype=prototype, byte_range=byte_range)

async def set(self, value: Buffer, byte_range: ByteRangeRequest | None = None) -> None:
"""
Write bytes to the store.

Parameters
----------
value : Buffer
The buffer to write.
byte_range : ByteRangeRequest, optional
The range of bytes to write. If None, the entire buffer is written.

Raises
------
NotImplementedError
If `byte_range` is not None, because Store.set does not support partial writes yet.
"""
if byte_range is not None:
raise NotImplementedError("Store.set does not have partial writes yet")
await self.store.set(self.path, value)

async def delete(self) -> None:
"""
Delete the key from the store.

Raises
------
NotImplementedError
If the store does not support deletion.
"""
await self.store.delete(self.path)

async def set_if_not_exists(self, default: Buffer) -> None:
"""
Store a key to ``value`` if the key is not already present.

Parameters
----------
default : Buffer
The buffer to store if the key is not already present.
"""
await self.store.set_if_not_exists(self.path, default)

async def exists(self) -> bool:
"""
Check if the key exists in the store.

Returns
-------
bool
True if the key exists in the store, False otherwise.
"""
return await self.store.exists(self.path)

def __truediv__(self, other: str) -> StorePath:
"""combine this store path with another path"""
return self.__class__(self.store, _dereference_path(self.path, other))

def __str__(self) -> str:
Expand All @@ -67,6 +133,19 @@ def __repr__(self) -> str:
return f"StorePath({self.store.__class__.__name__}, {str(self)!r})"

def __eq__(self, other: object) -> bool:
"""
Check if two StorePath objects are equal.

Returns
-------
bool
True if the two objects are equal, False otherwise.

Notes
-----
Two StorePath objects are considered equal if their stores are equal
and their paths are equal.
"""
try:
return self.store == other.store and self.path == other.path # type: ignore[attr-defined, no-any-return]
except Exception:
Expand All @@ -83,6 +162,50 @@ async def make_store_path(
mode: AccessModeLiteral | None = None,
storage_options: dict[str, Any] | None = None,
) -> StorePath:
"""
Convert a `StoreLike` object into a StorePath object.

This function takes a `StoreLike` object and returns a `StorePath` object. The
`StoreLike` object can be a `Store`, `StorePath`, `Path`, `str`, or `dict[str, Buffer]`.
If the `StoreLike` object is a Store or `StorePath`, it is converted to a
`StorePath` object. If the `StoreLike` object is a Path or str, it is converted
to a LocalStore object and then to a `StorePath` object. If the `StoreLike`
object is a dict[str, Buffer], it is converted to a `MemoryStore` object and
then to a `StorePath` object.

If the `StoreLike` object is None, a `MemoryStore` object is created and
converted to a `StorePath` object.

If the `StoreLike` object is a str and starts with a protocol, it is
converted to a RemoteStore object and then to a `StorePath` object.

If the `StoreLike` object is a dict[str, Buffer] and the mode is not None,
the `MemoryStore` object is created with the given mode.

If the `StoreLike` object is a str and starts with a protocol, the
RemoteStore object is created with the given mode and storage options.

Parameters
----------
store_like : StoreLike | None
The object to convert to a `StorePath` object.
mode : AccessModeLiteral | None, optional
The mode to use when creating the `StorePath` object. If None, the
default mode is 'r'.
storage_options : dict[str, Any] | None, optional
The storage options to use when creating the `RemoteStore` object. If
None, the default storage options are used.

Returns
-------
StorePath
The converted StorePath object.

Raises
------
TypeError
If the StoreLike object is not one of the supported types.
"""
from zarr.storage.remote import RemoteStore # circular import

used_storage_options = False
Expand Down
Loading