-
Notifications
You must be signed in to change notification settings - Fork 19
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Working with ProxySource: examples. #259
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Several comments. I think we need a team discussion about some of them.
src/blosc2/proxy.py
Outdated
>>> ndarray = blosc2.asarray(data) | ||
>>> proxy = blosc2.Proxy(ndarray) | ||
>>> full_data = proxy.fetch() | ||
>>> f"Full data cache: {full_data[:]}" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this would be more graphical if you fetch just an slice (say [slice(0, 3), slice(0,2)
, so that when data is printed, one can see that only that slice has been fetched (i.e. other values are uninitialized).
src/blosc2/proxy.py
Outdated
[12 13][14 15][16 17] | ||
[18 19]] | ||
>>> slice_data = proxy[0:2, :] | ||
>>> f"Slice data cache: {slice_data}" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here it makes little sense to show the slice, as the full array has been fetched already. Better put this above, as already suggested.
src/blosc2/proxy.py
Outdated
>>> self.chunks = [data.shape[i] for i in range(data.ndim)] | ||
>>> self.blocks = [data.shape[i] for i in range(data.ndim)] | ||
>>> self.dtype = data.dtype | ||
>>> f"Data shape: {self.shape}, Chunks: {self.chunks}" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Too many indentation here
src/blosc2/proxy.py
Outdated
>>> async def aget_chunk(self, nchunk): | ||
>>> await asyncio.sleep(0.1) | ||
>>> return self.data[nchunk] | ||
>>> # Class that inherits from blosc2.Proxy |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This comment is trivial, it can go away.
src/blosc2/proxy.py
Outdated
>>> import asyncio | ||
>>> class MyProxySource: | ||
>>> def __init__(self, data): | ||
>>> # If the source is multidimensional, it must have the attributes: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"the" -> "the next"
>>> ndarray = blosc2.asarray(data) | ||
>>> proxy = blosc2.Proxy(ndarray) | ||
>>> slice_1 = proxy[0:3, 0:3] | ||
>>> f"Slice 1: {slice_1}" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same, print the entire proxy values to see that only slice [:3, :3]
has been filled. If (10, 10)
is too large for the dump, use e.g. (5, 5)
.
src/blosc2/proxy.py
Outdated
"""The dtype of :paramref:`self` or None if the data is unidimensional""" | ||
"""The dtype of :paramref:`self` or None if the data is unidimensional | ||
|
||
Examples |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think these are trivial. I'd remove examples for properties like this.
src/blosc2/proxy.py
Outdated
"""The shape of :paramref:`self`""" | ||
"""The shape of :paramref:`self` | ||
|
||
Examples |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ditto
@@ -224,6 +329,16 @@ def vlmeta(self): | |||
See Also | |||
-------- | |||
:ref:`SChunk.vlmeta` | |||
|
|||
Examples |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ditto
@@ -240,6 +355,26 @@ def fields(self): | |||
See Also | |||
-------- | |||
:ref:`NDField` | |||
|
|||
Examples |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm unsure about this one. It brings some insight about how to use the fields
property, but on the other hand, this has already been documented in the original NDArray.fields
. Better a See also
section?
9ad6ae6
to
6ef1029
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With the new amendments, this looks good to me.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some type annotations should be restored. Other than a comment on the Proxy example, LGTM.
src/blosc2/proxy.py
Outdated
import blosc2 | ||
import numpy as np | ||
|
||
import blosc2 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No need to repeat import blosc2
src/blosc2/proxy.py
Outdated
@@ -50,7 +51,7 @@ def dtype(self) -> np.dtype: | |||
pass | |||
|
|||
@abstractmethod | |||
def get_chunk(self, nchunk: int) -> bytes: | |||
def get_chunk(self, nchunk): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Restore type annotations def get_chunk(self, nchunk: int) -> bytes:
src/blosc2/proxy.py
Outdated
@@ -226,7 +227,7 @@ def __init__(self, src: ProxySource or ProxyNDSource, urlpath: str = None, **kwa | |||
for key in vlmeta: | |||
self._schunk_cache.vlmeta[key] = vlmeta[key] | |||
|
|||
def fetch(self, item: slice | list[slice] = None) -> blosc2.NDArray | blosc2.schunk.SChunk: | |||
def fetch(self, item=None): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Restore type annotations
src/blosc2/proxy.py
Outdated
@@ -276,7 +272,7 @@ def fetch(self, item: slice | list[slice] = None) -> blosc2.NDArray | blosc2.sch | |||
|
|||
return self._cache | |||
|
|||
async def afetch(self, item: slice | list[slice] = None) -> blosc2.NDArray | blosc2.schunk.SChunk: | |||
async def afetch(self, item=None): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Dito
src/blosc2/proxy.py
Outdated
>>> class MyProxy(blosc2.Proxy): | ||
>>> def __init__(self, source): | ||
>>> super().__init__(source) | ||
>>> self._cache = source.data | ||
>>> # Asynchronous method to get the cache data | ||
>>> async def afetch(self, slice_=None): | ||
>>> return self._cache if slice_ is None else self._cache[slice_] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You do not need to create a class MyProxy
which inherits from blosc2.Proxy
. You can directly create the proxy with proxy = blosc2.Proxy(source)
""" | ||
# Populate the cache | ||
self.fetch(item) | ||
return self._cache[item] | ||
|
||
@property | ||
def dtype(self) -> np.dtype: | ||
"""The dtype of :paramref:`self` or None if the data is unidimensional""" | ||
def dtype(self): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Dito
return self._cache.dtype if isinstance(self._cache, blosc2.NDArray) else None | ||
|
||
@property | ||
def shape(self) -> tuple[int]: | ||
"""The shape of :paramref:`self`""" | ||
def shape(self): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Dito
src/blosc2/proxy.py
Outdated
return self._cache.shape if isinstance(self._cache, blosc2.NDArray) else len(self._cache) | ||
|
||
def __str__(self): | ||
return f"Proxy({self.src}, urlpath={self.urlpath})" | ||
|
||
@property | ||
def vlmeta(self) -> blosc2.schunk.vlmeta: | ||
def vlmeta(self): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Dito
src/blosc2/proxy.py
Outdated
@@ -378,7 +428,7 @@ def vlmeta(self) -> blosc2.schunk.vlmeta: | |||
return self._schunk_cache.vlmeta | |||
|
|||
@property | |||
def fields(self) -> dict: | |||
def fields(self): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Dito
src/blosc2/proxy.py
Outdated
@@ -404,7 +474,7 @@ def __init__(self, proxy: Proxy, field: str): | |||
self.shape = proxy.shape | |||
self.dtype = proxy.dtype | |||
|
|||
def __getitem__(self, item: slice | list[slice]) -> np.ndarray: | |||
def __getitem__(self, item: slice): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Dito
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have some comments.
src/blosc2/c2array.py
Outdated
>>> host = "https://demo.caterva2.net/" | ||
>>> root = "b2tests" | ||
>>> dir = "expr/" | ||
>>> name = "ds-0-10-linspace-float64-(True, True)-a1-(60, 60)d.b2nd" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd use regular examples in example
root.
src/blosc2/c2array.py
Outdated
>>> f"Shape of the remote array: {remote_array.shape}" | ||
>>> f"Chunks of the remote array: {remote_array.chunks}" | ||
>>> f"Blocks of the remote array: {remote_array.blocks}" | ||
>>> f"Dtype of the remote array: {remote_array.dtype}" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd replace this by just:
>>> remote_array.shape
(60, 60)
...
This pull request has been replaced by #267 |
No description provided.