Working with ProxySource: examples. #259

omaech · 2024-09-18T11:40:02Z

No description provided.

FrancescAlted

Several comments. I think we need a team discussion about some of them.

FrancescAlted · 2024-09-18T15:11:29Z

src/blosc2/proxy.py

+        >>> ndarray = blosc2.asarray(data)
+        >>> proxy = blosc2.Proxy(ndarray)
+        >>> full_data = proxy.fetch()
+        >>> f"Full data cache: {full_data[:]}"


I think this would be more graphical if you fetch just an slice (say [slice(0, 3), slice(0,2), so that when data is printed, one can see that only that slice has been fetched (i.e. other values are uninitialized).

FrancescAlted · 2024-09-18T15:13:18Z

src/blosc2/proxy.py

+            [12 13][14 15][16 17]
+            [18 19]]
+        >>> slice_data = proxy[0:2, :]
+        >>> f"Slice data cache: {slice_data}"


Here it makes little sense to show the slice, as the full array has been fetched already. Better put this above, as already suggested.

FrancescAlted · 2024-09-18T15:13:48Z

src/blosc2/proxy.py

+        >>>                 self.chunks = [data.shape[i] for i in range(data.ndim)]
+        >>>                 self.blocks = [data.shape[i] for i in range(data.ndim)]
+        >>>                 self.dtype = data.dtype
+        >>>                 f"Data shape: {self.shape}, Chunks: {self.chunks}"


Too many indentation here

FrancescAlted · 2024-09-18T15:15:47Z

src/blosc2/proxy.py

+        >>>       async def aget_chunk(self, nchunk):
+        >>>                 await asyncio.sleep(0.1)
+        >>>                 return self.data[nchunk]
+        >>> # Class that inherits from blosc2.Proxy


This comment is trivial, it can go away.

FrancescAlted · 2024-09-18T15:18:19Z

src/blosc2/proxy.py

+        >>> import asyncio
+        >>> class MyProxySource:
+        >>>       def __init__(self, data):
+        >>>                # If the source is multidimensional, it must have the attributes:


"the" -> "the next"

FrancescAlted · 2024-09-18T15:37:54Z

src/blosc2/proxy.py

+        >>> ndarray = blosc2.asarray(data)
+        >>> proxy = blosc2.Proxy(ndarray)
+        >>> slice_1 = proxy[0:3, 0:3]
+        >>> f"Slice 1: {slice_1}"


Same, print the entire proxy values to see that only slice [:3, :3] has been filled. If (10, 10) is too large for the dump, use e.g. (5, 5).

FrancescAlted · 2024-09-18T15:38:45Z

src/blosc2/proxy.py

-        """The dtype of :paramref:`self` or None if the data is unidimensional"""
+        """The dtype of :paramref:`self` or None if the data is unidimensional
+
+        Examples


I think these are trivial. I'd remove examples for properties like this.

FrancescAlted · 2024-09-18T15:38:56Z

src/blosc2/proxy.py

-        """The shape of :paramref:`self`"""
+        """The shape of :paramref:`self`
+
+        Examples


FrancescAlted · 2024-09-18T15:39:12Z

src/blosc2/proxy.py

@@ -224,6 +329,16 @@ def vlmeta(self):
        See Also
        --------
        :ref:`SChunk.vlmeta`
+
+        Examples


FrancescAlted · 2024-09-18T15:44:28Z

src/blosc2/proxy.py

@@ -240,6 +355,26 @@ def fields(self):
        See Also
        --------
        :ref:`NDField`
+
+        Examples


I'm unsure about this one. It brings some insight about how to use the fields property, but on the other hand, this has already been documented in the original NDArray.fields. Better a See also section?

FrancescAlted

With the new amendments, this looks good to me.

martaiborra

Some type annotations should be restored. Other than a comment on the Proxy example, LGTM.

martaiborra · 2024-09-23T08:24:50Z

src/blosc2/proxy.py

+import blosc2
 import numpy as np

 import blosc2


No need to repeat import blosc2

martaiborra · 2024-09-23T08:25:23Z

src/blosc2/proxy.py

@@ -50,7 +51,7 @@ def dtype(self) -> np.dtype:
        pass

    @abstractmethod
-    def get_chunk(self, nchunk: int) -> bytes:
+    def get_chunk(self, nchunk):


Restore type annotations def get_chunk(self, nchunk: int) -> bytes:

martaiborra · 2024-09-23T08:25:45Z

src/blosc2/proxy.py

@@ -226,7 +227,7 @@ def __init__(self, src: ProxySource or ProxyNDSource, urlpath: str = None, **kwa
            for key in vlmeta:
                self._schunk_cache.vlmeta[key] = vlmeta[key]

-    def fetch(self, item: slice | list[slice] = None) -> blosc2.NDArray | blosc2.schunk.SChunk:
+    def fetch(self, item=None):


Restore type annotations

martaiborra · 2024-09-23T08:26:41Z

src/blosc2/proxy.py

@@ -276,7 +272,7 @@ def fetch(self, item: slice | list[slice] = None) -> blosc2.NDArray | blosc2.sch

        return self._cache

-    async def afetch(self, item: slice | list[slice] = None) -> blosc2.NDArray | blosc2.schunk.SChunk:
+    async def afetch(self, item=None):


martaiborra · 2024-09-23T08:32:55Z

src/blosc2/proxy.py

+        >>> class MyProxy(blosc2.Proxy):
+        >>>     def __init__(self, source):
+        >>>               super().__init__(source)
+        >>>               self._cache = source.data
+        >>>     # Asynchronous method to get the cache data
+        >>>     async def afetch(self, slice_=None):
+        >>>               return self._cache if slice_ is None else self._cache[slice_]


You do not need to create a class MyProxy which inherits from blosc2.Proxy. You can directly create the proxy with proxy = blosc2.Proxy(source)

martaiborra · 2024-09-23T08:40:09Z

src/blosc2/proxy.py

        """
        # Populate the cache
        self.fetch(item)
        return self._cache[item]

    @property
-    def dtype(self) -> np.dtype:
-        """The dtype of :paramref:`self` or None if the data is unidimensional"""
+    def dtype(self):


martaiborra · 2024-09-23T08:40:16Z

src/blosc2/proxy.py

        return self._cache.dtype if isinstance(self._cache, blosc2.NDArray) else None

    @property
-    def shape(self) -> tuple[int]:
-        """The shape of :paramref:`self`"""
+    def shape(self):


martaiborra · 2024-09-23T08:40:25Z

src/blosc2/proxy.py

        return self._cache.shape if isinstance(self._cache, blosc2.NDArray) else len(self._cache)

    def __str__(self):
        return f"Proxy({self.src}, urlpath={self.urlpath})"

    @property
-    def vlmeta(self) -> blosc2.schunk.vlmeta:
+    def vlmeta(self):


martaiborra · 2024-09-23T08:40:31Z

src/blosc2/proxy.py

@@ -378,7 +428,7 @@ def vlmeta(self) -> blosc2.schunk.vlmeta:
        return self._schunk_cache.vlmeta

    @property
-    def fields(self) -> dict:
+    def fields(self):


martaiborra · 2024-09-23T08:41:07Z

src/blosc2/proxy.py

@@ -404,7 +474,7 @@ def __init__(self, proxy: Proxy, field: str):
        self.shape = proxy.shape
        self.dtype = proxy.dtype

-    def __getitem__(self, item: slice | list[slice]) -> np.ndarray:
+    def __getitem__(self, item: slice):


FrancescAlted

I have some comments.

FrancescAlted · 2024-09-24T06:57:27Z

src/blosc2/c2array.py

+        >>> host = "https://demo.caterva2.net/"
+        >>> root = "b2tests"
+        >>> dir = "expr/"
+        >>> name = "ds-0-10-linspace-float64-(True, True)-a1-(60, 60)d.b2nd"


I'd use regular examples in example root.

FrancescAlted · 2024-09-24T06:58:28Z

src/blosc2/c2array.py

+        >>> f"Shape of the remote array: {remote_array.shape}"
+        >>> f"Chunks of the remote array: {remote_array.chunks}"
+        >>> f"Blocks of the remote array: {remote_array.blocks}"
+        >>> f"Dtype of the remote array: {remote_array.dtype}"


I'd replace this by just:

>>> remote_array.shape (60, 60) ...

omaech · 2024-09-25T11:16:36Z

This pull request has been replaced by #267

FrancescAlted requested changes Sep 18, 2024

View reviewed changes

omaech added 2 commits September 19, 2024 13:35

Proxy examples

97c4d4c

Proxy examples: first corrections.

6ef1029

omaech force-pushed the proxy-examples branch from 9ad6ae6 to 6ef1029 Compare September 19, 2024 11:36

Merge branch 'main' into proxy-examples

0af76a5

FrancescAlted self-requested a review September 23, 2024 08:16

FrancescAlted approved these changes Sep 23, 2024

View reviewed changes

FrancescAlted requested a review from martaiborra September 23, 2024 08:18

martaiborra reviewed Sep 23, 2024

View reviewed changes

Proxy examples: second corrections.

f016ee8

FrancescAlted reviewed Sep 24, 2024

View reviewed changes

omaech and others added 15 commits September 24, 2024 13:01

Examples of C2Array and Proxy.

499d313

Unify cparams default values && add dataclass cparams

5f2d9e8

Move params defaults to storage.py && fix tests for new defaults

8a6fc22

Add DParams dataclass && avoid rewriting the defaults

68f6d49

Add Storage dataclass

f0cf2b6

Make SChunk.cparams and dparams to only accept dataclasses

8689a66

Further testing and fixing. WIP

afbc1f2

Cannot test tuner

57a5ae6

Move cparams and dparams out of storage

cb7a849

Further test dataclasses with ndarrays

9ecb2e0

Always use blosc2.nthreads as default value

5a8c269

Further testing

4537161

Do some fixes

5fce631

Proxy examples

3a43d2e

Proxy examples: second corrections.

eff3865

omaech closed this Sep 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Working with ProxySource: examples. #259

Working with ProxySource: examples. #259

omaech commented Sep 18, 2024

FrancescAlted left a comment

FrancescAlted Sep 18, 2024

FrancescAlted Sep 18, 2024

FrancescAlted Sep 18, 2024

FrancescAlted Sep 18, 2024

FrancescAlted Sep 18, 2024

FrancescAlted Sep 18, 2024

FrancescAlted Sep 18, 2024

FrancescAlted Sep 18, 2024

FrancescAlted Sep 18, 2024

FrancescAlted Sep 18, 2024

FrancescAlted left a comment

martaiborra left a comment

martaiborra Sep 23, 2024

martaiborra Sep 23, 2024

martaiborra Sep 23, 2024

martaiborra Sep 23, 2024

martaiborra Sep 23, 2024

martaiborra Sep 23, 2024

martaiborra Sep 23, 2024

martaiborra Sep 23, 2024

martaiborra Sep 23, 2024

martaiborra Sep 23, 2024

FrancescAlted left a comment

FrancescAlted Sep 24, 2024

FrancescAlted Sep 24, 2024

omaech commented Sep 25, 2024

Working with ProxySource: examples. #259

Working with ProxySource: examples. #259

Conversation

omaech commented Sep 18, 2024

FrancescAlted left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FrancescAlted left a comment

Choose a reason for hiding this comment

martaiborra left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FrancescAlted left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

omaech commented Sep 25, 2024