Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updated SZ support #213

Merged
merged 6 commits into from
Nov 25, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions doc/contribute.rst
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,34 @@ compression_opts: (**block_size**,)
Default 0 for a block size of 1GB.
It MUST be < 1.9 GB.

sz
..

compression_opts: (
**error_bound_mode** (int32),
**abs_error high** (big endian float64),
**abs_error low**,
**rel_error high** (big endian float64),
**rel_error low**,
**pw_rel_error high** (big endian float64),
**pw_rel_error low**,
**psnr high** (big endian float64),
**psnr low**,
)

The `set_local` function prepends:

- For **dim size** from 2 to 5:

(**dim size**, **data type**, **r1**, **r2**, **r3** (if **dim size**>=3), **r4** (if **dim size**>=4), **r5** (if **dim size**==5))

**rX** are set up to **dim size** (e.g., For **dim size** == 2 only **r1** and **r2** are used)

- For **dim size** == 1: **r1** is stored on 64 bits:

(**dim size**, **data type**, **r1 most-significant bytes**, **r1 least-significant bytes**)


zfp
...

Expand Down
7 changes: 7 additions & 0 deletions doc/usage.rst
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,13 @@ LZ4
:members:
:undoc-members:

SZ
==

.. autoclass:: SZ
:members:
:undoc-members:

Zfp
===

Expand Down
44 changes: 24 additions & 20 deletions src/hdf5plugin/_filters.py
Original file line number Diff line number Diff line change
Expand Up @@ -416,7 +416,7 @@ def __init__(self,
class SZ(_FilterRefClass):
"""``h5py.Group.create_dataset``'s compression arguments for using SZ filter.

For more details about the compressor `SZ <https://https://szcompressor.org/>`_.
For more details about the compressor `SZ <https://szcompressor.org/>`_.
It can be passed as keyword arguments:

.. code-block:: python
Expand All @@ -431,7 +431,7 @@ class SZ(_FilterRefClass):
This filter provides different modes:

- **Absolute** mode: To use, set the ``absolute`` argument.
It ensures that the resulting values will be within the absolute tolerance provided with the argument.
It ensures that the resulting values will be within the provided absolute tolerance.

.. code-block:: python

Expand All @@ -441,8 +441,8 @@ class SZ(_FilterRefClass):
**hdf5plugin.Zfp(absolute=0.1))

- **Relative** mode: To use, set the ``relative`` argument.
It ensures that the resulting values will be within the relative tolerance provided with the argument.
The tolerance will be computed by multiplying the the argument provided by the range of the data values.
It ensures that the resulting values will be within the provided relative tolerance.
The tolerance will be computed by multiplying the provided argument by the range of the data values.

.. code-block:: python

Expand All @@ -452,8 +452,7 @@ class SZ(_FilterRefClass):
**hdf5plugin.SZ(relative=0.01))

- **Point-wise relative** mode: To use, set the ``pointwise_relative`` argument.
It ensures that each grid point of the resulting values will be within the relative tolerance provided with the
argument.
It ensures that each grid point of the resulting values will be within the provided relative tolerance.

.. code-block:: python

Expand All @@ -466,34 +465,39 @@ class SZ(_FilterRefClass):
filter_name = "sz"
filter_id = SZ_ID

def __init__(self, absolute=None, relative=None, pointwise_relative=1e-05):
def __init__(self, absolute=None, relative=None, pointwise_relative=None):
if (absolute, relative, pointwise_relative).count(None) < 2:
raise TypeError("hdf5plugin.SZ() takes at most one not None argument")

# Get SZ encoding options
if absolute is not None:
sz_mode = 0
parameter = absolute
elif relative is not None:
sz_mode = 1
parameter = relative
elif pointwise_relative is not None:
sz_mode = 10
parameter = pointwise_relative
else:
raise TypeError("One of the options need to be provided: absolute, relative or pointwise_relative.")
sz_mode = 10
if pointwise_relative is None:
pointwise_relative = 1e-5
vasole marked this conversation as resolved.
Show resolved Hide resolved

packed_error = self.pack_error(parameter)
compression_opts = (sz_mode, *packed_error, *packed_error, *packed_error, *packed_error)
compression_opts = (
sz_mode,
*self.__pack_float64(absolute or 0.),
*self.__pack_float64(relative or 0.),
*self.__pack_float64(pointwise_relative or 0.),
*self.__pack_float64(0.), # psnr
)
Comment on lines -483 to +488
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Previous implementation was working because not all options are used depending on the mode, but this better matches the meaning of the compression_opts


logger.info(f"SZ mode {sz_mode} used.")
logger.info(f"filter options {compression_opts}")

self.filter_options = compression_opts

@staticmethod
def pack_error(error: float) -> tuple:
packed = struct.pack('<d', error) # Pack as IEEE 754 double
high = struct.unpack('<I', packed[0:4])[0] # Unpack high bits as unsigned int
low = struct.unpack('<I', packed[4:8])[0]
return low, high
def __pack_float64(error: float) -> tuple:
packed = struct.pack('>d', error) # Pack as big-endian IEEE 754 double
high = struct.unpack('>I', packed[0:4])[0] # Unpack most-significant bits as unsigned int
low = struct.unpack('>I', packed[4:8])[0] # Unpack least-significant bits as unsigned int
return high, low
Comment on lines -492 to +500
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here previous implementation was working but the C code state working with big endian, so let's match it too.

BTW, that would be good to check it on a big-endian machine.



class Zstd(_FilterRefClass):
Expand Down
3 changes: 1 addition & 2 deletions src/hdf5plugin/test.py
Original file line number Diff line number Diff line change
Expand Up @@ -337,8 +337,7 @@ def test_register_single_filter_by_id(self):
@unittest.skipUnless(BUILD_CONFIG.embedded_filters, "No embedded filters")
def test_register_all_filters(self):
"""Re-register embedded filters all at once"""
status = hdf5plugin.register()
self.assertTrue(status)
hdf5plugin.register()
for filter_name in BUILD_CONFIG.embedded_filters:
Comment on lines -340 to 341
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The assert fails if not all filters are embedded (by using HDF5PLUGIN_STRIP)

with self.subTest(name=filter_name):
self._simple_test(filter_name)
Expand Down