-
Notifications
You must be signed in to change notification settings - Fork 75
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allow zero-copy decompression by allowing bytes-like input #230
Conversation
@lgarrison thanks for submitting this! After reviewing your PR, I thought to myself, "oh this is great, but it will need some tests". So, I just had a look at the test suite, and it seems like this behavior may have already been supported: https://github.com/Blosc/python-blosc/blob/master/blosc/test.py#L81-L92 Is there something, perhaps about recent Python versions that disallowed this behavior at the Python level? |
Nope, you're absolutely right, it already works with |
@lgarrison interesting. It also seems like there are no tests for |
I added |
It looks good to me! |
Yes, the appveyor seems to have timed out. |
opened/closed to force CI retrigger. |
@FrancescAlted do you think it would be OK to skip trying to fix appveyor and use the passing of the other platforms as an indicator? |
Definitely. We have decided to start using Azure for CI for a while now, and the plan is to deprecate appveyor. |
@FrancescAlted yay! 😁 |
@lgarrison thank you again for your contribution! |
Thanks @esc and @FrancescAlted ! |
decompress()
anddecompress_ptr()
currently require the compressed input to be a Pythonbytes
object.bytes
objects are nice because they're immutable, but this comes at the cost of copying the data when constructing thebytes
. So, in Blosc, if you have compressed data in abytesarray
, or Numpy array, there's no way to decompress the data without incurring this cost. Since Blosc decompression can be nearly as fast as amemcpy()
, this performance hit is noticeable.This PR changes
decompress()
anddecompress_ptr()
to accept bytes-like input. Actually, at the CPython level,decompress()
already allowed bytes-like input, but this was disallowed by the Python wrapper, so I just made that check a little more permissive.I think this PR is a strict superset of the old behavior, and it passes
make test
on my local machine, but feedback is welcome!