Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Deserialization: zero-copy merge subframes when possible #5208

Merged
merged 21 commits into from
Nov 12, 2021
Merged

Deserialization: zero-copy merge subframes when possible #5208

merged 21 commits into from
Nov 12, 2021

Commits on Aug 12, 2021

  1. Zero-copy merge subframes with pointer arithmetic 

    Use NumPy to get the pointers of the memoryviews and do pointer arithmetic. This is way more reliable and self-contained.

TODO unit test for `merge_memoryviews` and deal with NumPy situation
    @gjoseph92
    gjoseph92 committed Aug 12, 2021
    Configuration menu
    Copy the full SHA
    0f90ee7 View commit details
    Browse the repository at this point in the history

  2. reorg and handle NumPy

    @gjoseph92
    gjoseph92 committed Aug 12, 2021
    Configuration menu
    Copy the full SHA
    c463a45 View commit details
    Browse the repository at this point in the history

  3. Use ctypes trick to eliminate NumPy dependency

    @gjoseph92
    gjoseph92 committed Aug 12, 2021
    Configuration menu
    Copy the full SHA
    a2f7243 View commit details
    Browse the repository at this point in the history

  4. tests; handle empty slices & base buffer type

    @gjoseph92
    gjoseph92 committed Aug 12, 2021
    Configuration menu
    Copy the full SHA
    cc3999d View commit details
    Browse the repository at this point in the history

  5. ValueErrors instead of asserts

    @gjoseph92
    gjoseph92 committed Aug 12, 2021
    Configuration menu
    Copy the full SHA
    78ca265 View commit details
    Browse the repository at this point in the history

Commits on Aug 13, 2021

  1. Fall back to NumPy for read-only buffers 

    It's very annoying: `from_buffer` requires the buffer to be writeable, because ctypes is inconsistently strict about const correctness (see https://bugs.python.org/issue11427#msg148586). I don't think there's any clever pointer-casting we could do as mentioned in that thread, since we need to access through the buffer protocol—we can't just cast `mv.obj` to `c_void_p`, since we need the pointer into the current location within the buffer, not just at the start of the buffer.

So for read-only buffers, we try to use NumPy, and if it's not available, we'll end up silently(!) falling back on copying.
    @gjoseph92
    gjoseph92 committed Aug 13, 2021
    Configuration menu
    Copy the full SHA
    ac3eaff View commit details
    Browse the repository at this point in the history

  2. don't need merge_subframes really

    @gjoseph92
    gjoseph92 committed Aug 13, 2021
    Configuration menu
    Copy the full SHA
    cc9cee3 View commit details
    Browse the repository at this point in the history

  3. skip test if numpy unavailable

    @gjoseph92
    gjoseph92 committed Aug 13, 2021
    Configuration menu
    Copy the full SHA
    22f246d View commit details
    Browse the repository at this point in the history

Commits on Aug 16, 2021

  1. Simplify repeated code per Guido's suggestion

    @gjoseph92
    gjoseph92 committed Aug 16, 2021
    Configuration menu
    Copy the full SHA
    439be17 View commit details
    Browse the repository at this point in the history

  2. future annotations and collections.abc

    @gjoseph92
    gjoseph92 committed Aug 16, 2021
    Configuration menu
    Copy the full SHA
    0032822 View commit details
    Browse the repository at this point in the history

  3. test error on non-memoryview

    @gjoseph92
    gjoseph92 committed Aug 16, 2021
    Configuration menu
    Copy the full SHA
    3315441 View commit details
    Browse the repository at this point in the history

Commits on Aug 17, 2021

  1. Don't disable exception chaining 

    Co-authored-by: crusaderky <crusaderky@gmail.com>
    @gjoseph92 @crusaderky
    gjoseph92 and crusaderky authored Aug 17, 2021
    Configuration menu
    Copy the full SHA
    da8ee57 View commit details
    Browse the repository at this point in the history

Commits on Oct 5, 2021

  1. Update distributed/protocol/utils.py 

    Co-authored-by: jakirkham <jakirkham@gmail.com>
    @gjoseph92 @jakirkham
    gjoseph92 and jakirkham authored Oct 5, 2021
    Configuration menu
    Copy the full SHA
    82ec2ca View commit details
    Browse the repository at this point in the history

  2. Move one_byte_carr into global

    @gjoseph92
    gjoseph92 authored Oct 5, 2021
    Configuration menu
    Copy the full SHA
    edaeb7d View commit details
    Browse the repository at this point in the history

Commits on Oct 7, 2021

  1. Merge remote-tracking branch 'origin/main' into pr/gjoseph92/5208-2

    @fjetter
    fjetter committed Oct 7, 2021
    Configuration menu
    Copy the full SHA
    ccea5c4 View commit details
    Browse the repository at this point in the history

  2. Fix formatting

    @fjetter
    fjetter committed Oct 7, 2021
    Configuration menu
    Copy the full SHA
    0d1e6d1 View commit details
    Browse the repository at this point in the history

Commits on Oct 8, 2021

  1. more ci

    @gjoseph92
    gjoseph92 committed Oct 8, 2021
    Configuration menu
    Copy the full SHA
    2e0ae4e View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'upstream/main' into serialization/zero-… 

    …copy-merge-memoryviews
    @gjoseph92
    gjoseph92 committed Oct 8, 2021
    Configuration menu
    Copy the full SHA
    269ef1f View commit details
    Browse the repository at this point in the history

Commits on Nov 1, 2021

  1. Merge remote-tracking branch 'upstream/main' into serialization/zero-… 

    …copy-merge-memoryviews
    @gjoseph92
    gjoseph92 committed Nov 1, 2021
    Configuration menu
    Copy the full SHA
    6574588 View commit details
    Browse the repository at this point in the history

  2. mypy

    @gjoseph92
    gjoseph92 committed Nov 1, 2021
    Configuration menu
    Copy the full SHA
    f141867 View commit details
    Browse the repository at this point in the history

Commits on Nov 3, 2021

  1. Merge remote-tracking branch 'upstream/main' into serialization/zero-… 

    …copy-merge-memoryviews
    @gjoseph92
    gjoseph92 committed Nov 3, 2021
    Configuration menu
    Copy the full SHA
    69e5cf2 View commit details
    Browse the repository at this point in the history