Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor batch results storage and publishing #1121

Merged
merged 20 commits into from
Jun 27, 2024
Merged

Refactor batch results storage and publishing #1121

merged 20 commits into from
Jun 27, 2024

Commits on Jun 24, 2024

  1. Refactor batch results storage: 

    - Limits total number of stored results to 100,000. Should this be configurable?
- Changes `.uuids` and `.errors` to be properties that are resolved at read time
- This was due to strange counting bug because of bad internal logic mapping grpc results to the data wrapper object
- Is this a potentially breaking change?
    @tsmith023
    tsmith023 committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    8c56a61 View commit details
    Browse the repository at this point in the history

  2. Revert API logic using different internal logic to handle uuids

    @tsmith023
    tsmith023 committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    a33b95e View commit details
    Browse the repository at this point in the history

  3. Finalise implementation for storing only 100k uuids

    @tsmith023
    tsmith023 committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    3ccbde0 View commit details
    Browse the repository at this point in the history

  4. Fix flake8 errors

    @tsmith023
    tsmith023 committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    68f7ade View commit details
    Browse the repository at this point in the history

Commits on Jun 25, 2024

  1. Add deprecation warning to .all_responses usage

    @tsmith023
    tsmith023 committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    3531985 View commit details
    Browse the repository at this point in the history

Commits on Jun 26, 2024

  1. Changes in response to review feedback

    @tsmith023
    tsmith023 committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    e97bb5f View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' of https://github.com/weaviate/weaviate-python-cl… 

    …ient into fix/batch-storing-of-results
    @tsmith023
    tsmith023 committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    1f6a498 View commit details
    Browse the repository at this point in the history

  3. Perform explicit copy of keys to avoid `RuntimeError: dictionary chan… 

    …ged size during iteration`
    @tsmith023
    tsmith023 committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    6b4b057 View commit details
    Browse the repository at this point in the history

  4. Add message to potentially flakey assertion

    @tsmith023
    tsmith023 committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    0857478 View commit details
    Browse the repository at this point in the history

  5. Fix all init of BatchObjectReturn to take OrderedDict args

    @tsmith023
    tsmith023 committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    bddec19 View commit details
    Browse the repository at this point in the history

  6. Revert back to just using dict over ordereddict

    @tsmith023
    tsmith023 committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    3a99358 View commit details
    Browse the repository at this point in the history

  7. Fix typo in list comprehension

    @tsmith023
    tsmith023 committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    78c1a31 View commit details
    Browse the repository at this point in the history

  8. Refactor results storage to be simpler and more performant

    @tsmith023
    tsmith023 committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    d717b3c View commit details
    Browse the repository at this point in the history

  9. Pass the original index from the batch top-down to ensure consistency

    @tsmith023
    tsmith023 committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    de899c9 View commit details
    Browse the repository at this point in the history

  10. Fix _all_responses updating

    @tsmith023
    tsmith023 committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    a076e18 View commit details
    Browse the repository at this point in the history

  11. Merge branch 'main' of https://github.com/weaviate/weaviate-python-cl… 

    …ient into fix/batch-storing-of-results
    @tsmith023
    tsmith023 committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    49cd443 View commit details
    Browse the repository at this point in the history

  12. Fix batch objects result add unit test

    @tsmith023
    tsmith023 committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    4f9050b View commit details
    Browse the repository at this point in the history

  13. Replace loop dict delete with reinstantiation due to flakes

    @tsmith023
    tsmith023 committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    5de78aa View commit details
    Browse the repository at this point in the history

  14. Sort keys correctly in test

    @tsmith023
    tsmith023 committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    b83e928 View commit details
    Browse the repository at this point in the history

  15. Add back looping deletion of old keys in uuids

    @tsmith023
    tsmith023 committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    7b48585 View commit details
    Browse the repository at this point in the history