Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor batch results storage and publishing #1121

Merged
merged 20 commits into from
Jun 27, 2024

Conversation

tsmith023
Copy link
Contributor

@tsmith023 tsmith023 commented Jun 24, 2024

  • Limits total number of stored results to 100,000. Should this be configurable?

Closes #1094

- Limits total number of stored results to 100,000. Should this be configurable?
- Changes `.uuids` and `.errors` to be properties that are resolved at read time
- This was due to strange counting bug because of bad internal logic mapping grpc results to the data wrapper object
- Is this a potentially breaking change?
@tsmith023 tsmith023 requested a review from dirkkul June 24, 2024 10:43
@codecov-commenter
Copy link

codecov-commenter commented Jun 24, 2024

Codecov Report

Attention: Patch coverage is 96.49123% with 2 lines in your changes missing coverage. Please review.

Project coverage is 94.58%. Comparing base (42fb96c) to head (3531985).
Report is 3 commits behind head on main.

Files Patch % Lines
weaviate/collections/classes/batch.py 92.59% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1121      +/-   ##
==========================================
- Coverage   94.68%   94.58%   -0.11%     
==========================================
  Files         190      191       +1     
  Lines       18932    18984      +52     
==========================================
+ Hits        17926    17956      +30     
- Misses       1006     1028      +22     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@tsmith023 tsmith023 requested a review from dirkkul June 25, 2024 16:16
weaviate/collections/classes/batch.py Outdated Show resolved Hide resolved
weaviate/collections/classes/batch.py Outdated Show resolved Hide resolved
weaviate/collections/classes/batch.py Outdated Show resolved Hide resolved
@tsmith023 tsmith023 requested a review from dirkkul June 26, 2024 09:04
@dirkkul dirkkul merged commit adb3e36 into main Jun 27, 2024
43 checks passed
@dirkkul dirkkul deleted the fix/batch-storing-of-results branch June 27, 2024 11:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Limit amount of succesfully added UUID that are kept in batching
3 participants