Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: make VectorStorage and DistCalculator static to generate better code #2355

Merged
merged 6 commits into from
May 22, 2024

Conversation

BubbleCal
Copy link
Contributor

@BubbleCal BubbleCal commented May 20, 2024

4.3647% faster

@BubbleCal
Copy link
Contributor Author

blocked by #2353

@codecov-commenter
Copy link

codecov-commenter commented May 20, 2024

Codecov Report

Attention: Patch coverage is 73.17073% with 11 lines in your changes are missing coverage. Please review.

Project coverage is 80.69%. Comparing base (21d6a25) to head (fea075a).
Report is 4 commits behind head on main.

Files Patch % Lines
rust/lance-index/src/vector/hnsw/builder.rs 81.25% 0 Missing and 3 partials ⚠️
rust/lance-index/src/vector/sq/storage.rs 40.00% 3 Missing ⚠️
rust/lance-index/src/vector/pq/storage.rs 75.00% 1 Missing ⚠️
rust/lance/src/index/vector/fixture_test.rs 0.00% 1 Missing ⚠️
rust/lance/src/index/vector/ivf.rs 0.00% 1 Missing ⚠️
rust/lance/src/index/vector/pq.rs 0.00% 1 Missing ⚠️
rust/lance/src/session/index_extension.rs 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2355      +/-   ##
==========================================
- Coverage   80.72%   80.69%   -0.04%     
==========================================
  Files         192      191       -1     
  Lines       56230    56355     +125     
  Branches    56230    56355     +125     
==========================================
+ Hits        45394    45478      +84     
- Misses       8213     8248      +35     
- Partials     2623     2629       +6     
Flag Coverage Δ
unittests 80.69% <73.17%> (-0.04%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: BubbleCal <bubble-cal@outlook.com>
Signed-off-by: BubbleCal <bubble-cal@outlook.com>
Signed-off-by: BubbleCal <bubble-cal@outlook.com>
@@ -216,7 +211,7 @@ impl HNSW {
ef: usize,
bitset: Option<RoaringBitmap>,
visited_generator: &mut VisitedGenerator,
storage: &dyn VectorStorage,
storage: &impl VectorStore,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By switching to impl, that makes this generic and increases generated code size. Do we think the speedup is worth it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this leads up to making distance calculator generic,
let me finish it and bench it first.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We only have a handful of vector storage types anyway I believe. 4.3% definitely seems worth it to me

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's measure the binary size before and after? but yeah, 4.3% for single point will turn into more when we enable batch query. If size increase is still reasonable, say 1-2 MB, I think we should go forward with this.

Signed-off-by: BubbleCal <bubble-cal@outlook.com>
Signed-off-by: BubbleCal <bubble-cal@outlook.com>
Signed-off-by: BubbleCal <bubble-cal@outlook.com>
@BubbleCal BubbleCal changed the title perf: make VectorStorage static to generate better code perf: make VectorStorage and DistCalculator static to generate better code May 21, 2024
@BubbleCal BubbleCal merged commit 56d5505 into lancedb:main May 22, 2024
22 checks passed
renovate bot referenced this pull request in spiraldb/vortex Jun 12, 2024
[![Mend
Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [lance](https://togithub.com/lancedb/lance) | dependencies | minor |
`0.10.16` -> `0.12.0` |

---

### Release Notes

<details>
<summary>lancedb/lance (lance)</summary>

### [`v0.12.1`](https://togithub.com/lancedb/lance/releases/tag/v0.12.1)

[Compare
Source](https://togithub.com/lancedb/lance/compare/v0.12.0...v0.12.1)

<!-- Release notes generated using configuration in .github/release.yml
at v0.12.1 -->

#### What's Changed

##### Bug Fixes 🐛

- fix: incorrect chunking was making lance datasets use too much RAM by
[@&#8203;westonpace](https://togithub.com/westonpace) in
[https://github.com/lancedb/lance/pull/2438](https://togithub.com/lancedb/lance/pull/2438)

**Full Changelog**:
lancedb/lance@v0.12.0...v0.12.1

### [`v0.12.0`](https://togithub.com/lancedb/lance/releases/tag/v0.12.0)

[Compare
Source](https://togithub.com/lancedb/lance/compare/v0.11.1...v0.12.0)

<!-- Release notes generated using configuration in .github/release.yml
at v0.12.0 -->

#### What's Changed

##### Breaking Changes 🛠

- feat: change dataset uri to return full qualified url instead of
object store path by [@&#8203;eddyxu](https://togithub.com/eddyxu) in
[https://github.com/lancedb/lance/pull/2416](https://togithub.com/lancedb/lance/pull/2416)

##### New Features 🎉

- feat: new shuffler by
[@&#8203;BubbleCal](https://togithub.com/BubbleCal) in
[https://github.com/lancedb/lance/pull/2404](https://togithub.com/lancedb/lance/pull/2404)
- feat: new index builder by
[@&#8203;BubbleCal](https://togithub.com/BubbleCal) in
[https://github.com/lancedb/lance/pull/2401](https://togithub.com/lancedb/lance/pull/2401)
- feat: stable row id manifest changes by
[@&#8203;wjones127](https://togithub.com/wjones127) in
[https://github.com/lancedb/lance/pull/2363](https://togithub.com/lancedb/lance/pull/2363)
- feat: once a table has been created with v1 or v2 format then it
should always use that format by
[@&#8203;westonpace](https://togithub.com/westonpace) in
[https://github.com/lancedb/lance/pull/2435](https://togithub.com/lancedb/lance/pull/2435)

##### Bug Fixes 🐛

- fix: fix file writer which was not writing page buffers in the correct
order by [@&#8203;westonpace](https://togithub.com/westonpace) in
[https://github.com/lancedb/lance/pull/2413](https://togithub.com/lancedb/lance/pull/2413)

##### Other Changes

- refactor: refactor logical decoders into "field decoders" by
[@&#8203;westonpace](https://togithub.com/westonpace) in
[https://github.com/lancedb/lance/pull/2407](https://togithub.com/lancedb/lance/pull/2407)
- refactor: rename use_experimental_writer to use_legacy_format by
[@&#8203;westonpace](https://togithub.com/westonpace) in
[https://github.com/lancedb/lance/pull/2433](https://togithub.com/lancedb/lance/pull/2433)
- refactor: minor refactor to allow I/O scheduler to be cloned in page
schedulers by [@&#8203;westonpace](https://togithub.com/westonpace) in
[https://github.com/lancedb/lance/pull/2432](https://togithub.com/lancedb/lance/pull/2432)

**Full Changelog**:
lancedb/lance@v0.11.1...v0.12.0

### [`v0.11.1`](https://togithub.com/lancedb/lance/releases/tag/v0.11.1)

[Compare
Source](https://togithub.com/lancedb/lance/compare/v0.11.0...v0.11.1)

<!-- Release notes generated using configuration in .github/release.yml
at v0.11.1 -->

#### What's Changed

##### New Features 🎉

- feat(java): support jdk8 by
[@&#8203;LuQQiu](https://togithub.com/LuQQiu) in
[https://github.com/lancedb/lance/pull/2362](https://togithub.com/lancedb/lance/pull/2362)
- feat: support kmode with hamming distance by
[@&#8203;eddyxu](https://togithub.com/eddyxu) in
[https://github.com/lancedb/lance/pull/2366](https://togithub.com/lancedb/lance/pull/2366)
- feat: row id index structures (experimental) by
[@&#8203;wjones127](https://togithub.com/wjones127) in
[https://github.com/lancedb/lance/pull/2303](https://togithub.com/lancedb/lance/pull/2303)
- feat: update merge_insert to add statistics for inserted, updated,
deleted rows by [@&#8203;raunaks13](https://togithub.com/raunaks13) in
[https://github.com/lancedb/lance/pull/2357](https://togithub.com/lancedb/lance/pull/2357)
- feat: define Flat index as a scan over VectorStorage by
[@&#8203;chebbyChefNEQ](https://togithub.com/chebbyChefNEQ) in
[https://github.com/lancedb/lance/pull/2380](https://togithub.com/lancedb/lance/pull/2380)
- feat: add some schema utility methods to the v2 reader/writer by
[@&#8203;westonpace](https://togithub.com/westonpace) in
[https://github.com/lancedb/lance/pull/2389](https://togithub.com/lancedb/lance/pull/2389)
- feat: general compression for value page buffer by
[@&#8203;niyue](https://togithub.com/niyue) in
[https://github.com/lancedb/lance/pull/2368](https://togithub.com/lancedb/lance/pull/2368)
- feat: make the index cache size (in bytes) available by
[@&#8203;westonpace](https://togithub.com/westonpace) in
[https://github.com/lancedb/lance/pull/2381](https://togithub.com/lancedb/lance/pull/2381)
- feat: add special uri scheme to use CloudFileReader for local fs by
[@&#8203;chebbyChefNEQ](https://togithub.com/chebbyChefNEQ) in
[https://github.com/lancedb/lance/pull/2402](https://togithub.com/lancedb/lance/pull/2402)
- feat: add encoder utilities for pushdown by
[@&#8203;westonpace](https://togithub.com/westonpace) in
[https://github.com/lancedb/lance/pull/2388](https://togithub.com/lancedb/lance/pull/2388)

##### Bug Fixes 🐛

- fix: concat batches before writing to avoid small IO slow down by
[@&#8203;chebbyChefNEQ](https://togithub.com/chebbyChefNEQ) in
[https://github.com/lancedb/lance/pull/2384](https://togithub.com/lancedb/lance/pull/2384)
- fix: low recall if the num partitions is more than num rows by
[@&#8203;BubbleCal](https://togithub.com/BubbleCal) in
[https://github.com/lancedb/lance/pull/2386](https://togithub.com/lancedb/lance/pull/2386)
- fix: f32 reduce_min for x86 by
[@&#8203;heiher](https://togithub.com/heiher) in
[https://github.com/lancedb/lance/pull/2385](https://togithub.com/lancedb/lance/pull/2385)
- fix: fix incorrect validation logic in updater by
[@&#8203;westonpace](https://togithub.com/westonpace) in
[https://github.com/lancedb/lance/pull/2408](https://togithub.com/lancedb/lance/pull/2408)

##### Performance Improvements 🚀

- perf: make VectorStorage and DistCalculator static to generate better
code by [@&#8203;BubbleCal](https://togithub.com/BubbleCal) in
[https://github.com/lancedb/lance/pull/2355](https://togithub.com/lancedb/lance/pull/2355)
- perf: optimize IO path for reading manifest by
[@&#8203;wjones127](https://togithub.com/wjones127) in
[https://github.com/lancedb/lance/pull/2396](https://togithub.com/lancedb/lance/pull/2396)

##### Other Changes

- refactor: make proto conversion fallible and not copy by
[@&#8203;wjones127](https://togithub.com/wjones127) in
[https://github.com/lancedb/lance/pull/2371](https://togithub.com/lancedb/lance/pull/2371)
- refactor: separate take and schema evolution impls to own files by
[@&#8203;wjones127](https://togithub.com/wjones127) in
[https://github.com/lancedb/lance/pull/2372](https://togithub.com/lancedb/lance/pull/2372)
- Revert "fix: concat batches before writing to avoid small IO slow down
([#&#8203;2384](https://togithub.com/lancedb/lance/issues/2384))" by
[@&#8203;chebbyChefNEQ](https://togithub.com/chebbyChefNEQ) in
[https://github.com/lancedb/lance/pull/2387](https://togithub.com/lancedb/lance/pull/2387)
- refactor: shuffle around v2 metadata sections to allow read-on-demand
statistics by [@&#8203;westonpace](https://togithub.com/westonpace) in
[https://github.com/lancedb/lance/pull/2400](https://togithub.com/lancedb/lance/pull/2400)

#### New Contributors

- [@&#8203;niyue](https://togithub.com/niyue) made their first
contribution in
[https://github.com/lancedb/lance/pull/2368](https://togithub.com/lancedb/lance/pull/2368)
- [@&#8203;heiher](https://togithub.com/heiher) made their first
contribution in
[https://github.com/lancedb/lance/pull/2385](https://togithub.com/lancedb/lance/pull/2385)

**Full Changelog**:
lancedb/lance@v0.11.0...v0.11.1

### [`v0.11.0`](https://togithub.com/lancedb/lance/releases/tag/v0.11.0)

[Compare
Source](https://togithub.com/lancedb/lance/compare/v0.10.18...v0.11.0)

<!-- Release notes generated using configuration in .github/release.yml
at v0.11.0 -->

#### What's Changed

##### Breaking Changes 🛠

- feat(rust)!: use BoxedError in Error::IO by
[@&#8203;broccoliSpicy](https://togithub.com/broccoliSpicy) in
[https://github.com/lancedb/lance/pull/2329](https://togithub.com/lancedb/lance/pull/2329)

##### New Features 🎉

- feat: add v2 support to fragment merge / update paths by
[@&#8203;westonpace](https://togithub.com/westonpace) in
[https://github.com/lancedb/lance/pull/2311](https://togithub.com/lancedb/lance/pull/2311)
- feat: add priority to I/O scheduler by
[@&#8203;westonpace](https://togithub.com/westonpace) in
[https://github.com/lancedb/lance/pull/2315](https://togithub.com/lancedb/lance/pull/2315)
- feat: add take_rows operation to the v2 file reader's python bindings
by [@&#8203;westonpace](https://togithub.com/westonpace) in
[https://github.com/lancedb/lance/pull/2331](https://togithub.com/lancedb/lance/pull/2331)
- feat: added example for reading and writing dataset in rust by
[@&#8203;raunaks13](https://togithub.com/raunaks13) in
[https://github.com/lancedb/lance/pull/2349](https://togithub.com/lancedb/lance/pull/2349)
- feat: new HNSW implementation by
[@&#8203;BubbleCal](https://togithub.com/BubbleCal) in
[https://github.com/lancedb/lance/pull/2353](https://togithub.com/lancedb/lance/pull/2353)
- feat: add fragment take / fixed-size-binary support to v2 format by
[@&#8203;westonpace](https://togithub.com/westonpace) in
[https://github.com/lancedb/lance/pull/2354](https://togithub.com/lancedb/lance/pull/2354)

##### Bug Fixes 🐛

- fix: recognize a simple expression like 'is_foo' as a scalar index
query by [@&#8203;westonpace](https://togithub.com/westonpace) in
[https://github.com/lancedb/lance/pull/2356](https://togithub.com/lancedb/lance/pull/2356)
- fix: rework list encoder to handle list-struct by
[@&#8203;westonpace](https://togithub.com/westonpace) in
[https://github.com/lancedb/lance/pull/2344](https://togithub.com/lancedb/lance/pull/2344)
- fix: minor bug fixes for v2 by
[@&#8203;westonpace](https://togithub.com/westonpace) in
[https://github.com/lancedb/lance/pull/2361](https://togithub.com/lancedb/lance/pull/2361)

##### Documentation 📚

- docs: clearify comments in table.proto -> message DataFragment ->
physical_rows by
[@&#8203;broccoliSpicy](https://togithub.com/broccoliSpicy) in
[https://github.com/lancedb/lance/pull/2346](https://togithub.com/lancedb/lance/pull/2346)

##### Performance Improvements 🚀

- perf: use the file metadata cache in scalar indices by
[@&#8203;westonpace](https://togithub.com/westonpace) in
[https://github.com/lancedb/lance/pull/2330](https://togithub.com/lancedb/lance/pull/2330)

##### Other Changes

- chore: remove `m_max` and `use_heuristic` params from HNSW builder by
[@&#8203;BubbleCal](https://togithub.com/BubbleCal) in
[https://github.com/lancedb/lance/pull/2336](https://togithub.com/lancedb/lance/pull/2336)
- fix(java): fix JNI jar loader issue by
[@&#8203;LuQQiu](https://togithub.com/LuQQiu) in
[https://github.com/lancedb/lance/pull/2340](https://togithub.com/lancedb/lance/pull/2340)
- ci: fix labeler permissions by
[@&#8203;wjones127](https://togithub.com/wjones127) in
[https://github.com/lancedb/lance/pull/2348](https://togithub.com/lancedb/lance/pull/2348)
- fix: rework decoding to fix bugs in nested struct decoding by
[@&#8203;westonpace](https://togithub.com/westonpace) in
[https://github.com/lancedb/lance/pull/2337](https://togithub.com/lancedb/lance/pull/2337)

#### New Contributors

- [@&#8203;broccoliSpicy](https://togithub.com/broccoliSpicy) made their
first contribution in
[https://github.com/lancedb/lance/pull/2346](https://togithub.com/lancedb/lance/pull/2346)
- [@&#8203;raunaks13](https://togithub.com/raunaks13) made their first
contribution in
[https://github.com/lancedb/lance/pull/2349](https://togithub.com/lancedb/lance/pull/2349)

**Full Changelog**:
lancedb/lance@v0.10.18...v0.11.0

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).

🚦 **Automerge**: Enabled.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View
repository job log
[here](https://developer.mend.io/github/spiraldb/vortex).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4zOTMuMCIsInVwZGF0ZWRJblZlciI6IjM3LjM5My4wIiwidGFyZ2V0QnJhbmNoIjoiZGV2ZWxvcCIsImxhYmVscyI6W119-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
AdamGS referenced this pull request in AdamGS/vortex Jun 14, 2024
[![Mend
Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [lance](https://togithub.com/lancedb/lance) | dependencies | minor |
`0.10.16` -> `0.12.0` |

---

### Release Notes

<details>
<summary>lancedb/lance (lance)</summary>

### [`v0.12.1`](https://togithub.com/lancedb/lance/releases/tag/v0.12.1)

[Compare
Source](https://togithub.com/lancedb/lance/compare/v0.12.0...v0.12.1)

<!-- Release notes generated using configuration in .github/release.yml
at v0.12.1 -->

#### What's Changed

##### Bug Fixes 🐛

- fix: incorrect chunking was making lance datasets use too much RAM by
[@&#8203;westonpace](https://togithub.com/westonpace) in
[https://github.com/lancedb/lance/pull/2438](https://togithub.com/lancedb/lance/pull/2438)

**Full Changelog**:
lancedb/lance@v0.12.0...v0.12.1

### [`v0.12.0`](https://togithub.com/lancedb/lance/releases/tag/v0.12.0)

[Compare
Source](https://togithub.com/lancedb/lance/compare/v0.11.1...v0.12.0)

<!-- Release notes generated using configuration in .github/release.yml
at v0.12.0 -->

#### What's Changed

##### Breaking Changes 🛠

- feat: change dataset uri to return full qualified url instead of
object store path by [@&#8203;eddyxu](https://togithub.com/eddyxu) in
[https://github.com/lancedb/lance/pull/2416](https://togithub.com/lancedb/lance/pull/2416)

##### New Features 🎉

- feat: new shuffler by
[@&#8203;BubbleCal](https://togithub.com/BubbleCal) in
[https://github.com/lancedb/lance/pull/2404](https://togithub.com/lancedb/lance/pull/2404)
- feat: new index builder by
[@&#8203;BubbleCal](https://togithub.com/BubbleCal) in
[https://github.com/lancedb/lance/pull/2401](https://togithub.com/lancedb/lance/pull/2401)
- feat: stable row id manifest changes by
[@&#8203;wjones127](https://togithub.com/wjones127) in
[https://github.com/lancedb/lance/pull/2363](https://togithub.com/lancedb/lance/pull/2363)
- feat: once a table has been created with v1 or v2 format then it
should always use that format by
[@&#8203;westonpace](https://togithub.com/westonpace) in
[https://github.com/lancedb/lance/pull/2435](https://togithub.com/lancedb/lance/pull/2435)

##### Bug Fixes 🐛

- fix: fix file writer which was not writing page buffers in the correct
order by [@&#8203;westonpace](https://togithub.com/westonpace) in
[https://github.com/lancedb/lance/pull/2413](https://togithub.com/lancedb/lance/pull/2413)

##### Other Changes

- refactor: refactor logical decoders into "field decoders" by
[@&#8203;westonpace](https://togithub.com/westonpace) in
[https://github.com/lancedb/lance/pull/2407](https://togithub.com/lancedb/lance/pull/2407)
- refactor: rename use_experimental_writer to use_legacy_format by
[@&#8203;westonpace](https://togithub.com/westonpace) in
[https://github.com/lancedb/lance/pull/2433](https://togithub.com/lancedb/lance/pull/2433)
- refactor: minor refactor to allow I/O scheduler to be cloned in page
schedulers by [@&#8203;westonpace](https://togithub.com/westonpace) in
[https://github.com/lancedb/lance/pull/2432](https://togithub.com/lancedb/lance/pull/2432)

**Full Changelog**:
lancedb/lance@v0.11.1...v0.12.0

### [`v0.11.1`](https://togithub.com/lancedb/lance/releases/tag/v0.11.1)

[Compare
Source](https://togithub.com/lancedb/lance/compare/v0.11.0...v0.11.1)

<!-- Release notes generated using configuration in .github/release.yml
at v0.11.1 -->

#### What's Changed

##### New Features 🎉

- feat(java): support jdk8 by
[@&#8203;LuQQiu](https://togithub.com/LuQQiu) in
[https://github.com/lancedb/lance/pull/2362](https://togithub.com/lancedb/lance/pull/2362)
- feat: support kmode with hamming distance by
[@&#8203;eddyxu](https://togithub.com/eddyxu) in
[https://github.com/lancedb/lance/pull/2366](https://togithub.com/lancedb/lance/pull/2366)
- feat: row id index structures (experimental) by
[@&#8203;wjones127](https://togithub.com/wjones127) in
[https://github.com/lancedb/lance/pull/2303](https://togithub.com/lancedb/lance/pull/2303)
- feat: update merge_insert to add statistics for inserted, updated,
deleted rows by [@&#8203;raunaks13](https://togithub.com/raunaks13) in
[https://github.com/lancedb/lance/pull/2357](https://togithub.com/lancedb/lance/pull/2357)
- feat: define Flat index as a scan over VectorStorage by
[@&#8203;chebbyChefNEQ](https://togithub.com/chebbyChefNEQ) in
[https://github.com/lancedb/lance/pull/2380](https://togithub.com/lancedb/lance/pull/2380)
- feat: add some schema utility methods to the v2 reader/writer by
[@&#8203;westonpace](https://togithub.com/westonpace) in
[https://github.com/lancedb/lance/pull/2389](https://togithub.com/lancedb/lance/pull/2389)
- feat: general compression for value page buffer by
[@&#8203;niyue](https://togithub.com/niyue) in
[https://github.com/lancedb/lance/pull/2368](https://togithub.com/lancedb/lance/pull/2368)
- feat: make the index cache size (in bytes) available by
[@&#8203;westonpace](https://togithub.com/westonpace) in
[https://github.com/lancedb/lance/pull/2381](https://togithub.com/lancedb/lance/pull/2381)
- feat: add special uri scheme to use CloudFileReader for local fs by
[@&#8203;chebbyChefNEQ](https://togithub.com/chebbyChefNEQ) in
[https://github.com/lancedb/lance/pull/2402](https://togithub.com/lancedb/lance/pull/2402)
- feat: add encoder utilities for pushdown by
[@&#8203;westonpace](https://togithub.com/westonpace) in
[https://github.com/lancedb/lance/pull/2388](https://togithub.com/lancedb/lance/pull/2388)

##### Bug Fixes 🐛

- fix: concat batches before writing to avoid small IO slow down by
[@&#8203;chebbyChefNEQ](https://togithub.com/chebbyChefNEQ) in
[https://github.com/lancedb/lance/pull/2384](https://togithub.com/lancedb/lance/pull/2384)
- fix: low recall if the num partitions is more than num rows by
[@&#8203;BubbleCal](https://togithub.com/BubbleCal) in
[https://github.com/lancedb/lance/pull/2386](https://togithub.com/lancedb/lance/pull/2386)
- fix: f32 reduce_min for x86 by
[@&#8203;heiher](https://togithub.com/heiher) in
[https://github.com/lancedb/lance/pull/2385](https://togithub.com/lancedb/lance/pull/2385)
- fix: fix incorrect validation logic in updater by
[@&#8203;westonpace](https://togithub.com/westonpace) in
[https://github.com/lancedb/lance/pull/2408](https://togithub.com/lancedb/lance/pull/2408)

##### Performance Improvements 🚀

- perf: make VectorStorage and DistCalculator static to generate better
code by [@&#8203;BubbleCal](https://togithub.com/BubbleCal) in
[https://github.com/lancedb/lance/pull/2355](https://togithub.com/lancedb/lance/pull/2355)
- perf: optimize IO path for reading manifest by
[@&#8203;wjones127](https://togithub.com/wjones127) in
[https://github.com/lancedb/lance/pull/2396](https://togithub.com/lancedb/lance/pull/2396)

##### Other Changes

- refactor: make proto conversion fallible and not copy by
[@&#8203;wjones127](https://togithub.com/wjones127) in
[https://github.com/lancedb/lance/pull/2371](https://togithub.com/lancedb/lance/pull/2371)
- refactor: separate take and schema evolution impls to own files by
[@&#8203;wjones127](https://togithub.com/wjones127) in
[https://github.com/lancedb/lance/pull/2372](https://togithub.com/lancedb/lance/pull/2372)
- Revert "fix: concat batches before writing to avoid small IO slow down
([#&#8203;2384](https://togithub.com/lancedb/lance/issues/2384))" by
[@&#8203;chebbyChefNEQ](https://togithub.com/chebbyChefNEQ) in
[https://github.com/lancedb/lance/pull/2387](https://togithub.com/lancedb/lance/pull/2387)
- refactor: shuffle around v2 metadata sections to allow read-on-demand
statistics by [@&#8203;westonpace](https://togithub.com/westonpace) in
[https://github.com/lancedb/lance/pull/2400](https://togithub.com/lancedb/lance/pull/2400)

#### New Contributors

- [@&#8203;niyue](https://togithub.com/niyue) made their first
contribution in
[https://github.com/lancedb/lance/pull/2368](https://togithub.com/lancedb/lance/pull/2368)
- [@&#8203;heiher](https://togithub.com/heiher) made their first
contribution in
[https://github.com/lancedb/lance/pull/2385](https://togithub.com/lancedb/lance/pull/2385)

**Full Changelog**:
lancedb/lance@v0.11.0...v0.11.1

### [`v0.11.0`](https://togithub.com/lancedb/lance/releases/tag/v0.11.0)

[Compare
Source](https://togithub.com/lancedb/lance/compare/v0.10.18...v0.11.0)

<!-- Release notes generated using configuration in .github/release.yml
at v0.11.0 -->

#### What's Changed

##### Breaking Changes 🛠

- feat(rust)!: use BoxedError in Error::IO by
[@&#8203;broccoliSpicy](https://togithub.com/broccoliSpicy) in
[https://github.com/lancedb/lance/pull/2329](https://togithub.com/lancedb/lance/pull/2329)

##### New Features 🎉

- feat: add v2 support to fragment merge / update paths by
[@&#8203;westonpace](https://togithub.com/westonpace) in
[https://github.com/lancedb/lance/pull/2311](https://togithub.com/lancedb/lance/pull/2311)
- feat: add priority to I/O scheduler by
[@&#8203;westonpace](https://togithub.com/westonpace) in
[https://github.com/lancedb/lance/pull/2315](https://togithub.com/lancedb/lance/pull/2315)
- feat: add take_rows operation to the v2 file reader's python bindings
by [@&#8203;westonpace](https://togithub.com/westonpace) in
[https://github.com/lancedb/lance/pull/2331](https://togithub.com/lancedb/lance/pull/2331)
- feat: added example for reading and writing dataset in rust by
[@&#8203;raunaks13](https://togithub.com/raunaks13) in
[https://github.com/lancedb/lance/pull/2349](https://togithub.com/lancedb/lance/pull/2349)
- feat: new HNSW implementation by
[@&#8203;BubbleCal](https://togithub.com/BubbleCal) in
[https://github.com/lancedb/lance/pull/2353](https://togithub.com/lancedb/lance/pull/2353)
- feat: add fragment take / fixed-size-binary support to v2 format by
[@&#8203;westonpace](https://togithub.com/westonpace) in
[https://github.com/lancedb/lance/pull/2354](https://togithub.com/lancedb/lance/pull/2354)

##### Bug Fixes 🐛

- fix: recognize a simple expression like 'is_foo' as a scalar index
query by [@&#8203;westonpace](https://togithub.com/westonpace) in
[https://github.com/lancedb/lance/pull/2356](https://togithub.com/lancedb/lance/pull/2356)
- fix: rework list encoder to handle list-struct by
[@&#8203;westonpace](https://togithub.com/westonpace) in
[https://github.com/lancedb/lance/pull/2344](https://togithub.com/lancedb/lance/pull/2344)
- fix: minor bug fixes for v2 by
[@&#8203;westonpace](https://togithub.com/westonpace) in
[https://github.com/lancedb/lance/pull/2361](https://togithub.com/lancedb/lance/pull/2361)

##### Documentation 📚

- docs: clearify comments in table.proto -> message DataFragment ->
physical_rows by
[@&#8203;broccoliSpicy](https://togithub.com/broccoliSpicy) in
[https://github.com/lancedb/lance/pull/2346](https://togithub.com/lancedb/lance/pull/2346)

##### Performance Improvements 🚀

- perf: use the file metadata cache in scalar indices by
[@&#8203;westonpace](https://togithub.com/westonpace) in
[https://github.com/lancedb/lance/pull/2330](https://togithub.com/lancedb/lance/pull/2330)

##### Other Changes

- chore: remove `m_max` and `use_heuristic` params from HNSW builder by
[@&#8203;BubbleCal](https://togithub.com/BubbleCal) in
[https://github.com/lancedb/lance/pull/2336](https://togithub.com/lancedb/lance/pull/2336)
- fix(java): fix JNI jar loader issue by
[@&#8203;LuQQiu](https://togithub.com/LuQQiu) in
[https://github.com/lancedb/lance/pull/2340](https://togithub.com/lancedb/lance/pull/2340)
- ci: fix labeler permissions by
[@&#8203;wjones127](https://togithub.com/wjones127) in
[https://github.com/lancedb/lance/pull/2348](https://togithub.com/lancedb/lance/pull/2348)
- fix: rework decoding to fix bugs in nested struct decoding by
[@&#8203;westonpace](https://togithub.com/westonpace) in
[https://github.com/lancedb/lance/pull/2337](https://togithub.com/lancedb/lance/pull/2337)

#### New Contributors

- [@&#8203;broccoliSpicy](https://togithub.com/broccoliSpicy) made their
first contribution in
[https://github.com/lancedb/lance/pull/2346](https://togithub.com/lancedb/lance/pull/2346)
- [@&#8203;raunaks13](https://togithub.com/raunaks13) made their first
contribution in
[https://github.com/lancedb/lance/pull/2349](https://togithub.com/lancedb/lance/pull/2349)

**Full Changelog**:
lancedb/lance@v0.10.18...v0.11.0

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).

🚦 **Automerge**: Enabled.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View
repository job log
[here](https://developer.mend.io/github/spiraldb/vortex).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4zOTMuMCIsInVwZGF0ZWRJblZlciI6IjM3LjM5My4wIiwidGFyZ2V0QnJhbmNoIjoiZGV2ZWxvcCIsImxhYmVscyI6W119-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants