add codebook passing and pq/opq dim overwrite. #288

jinwei14 · 2023-03-29T03:03:19Z

Does this PR have a descriptive title that could go in our release notes?
Does this PR add any new dependencies?
Does this PR modify any existing APIs?
- Is the change to the API backwards compatible?
Should this result in any changes to our documentation, either updating existing docs or adding new ones?

Reference Issues/PRs

What does this implement/fix? Briefly explain your changes.

this change will pass in a pretrained codebook prefix. for pq and OPQ part it will load the pretrained roatation matrix and pivots.
changed the default max PQ dim from 256 to 384 to adopt richer embeddings.
we will need to update readme for new parameters if this PR is accepted.

Any other comments?

…osoft#225) - add code for two variants of filtered index, readme and CI tests - add utils for synthetic label generation and CI tests. * Add co-authors Co-authored-by: ravishankar <rakri@microsoft.com> Co-authored-by: Varun Sivashankar <t-varunsi@microsoft.com> --------- Co-authored-by: ravishankar <rakri@microsoft.com> Co-authored-by: David Kaczynski <dkaczynski@microsoft.com> Co-authored-by: Siddharth Gollapudi <t-gollapudis@microsoft.com> Co-authored-by: Neelam Mahapatro <nmahapatro@microsoft.com> Co-authored-by: Harsha Vardhan Simhadri <harshasi@microsoft.com> Co-authored-by: Harsha Vardhan Simhadri <harsha-simhadri@users.noreply.github.com> Co-authored-by: REDMOND\patelyash <patelyash@microsoft.com> Co-authored-by: Varun Sivashankar <t-varunsi@microsoft.com>

…crosoft#236) * Rather than sift through all the *.cpp and *.h in the root directory, we're looking for only the sources in our main repository for formatting. Git submodules are excluded * Removing the --Werror flag only until we actually format all of the code in a future commit * We're choosing to base our style on the Microsoft style guide and not make any changes * Running format action on source code. Settling on Google styling. Settled on '.clang-format' instead of '_clang-format'. Fixed instructions such that only clang-format 12 is installed (13 changes SortIncludes options from true/false to a trinary set of options, none of which include the word 'false') * Enabling error on malformatted file * Revert "Enabling error on malformatted file" This reverts commit fa33e82. * Revert "Running format action on source code. Settling on Google styling. Settled on '.clang-format' instead of '_clang-format'. Fixed instructions such that only clang-format 12 is installed (13 changes SortIncludes options from true/false to a trinary set of options, none of which include the word 'false')" This reverts commit e0281be. * Trying again; formatting rules based on Google rules, disables sorting includes as that breaks us, and enabling check on build. * Somehow this was missed in the mass format. Formatting include/distance.h. * Manually fixing the formatting because clang-format wouldn't, but WOULD flag it as invalid

Fix typo in SSD index readme

Remove warnings affecting internal build pipelines --------- Co-authored-by: Yiyong Lin <yiyolin@microsoft.com>

* Add support for multiple frozen points * Add the missing parameters to the constructor.

* Added filtered disk index readme

* Transferring Varun's chagges from external fork with squash merge * generating multiple gt's for each filter label + search with multiple filter labels (code cleanup) * supporting no-filter + one filter label + filter label file (multiple filters) while computing GT * generating multiple gt's + refactoring code for readability & cleanliness * adding more tests for filtered search * updating pr-test to test filtered cases * lowering recall requirement for disk index * transferred functions to filter_utils * adding more test for build and search without universal label * adding one_per_point distribution to generate_synthetic_labels + cleaning up artifacts after compute gt+ removing minor errors * refactoring search_disk_index to use a query filter vector --------- Co-authored-by: patelyash <patelyash@microsoft.com> Co-authored-by: Varun Sivashankar <t-varunsi@microsoft.com>

…ailed

src/disk_utils.cpp

tests/build_disk_index.cpp

PhilipBAdams · 2023-03-29T17:05:05Z

One other thing to consider, do you want to support overriding codebook for disk PQ too? We don't currently use disk PQ in production

jinwei14 · 2023-03-30T03:21:23Z

One other thing to consider, do you want to support overriding codebook for disk PQ too? We don't currently use disk PQ in production

Idk, what do you think? shall I also do it in diskpq?

include/pq.h

src/disk_utils.cpp

tests/build_disk_index.cpp

harsha-simhadri · 2023-03-31T16:55:11Z

One other thing to consider, do you want to support overriding codebook for disk PQ too? We don't currently use disk PQ in production

Idk, what do you think? shall I also do it in diskpq?

I am OK with leaving this out unless you need this too.

src/disk_utils.cpp

src/pq.cpp

src/index.cpp

tests/build_disk_index.cpp

include/pq.h

src/disk_utils.cpp

jinwei14 and others added 14 commits March 29, 2023 10:59

add codebook passing and pq/opq dim overwrite.

68f1443

Update SSD_index.md (microsoft#258)

cd8bee3

Fix typo in SSD index readme

Add filter-diskann paper link to readme (microsoft#275)

4320bad

Update README.md (microsoft#277)

6c7c2b3

update citation (microsoft#281)

936922c

Some fixes to pass internal building pipeline (microsoft#282)

066b9ed

Remove warnings affecting internal build pipelines --------- Co-authored-by: Yiyong Lin <yiyolin@microsoft.com>

Add support for multiple frozen points (microsoft#283)

09e8404

* Add support for multiple frozen points * Add the missing parameters to the constructor.

Added filtered disk index readme (microsoft#276)

162d1ea

* Added filtered disk index readme

udpate merging code

308b377

merge commit

5444f79

Using boost program options under Visual Studio MSVC 14.0 Assertion f…

6c1a175

…ailed

PhilipBAdams reviewed Mar 29, 2023

View reviewed changes

src/disk_utils.cpp Outdated Show resolved Hide resolved

PhilipBAdams reviewed Mar 29, 2023

View reviewed changes

tests/build_disk_index.cpp Outdated Show resolved Hide resolved

jinwei14 added 8 commits March 30, 2023 11:33

some commts and rewriting

0f5ec26

add back LF which might be confict with MSVC 14.0

918d11f

clang formating change

8d2a007

clang formating

9b46adc

revert back to Lf

dde79a3

unexpected failure on UT re-try

a7c8db6

adding default string to the path

6dd780c

fix reference issue

86ee064

harsha-simhadri requested a review from gopalrs March 31, 2023 04:08

gopalrs requested changes Mar 31, 2023

View reviewed changes

merge all new PR change into my branch

7b1553d

jinwei14 added 2 commits April 3, 2023 08:08

clang format

6833255

bump it up to 512 for MAX_PQ_CHUNKS

29f53d0

jinwei14 requested a review from gopalrs April 3, 2023 01:15

harsha-simhadri reviewed Apr 3, 2023

View reviewed changes

src/disk_utils.cpp Show resolved Hide resolved

harsha-simhadri reviewed Apr 3, 2023

View reviewed changes

src/pq.cpp Show resolved Hide resolved

harsha-simhadri reviewed Apr 3, 2023

View reviewed changes

src/index.cpp Outdated Show resolved Hide resolved

harsha-simhadri reviewed Apr 3, 2023

View reviewed changes

tests/build_disk_index.cpp Show resolved Hide resolved

default codebook prefix value pass in for generate_quantized_data

27d1653

jinwei14 requested a review from harsha-simhadri April 3, 2023 01:46

add check for disabling both -B and -QD pass in

a1792e4

gopalrs approved these changes Apr 3, 2023

View reviewed changes

include/pq.h Show resolved Hide resolved

src/disk_utils.cpp Show resolved Hide resolved

gopalrs merged commit 4c8041b into microsoft:main Apr 3, 2023

jinwei14 deleted the codebookPassin branch April 3, 2023 06:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add codebook passing and pq/opq dim overwrite. #288

add codebook passing and pq/opq dim overwrite. #288

jinwei14 commented Mar 29, 2023 •

edited

Loading

PhilipBAdams commented Mar 29, 2023

jinwei14 commented Mar 30, 2023

harsha-simhadri commented Mar 31, 2023

add codebook passing and pq/opq dim overwrite. #288

add codebook passing and pq/opq dim overwrite. #288

Conversation

jinwei14 commented Mar 29, 2023 • edited Loading

Reference Issues/PRs

What does this implement/fix? Briefly explain your changes.

Any other comments?

PhilipBAdams commented Mar 29, 2023

jinwei14 commented Mar 30, 2023

harsha-simhadri commented Mar 31, 2023

jinwei14 commented Mar 29, 2023 •

edited

Loading