Skip to content
This repository has been archived by the owner on Aug 16, 2023. It is now read-only.

Refine mmap implement for HNSW #1015

Merged
merged 1 commit into from
Jul 31, 2023
Merged

Refine mmap implement for HNSW #1015

merged 1 commit into from
Jul 31, 2023

Conversation

yah01
Copy link
Member

@yah01 yah01 commented Jul 28, 2023

/kind bug
fix #1005

@sre-ci-robot
Copy link

@yah01: The label(s) kind/bug cannot be applied, because the repository doesn't have them.

In response to this:

/kind bug
fix #1005

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@mergify mergify bot added the dco-passed DCO check passed. label Jul 28, 2023
@yah01
Copy link
Member Author

yah01 commented Jul 28, 2023

Also add tests for mmap

@yah01 yah01 changed the title Unmap L2 norm data for HNSW with COSINE metric type Refine mmap implement for HNSW Jul 30, 2023
@yah01
Copy link
Member Author

yah01 commented Jul 30, 2023

After trying, to map the whole file may be the simplest way, but we keep the graph in memory by copying it, to avoid much performance downgrade

@yah01
Refine mmap implement for HNSW
5731106 
Signed-off-by: yah01 <yang.cen@zilliz.com>
@yah01
Copy link
Member Author

yah01 commented Jul 30, 2023

Also add tests for all index support mmap

@mergify mergify bot added the ci-passed label Jul 30, 2023
liliu-z
liliu-z reviewed Jul 31, 2023
View reviewed changes
src/io/fileIO.h Show resolved Hide resolved
thirdparty/hnswlib/hnswlib/hnswalg.h Show resolved Hide resolved
@liliu-z
Copy link
Member

liliu-z commented Jul 31, 2023

/lgtm
/approve

@sre-ci-robot
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: liliu-z, yah01

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@sre-ci-robot sre-ci-robot merged commit b7d0b0a into milvus-io:main Jul 31, 2023
7 checks passed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@liliu-z liliu-z liliu-z left review comments

@cydrain cydrain Awaiting requested review from cydrain

@foxspy foxspy Awaiting requested review from foxspy

Assignees

@liliu-z liliu-z

Labels
approved ci-passed dco-passed DCO check passed. lgtm size/XL
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

HNSW throws exception while releasing in the case of mmap enabled
3 participants
@yah01 @sre-ci-robot @liliu-z