Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create vectordb retrieval #49

Merged
merged 24 commits into from
Jan 24, 2024
Merged

Create vectordb retrieval #49

merged 24 commits into from
Jan 24, 2024

Conversation

bwook00
Copy link
Contributor

@bwook00 bwook00 commented Jan 24, 2024

  • Implement vectordb retrieval using chromadb (the code itself is similar to bm25).
  • Add the ability to use vectordb in retrieval/base.py.
  • push chroma file for test code as well
    (changed the distance metric when creating a chromadb collection to enable 'cosine similarity search'.
    For that db, ingest the corpus.parquet file in the sample project folder)

-> I'll also create a function to ingest vectordb and add it to evaluator.py, then close Feature/#21.

autorag/nodes/retrieval/base.py Outdated Show resolved Hide resolved
autorag/nodes/retrieval/base.py Outdated Show resolved Hide resolved
autorag/nodes/retrieval/base.py Outdated Show resolved Hide resolved
autorag/nodes/retrieval/base.py Outdated Show resolved Hide resolved
autorag/nodes/retrieval/vectordb.py Outdated Show resolved Hide resolved
autorag/utils/util.py Show resolved Hide resolved
tests/autorag/nodes/retrieval/test_vectordb.py Outdated Show resolved Hide resolved
autorag/nodes/retrieval/vectordb.py Show resolved Hide resolved
@bwook00 bwook00 requested a review from vkehfdl1 January 24, 2024 15:45
# Conflicts:
#	autorag/__init__.py
#	requirements.txt
vkehfdl1
vkehfdl1 previously approved these changes Jan 24, 2024
Copy link
Contributor

@vkehfdl1 vkehfdl1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quickly add ingest code and get rid of all chroma resources haha

Copy link
Contributor

@Eastsidegunn Eastsidegunn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Eastsidegunn
Eastsidegunn previously approved these changes Jan 24, 2024
Copy link
Contributor

@Eastsidegunn Eastsidegunn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@vkehfdl1 vkehfdl1 enabled auto-merge (squash) January 24, 2024 16:22
# Conflicts:
#	requirements.txt
Eastsidegunn
Eastsidegunn previously approved these changes Jan 24, 2024
Copy link
Contributor

@Eastsidegunn Eastsidegunn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@vkehfdl1 vkehfdl1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@vkehfdl1 vkehfdl1 merged commit 654f540 into main Jan 24, 2024
3 checks passed
@vkehfdl1 vkehfdl1 deleted the Feature/#21 branch January 24, 2024 17:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants