Add Qdrant as a long memory backend. #1515

wangxuqi · 2023-04-15T08:55:27Z

Hi there!
For this pull request, I have read your announcement carefully.

Background

As I notice there are many vector databases provided in chatgpt-retrieve-plugin.
I think all the vector databases can be used by you as a long-term memory. So this is the first commit to add the most easier used one 'Qdrant'. And I am willing to implement them all!

Changes

This commit only "Add Qdrant as a long memory backend."
what I do is:

Implement QdrantMemory based on MemoryProviderSingleton
add 'qdrant' option in "memory/init.py"
add 'qdrant-client' in requirements.txt
add unit test and integration test in 'qdrant_memory_test.py' and 'qdrant_memory_tests.py'

Documentation

The implement is very clear in-code comments. I just implement all your API and test all them to work.

def add(self, data)
def get(self, data)
def clear(self)
def get_relevant(self, data, num_relevant=5)
def get_stats(self)

Test Plan

Unit test in 'qdrant_memory_test.py'
IntegrationTest in 'qdrant_memory_tests.py'
Test step: Lauch a Qdrant docker instance docker run -p "6333:6333" -p "6334:6334" qdrant/qdrant:v1.0.3 and then just run all the related cases.
I've also tested with the following instructions:

Enter up to 5 goals for your AI:  For example: Increase net worth, Grow Twitter Account, Develop and manage multiple businesses autonomously'
Enter nothing to load defaults, enter nothing when finished.
Goal 1: Visit a website which provides today's news.
Goal 2: Summarize the articles and write the results to text files
Goal 3: Terminate
Goal 4: 
Using memory of type: QdrantMemory
SUMMARIZE TODAY'S LEADING NEWS THOUGHTS:  I need to determine which news website to visit and then summarize 
...

PR Quality Checklist

My pull request is atomic and focuses on a single change.
I have thoroughly tested my changes with multiple different prompts.
I have considered potential risks and mitigations for my changes.
I have documented my changes clearly and comprehensively.
I have not snuck in any "extra" small tweaks changes

nponeccop · 2023-04-15T13:34:04Z

@wangxuqi There are conflicts now

wangxuqi · 2023-04-15T15:44:24Z

@wangxuqi There are conflicts now

The conflict is caused by another long memory 'Milvus' merged. I have resolved the conflict.

wangxuqi · 2023-04-15T16:35:55Z

@nponeccop As I solve the conflict , I find the merged 'Milvus memory' code is very poor. With no tests and even not worked. Also the package import is wrong and the milvs.py is not under the correct directory.
Please review the pull request again #801 or I will fixed it later.

wangxuqi · 2023-04-16T02:31:32Z

Fix conflict with another merged PR 1658.
As I see you have add a potential plugin label. I think it maybe a better way to fit all chatgpt-retrieve-plugin. vector databases. Try to do this later.

nponeccop · 2023-04-16T15:24:21Z

README.md

@@ -342,19 +343,6 @@ export PINECONE_ENV="<YOUR_PINECONE_REGION>" # e.g: "us-east4-gcp"
 export MEMORY_BACKEND="pinecone"
 ```

-### Milvus Setup


Don't remove milvus

Milvus Setup is duplicated. You can see line 367 to 377

wangxuqi · 2023-04-17T01:54:24Z

Ops! Conflict with README.md again. resolve it. Hope this time no more conflicts.

wangxuqi · 2023-04-17T02:52:34Z

CI broken fixed by #2061

github-actions · 2023-04-18T23:21:10Z

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

github-actions · 2023-04-19T02:04:52Z

Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.

github-actions · 2023-04-22T12:04:50Z

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

github-actions · 2023-04-22T15:45:32Z

Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.

codecov · 2023-04-22T15:48:51Z

Codecov Report

Patch coverage: 14.06% and project coverage change: -0.46 ⚠️

Comparison is base (8e8a5a1) 36.22% compared to head (0457470) 35.77%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1515      +/-   ##
==========================================
- Coverage   36.22%   35.77%   -0.46%     
==========================================
  Files          60       61       +1     
  Lines        2849     2913      +64     
  Branches      471      477       +6     
==========================================
+ Hits         1032     1042      +10     
- Misses       1755     1808      +53     
- Partials       62       63       +1

Impacted Files	Coverage Δ
autogpt/memory/qdrant.py	`4.00% <4.00%> (ø)`
autogpt/memory/__init__.py	`43.07% <41.66%> (-0.32%)`	⬇️
autogpt/config/config.py	`75.00% <100.00%> (+0.35%)`	⬆️

... and 1 file with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

github-actions · 2023-04-26T18:21:05Z

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

p-i- · 2023-05-05T00:56:09Z

This is a mass message from the AutoGPT core team.
Our apologies for the ongoing delay in processing PRs.
This is because we are re-architecting the AutoGPT core!

For more details (and for infor on joining our Discord), please refer to:
https://github.com/Significant-Gravitas/Auto-GPT/wiki/Architecting

ntindle · 2023-05-25T16:00:09Z

Hey, I've marked this as don't merge until the Memory Fixes are in. Sorry to keep it on hold longer, just no real way to test/validate functionality until the fixes are in

wangxuqi · 2023-05-26T03:32:46Z

Hey, I've marked this as don't merge until the Memory Fixes are in. Sorry to keep it on hold longer, just no real way to test/validate functionality until the fixes are in

As I see the discussion #4280. So close the pr for now. Thanks for your attention and reply. Wish you all good.

nponeccop previously approved these changes Apr 15, 2023

View reviewed changes

wangxuqi dismissed nponeccop’s stale review via e1941e2 April 15, 2023 15:40

wangxuqi force-pushed the qdrant_memory branch from 85f2072 to e1941e2 Compare April 15, 2023 15:40

wangxuqi requested a review from nponeccop April 15, 2023 15:52

wangxuqi force-pushed the qdrant_memory branch from e1941e2 to 51c7c11 Compare April 15, 2023 15:53

nponeccop previously approved these changes Apr 15, 2023

View reviewed changes

richbeales added the needs discussion To be discussed among maintainers label Apr 15, 2023

wangxuqi mentioned this pull request Apr 15, 2023

Fix Milvus as a long-term memory backend. #1658

Merged

5 tasks

nponeccop mentioned this pull request Apr 15, 2023

PR Batch 6 #1699

Closed

1 task

richbeales added the potential plugin This may fit better into our plugin system. label Apr 15, 2023

wangxuqi dismissed nponeccop’s stale review via bed39c8 April 16, 2023 02:24

wangxuqi force-pushed the qdrant_memory branch from 3fa5254 to bed39c8 Compare April 16, 2023 02:24

wangxuqi requested a review from nponeccop April 16, 2023 02:57

wangxuqi force-pushed the qdrant_memory branch from bed39c8 to c1cb33f Compare April 16, 2023 09:46

nponeccop suggested changes Apr 16, 2023

View reviewed changes

wangxuqi force-pushed the qdrant_memory branch from c1cb33f to 475b2d5 Compare April 16, 2023 15:37

wangxuqi requested a review from nponeccop April 16, 2023 15:40

nponeccop previously approved these changes Apr 16, 2023

View reviewed changes

wangxuqi dismissed nponeccop’s stale review via 2f1c605 April 17, 2023 01:58

wangxuqi force-pushed the qdrant_memory branch 2 times, most recently from 2f1c605 to 4d84b25 Compare April 17, 2023 02:34

nponeccop mentioned this pull request Apr 17, 2023

PR Batch 7 #2256

Closed

1 task

wangxuqi requested a review from nponeccop April 18, 2023 11:12

github-actions bot added the conflicts Automatically applied to PRs with merge conflicts label Apr 18, 2023

wangxuqi force-pushed the qdrant_memory branch from a457928 to 8d22fd5 Compare April 19, 2023 02:04

github-actions bot removed the conflicts Automatically applied to PRs with merge conflicts label Apr 19, 2023

wangxuqi force-pushed the qdrant_memory branch from 2e2c1f7 to c2d816d Compare April 19, 2023 02:18

github-actions bot added the conflicts Automatically applied to PRs with merge conflicts label Apr 22, 2023

wangxuqi added 4 commits April 22, 2023 23:29

Add Qdrant as a long memory backend.

6914d89

Remove qdrant-client from requirements.txt to make it lazy load.

fe8030e

Add AnalyticDB as a long-term memory backend. Format to meet lint CI.

4a44808

Move README about Qdrant in to memory.md

0457470

wangxuqi force-pushed the qdrant_memory branch from c2d816d to 0457470 Compare April 22, 2023 15:45

github-actions bot removed the conflicts Automatically applied to PRs with merge conflicts label Apr 22, 2023

github-actions bot added the size/xl label Apr 22, 2023

github-actions bot added the conflicts Automatically applied to PRs with merge conflicts label Apr 26, 2023

wangxuqi closed this May 26, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Qdrant as a long memory backend. #1515

Add Qdrant as a long memory backend. #1515

wangxuqi commented Apr 15, 2023 •

edited

Loading

nponeccop commented Apr 15, 2023

wangxuqi commented Apr 15, 2023 •

edited

Loading

wangxuqi commented Apr 15, 2023

wangxuqi commented Apr 16, 2023

nponeccop Apr 16, 2023

wangxuqi Apr 16, 2023

wangxuqi commented Apr 17, 2023

wangxuqi commented Apr 17, 2023

github-actions bot commented Apr 18, 2023

github-actions bot commented Apr 19, 2023

github-actions bot commented Apr 22, 2023

github-actions bot commented Apr 22, 2023

codecov bot commented Apr 22, 2023

github-actions bot commented Apr 26, 2023

p-i- commented May 5, 2023

ntindle commented May 25, 2023

wangxuqi commented May 26, 2023

Add Qdrant as a long memory backend. #1515

Add Qdrant as a long memory backend. #1515

Conversation

wangxuqi commented Apr 15, 2023 • edited Loading

Background

Changes

Documentation

Test Plan

PR Quality Checklist

nponeccop commented Apr 15, 2023

wangxuqi commented Apr 15, 2023 • edited Loading

wangxuqi commented Apr 15, 2023

wangxuqi commented Apr 16, 2023

nponeccop Apr 16, 2023

Choose a reason for hiding this comment

wangxuqi Apr 16, 2023

Choose a reason for hiding this comment

wangxuqi commented Apr 17, 2023

wangxuqi commented Apr 17, 2023

github-actions bot commented Apr 18, 2023

github-actions bot commented Apr 19, 2023

github-actions bot commented Apr 22, 2023

github-actions bot commented Apr 22, 2023

codecov bot commented Apr 22, 2023

Codecov Report

github-actions bot commented Apr 26, 2023

p-i- commented May 5, 2023

ntindle commented May 25, 2023

wangxuqi commented May 26, 2023

wangxuqi commented Apr 15, 2023 •

edited

Loading

wangxuqi commented Apr 15, 2023 •

edited

Loading