Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add External VectorDB Connections #872

Merged
merged 12 commits into from
Oct 23, 2024
Merged

Add External VectorDB Connections #872

merged 12 commits into from
Oct 23, 2024

Commits on Oct 19, 2024

  1. fix parse_ocr.yaml typo

    jeffrey committed Oct 19, 2024
    Configuration menu
    Copy the full SHA
    44fd77b View commit details
    Browse the repository at this point in the history

Commits on Oct 20, 2024

  1. Add BaseVectorDB schema only for AutoRAG and replace original chroma …

    …collection to Chroma vector store instance. (#871)
    
    * add Chroma wrapper for vectorstore compatibility
    
    * Make my own Chroma class
    
    * refactor get_id_scores
    
    * replace chroma to VectorStore instance with async
    
    * use vector store embedding at vectordb module
    
    * optimize imports
    
    ---------
    
    Co-authored-by: jeffrey <vkefhdl1@gmail.com>
    vkehfdl1 and jeffrey authored Oct 20, 2024
    Configuration menu
    Copy the full SHA
    d20806e View commit details
    Browse the repository at this point in the history
  2. Merge branch 'main' into Feature/#830

    jeffrey committed Oct 20, 2024
    Configuration menu
    Copy the full SHA
    4dd1c16 View commit details
    Browse the repository at this point in the history

Commits on Oct 21, 2024

  1. Load vector store configuration from YAML file and use it on the vect…

    …ordb module. (#874)
    
    * make load_yaml_config for easier retrieve config yaml file
    
    * load vectordb instance from config YAML file
    
    * load vectordb from YAML in the vectordb module
    
    * fixed wrong annotation
    
    ---------
    
    Co-authored-by: jeffrey <vkefhdl1@gmail.com>
    vkehfdl1 and jeffrey authored Oct 21, 2024
    Configuration menu
    Copy the full SHA
    5475fdb View commit details
    Browse the repository at this point in the history

Commits on Oct 22, 2024

  1. Make Evaluator with new external DB works (#877)

    * add filter from corpus data and filter from retrieval_gt for optional ingest
    
    * make Evaluator with external vectordb working.
    
    ---------
    
    Co-authored-by: jeffrey <vkefhdl1@gmail.com>
    vkehfdl1 and jeffrey authored Oct 22, 2024
    Configuration menu
    Copy the full SHA
    049e643 View commit details
    Browse the repository at this point in the history

Commits on Oct 23, 2024

  1. Implement Mivlus instance and test it (done)

    jeffrey committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    0baa503 View commit details
    Browse the repository at this point in the history
  2. Implement milvus and test it (#879)

    Co-authored-by: jeffrey <vkefhdl1@gmail.com>
    vkehfdl1 and jeffrey authored Oct 23, 2024
    Configuration menu
    Copy the full SHA
    b81cbe5 View commit details
    Browse the repository at this point in the history
  3. Use vectordb dictionary at extracted yaml file (#880)

    * extract vectordb and save it to extracted yaml as also
    
    * add delete collection method for milvus
    
    * edit hybrid testing for new vectordb
    
    * optimize imports at test_hybrid_cc.py
    
    ---------
    
    Co-authored-by: jeffrey <vkefhdl1@gmail.com>
    vkehfdl1 and jeffrey authored Oct 23, 2024
    Configuration menu
    Copy the full SHA
    c103dc4 View commit details
    Browse the repository at this point in the history
  4. Documentation for new external Vector DB connection (#881)

    * add vector db documentations (milvus and chroma)
    
    * change documentation file for vectordb
    
    * Fix all sample config YAML file to use vectordb
    
    ---------
    
    Co-authored-by: jeffrey <vkefhdl1@gmail.com>
    vkehfdl1 and jeffrey authored Oct 23, 2024
    Configuration menu
    Copy the full SHA
    5574864 View commit details
    Browse the repository at this point in the history
  5. resolve error at test_run_retrieval_node.py

    jeffrey committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    d5a8cb2 View commit details
    Browse the repository at this point in the history
  6. skip milvus test code

    jeffrey committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    cb64190 View commit details
    Browse the repository at this point in the history
  7. resolve test error

    jeffrey committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    bec9759 View commit details
    Browse the repository at this point in the history