community: add Needle retriever and document loader integration

[JANHMS]

• PR title: "community: add Needle retriever and document loader integration"

  • Where "package" is whichever of langchain, community, core, etc. is being modified. Use "docs: ..." for purely docs changes, "infra: ..." for CI changes.
  • Example: "community: add foobar LLM"

• PR message: Delete this entire checklist and replace with

  • Description: This PR adds a new integration for Needle, which includes:
    • NeedleRetriever: A retriever for fetching documents from Needle collections.
    • NeedleLoader: A document loader for managing and loading documents into Needle collections.
    • Example notebooks demonstrating usage have been added in:
      • docs/docs/integrations/retrievers/needle.ipynb
      • docs/docs/integrations/document_loaders/needle.ipynb.
  • Dependencies: The needle-python package is required as an external dependency for accessing Needle's API. It has been added to the extended testing dependencies list.
  • Twitter handle: Feel free to mention me if this PR gets announced: needlexai.

• Add tests and docs: If you're adding a new integration, please include

  1. Unit tests have been added for both NeedleRetriever and NeedleLoader in libs/community/tests/unit_tests. These tests mock API calls to avoid relying on network access.
  2. Example notebooks have been added to docs/docs/integrations/, showcasing both retriever and loader functionality.

• Lint and test: Run make format, make lint, and make test from the root of the package(s) you've modified. See contribution guidelines for more: https://python.langchain.com/docs/contributing/

  • make format: Passed
  • make lint: Passed
  • make test: Passed (requires needle-python to be installed locally; this package is not added to LangChain dependencies).

Additional guidelines:

• Optional dependencies are imported only within functions.
• No dependencies have been added to pyproject.toml files except for those required for unit tests.
• The PR does not touch more than one package.
• Changes are fully backwards compatible.
• Community additions are not re-imported into LangChain core.

If no one reviews your PR within a few days, please @-mention one of baskaryan, efriis, eyurtsev, ccurme, vbarda, hwchase17.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
langchain	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Nov 16, 2024 6:16pm