-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Add LinkContentFetcher
Haystack 2.0 component
#5724
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Implementation is solid but I think it can be simplified a lot. Sorry that I left many comments, I tried to provide the code where possible.
6d43a0c
to
89ed8f7
Compare
LinkContentFetcher
Haystack 2.0 component
I've just realized that there's no proper way to store the binary returned with the response into the current Document class. As this is not the first time we have trouble with the limitations of Document, I'm going to first expand the Document class and then wrap up this PR. |
…into new_content_fetcher
Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>
Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>
Pull Request Test Coverage Report for Build 6245834225
💛 - Coveralls |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🚀
Why:
This PR introduces the
LinkContentFetcher
component to Haystack 2.0. This component is responsible for fetching content from a given URL and converting it into aDocument
object, which can then be used within the Haystack 2.0 pipeline.Part of #5614
What:
A new
LinkContentFetcher
component has been added to the newly createdfetchers
package. This package is intended to be the future home for all components that fetch data via network calls.How can it be used:
Here's a code snippet demonstrating how to use the new
LinkContentFetcher
component:How did you test it:
Unit tests have been added to cover the new component. Additionally, manual tests were conducted, as described in the "How can it be used" section above.
Notes For Reviewer:
Please ensure that the new component and its unit tests are correctly implemented.