Feature : Support for Hugging Face streaming dataset

[azamiftikhar1000]

#2864

Enhanced the Hugging Face loader to support streaming-enabled datasets, expanding its capabilities.

[vowelparrot]

Thanks for the PR! I believe @eyurtsev is working on an iterable data loading interface - we'll try to make sure this fits in with what we're building towards before landing

[azamiftikhar1000]

Thanks for the PR! I believe @eyurtsev is working on an iterable data loading interface - we'll try to make sure this fits in with what we're building towards before landing

Make sense, thank you!

[eyurtsev]

Hi @azamiftikhar1000,

A lazy iteration method was added to the interface: #3659

Notable change is that this method does not have a text splitter. I'm thinking about adding text splitting and batching via a composition, so it won't be included by default when doing lazy iteration.

For context it's a part the following sequence of changes: #2833 (comment)

[azamiftikhar1000]

Hi @azamiftikhar1000,

A lazy iteration method was added to the interface: #3659

Notable change is that this method does not have a text splitter. I'm thinking about adding text splitting and batching via a composition, so it won't be included by default when doing lazy iteration.

For context it's a part the following sequence of changes: #2833 (comment)

Actually, it makes a lot of sense to have a new method for handling lazy loading and to implement lazy loading for all iterable loaders, not just Hugging Face. Thanks for that!

[eyurtsev]

lazy load now exists on the interface, if anyone wants can also pass through the streaming flag to allow loading without the download