-
Notifications
You must be signed in to change notification settings - Fork 15.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add progress bar to filesystemblob loader, update pytest config for unit tests #4212
Conversation
|
||
|
||
def _make_iterator( | ||
length_func: Callable[[], int], show_progress: bool = False |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
out of curiosity can length_func not just be called when passed into this function (so arg would just be length
instead of length_func)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It forces us to iterate through all the files on the file system, which is unnecessary if we're not using tqdm to show the progress. So we pass a function so we can calculate the length on the condition that we want to use tqdm
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
now that's big brain 😻
…nit tests (langchain-ai#4212) This PR adds: * Option to show a tqdm progress bar when using the file system blob loader * Update pytest run configuration to be stricter * Adding a new marker that checks that required pkgs exist
…nit tests (langchain-ai#4212) This PR adds: * Option to show a tqdm progress bar when using the file system blob loader * Update pytest run configuration to be stricter * Adding a new marker that checks that required pkgs exist
This PR adds: