Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up expand_directories function #17

Merged
merged 6 commits into from
Oct 14, 2022

Conversation

mazulo
Copy link
Owner

@mazulo mazulo commented Oct 14, 2022

In a directory with 1411 valid files, the new implementation using os.scandir took simply 8ms. The previous implementation for the same project would found more files (6692) but most of them not valid ones.

Here are the results using different implementations:

❯ python second_performance_test.py
os.walk                 took 20 ms. Found files: 1411
os.walk-glob    took 30 ms. Found files: 1399
glob.glob               took 26 ms. Found files: 1385
glob.iglob              took 27 ms. Found files: 1385
pathlib.rglob   took 47 ms. Found files: 1406
find_files              took 39 ms. Found files: 1411
fast_scandir    took 8 ms. Found files: 1411
run_previous_os_walk_from_the_project   took 22 ms. Found files: 6692

and here is the code for this: https://gist.github.com/mazulo/f1fe5047fda3ab3a8f965ddfb7e39f2a

@mazulo mazulo self-assigned this Oct 14, 2022
@mazulo mazulo merged commit e108263 into main Oct 14, 2022
@mazulo mazulo deleted the mazulo/speed-up-expand-directories-function branch October 14, 2022 16:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant