This fork was created to investigate the slow performance of youtube-dl with harbour-ytplayer and all support for other sites than YouTube has been removed.
All bugs are considered as features, until you can reproduce them using the official release :)
First I was generally suspecting Python, because simply running youtube-dl --version took a long time to complete. There could not be any network delays that caused the slow execution. Another thought was that perhaps the large code base with tens of supported sited would slow the execution down. With these in the back of my head, I started my work.
As I am not very familiar with Python, I also discovered that the the youtube-dl binary is simply a zipped collection of .py files. As I tested various ways to run the scripy, I came into the conclusion that the zip file must be decompressed and compiled every single time the script is called. This adds a lot of overhead to the execution.
I found out that Nuitka can turn Python into C++, so I gave that a try. I tested both the "full" version and the stripped down version. The command used to compile the binary on all platforms was python3 -m nuitka --follow-imports __main__.py.
As youtube-dl is used to only access YouTube in the project, I started removing support for everything but YouTube. This took a little trial and error, but wasn't too hard. Basically:
- Remove unnecessary extractors
- Remore references to the removed extractors from extractors.py and generic.py
make
You can check the git changes for details if you are interested.
The version tested is 2020.02.16. I ran the tests on the devices using local terminal, running the command time python3 ./youtube-dl for the scripts and time ./python-dl.bin for the Nuitka compiled versions. For the unzipped Python script runs, the first run always generates the *.pyc files, and this run takes more time. Naturally, these first runs were discarded. With that out of the way; I ran the command five more times in a row and took the average.
My desktop computer was running up-to-date Manjaro Linux with kernel 5.5.2-1. All other devices were running Sailfish OS 3.2.1.
First, the execution times in seconds:
| Device \ Variant | ytdl-full.py zipped | ytdl-lite.py zipped | ytdl-full unzipped | ytdl-lite unzipped | ytdl-full Nuitka | ytdl-lite Nuitka |
|---|---|---|---|---|---|---|
| AMD Ryzen 7 2700 | 1.111 | 0.275 | 0.114 | 0.114 | 0.219 | 0.116 |
| Sony Xperia XA2 Ultra | 6.051 | 1.741 | 1.399 | 0.516 | 1.287 | 0.643 |
| Sony Xperia Z3 Compact Tablet | 6.366 | 1.658 | 1.658 | 0.666 | 1.369 | 0.715 |
| Jolla Phone | 12.17 | 3.138 | 2.976 | 1.218 | 2.422 | 1.195 |
And then the same data as percentages. For each row, 100% is the largest execution time, which is the vanilla, zipped full version of youtube-dl.
| Device \ Variant | ytdl-full.py zipped | ytdl-lite.py zipped | ytdl-full unzipped | ytdl-lite unzipped | ytdl-full Nuitka | ytdl-lite Nuitka |
|---|---|---|---|---|---|---|
| AMD Ryzen 7 2700 | 100% | 25% | 10% | 10% | 20% | 10% |
| Sony Xperia XA2 Ultra | 100% | 29% | 23% | 9% | 21% | 11% |
| Sony Xperia Z3 Compact Tablet | 100% | 26% | 26% | 10% | 22% | 11% |
| Jolla Phone | 100% | 26% | 24% | 10% | 20% | 10% |
The desktop computer is by far the fastest, as expected. Interestingly, just uncompressing the full version gives the maximum reduction in execution time, whereas the mobile devices still benefit greatly using the uncompressed lite version.
For the mobile devices, the results are practically identical. Just uncompressing the vanilla Python zip cuts ~75% off the execution time, which is very close with using the zipped lite version. The best time was achieved with unzipped lite version, with ~90% of the execution time cut off.
Let's look at the disk space usage for all cases:
| ytdl-full zip | ytdl-lite zip | ytdl-full unc. | ytdl-lite unc. | Nuitka full armh7l / amd64 | Nuitka lite armh7l / amd64 |
| 1.7 MB | 0.25 MB | 5.2 MB | 1.0 MB | 26.3 MB / 38.6 MB | 4.9 MB / 7.7 MB |
I uploaded all the binaries produces by Nuitka as compiling them may be cumbersome. They have not been tested in any way (except the version string query) so please proceed with caution.
As a conclusion, I'll prefer the uncompressed lite version of youtube-dl in my project.