YT-DLP support

[mujunk]

Could you please consider this fork since youtube_dl maintenance is very slow and this one envelopes youtube_dl anyways
https://github.com/yt-dlp/yt-dlp

[nullket]

+1 as this would also solve the buffering throttle issue (see https://code.videolan.org/videolan/vlc/-/issues/26174#note_286445). Currently no YouTube Video coming from send to Kodi is watchable for me as buffering always stops the playback.

[nullket]

I did a quick local prototype and it worked perfectly (without buffering or any other issues):

• Placed the subfolder yt_dlp of the dlp repo in the sendtokodi addon folder
• Changed

  plugin.video.sendtokodi/service.py

  Line 45 in a8d39ca

  from youtube_dl import YoutubeDL

  to

from yt_dlp import YoutubeDL

[firsttris]

should i provide 2 addons with different YoutubeDL versions? one vanilla and one for yt_dlp?

[nullket]

At this point yt_dlp has proven to be a stable fork but yt_dl is not completely dead yet.

IMHO, I would simply include both in one addon version (this bloats "just" another 13 MB) and let the user decide via kodi addon settings:

• default yt_dlp but yt_dl can be choosen (or other way round if you prefer, it is your project)
• as the most function calls are still the same you could then simply do something linke

if xbmcplugin.getSetting("use_legacy_yt_dl"):
    from youtube_dl import YoutubeDL 
else:
    from yt_dlp import YoutubeDL

This could make the maintaining a bit easier and the user experience better (lets be honest, must of us just want things to work and not dig in to this discussion here and figure out that we need to manually install another version of this plugin so their youtube videos play without constant buffering). Also I am not quite sure how your App works but I think another addon version would need to either have the same name (which would require a different repo or checking that not vanilla and the other version is installed) OR changes to you app so it detects a version with a different name (This was the case with the python3 version right?)

[darnir]

I agree with nullket here. It would be nice to simply package both yt_dl and yt_dlp in the same plugin.
This will "just work" for almost everyone. You can set the default to yt_dl if you don't want to change things for people from under them.

[firsttris]

thats a very nice idea thx @nullket.
i will implement it as soon as i find time, but pull-requests are very welcome.

i also need to migrate my travis ci script to github actions, since travis was shut down for free use.

currently the addon is not re-building at all.

[darnir]

I can try to provide a pull request. Let me look into it

[firsttris]

i fixed the ci build.

if someone is providing a pull-request for the python code (to extend it as @nullket proposed) i would in return extend the ci job to also package the yt-dlp fork

[nullket]

Hey @firsttris whats your opinion on including youtube-dl/dlp both as git submodules with defined commits stating their version instead of having the github action pull the newest commit?

The benefits would be:

• Version control on the release side (and tested in development before)
• Your local plugin folder would (almost see below) resemble the release version
• The github action would only had to remove the .git folder, add the version etc. (like it already does right now)
• Adding or removing an alternate youtube-dl fork would be quite easy

Both repo folders - added as submodules - take about 8 MB of space each. The python import in the service.py would then for example be from lib.youtube-dlp.yt_dlp import .. instead of from yt_dlp import ...

Just an idea. Interested in your opinion.

[firsttris]

hey @nullket

i think thats an interesting idea to consider, which would make the CI script alot easier and cleaner.

never worked with git submodules but my understanding is that is like a reference to the other repo.

[nullket]

never worked with git submodules but my understanding is that is like a reference to the other repo.

Yes that is correct. They are gits in the main repo where the main repo simply saves which commit number the submodule should be.

Adding one

• You simply insert a new one with git submodule add <git-url> local/path/where/to/save
• Go back to your projects root to add and commit this submodule in the main repo with git add local/path/where/to/save && git commit -m "Added xyz as submodule".

Want to upgrade a specific submodule to its latest commit?
That is easy, just go to the folder and pull the changes. After that tell the main repo about those changes.

• cd local/path/where/to/save && git pull
• cd main/repo && git add local/path/where/to/save && git commit -m "Udated xyz submodule".

Pitfall: New clone of the main repo or having an old clone

• When cloning a fresh copy of the main repo, submodules are per default not automatically cloned (the folder simply stays empty)
• Cloning of your repo should be done with git clone --recursive <url>
• If one forgot the --recursive you can always do a git submodule init in the main repo

I created a PR if you agree on switching but the workflow is not testable for me, see #37

[firsttris]

good work, i merged your PR and it worked like a charm.

i also simplified the build-script.sh, it was a bit redundant. now the script is parameterized.

[nullket]

Perfect :) Yep, your changes to the build script make totally sense. One could even split building and releasing into two steps (or one step with an argument to the script) so new PRs (and maybe even forks) could build a zip file (stored as a GitHub workflow artifacts) and one could easily test those „addons“ before merging. But I feel like I am over-engineering the process here a bit, based on how not so often you get PRs (I am currently just hyped poking around the Kodi Python api for the first time)

[firsttris]

the main idea is that the pipeline is autonomous releasing new versions based on a scheduler. wihtout human interference.

[firsttris]

still missing a pull-request to change service.py to either import youtube-dl or yt-dlp based on the kodi settings (as @nullket proposed), before we can close this issue.

[anohren]

Actually the ytdl mega thread about YouTube throttling is writing patches on a daily basis now and seems to lean towards pulling the dlp changes to fix that. If I understood correctly.

What other benefits will ytdlp have?

[nullket]

They are writing patches but they do not seem to be merged. Looks like the maintainers are quite busy. Apart from YouTube this can be seen for other sources as well (broken downloader, wrong quality etc.). YouTube-dl is fine, it is just too often outdated. Ignoring all the other features we do not need, the amount of updates ytdlp gets is its biggest advantage.

To sum it up: both should be available and let the SendToKodi users choose in case they prefer one more than the other. I would just say that the default one should always be that one that works (the most).

[anohren]

I haven't checked out ytdlp, but updates sounds good.

[pitsi]

Can you please make yt-dlp the default resolver or at least offer a way to select between yt-dlp and (the now obsolete) youtube-dl?

The throttling in youtube videos makes the use of sendtokodi a bad experience.
Almost almost every single app (e.g. mpv) / script (e.g. youtube-viewer by trizen) / addon (e.g. kodi's youtube addon) has addressed this months ago by either making changes in its code or by using yt-dlp in favor of youtube-dl.
And what is even weirder is that throttling happens even if I send the video from an invidious instance instead of youtube. It behaves as if the video url is redirected to youtube and then resolved and played from there.

[firsttris]

@nullket how far are we with choosing resolver?

[nullket]

Can you please make yt-dlp the default resolver or at least offer a way to select between yt-dlp and (the now obsolete) youtube-dl?

We needed to sort out some things in repo/release structure first. Those bigger changes were made just last week and we wanted to see if the automatic building and releasing pipeline works before we mix up two major changes. But you are right. I am currently a bit busy at work (important deadline ahead) but I hope get this done by next week. Actually I have it is already working but the settings are dangled (missing strings in the menu) and we need a solution to support youtube-dl for the old python2 users. It would be awesome to detect if the script is executed on python2 (easy) and than change the kodi setting to youtube-dl (
don't know if this is possible). Otherwise the python2 version could have a patch file (which will be applied during build on the master branch) where the settings would be erased and it defaults to youtube-dl. Ideas are welcome.

And what is even weirder is that throttling happens even if I send the video from an invidious instance instead of youtube. It behaves as if the video url is redirected to youtube and then resolved and played from there.

I don't know how invidious is working. And I would guess that youtube-dl simply transforms invidious urls to youtube urls. I mean in the end it is the same backend and thus the same api.

[pitsi]

You are right about that detail for python2. However, sendtokodi's behavior is the same on either kodi 19 (main pc running debian testing/unstable and python 3 for the last ~1 year) or kodi 18 (old laptop running libreelec 9.2.6 and old tvbox running coreelec 9.2.8).

About that last part about invidious, you are right.

$ youtube-dl -v https://yewtu.be/watch?v=TtsglXhbxno
[debug] System config: []
[debug] User config: ['--no-cache-dir', '--restrict-filenames']
[debug] Custom config: []
[debug] Command-line args: ['-v', 'https://yewtu.be/watch?v=TtsglXhbxno']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.06.06
[debug] Python version 3.9.9 (CPython) - Linux-5.10.0-6-amd64-x86_64-with-glibc2.32
[debug] exe versions: ffmpeg 4.4.1, ffprobe 4.4.1
[debug] Proxy map: {}
[youtube] TtsglXhbxno: Downloading webpage
[debug] Default format spec: bestvideo+bestaudio/best
WARNING: Requested formats are incompatible for merge and will be merged into mkv.
[debug] Invoking downloader on 'https://rr4---sn-vuxbavcx-5uiz.googlevideo.com/videoplayback?expire=1639522415&ei=D8y4YZmrIIiJvdIPpJGLgAE&ip=87.202.1.96&id=o-AN2E3yK2UJ2xfVb7ynRimT9nfhEKGoslxkNsjAF9QgGn&itag=313&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C271%2C278%2C313%2C394%2C395%2C396%2C397%2C398%2C399%2C400%2C401&source=youtube&requiressl=yes&mh=OW&mm=31%2C29&mn=sn-vuxbavcx-5uiz%2Csn-4g5ednsr&ms=au%2Crdu&mv=m&mvi=4&pl=19&initcwndbps=852500&vprv=1&mime=video%2Fwebm&ns=lO2uiEJVbslgc4mFFcxn-WQG&gir=yes&clen=1491686079&dur=1783.081&lmt=1638657983259313&mt=1639500309&fvip=4&keepalive=yes&fexp=24001373%2C24007246&c=WEB&txp=5532432&n=1yLR53SZ86ea4o&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRQIhAJ8AqM2SDs5F3mRIFLah9yAxCAM4MVxYS85gjkh9l3epAiB1PZDjGDA3CeTqGIP4I8RAqseEIhTxeMJ83zgXJ3549Q%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRAIgGZtqKgMvcSl2reBOrCp1SZ6piPds5ttLb0i7GKrrBw4CICAx0iV18demzNSrmJrjT9g06kuBTBCEeSgkpMX5pAqz'
[download] Destination: Trying_to_do_Simple_Tasks_on_Linux_lol-TtsglXhbxno.f313.webm
[download]   0.1% of 1.39GiB at 74.69KiB/s ETA 05:24:51^C
ERROR: Interrupted by user

[nullket]

You are right about that detail for python2. However, sendtokodi's behavior is the same on either kodi 19

Yes but youtube-dlp is actually python3 only. So after enabling youtube-dlp (by default) the addon would be simply broken for python2 users.

[pitsi]

You are right again :(
The following output is from one of the *elec systems that are python2 only.

# ./yt-dlp https://youtube.com/watch?v=TtsglXhbxno
env: can't execute 'python3': No such file or directory

[pitsi]

I have an idea I would like to share and it has to do with youtube videos only.
Instead of resolving the video url with youtube-dl or yt-dlp and then sending it to kodi for playback, how about making the addon play the url with the original youtube addon for kodi? It has been patched for that throttling issue since 6.8.18 and playback should work properly.

[nullket]

how about making the addon play the url with the original youtube addon for kodi
This would require:

• detected if the plugin is installed at all (I assume most people have it but not all)
• detect if the URL is for youtube
• send the url to the addon via plugin://plugin.video.youtube/play/?video_id=$VIDEOID

Those would be quite a few things to do and it would kill the universal approach of this addon in a way. But what I like about it that it solves the problem for python2 users and also enables youtube in a higher quality (as the youtube addon uses adaptive stream and sendtokodi does not). @firsttris should this be added to the roadmap? Could be integrated as an optional setting ("play youtube videos in the youtube addon") in addtion to the youtube-dlp resolver?

[firsttris]

no i'm against that.

our addon will only do one thing and try to be good at it.
use youtube_dl oder dlp to resolve links and send them to kodi video player.

thats our usecase.

i dont want to create a egg-laying woolly milk sow
(i dont know if you can even translate it like this)

my understandig is with dlp we will solve the throtteling issue.
probably at some point it will also be solved by youtube_dl

but the focus of youtube_dl is not just youtube, its a multi site resolver/parser.

[firsttris]

also as @nullket mentioned any client can just send a request to plugin://plugin.video.youtube/play/?video_id=$VIDEOID
to trigger the youtube addon.

there would be no need to change this addon.

i remember our IOS App had such an option but i think we removed it at some point?

kore for android maybe has such an option as well?

i would also approve to change the chrome extension with such an option:
https://github.com/firsttris/chrome.sendtokodi/blob/master/src/components/Popup.jsx#L40
but i dont have time to implement it myself.

[pitsi]

probably at some point it will also be solved by youtube_dl

About that. This will never happen. Youtube-dl has not been updated for 6+ months now and there is even this sticky in their reddit that devs encourage people to move to yt-dlp. There are also similar discussions about it on reddit and on their github.
https://old.reddit.com/r/youtubedl/comments/qfbyal/read_slow_youtube_downloads/

Moreover, apps that used youtube-dl as their backend for downloading (clipgrab, media downloader etc) or playback (mpv etc) have now moved to yt-dlp.

As for me, I have to admit that I moved to sendtokodi from a similar addon that uses either resolveurl or script.module.youtube.dl (plugin.video.playthis by anxdpanic) because its "version" (or way of implementaion) of youtube-dl simply works better for many sites.

[nullket]

our addon will only do one thing and try to be good at it.

I like this (Unix philosophy)

i dont want to create a egg-laying woolly milk sow

I like this even more :D

i remember our IOS App had such an option but i think we removed it at some point?

Totally agree that would be a good place where this would need to go! I have no idea of app programming and it is closed source anyways but maybe one day you have the time and the mood to implement this. Until then I would say that youtube-dlp solves the problems for now. I am working on it!

[anohren]

i remember our IOS App had such an option but i think we removed it at some point?

Totally agree that would be a good place where this would need to go!

It has all the appearance of being present in the iOS app from where I'm standing. Every supported add-on is listed in the app?

[pitsi]

So, a new version of youtube-dl was released today... The changelog does not mention anything about the throttling issue and most comments on reddit prove it was not fixed.
Moreover, a core dev has put himself to its inactive devs, so I assume we can now consider it a dead project.

[anohren]

Interesting. Do you know if anyone gave a specific reason? It's interesting to know how projects come to stagnate.

[anohren]

Sorry, hit the wrong button there 🙂 Github design team still excelling at button placement.

[pitsi]

This is the pull request for youtube-dl that fixes the throttling issue on youtube. It seems it will not be merged in upstream youtube-dl anytime soon. Can you patch the relevant file in your addon so as to finally get rid of that annoyance?
ytdl-org/youtube-dl#30184

[firsttris]

i think we could switch to his branch
https://github.com/dirkf/youtube-dl/tree/df-youtube-unthrottle-patch

it seems he continously pulls in fixes, maybe it will be better then the official repo for the time beeing. till the situation changes.

what do you other guys think?

we just need to change this line to pull from this branch:
https://github.com/firsttris/plugin.video.sendtokodi/blob/master/.github/workflows/update-libs.yml#L16

[pitsi]

If it works with python2, I am ok with it. I just hope his fork does not get too many bugs.

[pitsi]

Merry Christmas to everyone.
Does anyone know (or can anyone verify) that the patch for youtube's throttling has been merged to dirkf's fork of youtube-dl? Every time there is an update on sendtokodi, I send a youtube video over to check if it is fixed and so far it has the same behavior as the original youtube-dl.

[nullket]

i think we could switch to his branch
https://github.com/dirkf/youtube-dl/tree/df-youtube-unthrottle-patch

We could indeed but we would need to trust the author (not so much because of bugs but due to our auto update process we also created a possibility to smuggle in arbitrary code). But he is quite active on the youtube-dl repo and I think we should simply swap the git url as you proposed.

Also, I have created a new version of sendtokodi which uses youtube-dlp as default in kodi v19+ and should use youtube-dl on python2 systems. @pitsi and others: I would love to get some beta feedback on this version. Download and installation instructions can be found in the last paragraph of this comment.

[pitsi]

Please be patient :)
I downloaded the zip you mentioned above and (using file-roller) I extracted (or at least I tried) the 2 folders on my desktop. The folders were created, but they are empty. And there is also a file outside of them, that has literally no permissions at all. Am I doing something wrong?

$ ls -l plugin.video.sendtokodi-1.0.b1.zip 
---------- 1 x x 9718721 Dec 29 15:34 plugin.video.sendtokodi-1.0.b1.zip

---edit
I tried it with unzip
unzip prebuilt-1.0.b1.zip
on the system that runs libreelec 9.2.6. The 2 files extracted in their relevant folders, but the permissions issue is still there

# ls -l Matrix/plugin.video.sendtokodi-1.0.b1.zip Leia/plugin.video.sendtokodi-1.0.b1.zip
----------    1 root     root       9718723 Dec 29 15:34 Leia/plugin.video.sendtokodi-1.0.b1.zip
----------    1 root     root       9718721 Dec 29 15:34 Matrix/plugin.video.sendtokodi-1.0.b1.zip

It installed with no issue though. I am off to test it.

---re-edit
It works like it used to on kodi 18, although yt-dlp is selected as the default resolver. I assume it falls back to youtube-dl. The throttling issue is still present, although it happens very few times during playback.
I tried unzip on the desktop too, but I get the same behavior, plus a permission error when I try to access each folder.

[nullket]

Yep that was on me. I zipped with the files belonging to a non default user. I corrected the zip but you figured out how to repair it as well :)

[pitsi]

And it works as it should on kodi 19. It using yt-dlp because I see no throttling.
The problem is... I can't find it under my addons (settings > addons > my addons > all) so as to check its settings, but it is installed

$ ls .kodi/addons/ | grep send
plugin.video.sendtokodi

---edit
Nvm, I restarted kodi and it showed up. Changing the setting to youtube-dl gets the throttling back, so I can say it definitely works as it should.
For the next update, can you please make the zip file have proper permissions? I lost more time to change them and make the files and folders visible through my file manager and kodi than to test the addon itself. Thank you in advance.

[firsttris]

@nullket pushed plugin.video.sendtokodi-0.9.28.zip

which now includes yt-dlp as default resolver for python3 version, python2 still uses youtube-dl.

there is also a addon-settings to switch the resolver.

[pitsi]

Is this version any different than 1.0b1 I tested a few days ago? I wanto to remove it and install the one from the repo so as to get updates automatically.

[nullket]

Yes and no. It is just different in terms of a newer yt-dlp version but that one brought some important improvements for youtube. I recommend to uninstall the beta and revert to the release of our repo instead.

I will close the issue as the initial support has been provided. In case of problems, feature request, etc. feel free to open another issue (or even report back here).

[pitsi]

This is the pull request for youtube-dl that fixes the throttling issue on youtube. It seems it will not be merged in upstream youtube-dl anytime soon. Can you patch the relevant file in your addon so as to finally get rid of that annoyance? ytdl-org/youtube-dl#30184

About that. As it seems, the patch has been merged in dirkf's fork of youtube-dl. However, the throttling is still there. I want to troubleshoot it a bit more, so as to know if the problem is on kodi itself. Ideally, I want to run it as a plain executable on my pc and see what happens.
But how will I do it? There are no binaries for his fork. Any ideas are welcome.

[nullket]

About that. As it seems, the patch has been merged in dirkf's fork of youtube-dl

He did not merge it but he created the patch himself within the fork. That is the usual git(hub) contribution way: When you want to contribute, you fork a project, create a new branch within this fork, do your work and create a pull request to the master/main branch of the original repo.

However, the throttling is still there

Sure. We still use the official youtube-dl version which is still broken. Nobody changed to dirkf's fork and his PR is not merged (thus the official version still being broken). Now you might ask why we have not simply changed to dirks branch. The answer is quite easy: We have an automatic update process going on. So whenever dirk would add something to his branch a new sendtokodi addon version would be created. This sounds good so far. But this could also be very easy be exploited to bring potentially harmful code to all the python2 users using this plugin. This mechanism was fine for youtube-dl because it was a well reviewed repo. I am not saying dirk can not be trusted but it is simply not that easy. We could simply switch to his fork but imho a better way would be to patch youtube-dl ourselves (using dirks work by e.g. forking his fork under our namespace). @firsttris what do you think? In the end it is your project? An alternative is sending urls directly to the youtube addon from the app instead (like it was at some point?)

I want to troubleshoot it a bit more, so as to know if the problem is on kodi itself

The problem is not within kodi the problem is that youtube-dl is still broken. While the problem is solved for python3 users it simply remains for those who have not upgraded yet. I don't know your system and of course the choice is totally yours but it might be less a hassle to just update kodi (or your whole OS if you use something like libreELEC) to kodi v19+.

Ideally, I want to run it as a plain executable on my pc and see what happens.

The whole thing is basically just python. There is no need for a binary as long as you have a python interpreter installed on your computer.

Any ideas are welcome.

In my opinion you should simply upgrade to a newer kodi version.

[pitsi]

I do not remember how to multiquote on github, so excuse me for the following.

I did not know that the original youtube-dl is still used. I thought you switched to the fork and that the noumerous (almost daily) releases of sendtokodi are because of the new pathes that the fork gets.
As for the patching, I also suggested it a few weeks ago. Now that I know it is not done yet, I may try it myself.

About testing his fork. I thought I had to build something. Anyway, I cloned his repo and the branch that (supposedly) contains the patch (df-youtube-unthrottle-patch), run youtube-dl to download something and the throttling is still there

$ ./youtube-dl https://www.youtube.com/watch?v=C75GfC1y_nw
[youtube] C75GfC1y_nw: Downloading webpage
[youtube] C75GfC1y_nw: Downloading player 8da38e9a
[download] Destination: Anser_-_Anthropos_Idios_Official_video_clip_Prod._by_Eversor-C75GfC1y_nw.f313.webm
[download]   0.1% of 372.49MiB at 61.60KiB/s ETA 01:43:04^C
ERROR: Interrupted by user

As for the why I do not upgrade to a newer version of kodi. I have 5 devices that run kodi. Excluding my pc that runs kodi 19.3 under debian unstable x64, the remaining 4 all run *elec 9.2.x ones with kodi 18.9. These are

• an old x64 laptop that runs libreelec 9.2.6
• an rpi 3b+ that runs libreelec 9.2.8
• an old tvbox with s905 that runs coreelec legacy 9.2.8
• an old tvbox with s805 that runs libreelec-aml 9.2.8.x

I will upgrade the laptop to libreelec 10 once I sort out some playback issues it has on kodi 19. The os will be installed from scratch and it will have to be reconfigured from scratch, so it is a lot of time to dedicate to it.
The rpi will be upgraded to libreelec 10 (or newer) once the libreelec team announces a non-nightly version for it. So far, there is none and they admitted they have focused on rpi4, leaving the old models behind. Like the laptop, the os will also be installed and reconfigured from scratch.
As for the tvboxes, they can not be upgraded past those versions because their devs have ended support for them due to the lack of drivers of their socs for newer versions of the kernel etc and I totally respect that.

[nullket]

As for the patching, I also suggested it a few weeks ago. Now that I know it is not done yet, I may try it myself.

The suggestion is one part - someone doing it in their free time the other ;)

noumerous (almost daily) releases

That is actually all the updates and new features for youtube-dlp

and the throttling is still there

Out of curiosity I tried it myself and it works just fine. I assume whatever you were running was not the actual patched version.

git clone https://github.com/dirkf/youtube-dl.git
cd youtube_dl/
git checkout df-youtube-unthrottle-patch
python -m youtube_dl https://www.youtube.com/watch?v=C75GfC1y_nw
...
[download]  26.8% of 372.49MiB at  9.08MiB/s ETA 00:30

Anyways, at this point I would say we have spammed all others here in this issue enough with notification emails. Feel free to open a seperate youtube-dl legacy issue or even a PR with a solution.

[pitsi]

I am not an expert in git commands like you, so I downloaded the zip file from this branch here, extracted it to a folder, switched to the bin folder and then run youtube-dl from there.
https://github.com/dirkf/youtube-dl/tree/df-youtube-unthrottle-patch

Last offtopic of the day, sorry.

[nullket]

I have provided a workaround for python2 which was published just now. So all you need to do is to update the addon. In case of problems/comments please open a new issue.

#49