Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[YouTube] Randomly slow youtube download speed #29326

Closed
triplesixman opened this issue Jun 17, 2021 · 342 comments · Fixed by yt-dlp/yt-dlp#1437
Closed

[YouTube] Randomly slow youtube download speed #29326

triplesixman opened this issue Jun 17, 2021 · 342 comments · Fixed by yt-dlp/yt-dlp#1437

Comments

@triplesixman
Copy link

triplesixman commented Jun 17, 2021

Checklist

  • [*] I'm reporting a broken site support
  • [*] I've verified that I'm running youtube-dl version 2021.06.06
  • [*] I've checked that all provided URLs are alive and playable in a browser
  • [*] I've checked that all URLs and arguments with special characters are properly quoted or escaped
  • [*] I've searched the bugtracker for similar issues including closed ones

Verbose log

root@server:~# youtube-dl https://youtu.be/8PecfdkEM2Y --source-address 64.31.22.34 --verbose
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'https://youtu.be/8PecfdkEM2Y', u'--source-address', u'64.31.22.34', u'--verbose']
WARNING: Assuming --restrict-filenames since file system encoding cannot encode all characters. Set the LC_ALL environment variable to fix this.
[debug] Encodings: locale ANSI_X3.4-1968, fs ANSI_X3.4-1968, out ANSI_X3.4-1968, pref ANSI_X3.4-1968
[debug] youtube-dl version 2021.06.06
[debug] Python version 2.7.13 (CPython) - Linux-4.9.0-15-amd64-x86_64-with-debian-9.13
[debug] exe versions: ffmpeg 4.1.2, ffprobe 4.1.2, phantomjs 2.1.1
[debug] Proxy map: {}
[youtube] 8PecfdkEM2Y: Downloading webpage
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on u'https://r2---sn-ab5l6n67.googlevideo.com/videoplayback?expire=1623953531&ei=GzzLYMyiKojn8wSQ-42YBA&ip=64.31.22.34&id=o-AMDSXCc14P0ndQaRCkihVxb1SwdClDAhUbq8xP8nD2ss&itag=313&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C271%2C278%2C313%2C394%2C395%2C396%2C397%2C398%2C399%2C400%2C401&source=youtube&requiressl=yes&mh=e_&mm=31%2C26&mn=sn-ab5l6n67%2Csn-vgqsrnee&ms=au%2Conr&mv=m&mvi=2&pl=24&initcwndbps=14958750&vprv=1&mime=video%2Fwebm&ns=gA0RPx_S5d5eK1Qi3zH6xdcF&gir=yes&clen=202533160&dur=299.160&lmt=1617981974189982&mt=1623931501&fvip=2&keepalive=yes&fexp=24001373%2C24007246&c=WEB&txp=5532432&n=0B2xjorVelV0Xu6Hq&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRQIhAI7W7dT0pYaOQgxn1mHYX3js6NByrqiykD9fsPJs3kAXAiApHiHXVdDMO1k6OKyg2sAb1PMyMO1jfgtZV5R-7frcpw%3D%3D&sig=AOq0QJ8wRgIhAPdD-seXFXT-yOEqoIqCQfPnRqMLASvU8SbymG5TPpNhAiEA1pVFpS3hDYqTVe1ia5sDOi9RaPf3BCuT94XB-vICq_E='
[download] Destination: Hatik_-_daron_des_demain_session_acoustique-8PecfdkEM2Y.f313.webm
[download]  18.4% of 193.15MiB at 77.46KiB/s ETA 34:44Terminated

While the download was in progress, I ran the exact same command in another terminal, another folder, and the download was completed in a few seconds:

root@server:~/test# youtube-dl https://youtu.be/8PecfdkEM2Y --source-address 64.31.22.34 --verbose
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'https://youtu.be/8PecfdkEM2Y', u'--source-address', u'64.31.22.34', u'--verbose']
WARNING: Assuming --restrict-filenames since file system encoding cannot encode all characters. Set the LC_ALL environment variable to fix this.
[debug] Encodings: locale ANSI_X3.4-1968, fs ANSI_X3.4-1968, out ANSI_X3.4-1968, pref ANSI_X3.4-1968
[debug] youtube-dl version 2021.06.06
[debug] Python version 2.7.13 (CPython) - Linux-4.9.0-15-amd64-x86_64-with-debian-9.13
[debug] exe versions: ffmpeg 4.1.2, ffprobe 4.1.2, phantomjs 2.1.1
[debug] Proxy map: {}
[youtube] 8PecfdkEM2Y: Downloading webpage
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on u'https://r2---sn-ab5l6n67.googlevideo.com/videoplayback?expire=1623953574&ei=RjzLYLyWBIHGhwak2IzwBg&ip=64.31.22.34&id=o-AFgzn9Zdn1KSGbuO09ZkpRma9GzWqWUApkavXXN93_F6&itag=313&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C271%2C278%2C313%2C394%2C395%2C396%2C397%2C398%2C399%2C400%2C401&source=youtube&requiressl=yes&mh=e_&mm=31%2C29&mn=sn-ab5l6n67%2Csn-ab5szne7&ms=au%2Crdu&mv=m&mvi=2&pl=24&initcwndbps=14958750&vprv=1&mime=video%2Fwebm&ns=hoSqVT_3ust7ILej5iYoT40F&gir=yes&clen=202533160&dur=299.160&lmt=1617981974189982&mt=1623931501&fvip=2&keepalive=yes&fexp=24001373%2C24007246&c=WEB&txp=5532432&n=l6PFqwM4uREk1JKwP&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRQIhAJU5426qtqf6BwLiB48OKkcK_ATe_S9jDPYAVbttM7T1AiBoVGwb1ZBagaiUyKeVGLv562cloZeh5xBT2lFZx61gyQ%3D%3D&sig=AOq0QJ8wRAIgWSAyj1JyqoTHFWMdJ04gjcIDJ8tFryw5sNsf5soVaPQCIE81_29FoDoAOeRv2_hdcnBi2-4XxvlGvbX9YNajLi6y'
[download] Destination: Hatik_-_daron_des_demain_session_acoustique-8PecfdkEM2Y.f313.webm
[download] 100% of 193.15MiB in 00:04
[debug] Invoking downloader on u'https://r2---sn-ab5l6n67.googlevideo.com/videoplayback?expire=1623953574&ei=RjzLYLyWBIHGhwak2IzwBg&ip=64.31.22.34&id=o-AFgzn9Zdn1KSGbuO09ZkpRma9GzWqWUApkavXXN93_F6&itag=251&source=youtube&requiressl=yes&mh=e_&mm=31%2C29&mn=sn-ab5l6n67%2Csn-ab5szne7&ms=au%2Crdu&mv=m&mvi=2&pl=24&initcwndbps=14958750&vprv=1&mime=audio%2Fwebm&ns=hoSqVT_3ust7ILej5iYoT40F&gir=yes&clen=5324207&dur=299.201&lmt=1617980855722369&mt=1623931501&fvip=2&keepalive=yes&fexp=24001373%2C24007246&c=WEB&txp=5531432&n=l6PFqwM4uREk1JKwP&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAIqCBWf7PmHH8y1wnG8QvB-0vxKzRG26qCAIWdgOAT1PAiEAwTSl4J0e9L7emiYUDKV_YjfApo2gchge3iVfrYH76lo%3D&sig=AOq0QJ8wRQIgODhcEL0uT0u1nXP41IARrB63CfmpDmUzl6HhPwrXOTwCIQD-OUE152N7yzYXcgU_tAPCP0YdRdfVyFlHE4kIYyyoew=='
[download] Destination: Hatik_-_daron_des_demain_session_acoustique-8PecfdkEM2Y.f251.webm
[download] 100% of 5.08MiB in 00:00
[ffmpeg] Merging formats into "Hatik_-_daron_des_demain_session_acoustique-8PecfdkEM2Y.webm"
[debug] ffmpeg command line: ffmpeg -y -loglevel 'repeat+info' -i 'file:Hatik_-_daron_des_demain_session_acoustique-8PecfdkEM2Y.f313.webm' -i 'file:Hatik_-_daron_des_demain_session_acoustique-8PecfdkEM2Y.f251.webm' -c copy -map '0:v:0' -map '1:a:0' 'file:Hatik_-_daron_des_demain_session_acoustique-8PecfdkEM2Y.temp.webm'
Deleting original file Hatik_-_daron_des_demain_session_acoustique-8PecfdkEM2Y.f313.webm (pass -k to keep)
Deleting original file Hatik_-_daron_des_demain_session_acoustique-8PecfdkEM2Y.f251.webm (pass -k to keep)

Description

Hello,

Important: The problem is random, maybe 1 chance on 8 to produce it. You have to download several videos in a row (about 10) to notice it.

Since a few weeks, randomly a youtube video can be slowed down to 48 Kio/s, so it takes 5-10 minutes to download a short video of 5 minutes instead of 4, 5 seconds, often the download does not succeed and stops after a few minutes.

This happens on several servers, several internet providers as well as with my private connection.

I even managed to launch a second download in parallel when the first one was taking time, the second one finished in 1 seconds, the first one in 5 minutes and was interrupted before the end. Same video, same connection, same command. (tested only with ipv4 because I don't have ipv6 on my servers or with my internet connection).

Attached is an extraction of the results with --dump-pages.
ko.txt
ok.txt

@inkuxuan
Copy link

I'm having exactly the same issue.

@shimiaoyerin
Copy link

我也注意到了,我一直是配合aria2使用的,只是也和你一样的问题,希望更多人提供有帮助的信息

@joro1
Copy link

joro1 commented Jun 18, 2021

Same issue, i.e. also noticing downloads are much slower than "normal", even short videos taking extremely long times.

@JJ840
Copy link

JJ840 commented Jun 18, 2021

Same issue, restarting the download keeps it working fine for a while but then it happens again. I would think throttling perhaps, but I have Starlink and they sure as shit don't throttle. Perhaps it's something on youtube's end?

edit: Curiously, I don't think it happens after I switched on a vpn to test it...

@neural-nut
Copy link

same here, this is driving me out of my mind

@srett
Copy link

srett commented Jun 18, 2021

Same issue here. When this happens, speed is like in OP's case around 77kb/s

From the 2018 ticket #15271 I found that --http-chunk-size 10M could mitigate the issue, but it doesn't help in this case; either no chunking happens, or the chunking doesn't fix it. No idea how to verify that it actually does what it's supposed to do.

@Lesmiscore

This comment has been minimized.

@triplesixman
Copy link
Author

triplesixman commented Jun 19, 2021

I sometimes face same issue.

I think either YouTube or your internet provider is throttling connection between YT and youtube-dl.
In addition, YouTube may transcode video while serving. I guess this is likely the cause if no one does throttling. (see https://blog.youtube/inside-youtube/new-era-video-infrastructure )

For both cases, youtube-dl can't do anything to fix.

Are you saying that youtube-dl will be obsolete with youtube.com because of this update when it spreads to all videos?

Honestly I have my doubts. I think it's more of a bug or maybe related to this update, but I imagine (and hope) that the contributors of youtube-dl will be able to find a solution, as they did in the last 15 years of existence.

@srett
Copy link

srett commented Jun 19, 2021

@nao20010128nao What I found suspicious is the download speed - for me - always being around 77kb/s if this happens. This is way to slow even for playback, and it happens independently of the format. f137 takes forever like this. If this was purely a problem on youtube's side, I'd expect to hit this problem in the browser too at times, but I never experience buffering.
My hunch would be some change on youtube's side, either accidental or nefarious, that leads to this problem.

@demget
Copy link

demget commented Jun 19, 2021

The temporal inaccurate workaround which can help in dealing with automatic downloads:

def speed_check(s):
    speed = s.get('speed')
    ready = s.get('downloaded_bytes', 0)
    total = s.get('total_bytes', 0)
    
    if speed and speed <= 77 * 1024 and ready >= total * 0.1:
        # if the speed is less than 77 kb/s and we have 
        # at least one tenths of the video downloaded
        raise DownloadError('Abnormal downloading speed drop.')

ydl.add_progress_hook(speed_check)

@nikooo777
Copy link

The temporal inaccurate workaround which can help in dealing with automatic downloads:

def speed_check(s):
    speed = s.get('speed')
    ready = s.get('downloaded_bytes', 0)
    total = s.get('total_bytes', 0)
    
    if speed and speed <= 77 * 1024 and ready >= total * 0.1:
        # if the speed is less than 77 kb/s and we have 
        # at least one tenths of the video downloaded
        raise DownloadError('Abnormal downloading speed drop.')

ydl.add_progress_hook(speed_check)

would you then retry the video until it's fast?

@demget
Copy link

demget commented Jun 20, 2021

@nikooo777 I have some kind of proxy in front of the Python downloader and it decides about retrying or not, so no need in doing it here, though it really makes sense for someone.

@danny-wu
Copy link

danny-wu commented Jun 21, 2021

I can confirm I am also experiencing this, since July 14th. I am either noticing ~77kb/s, or ~50kb/s.

The throttling is consistent across all of my servers and seems to only occur if I download a lot of videos; making me think it is an intentional form of throttling on youtube's side.

I have increased the timeouts and reduced the frequency of my throttling, and I am noticing far less throttles. If you are hitting this, I recommend you increase your timeouts.

@triplesixman
Copy link
Author

@danny-wu In my case, this happens even during the first download on my private internet connection. Also noticed on a linux server.

@Tinase-nau
Copy link

Tinase-nau commented Jun 21, 2021

I think YT may be throttling for profit reasons and/or they may be having technical difficulties and/or they are upgrading and/or censorship reasons.
A few days ago some videos at YT started buffering. Videos were allowed to be played at quality between 144p - 360p. But not all videos have the issue. Then I thought to try downloading the videos with YT-DL: "Blocked Drain 506" downloading at 40.91KiB/s, with several "Got server HTTP error: [WinError 10054] An existing connection was forcibly closed by the remote host." messages.
Ofc the download script downloads the highest possible quality. The throttling starts from 720p. At 480p I could actually watch the video without buffering.

@JJ840
Copy link

JJ840 commented Jun 21, 2021

From what I understand, if youtube is throttling downloads aria2c might help (https://wiki.archlinux.org/title/Youtube-dl#Faster_downloads)? I have absolutely no experience with it (if someone does and could link something that would be greatly appreciated) but I think it might be worth a try if someone here does have it set up.

@petaire92
Copy link

yep ! same here ! it works flawlessly on macOS but the build on Ubuntu gets slow randomly, also around 76/77kbps...

@danny-wu
Copy link

danny-wu commented Jun 21, 2021 via email

@Tinase-nau
Copy link

Win10 x64 here.
But I would like to know if really MacOS is unaffected?

@alexmerkel
Copy link
Contributor

But I would like to know if really MacOS is unaffected?

Nope, macOS 11 here and I have the same issue!

@petaire92
Copy link

Win10 x64 here.
But I would like to know if really MacOS is unaffected?

Well I have a big sur build that works perfectly with multiple dl per days and works flawlessly. But maybe it’s because those are newly added videos ?

@Tinase-nau
Copy link

Win10 x64 here.
But I would like to know if really MacOS is unaffected?

Well I have a big sur build that works perfectly with multiple dl per days and works flawlessly. But maybe it’s because those are newly added videos ?

Seems so. I don't notice buffering or quality loss on new videos. Buffering/throttling on several months old videos only. Maybe they introduced some new encoding codec and are reencoding old videos and new videos are already good? Maybe so.

@oneandonlyjason
Copy link

Seems like the Same Problem here on a Debian 10 Machine with youtube-dl. I can download a few Video normal and then it gets so slow that there is no noticable Network Activity.

@rubiksw
Copy link

rubiksw commented Jun 21, 2021

Same issue here on Linux.

@on1razor
Copy link

I have a similar problem, I thought that youtube was blocking me by ip, so I started uploading via vpn, but the problem repeats

@danny-wu
Copy link

There were helpful comments on this ticket from a user with initials A.W.; unfortunately it looks like they got deleted. I saw them because I have email notifications.

Has anyone made further progress on the query parameter investigation?

@danny-wu
Copy link

danny-wu commented Jun 22, 2021 via email

zarino added a commit to zarino/dotfiles that referenced this issue Dec 30, 2021
youtube-dl is larely dormant, with only infrequent updates
(https://news.ycombinator.com/item?id=29590501). In particular,
youtube-dl *still* hasn’t fixed a six month old bug that causes YouTube
downloads to be dramatically throttled, rendering it largely useless
(ytdl-org/youtube-dl#29326).

yt-dlp is youtube-dl’s spritual successor and appears better maintained
at the moment (ytdl-org/youtube-dl#30317).
xdanger added a commit to xdanger/dotfiles that referenced this issue Jan 9, 2022
@dzg
Copy link

dzg commented Jan 27, 2022

macOS here, on version 2021.12.17, and YouTube downloads are extremely slow, <100kbps

@ewtoombs
Copy link

Why is this closed? This is super obviously still a problem.

@ewtoombs
Copy link

Here's my fix: #28859 (comment)

@fireattack
Copy link

It was fixed long time ago. Just use the master source code. And it (together with fix) has been iterated so many times afterwards that your fix probably isn't relevant anyway.

If you still have issue using the master branch, please open a new issue and detail instead of necroing old post.

@dirkf
Copy link
Contributor

dirkf commented Jun 16, 2022

#28859 (comment)

@ewtoombs
Copy link

ewtoombs commented Jun 19, 2022

It was fixed long time ago. Just use the master source code. And it (together with fix) has been iterated so many times afterwards that your fix probably isn't relevant anyway.

If you still have issue using the master branch, please open a new issue and detail instead of necroing old post.

Well, you should really have read my fix, because it is not a diff to this project. It is a self-contained example that demonstrates the method, which is quite different from the one youtube-dl actually adopted.

@ewtoombs
Copy link

ewtoombs commented Jun 19, 2022

And I will necro the hell out of any "closed" issue, for as long as it isn't actually closed. All I saw was one failed PR. I think it was fair to me to assume, then, that this issue was marked as closed spuriously. Now, I see that is not the case, but I stand by my methods.

Anyway, thanks for integrating a fix. You are too late, though. I already made my own client, and I like it better, lololoлололо:trollface:لُلُلُلُلُلُلُلُلٌ. I can use it to grep youtube comments and search results.

@leohub2016
Copy link

Problem happened again, with yt-dlp, for youtube and twitch. Does it happen for someone else?

@dirkf
Copy link
Contributor

dirkf commented Jun 22, 2022

  1. yt-dlp => https://github.com/yt-dlp/yt-dlp/issues
  2. If the issue affects twitch it won't be the same problem
  3. If it affects two unrelated sites it's probably something to do with your network access.

@ytdl-org ytdl-org locked as resolved and limited conversation to collaborators Jun 22, 2022
dirkf added a commit to dirkf/youtube-dl that referenced this issue Jun 27, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet