Randomly slow youtube download speed

[a26364177]

Hello,

Important: The problem is random, maybe 1 chance on 8 to produce it. You have to download several videos in a row (about 10) to notice it.

Since a few weeks, randomly a youtube video can be slowed down to 48 Kio/s, so it takes 5-10 minutes to download a short video of 5 minutes instead of 4, 5 seconds, often the download does not succeed and stops after a few minutes.

This happens on several servers, several internet providers as well as with my private connection.

base on this: ytdl-org/youtube-dl#29326

[redbrain]

I've experienced this with other ytdl libraries as well, such as the one linked above.
If they find a way to fix it, I propose we implement the same. Until then, I don't think there's much to do.

[gatecrasher777]

Been looking at this. What I have found is that when there is a slow download, and I cancel downloading and restart some seconds later it then downloads at full steam.

So what is needed is some kind of download speed threshold below which the download is cancelled and after a short pause restarted. But it also needs a counter, so that the number of restarts is limited to prevent an infinite restart loop.

In the thread https://github.com/ytdl-org/youtube-dl/issues/29326 there is the suggestion of manipulation of the n parameter, but if you look at the Dea function it is extremely obfuscated, much more so than the decipher methods. I would think you would have to execute the original code in base.js directly for this to be a solution.

Edit: Not easy to test. I used a threshold of 100KB/s. The problem seems to be completely random. Restarting immediately will result in another slow download. For most cases a 20-30 second delay before restarting the download seems to work. But I had one instance where it took over a minute before it would download at full speed. Still testing...

[hwgilbert16]

I’ve encountered the same issue as well in my development of youtube-archiver. I initially thought that the issue was isolated to a codec, and I eliminated the one problem child, but then I saw it randomly happened to any of them.

My best guess is it may be something on YouTube’s end, perhaps it finds something fishy with the request to googlevideo and ratelimits it to prevent a bot attack. The best mitigation for now it seems is to monitor the readable stream. If the completion rate is too slow, cancel the download and restart it.

[LuanRT]

[EDIT] This solution is no longer valid

This was giving me a lot of headaches a while ago. I found a way to mitigate it doe, not the most elegant way but it works fine for me. However, if you're on a slow internet connection you might end up in an infinite loop, so I suggest you to play around with the code a bit.

const downloadVideo = (link) => {
  const output_video = './temp/video.mp4';
  return new Promise(function (resolve, reject) {
    const start = () => {
      const stream = ytdl(link, {
        filter: (format) => format.container === "mp4",
        requestOptions: {
          headers: {
            cookie: YT_COOKIE
          },
        },
      });

      stream.pipe(fs.createWriteStream(output_video));

      stream.once("response", () => {
        starttime = Date.now();
      });

      stream.on("progress", (chunkLength, downloaded, total) => {
        const percent = downloaded / total;
        const downloaded_minutes = (Date.now() - starttime) / 1000 / 60;
        const estimated_download_time = downloaded_minutes / percent - downloaded_minutes;

        // if the estimated download time is more than 1.5 minutes then we cancel and restart the download, this value works fine for me but you may need to change it based on your server/internet speed.
        if (estimated_download_time.toFixed(2) >= 1.5) {
          console.warn("Seems like YouTube is limiting our download speed, restarting the download to mitigate the problem..");
          stream.destroy();
          start();          
        }
      });

      stream.on("end", () => {
        resolve(output_video);
      });

      stream.on("error", (error) => {
        reject(error);
      });
    };
    start();
  });
};

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[gatecrasher777]

To overcome this throttling problem, ytdl-core may have to run the n transform function in the base.js player code. I do this in my innertube library and do not experience any throttled downloads. The n transform is a lot more complex than the signature decipher function. I've seen other libraries try to emulate the n transform but it differs substantially from yt player to yt player, so this approach is very cumbersome and not very reliable. Another successful approach I have seen is to emulate an android phone, where this throttling is apparently not being applied (perhaps a reward for using Google's OS).

[PrestonN]

Hey there. Is there a chance of getting an update to this issue? I've been noticing this a lot more recently and seems to happen with most videos I try. I may try to make a patch myself if I have time however I imagine my progress would end up being slow compared to someone else who is more familiar with the issue / code base.

[LuanRT]

To overcome this throttling problem, ytdl-core may have to run the n transform function in the base.js player code. I do this in my innertube library and do not experience any throttled downloads. The n transform is a lot more complex than the signature decipher function. I've seen other libraries try to emulate the n transform but it differs substantially from yt player to yt player, so this approach is very cumbersome and not very reliable. Another successful approach I have seen is to emulate an android phone, where this throttling is apparently not being applied (perhaps a reward for using Google's OS).

I have also developed an Innertube based library, and your implementation helped me a lot while working on it, thanks! Plus I can confirm that yes, transforming the n value fixes the throttling problem.

[qwertfisch]

Another successful approach I have seen is to emulate an android phone, where this throttling is apparently not being applied

This seems to be the approach currently working on yt-dlp:

$ yt-dlp https://www.youtube.com/watch?v=f8illUOlHzo
[youtube] f8illUOlHzo: Downloading webpage
[youtube] f8illUOlHzo: Downloading android player API JSON

[VVD]

Both yt-dlp and youtube-dl are constantly slow last several days.

[gatecrasher777]

My understanding is that the yt-dlp android workaround only works on videos that can be embedded. It also requires a move to using the innertube api for getInfo.

[t0pology]

Running into the same problem :( consistent 50-60 kbps download speeds, even after setting a higher rate via the -r flag.

[antoniosap]

same here (italy, turin)

[gatecrasher777]

This problem used to be random, but now seems to be on every video I try without solving the n-transform challenge.

I have made a PR. Hope it can be accepted. The alternative to running the Google code directly is to attempt to emulate the n-parameter challenge code. But it is hugely more complex than the decipher code and way above my ability.

Looking around at other YT projects, those emulation attempts have generally worked for a few days and then failed with newer player versions.

Projects that are working around the problem are using the ANDROID fix, but then they are having to jump through some hoops to overcome the limitations of the ANDROID fix. And it seems almost inevitable to me that Google will close this bypass soon.

[LuanRT]

This problem used to be random, but now seems to be on every video I try without solving the n-transform challenge.

I have made a PR. Hope it can be accepted. The alternative to running the Google code directly is to attempt to emulate the n-parameter challenge code. But it is hugely more complex than the decipher code and way above my ability.

Looking around at other YT projects, those emulation attempts have generally worked for a few days and then failed with newer player versions.

Projects that are working around the problem are using the ANDROID fix, but then they are having to jump through some hoops to overcome the limitations of the ANDROID fix. And it seems almost inevitable to me that Google will close this bypass soon.

Yup,
Was testing ytdl here too and now every video is being throttled. To be honest executing the N transformation algorithm directly is the only way to fix it, as you said, emulating it won't work because it changes every time a new player is released.

Edit:
I have been analyzing its patterns and realized it is indeed possible to emulate the n token algorithm without executing any code directly. For those working on #1022, you can take a look at my implementation here (tested with 20 different players).

[thlytras]

Same here... Download speeds from youtube consistently at about 40 kb/s. :(

[kolt54321]

Here too. Would love a fix.

[LillieH1000]

YouTube/Google has made a change to make youtube-dl downloading incrediby slow, the only fix is the fork known as yt-dlp but that doesn't have a node version yet.

[trivita]

yt-dlp has no such issue
both youtube-dl and you-get are affected by this

[PeepoFrog]

same problem. Using youtube-dl with smtTube, cant watch any video in realtime

[Wisewolk]

Yup I have the same issue, the download speed never exceeds 82kB/s which is very problematic
I know that 2 weeks ago, I already have noticed that it was kinda slow, but when I stopped, waited 5-10 sec and then redid the downloading, it was going fast again. However, this technique is not working anymore..

[LillieH1000]

Yup I have the same issue, the download speed never exceeds 82kB/s which is very problematic

I know that 2 weeks ago, I already have noticed that it was kinda slow, but when I stopped, waited 5-10 sec and then redid the downloading, it was going fast again. However, this technique is not working anymore..

#964 (comment) You need to use yt-dlp now.

[VVD]

Yup I have the same issue, the download speed never exceeds 82kB/s which is very problematic
I know that 2 weeks ago, I already have noticed that it was kinda slow, but when I stopped, waited 5-10 sec and then redid the downloading, it was going fast again. However, this technique is not working anymore..

#964 (comment) You need to use yt-dlp now.

yt-dlp affected too. :-(

[hwgilbert16]

This is likely caused by an update to YouTube's API if it's affecting multiple interpretations of this.

[cstout1]

After updating yt-dlp to version 2021.10.10 my downloads processed very quickly.

[LillieH1000]

Yup I have the same issue, the download speed never exceeds 82kB/s which is very problematic
I know that 2 weeks ago, I already have noticed that it was kinda slow, but when I stopped, waited 5-10 sec and then redid the downloading, it was going fast again. However, this technique is not working anymore..

#964 (comment) You need to use yt-dlp now.

yt-dlp affected too. :-(

That means you are running an outdated yt-dlp, the latest yt-dlp has this issue fixed.

[VVD]

Yup I have the same issue, the download speed never exceeds 82kB/s which is very problematic
I know that 2 weeks ago, I already have noticed that it was kinda slow, but when I stopped, waited 5-10 sec and then redid the downloading, it was going fast again. However, this technique is not working anymore..

#964 (comment) You need to use yt-dlp now.

yt-dlp affected too. :-(

That means you are running an outdated yt-dlp, the latest yt-dlp has this issue fixed.

ytdl-org/youtube-dl#30102 (comment)

[Taposan0825]

Hello.
I am experiencing the same problem.
I think this problem is because YouTube has put a 480Kbps limit on data downloads.

[i0xHeX]

Hello. I am experiencing the same problem. I think this problem is because YouTube has put a 480Kbps limit on data downloads.

IDM downloads the same video much faster, but always with a constant rate (~700 KB/s), sometimes faster. My overall internet speed limit is >5 MB/s

[gatecrasher777]

IDM downloads the same video much faster, but always with a constant rate (~700 KB/s), sometimes faster.

That's because IDM has multithreaded downloads, so downloading 10 segments simultaneously = 10x70 KB/s =700 KB/s.

The innertube library ytcog and CLI ytcog-dl are unthrottled using the same n-parameter solution as proposed in PR #1022. Might help some people while they wait for a fix.

[Firsh]

I'm interested in what makes yt-dlp work, while we wait. It's a long shot but could it be that YouTube has multiple ways of delivery and our format choices correspond with a way they don't prefer? Like what if the speeds we get are because of connecting to an obscure origin datacenter and for files they turned off the CDN for. When visiting the website it's fast not because of multiple threads but because of a different format maybe. I'm guessing but maybe while youtube-dl prefers the separate video and audio file way/format, perhaps YT switched to m3u8 segmenting but the previous is still available just not "near you" on a CDN?

[Wisewolk]

I have noticed yesterday that it went from 50 kB/s to a around 8MB/s for a fragment of time. However it was kinda broken because everytime it was going fast, it seems like youtube-dl was crashing
My terminal was displaying this :
[download] 11.0% of ~82.69MiB at 8.52MiB/s ETA 00:58[download] Got server HTTP error: HTTP Error 404: Not Found. Retrying fragment 11 (attempt 1 of 10)...

Is this can help you understand what is going on or not?

PS : I have noticed that it was happening 2 weeks ago, before the "limitation" of 50kB/s was set
2nd PS : I also have noticed some 403 errors when i'm downloading yt stuff with youtube-dl but I can go through when I'm retrying the downloading

[drinkspiller]

Want to use/test this fix immediately before it is merged into a release? Run this to install the PR that addresses this issue:

npm i fent/node-ytdl-core#pull/1022/head
rm -rf node_modules && npm i

[stefankiebacher]

Same in Karlsruhe (Germany) between 50-80 KiB/s though 100MiB stable connection.
[download] 75.5% of 2.28MiB at 52.22KiB/s ETA 00:10

[itseyup]

Same in my United States server. between 50-80 KiB/s though 2000MiB stable connection.

[blackjyn]

same here , it stucks at 40-50KiB/s with 1000+MiB super stable connection.
I think it is a bug

[Sansekai]

now version 4.10.0 too slow

[Cherubrine]

Same for me since yesterday evening (Austria (Main IP) / Germany (ISP Proxies) / Netherlands (ISP Proxies))

[gatecrasher777]

This is a new but small obstacle from YouTube in their base player files.

PR #1055

[TimeForANinja]

yep, fixed with #1055
gonna freeze this now - so pls open a new issue when experiencing this issue again