Having issues with the Cipher class, but not with any specific implementation the app uses

[danelynnn]

So I've been tinkering around with the Cipher class and how it processes stream data with 403 links, and while I am not smart enough to actually learn it, I have been trying to use it on a couple streamingData dictionaries I've found myself, outside of the library, through the WEB client. For some reason, running the same functions as found in apply_signature doesn't actually make the links valid, and not 403 (it's actually changing the 'n' parameter, but it's still invalid). The same links produced and processed within the library are actually working, though, even without poToken support.

My previous setup was using the iOS client, which doesn't need to go through all this hassle, as those links don't actually produce 403's, but I am wanting to fight through using the WEB client and to figure it out, but no luck. How could I go about debugging this?

[danelynnn]

I've noticed a lot of the Cipher class and setup was written by @felipeucelli; do you happen to be able to help me out with this? any help is greatly appreciated :]

[felipeucelli]

Hello @danelynnn

The Cipher class works to find the signature functions of the 'n' parameter and the siganture cipher in the base.js JavaScript. When found correctly, it sends the name of the function and the parameter to be decoded to JsInterp, which will interpret the JavaScript function and return the decoded value.

Previously, the lack of the 'n' parameter only caused a limit of 30Kbps on the download rate, but after the addition of PoToken it also became necessary.

To obtain a functional stream, in addition to decoding these parameters, you also need to send the base.js signatureTimestamp value to the API request. If not sent, streams will only be valid for a few seconds before returning a 403 error.

Clients currently requiring this decryption are: WEB, WEB_EMBED, WEB_MUSIC, WEB_CREATOR, WEB_SAFARI, MWEB and TV_EMBED.

YouTube changes base.js on average every 4 days. So you need to use the correct base.js to get the stream.

Each stream has an expiration timestamp, that is, they expire after a certain time, requiring a new stream to be requested.

PoToken clients are those that need the proof of origin token to work. If this token is not passed, only the itag 18 stream will work. Currently the PoToken clients are: WEB and ANDROID.

If you want to use a streamData from outside pytubefix, you need to make sure it is not expired, the version of base.js, whether the signatureTimestamp was sent in the request and the correct decryption in the 'n' parameter and the siganture cipher.

I don't recommend using the WEB client to obtain streams because of the need for PoToken.

[danelynnn]

That all makes a lot of sense, thank you so much for getting back to me. From what I've gathered, it does seem like I have used the Cipher class correctly, but there are a couple other things you've pointed out that I haven't done (this signatureTimestamp value, included). I can figure out how that works.

That poToken clarification helps so much as well, mostly because I think I actually was only confirming the success of the itag 18 non-adaptive format, and for my use case I am actually wanting to use one of the adaptive formats to more easily isolate audio.

I do appreciate so much of the help, and if you don't mind, I might have some more questions that I might continue in this thread.

Regarding your last note, I do understand, and I have actually gotten a working version of all of this that uses the iOS client, seemingly the only one that seems reliable, but I'm honestly a bit worried about it also failing, as the ANDROID_TESTSUITE client has recently. I'm figuring that going through all this obfuscation is actually a really good idea, and if it works, it could be a bit more reliable for the future.

[felipeucelli]

YouTube has been working tirelessly to block third-party apps, so I think it's only a matter of time before we lose a lot of clients and the ones that remain will need PoToken.

I would like to implement PoToken generation natively in pytubefix, but it is difficult work considering that DOM manipulation is necessary.

If you're interested, I found this project that interacts with the botguard API and can generate the PoToken correctly: https://github.com/LuanRT/BgUtils

[danelynnn]

Yeah, that's kinda what I've been seeing. It's why I'm trying to actually get the more obfuscated clients to work, because it feels like they've already put a lot of effort into those so it seems safer, but maybe that's just me.

I've also seen, and I think you've mentioned this somewhere, but https://www.npmjs.com/package/youtube-po-token-generator. It's in node, which makes it that much harder to implement in a Python app, but it was my initial plan with regards to generating these.

Say, would you mind if I linked my personal project repo and the methods I have, and get some help regarding what I've been missing? I'm still a bit stuck w/ regards to the WEB implementation, and I am trying to get it worked out but I'm still very slow as I've been mostly having to comb through most of the myriad of functions within the pytubefix repo.

[felipeucelli]

I don't know if I understand exactly what you meant. But if you want help understanding how WEB client stream generation works, yes of course, I'll help you

[danelynnn]

Thanks a lot! Will do, I'll see what I can do and return back with questions!