TikTok: youtube-dl hangs after "Requesting Header"

[hessijames79]

• I'm reporting a broken site support issue
• I've verified that I'm running youtube-dl version 2021.12.17
• I've checked that all provided URLs are alive and playable in a browser
• I've checked that all URLs and arguments with special characters are properly quoted or escaped
• I've searched the bugtracker for similar bug reports including closed ones
• I've read bugs section in FAQ

Verbose log

$ youtube-dl -v 'https://vm.tiktok.com/ZMLesneqK/'
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', 'https://vm.tiktok.com/ZMLesneqK/']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.9.2 (CPython) - Linux-5.10.0-10-amd64-x86_64-with-glibc2.31
[debug] exe versions: ffmpeg 4.4, ffprobe 4.4, rtmpdump 2.4
[debug] Proxy map: {}
[generic] ZMLesneqK: Requesting header

Description

After Requesting header, youtube-dl hangs forever.

[dirkf]

This looks like a wider occurrence of this noted in PR #30479:

Also, extraction could fail with a timeout (Error 60 in Windows, SSLError('The read operation timed out',) in Linux) or connection reset (Error 54 in Windows) due to some weird blocking by whatever fronts TikTok's pages (Akamai, apparently).

OP's URL isn't known to the TT extractor, so the error occurs in the generic extractor. However, even if the TT extractor does handle the URL the same error occurs.

More research needed.

The tool illustrated (now) below uses an API that doesn't have this problem. If, as with some YT APIs, this API can't currently be changed because of unmodifiable deployed clients, the API may be a safer target.

[dirkf]

As to the headers:

• yt-dl picks a random UA for each session, but it can be overridden; as detailed UAs are being deprecated yt-dl should eventually just send something like Mozilla/5.0 (platform stuff) (like Gecko), but the time hasn't come yet;
• the Accept header is probably unnecessary, as is Accept-Charset;
• the Cookie comes from the 302 Redirect response to GET https://vm.tiktok.com/ZMLesneqK/, and actually is the reason why the yt-dl TT extractor makes a dummy request before trying to fetch the TT page itself, as TT gave 403 otherwise;
• the others seem to be reasonable, and none are incorrect according to protocol.

This piece of utils.py is relevant:

std_headers = {
    'User-Agent': random_user_agent(),
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Encoding': 'gzip, deflate',
    'Accept-Language': 'en-us,en;q=0.5',
}

I wasn't been able to break through the block even by clearing headers deep in the Python HTTP routines, whereas apparently similar requests formulated with curl or wget are OK. Presumably whatever MITM Proxy does also passes the block.

yt-dlp uses the aweme API.

[hessijames79]

Hi!

Apparently, changing the user agent is the only change needed to fix it:

youtube-dl -v --user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0" "https://vm.tiktok.com/ZMLesneqK/"

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', '--user-agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0', 'https://vm.tiktok.com/ZMLesneqK/']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.9.2 (CPython) - Linux-5.10.0-10-amd64-x86_64-with-glibc2.31
[debug] exe versions: ffmpeg 4.4, ffprobe 4.4, rtmpdump 2.4
[debug] Proxy map: {}
[generic] ZMLesneqK: Requesting header
[redirect] Following redirect to https://www.tiktok.com/@eddysayi/video/7054218882072055046?_d=secCgwIARCbDRjEFSACKAESPgo8Ny%2BRiWmH3OT6VFDDOKQe%2FjdU6QNg2QvER6kPZv8%2BV8iBx6SaSGyiLedX5PcIDqwq0EOwLe84aH6c7rDnGgA%3D&checksum=f558941cd65da22b6359be076ef87f6ffe1d4188d600d52331e2263cc6baa1a8&language=de&preview_pb=0&sec_user_id=MS4wLjABAAAAOkEM5UFx0tM8hhBkjW8D1xAadF1RU12kEXRBL0Kub2pz70wmTYEXuYh8MdtS57Vj&share_app_id=1233&share_item_id=7054218882072055046&share_link_id=EA78C111-0165-44E0-9A7F-D0FC02C4B110&source=h5_m&timestamp=1642491289&tt_from=copy&u_code=dh6jgh57j8hf20&user_id=6931555450693420037&utm_campaign=client_share&utm_medium=ios&utm_source=copy&_r=1
[TikTok] 7054218882072055046: Downloading webpage
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'https://v16-webapp.tiktok.com/1ffb32527bc5dff792ca125bb76ac0f8/61e84c1c/video/tos/useast2a/tos-useast2a-ve-0068c004/5275f8639ba84d90999c6371ae527141/?a=1988&br=1370&bt=685&cd=0%7C0%7C0&ch=0&cr=0&cs=0&dr=0&ds=2&er=&ft=Yu12_F5qkag3-I&l=202201191136110101920580770DA82470&lr=tiktok_m&mime_type=video_mp4&net=0&pl=0&qs=0&rc=M3A8ODs6ZnhzOjMzNzczM0ApO2c4PDs2Zzs0NzdlNzRnZmdqM2gucjRnb2ZgLS1kMTZzc19iMzQzLV5gY182XmNgLV46Yw%3D%3D&vl=&vr='
[download] EddY-7054218882072055046.mp4 has already been downloaded

[download] 100% of 1.49MiB

Patrick

[dirkf]

This works fine for me:

Thanks, I confirm that for Py 2.7.17/3.5.5/3.9.7, so some junk header must be confusing things.

[Menard01]

To download tiktok videos, you have only to right click and "Download the video" #noobs
(No need a youtube-dl here)

[dirkf]

Yes, but you already told us that, and the information was noted, and then your post magically vanished...

[fuzzy76]

FWIW, I see this failing with "Unable to extract data" after adding the user-agent.

ERROR: Unable to extract data; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/local/Cellar/youtube-dl/2021.12.17/libexec/lib/python3.10/site-packages/youtube_dl/YoutubeDL.py", line 815, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/Cellar/youtube-dl/2021.12.17/libexec/lib/python3.10/site-packages/youtube_dl/YoutubeDL.py", line 836, in __extract_info
    ie_result = ie.extract(url)
  File "/usr/local/Cellar/youtube-dl/2021.12.17/libexec/lib/python3.10/site-packages/youtube_dl/extractor/common.py", line 534, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/Cellar/youtube-dl/2021.12.17/libexec/lib/python3.10/site-packages/youtube_dl/extractor/tiktok.py", line 110, in _real_extract
    page_props = self._parse_json(self._search_regex(
  File "/usr/local/Cellar/youtube-dl/2021.12.17/libexec/lib/python3.10/site-packages/youtube_dl/extractor/common.py", line 1012, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
youtube_dl.utils.RegexNotFoundError: Unable to extract data; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

[fuzzy76]

@89z the one in the issue description.

fuzzy76@laptop ~ % youtube-dl -v --user-agent 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0' 'https://vm.tiktok.com/ZMLesneqK/'
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', '--user-agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0', 'https://vm.tiktok.com/ZMLesneqK/']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.10.2 (CPython) - macOS-12.3.1-x86_64-i386-64bit
[debug] exe versions: ffmpeg 5.0.1, ffprobe 5.0.1, rtmpdump 2.4
[debug] Proxy map: {}
[generic] ZMLesneqK: Requesting header
[redirect] Following redirect to https://www.tiktok.com/@eddysayi/video/7054218882072055046?_d=secCgwIARCbDRjEFSACKAESPgo8Ny%2BRiWmH3OT6VFDDOKQe%2FjdU6QNg2QvER6kPZv8%2BV8iBx6SaSGyiLedX5PcIDqwq0EOwLe84aH6c7rDnGgA%3D&checksum=f558941cd65da22b6359be076ef87f6ffe1d4188d600d52331e2263cc6baa1a8&language=de&preview_pb=0&sec_user_id=MS4wLjABAAAAOkEM5UFx0tM8hhBkjW8D1xAadF1RU12kEXRBL0Kub2pz70wmTYEXuYh8MdtS57Vj&share_app_id=1233&share_item_id=7054218882072055046&share_link_id=EA78C111-0165-44E0-9A7F-D0FC02C4B110&source=h5_m&timestamp=1642491289&tt_from=copy&u_code=dh6jgh57j8hf20&user_id=6931555450693420037&utm_campaign=client_share&utm_medium=ios&utm_source=copy&_r=1
[TikTok] Setting up session
[TikTok] 7054218882072055046: Downloading webpage
ERROR: Unable to extract data; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/local/Cellar/youtube-dl/2021.12.17/libexec/lib/python3.10/site-packages/youtube_dl/YoutubeDL.py", line 815, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/Cellar/youtube-dl/2021.12.17/libexec/lib/python3.10/site-packages/youtube_dl/YoutubeDL.py", line 836, in __extract_info
    ie_result = ie.extract(url)
  File "/usr/local/Cellar/youtube-dl/2021.12.17/libexec/lib/python3.10/site-packages/youtube_dl/extractor/common.py", line 534, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/Cellar/youtube-dl/2021.12.17/libexec/lib/python3.10/site-packages/youtube_dl/extractor/tiktok.py", line 110, in _real_extract
    page_props = self._parse_json(self._search_regex(
  File "/usr/local/Cellar/youtube-dl/2021.12.17/libexec/lib/python3.10/site-packages/youtube_dl/extractor/common.py", line 1012, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
youtube_dl.utils.RegexNotFoundError: Unable to extract data; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
fuzzy76@laptop ~ %