[Loom] Add new extractor #28039

wongyiuhang · 2021-02-01T08:05:27Z

Please follow the guide below

You will be asked some questions, please read them carefully and answer honestly
Put an x into all the boxes [ ] relevant to your pull request (like that [x])
Use Preview tab to see how your pull request will actually look like

Before submitting a pull request make sure you have:

Searched the bugtracker for similar pull requests
Read adding new extractor tutorial
Read youtube-dl coding conventions and adjusted the code to meet them
Covered the code with tests (note that PRs without tests will be REJECTED)
Checked the code with flake8

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

I am the original author of this code and I am willing to release it under Unlicense
I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

Bug fix
Improvement
New extractor
New feature

Description of your pull request and other information

In response to a site request #27957, this new extractor is written for loom.com.

Closes #27957

… test/test_unicode_literals.py

dstftw · 2021-02-13T22:38:29Z

youtube_dl/extractor/loom.py

+    def _extract_video_info_json(self, webpage, video_id):
+        info = self._html_search_regex(
+            r'window.loomSSRVideo = (.+?);',
+            webpage,
+            'info')
+        return self._parse_json(info, 'json', js_to_json)
+
+    def _get_url_by_id_type(self, video_id, type):
+        request = compat_urllib_request.Request(
+            self._BASE_URL + 'api/campaigns/sessions/' + video_id + '/' + type,
+            {})
+        json_doc = self._download_json(request, video_id)
+        return (url_or_none(json_doc.get('url')), json_doc.get('part_credentials'))


Updated at 34e6a6b

dstftw · 2021-02-13T22:38:45Z

youtube_dl/extractor/loom.py

+        request = compat_urllib_request.Request(
+            self._BASE_URL + 'api/campaigns/sessions/' + video_id + '/' + type,
+            {})


Move into _download_json.

Updated at 70b8045

dstftw · 2021-02-13T22:39:23Z

youtube_dl/extractor/loom.py

+    def _get_m3u8_formats(self, url, video_id, credentials):
+        format_list = self._extract_m3u8_formats(url, video_id)
+        for item in format_list:
+            item['protocol'] = 'm3u8_native'
+            item['url'] += '?' + credentials
+            item['ext'] = 'mp4'
+            item['format_id'] = 'hls-' + str(item.get('height', 0))
+            item['extra_param_to_segment_url'] = credentials
+        return format_list


Updated at 34e6a6b

dstftw · 2021-02-13T22:39:40Z

youtube_dl/extractor/loom.py

+            ext = self._search_regex(
+                r'\.([a-zA-Z0-9]+)\?',
+                url, 'ext', default=None)
+            if(ext != 'm3u8'):


Updated at 34e6a6b

dstftw · 2021-02-13T22:39:51Z

youtube_dl/extractor/loom.py

+            ext = self._search_regex(
+                r'\.([a-zA-Z0-9]+)\?',
+                url, 'ext', default=None)


Read coding conventions.

For this part, I may need to extract the file extension from a url.

Would you prefer a relaxed regex \.([^.?]+)\??

Or HEAD [URL] and extract the extension from content-type header with mimetype2ext(mt)?

dstftw · 2021-02-13T22:40:09Z

youtube_dl/extractor/loom.py

+                    'width': try_get(info, lambda x: x['video_properties']['width']),
+                    'height': try_get(info, lambda x: x['video_properties']['height'])


int_or_none

Updated at 29c4168

dstftw · 2021-02-13T22:40:29Z

youtube_dl/extractor/loom.py

+
+        return {
+            'id': info.get('id'),
+            'title': info.get('name'),


Mandatory. Read coding conventions.

For the id, I may provide a fallback value from the url. However, the title does not have another fallback source, other than the embedded JSON.

Any advice?

Afterthoughts:
Is that okay if use use [video_id] or the word Loom as the fallback title?

dstftw · 2021-02-13T22:40:59Z

youtube_dl/extractor/loom.py

+
+        for i in range(len(folder_info['entries'])):
+            video_id = folder_info['entries'][i]
+            folder_info['entries'][i] = LoomIE(self._downloader)._real_extract(url_or_none(self._BASE_URL + 'share/' + video_id))


url_result.

Updated at 1b2651e

Removed if statement parentheses

wongyiuhang · 2021-02-24T20:02:07Z

youtube_dl/extractor/loom.py

+
+            ext = self._search_regex(
+                r'\.([a-zA-Z0-9]+)\?',
+                url, 'ext', default=None)
+            if ext != 'm3u8':
+                formats.append({
+                    'url': url,
+                    'ext': ext,
+                    'format_id': type,
+                    'width': int_or_none(try_get(info, lambda x: x['video_properties']['width'])),
+                    'height': int_or_none(try_get(info, lambda x: x['video_properties']['height']))
+                })
+            else:
+                credentials = compat_urllib_parse_urlencode(part_credentials)
+                m3u8_formats = self._extract_m3u8_formats(url, video_id)
+                for item in m3u8_formats:
+                    item['protocol'] = 'm3u8_native'
+                    item['url'] += '?' + credentials
+                    item['ext'] = 'mp4'
+                    item['format_id'] = 'hls-' + str(item.get('height', 0))
+                    item['extra_param_to_segment_url'] = credentials
+                for i in range(len(m3u8_formats)):
+                    formats.insert(
+                        (-1, len(formats))[i == len(m3u8_formats) - 1],
+                        m3u8_formats[i])


octet-stream support required
#27957 (comment)

Fred-Vatin · 2021-03-18T20:01:45Z

This has been already merged ?

I try to download a video here. When I try with the MPD link I can detect, YTDL returns a 403 error.

Are you able to download it ?

wongyiuhang · 2021-03-19T04:45:15Z

This has been already merged ?

I try to download a video here. When I try with the MPD link I can detect, YTDL returns a 403 error.

Are you able to download it ?

This PR has not completed the code reviewing yet. Also, I have yet implemented the *.mpd support yet...🙈

rememberlenny · 2021-09-24T19:09:16Z

@dstftw or @wongyiuhang can I help move this along?

wongyiuhang · 2021-09-25T06:21:42Z

@dstftw or @wongyiuhang can I help move this along?

Yes, of course. I'm sorry for holding the pull request. Is there anything that I need to do? 👀

cellarpub · 2021-11-06T07:34:10Z

@wongyiuhang since the PR has not yet been merged I have tried to git clone your repo and checkout to 'loom' branch. Then I have installed with 'pip3 install -e .' but downloading the link above does not work.
I am getting the following error:

ERROR: Unsupported URL: https://www.loom.com/share/384b27b953714dc19aba4768643038bd

This is my youtube-dl -v:

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.02.10
[debug] Git HEAD: 360d5f0da
[debug] Python version 3.9.7 (CPython) - Linux-5.15.0-1-MANJARO-x86_64-with-glibc2.33
[debug] exe versions: ffmpeg 4.4, ffprobe 4.4, rtmpdump 2.4
[debug] Proxy map: {}
Usage: youtube-dl [OPTIONS] URL [URL...]

mysticaltech · 2022-01-31T21:27:20Z

Hey folks, you were almost there!

coolbaluk · 2022-02-02T16:28:22Z

I suggest an optimisation, which is to use the transcoded_url if it exits, and then the raw_url if not and then do the hls/dash stitching.

Here's how to the *.mpd @wongyiuhang

            if ext == 'mpd':
                credentials = compat_urllib_parse_urlencode(part_credentials)
                mpd_formats = self._extract_mpd_formats(
                    url, video_id)
                for item in mpd_formats:
                    for f in item['fragments']:
                        f['path'] += '?' + credentials
                        self.to_screen(f)
                    item['protocol'] = 'http_dash_segments'
                    item['url'] += '?' + credentials
                    item['ext'] = 'mp4'
                    item['format_id'] = 'dash-' + str(item.get('height', 0))
                for i in range(len(mpd_formats)):
                    formats.insert(
                        (-1, len(formats))[i == len(mpd_formats) - 1],
                        mpd_formats[i])

I'm happy to give a hand, what else is needed to get this one through ? @dstftw

Anyone else that we can tag ?

Release 2021.12.17

alfonsrv · 2022-04-03T16:49:49Z

I checked out @wongyiuhang's code from his loom branch and it doesn't work (anymore). Merely downloads a 4kb mp4. Somebody on Reddit suggests having to download the manifest.

Also, embedded URLs are invalid currently and look like this https://www.loom.com/embed/1ae0b5c204b14f5881f0a826cbc7b3b9

upintheairsheep · 2022-11-08T17:24:36Z

Hello, make sure not to forget about this.

ryanhugh · 2023-02-26T20:29:22Z

Is anyone available to keep moving this along? Support for loom would be great

upintheairsheep · 2023-02-27T03:08:59Z

Is anyone available to keep moving this along? Support for loom would be great

It works perfectly

upintheairsheep · 2023-02-27T03:09:46Z

https://archive.org/details/Loom-6670e3eba3c84dc09ada8306c7138075

upintheairsheep · 2023-02-27T03:10:09Z

We just need to add a little more metadata from mine and we done

upintheairsheep · 2023-02-27T03:10:47Z

wongyiuhang#1

dirkf · 2023-06-02T04:40:12Z

The first test for LoomFolderIE is giving 404 on JSON download. The API URL with folders/.../by_name seems not to be supported now. But the folder structure can be traversed with the folders/....

[Loom] Add new extractor

2302f32

wongyiuhang marked this pull request as ready for review February 1, 2021 10:02

cypheron mentioned this pull request Feb 3, 2021

Evaluation / overview of new proposed extractors / sites #28054

Open

wongyiuhang added 3 commits February 4, 2021 00:33

Merge branch 'master' into loom

14df8ad

[Loom] Update: Move related member functions into LoomIE

918f4f3

[Loom] Add: Additional playlist extractor for folder support

287e710

wongyiuhang force-pushed the loom branch from 7190dc6 to 287e710 Compare February 3, 2021 16:37

[Loom] Update: Change test case to avoid a false-positive result from…

c9f3667

… test/test_unicode_literals.py

wongyiuhang mentioned this pull request Feb 3, 2021

Site request: Support for loom.com #27957

Open

5 tasks

dstftw requested changes Feb 13, 2021

View reviewed changes

dstftw added the pending-fixes label Feb 13, 2021

wongyiuhang added 5 commits February 25, 2021 03:20

[Loom] Moved functions to inline

34e6a6b

Removed if statement parentheses

[Loom] Add missing parsing function

29c4168

[Loom] Add fallback to mandatory attribute

81bd98a

[Loom] Move request back into _download_json

70b8045

[Loom] Use url_result instead

1b2651e

wongyiuhang commented Feb 24, 2021

View reviewed changes

[Loom] Add url_or_none back

e218b26

Merge tag '2021.12.17' into loom

adc6e01

Release 2021.12.17

gamer191 mentioned this pull request May 14, 2022

Loom yt-dlp/yt-dlp#3715

Closed

7 tasks

dirkf force-pushed the master branch from 01bf89e to 4c6fba3 Compare August 26, 2022 07:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Loom] Add new extractor #28039

[Loom] Add new extractor #28039

wongyiuhang commented Feb 1, 2021 •

edited

Loading

dstftw Feb 13, 2021

wongyiuhang Feb 24, 2021

dstftw Feb 13, 2021

wongyiuhang Feb 24, 2021

dstftw Feb 13, 2021

wongyiuhang Feb 24, 2021

dstftw Feb 13, 2021

wongyiuhang Feb 24, 2021

dstftw Feb 13, 2021

wongyiuhang Feb 24, 2021

dstftw Feb 13, 2021

wongyiuhang Feb 24, 2021

dstftw Feb 13, 2021

wongyiuhang Feb 24, 2021 •

edited

Loading

dstftw Feb 13, 2021

wongyiuhang Feb 24, 2021

wongyiuhang Feb 24, 2021

Fred-Vatin commented Mar 18, 2021

wongyiuhang commented Mar 19, 2021

rememberlenny commented Sep 24, 2021

wongyiuhang commented Sep 25, 2021

cellarpub commented Nov 6, 2021

mysticaltech commented Jan 31, 2022

coolbaluk commented Feb 2, 2022

alfonsrv commented Apr 3, 2022

upintheairsheep commented Nov 8, 2022

ryanhugh commented Feb 26, 2023

upintheairsheep commented Feb 27, 2023

upintheairsheep commented Feb 27, 2023

upintheairsheep commented Feb 27, 2023

upintheairsheep commented Feb 27, 2023

dirkf commented Jun 2, 2023

		'width': try_get(info, lambda x: x['video_properties']['width']),
		'height': try_get(info, lambda x: x['video_properties']['height'])

[Loom] Add new extractor #28039

Are you sure you want to change the base?

[Loom] Add new extractor #28039

Conversation

wongyiuhang commented Feb 1, 2021 • edited Loading

Please follow the guide below

Before submitting a pull request make sure you have:

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

What is the purpose of your pull request?

Description of your pull request and other information

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wongyiuhang Feb 24, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Fred-Vatin commented Mar 18, 2021

wongyiuhang commented Mar 19, 2021

rememberlenny commented Sep 24, 2021

wongyiuhang commented Sep 25, 2021

cellarpub commented Nov 6, 2021

mysticaltech commented Jan 31, 2022

coolbaluk commented Feb 2, 2022

alfonsrv commented Apr 3, 2022

upintheairsheep commented Nov 8, 2022

ryanhugh commented Feb 26, 2023

upintheairsheep commented Feb 27, 2023

upintheairsheep commented Feb 27, 2023

upintheairsheep commented Feb 27, 2023

upintheairsheep commented Feb 27, 2023

dirkf commented Jun 2, 2023

wongyiuhang commented Feb 1, 2021 •

edited

Loading

wongyiuhang Feb 24, 2021 •

edited

Loading