Update video_id regexp to account for recent abc.com changes (fixes #25197 and #25207) #25216

tmthywynn8 · 2020-05-10T03:29:54Z

Please follow the guide below

You will be asked some questions, please read them carefully and answer honestly
Put an x into all the boxes [ ] relevant to your pull request (like that [x])
Use Preview tab to see how your pull request will actually look like

Before submitting a pull request make sure you have:

At least skimmed through adding new extractor tutorial and youtube-dl coding conventions sections
Searched the bugtracker for similar pull requests
Checked the code with flake8

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

I am the original author of this code and I am willing to release it under Unlicense
I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

Bug fix
Improvement
New extractor
New feature

Description of your pull request and other information

This pull request fixes #25197 and #25207 by accounting for the new changes on abc.com—first noticed on Wednesday, May 6. Now, the unique video ID can be found by searching for videoIdCode; note that only one instance will be found), potentially avoiding any false positives, e.g., other video IDs referenced on the page.

The solution was to change the regexp on line 141 and simplifying it for the new name on the video's source. The Rookie video in the examples dictionary can be used to verify the new changes.

Azmaeus

I verified this fix allows downloading from abc.com. I reviewed the code change, it looks fine to me.

cliss · 2020-06-05T22:00:16Z

Bump; there have been two releases since this PR was opened; any chance we can see it land in the next release, please? 😇

dstftw · 2020-06-05T22:05:25Z

youtube_dl/extractor/go.py

@@ -138,7 +138,7 @@ def _real_extract(self, url):
                    # from http://freeform.go.com/shows/shadowhunters/episodes/season-2/1-this-guilty-blood
                    r'data-video-id=["\']*(VDKA\w+)',
                    # https://abc.com/shows/the-rookie/episode-guide/season-02/03-the-bet
-                    r'\b(?:video)?id["\']\s*:\s*["\'](VDKA\w+)'


Do not touch old patterns.

Sorry for not minding my own business, but it looks like your requested change was implemented by @tmthywynn8 a couple of months ago. I'm not that familiar with the GitHub workflow, but it looks like this is just waiting for your approval.

Yes. I made the necessary changes in commit 783d296 and requested another review to make sure that I fulfilled the requirements. I'm not sure what the procedure is suppose to be, so maybe requesting another review was the wrong thing to do

I'm not sure yet but I think removing the original pattern (by replacing it with the new pattern) fixed the problem, and putting it back has re-broken it. My suspicion is that both patterns are matched on the page in question but one returns an invalid ID. Or perhaps a valid ID that nevertheless yields an error when the code later tries to retrieve it.

If I'm right, undoing the requested change will get it working again. But that's not really a solution, and again I'm not sure yet.

Sorry it took me so long to get to this, but I just tested #25216 and it works for me as-is. So my theory above is completely wrong.

Leaves original pattern untouched.

magnum32 · 2020-06-17T21:16:38Z

the fix still has not been added to youtube-dl. what is going on?

DarF234 · 2020-07-02T06:21:51Z

May I respectfully inquire as to what is delaying this fix? ABC is a very popular site and has the habit of reducing retention of free and logged-in content at seemingly random times. A timely fix would be most welcome. I am assuming (possibly incorrectly) that dstftw's request is the one we are waiting on, as he has been handling this issue.
All the patches I've found don't correspond with the code. So the only solution seems to be here.
I normally have high praise for the rapid updates to your software. But I believe this is the longest period of time I have seen without an update.

chrisallen · 2020-07-16T09:31:29Z

Could we please get this merged?

DarF234 · 2020-07-18T09:07:33Z

The problem has changed. I was stupid enough to not check logs until today. I'm no longer getting the Adobe Pass error. Now I'm getting "ExtractorError: Go said: 1025:Unable to retrieve required files for playback. Please try again in a few minutes."
An error that was assumed to be fixed in early may. This probably explained why the program worked that one evening. So the latest version may not have the Adobe Pass error, but it definitely has the one above. I won't be able to post an issue for more than 16 hours. If someone reading this can, please run a test on any full ABC show (clips download okay) that's not locked. If you get error 1025, if there's nothing open on it already, please post an issue ASAP! You'll be doing us all a favor by getting this error on the right track.
It it is an open issue, please provide a URL to it so we can follow it.

kucksdorfs · 2021-01-22T15:15:55Z

What's going on with this PR? This has been implemented in youtube-dlc months ago: blackjack4494#149 (comment)

liamengland1 · 2021-01-22T15:51:13Z

It will not be merged until maintainers fly to US, purchase a cable package, and test it themselves. And no, I'm not joking (well maybe a little bit).

kucksdorfs · 2021-01-22T15:58:16Z

It will not be merged until maintainers fly to US, purchase a cable package, and test it themselves. And no, I'm not joking (well maybe a little bit).

Would SSH access to a Raspberry Pi on a US ISP work?

rautamiekka · 2021-01-22T22:45:03Z

It will not be merged until maintainers fly to US, purchase a cable package, and test it themselves. And no, I'm not joking (well maybe a little bit).

Would SSH access to a Raspberry Pi on a US ISP work?

Don't you need to play the vids too ?

kucksdorfs · 2021-01-23T14:24:54Z

It will not be merged until maintainers fly to US, purchase a cable package, and test it themselves. And no, I'm not joking (well maybe a little bit).

Would SSH access to a Raspberry Pi on a US ISP work?

Don't you need to play the vids too ?

I think this could be handled by rsync or X11 forwarding (depending on how testing would need to get done). Or you know, whoever provides the SSH endpoint could also make sure the mp4 (or whatever) file could make sure the file is playable.

owenthewizard · 2021-02-22T17:08:08Z

I have a subscription and would be willing to provide any needed access to the maintainers. I'm located in the US.

kucksdorfs · 2021-02-22T20:09:37Z

I don't have the login, but I also have a residential IP address in the United States.

* https://github.com/ytdl-org/youtube-dl: [youtube] Remove unused code [go] Improve video id extraction (closes ytdl-org#25207, closes ytdl-org#25216, closes ytdl-org#26058) [test_execution] Add test for lazy extractors (refs ytdl-org#28780) [test_youtube_misc] Move YoutubeIE.extract_id test into separate module [youtube] Fix lazy extractors (closes ytdl-org#28780) [bbc] Extract full description from __INITIAL_DATA__ (refs ytdl-org#28774) [bbc] Extract description and timestamp from __INITIAL_DATA__ (ytdl-org#28774)

Update go.py

5a3e2a4

tmthywynn8 changed the title ~~Update video_id regexp to account for recent abc.com changes (fixes #25207)~~ Update video_id regexp to account for recent abc.com changes (fixes #25197 and #25207) May 10, 2020

sftman18 mentioned this pull request May 10, 2020

[Go] Abc.com giving Adobe Pass error #25197

Closed

5 tasks

Azmaeus approved these changes May 25, 2020

View reviewed changes

dstftw added the geo-restricted label Jun 5, 2020

dstftw requested changes Jun 5, 2020

View reviewed changes

dstftw added the pending-fixes label Jun 5, 2020

Update go.py

783d296

Leaves original pattern untouched.

tmthywynn8 requested a review from dstftw June 7, 2020 03:18

rpvcg mentioned this pull request Jun 20, 2020

Still getting Adobe Pass errors on Browser-playable content #25713

Closed

5 tasks

DarF234 mentioned this pull request Jul 19, 2020

Error 1025 on abc.com full shows. NOT a duplicate of #25207 or fixed by #25216 #26058

Closed

5 tasks

tony-izzo mentioned this pull request Jul 30, 2020

[Solved] ABC [GO]: Unable to retrieve required files for playback. #25207

Closed

5 tasks

dstftw force-pushed the master branch 2 times, most recently from 5e26784 to da2069f Compare September 13, 2020 13:50

kucksdorfs mentioned this pull request Sep 15, 2020

Issues with ABC (US American Broadcast - go.py) blackjack4494/youtube-dlc#113

Open

5 tasks

kucksdorfs mentioned this pull request Sep 27, 2020

Added regex for ABC.com site. blackjack4494/youtube-dlc#149

Merged

9 tasks

dstftw closed this in ac19c3a Apr 20, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update video_id regexp to account for recent abc.com changes (fixes #25197 and #25207) #25216

Update video_id regexp to account for recent abc.com changes (fixes #25197 and #25207) #25216

tmthywynn8 commented May 10, 2020 •

edited

Loading

Azmaeus left a comment

cliss commented Jun 5, 2020

dstftw Jun 5, 2020

tony-izzo Aug 17, 2020

tmthywynn8 Aug 18, 2020

tony-izzo Aug 18, 2020 •

edited

Loading

tony-izzo Sep 3, 2020

magnum32 commented Jun 17, 2020

DarF234 commented Jul 2, 2020

chrisallen commented Jul 16, 2020

DarF234 commented Jul 18, 2020

kucksdorfs commented Jan 22, 2021

liamengland1 commented Jan 22, 2021 •

edited

Loading

kucksdorfs commented Jan 22, 2021

rautamiekka commented Jan 22, 2021

kucksdorfs commented Jan 23, 2021

owenthewizard commented Feb 22, 2021

kucksdorfs commented Feb 22, 2021

Update video_id regexp to account for recent abc.com changes (fixes #25197 and #25207) #25216

Update video_id regexp to account for recent abc.com changes (fixes #25197 and #25207) #25216

Conversation

tmthywynn8 commented May 10, 2020 • edited Loading

Please follow the guide below

Before submitting a pull request make sure you have:

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

What is the purpose of your pull request?

Description of your pull request and other information

Azmaeus left a comment

Choose a reason for hiding this comment

cliss commented Jun 5, 2020

dstftw Jun 5, 2020

Choose a reason for hiding this comment

tony-izzo Aug 17, 2020

Choose a reason for hiding this comment

tmthywynn8 Aug 18, 2020

Choose a reason for hiding this comment

tony-izzo Aug 18, 2020 • edited Loading

Choose a reason for hiding this comment

tony-izzo Sep 3, 2020

Choose a reason for hiding this comment

magnum32 commented Jun 17, 2020

DarF234 commented Jul 2, 2020

chrisallen commented Jul 16, 2020

DarF234 commented Jul 18, 2020

kucksdorfs commented Jan 22, 2021

liamengland1 commented Jan 22, 2021 • edited Loading

kucksdorfs commented Jan 22, 2021

rautamiekka commented Jan 22, 2021

kucksdorfs commented Jan 23, 2021

owenthewizard commented Feb 22, 2021

kucksdorfs commented Feb 22, 2021

tmthywynn8 commented May 10, 2020 •

edited

Loading

tony-izzo Aug 18, 2020 •

edited

Loading

liamengland1 commented Jan 22, 2021 •

edited

Loading