Extractor for youtube mix (auto-generated playlist) #280

XiangRongLin · 2020-03-07T15:16:34Z

I carefully read the contribution guidelines and agree to them.
I did test the API against NewPipe.
I agree to ASAP create a PULL request for NewPipe for making in compatible when I changed the api.

Problem

As describes in the issue in the NewPipe repo, when sharing a youtube mix to newpipe, it crashes. Reason for that is that the linkhandler receives (1) https://www.youtube.com/watch?v=QXCE3Betyug&list=RDQXCE3Betyug and converts it to (2) https://www.youtube.com/playlist?list=RDQXCE3Betyug , which loads a basicly empty page.

Changes

The LinkHandler doesn't convert mix urls.
New PlaylistExtractor to extract playlist information from the url (1).
The YoutubeService uses the new extractor if it is a mix.

Testing

I tested the changes on my android 10 and 7.1 devices and a 5.0 emulator, by:

sharing a link from my browser to NewPipe
saving the playlist and opening it from the bookmarks fragment
playling the playlist (didn't play on 5.0 emulator, but i can't play ANY video here)

@Iamdeadlyz could you test the apk.
app-debug.zip

What's missing

Mixes are generated based on the user. Meaning if I open the site once while logged in and once while in incognito mode, it gives me different playlists.
The size of a mix is infinity, but getStreamCount returns a long. So i don't really know how to handle that. I would let that method return -1 and let the frontend interpret that as infinity.
Check if "My Mix" is getting handled correctly. Here the list id starts with "RDMM".

Notes

In the mix url (1) above you can see the playlist id is just the video id with "RD" at the start.
All mix urls that i found so far are like that, where the playlist id is "RD" + video id of the first video. I assume RD stands for radio (Playlist ids start with "PL" instead)
Since i can't reopen PR (Extractor for youtube mix (auto-generated playlist) #253) because i force pushed, i'll add the comments from @B0pol here:

Uploader name should be YouTube
Uploader avatar url should point to the YouTube logo

I noticed that there are quite a few duplicates if you continue scrolling down in a mix, especially the first video of the mix is repeated often.

Related PR: #276
Depends on: TeamNewPipe/NewPipe#3243

Iamdeadlyz · 2020-03-07T16:16:04Z

Hi @XiangRongLin

Thanks for the ping! The debug app works well with the auto-generated playlist. I can confirm that the issue is now fixed.

PeterHindes · 2020-03-12T20:33:26Z

Hey @XiangRongLin isn't "my mix" based on your youtube profile? How would that work in new-pipe at all?

XiangRongLin · 2020-03-12T20:42:19Z

@PeterHindes It just queues videos in the same genre. It's not very good at selecting them but it still works.

And currently you can only share mix urls to newpipe and not start a mix from newpipe itself, so you will always have a starting video on which the mix is based on.

I also just realized that i can't install my debug apk for whatever reason. I'll look into it on the weekend.

Stypox · 2020-03-13T13:47:40Z

I also just realized that i can't install my debug apk for whatever reason. I'll look into it on the weekend

I don't know why, but sometimes it happens. You can fix this by clicking "Clean Project" and rebuilding normally.

wb9688 · 2020-03-13T20:45:02Z

@Stypox: I think this happens when you compile the app to run the activity again on your phone, if you just do a full build (no need to clean), it'll work again.

Stypox · 2020-03-16T19:50:26Z

The size of a mix is infinity, but getStreamCount returns a long. So i don't really know how to handle that. I would let that method return -1 and let the frontend interpret that as infinity.

On IRC we are discussing if it would be a good idea to return -2 in these cases, that would mean "unknown number bigger than 100" (in this case infinity). -1 already means "unknown".

XiangRongLin · 2020-03-17T06:05:30Z

@Stypox
Thanks for the update.

In that case we could use static constants, so we don't have magic numbers floating around.

Stypox

Thank you for the effort! :-D
Code looks good

In that case we could use static constants, so we don't have magic numbers floating around.

Yeah, you could add a constant, something like "MORE_THAN_100", equal to -2. Then you would have to open another PR in the app to handle that correctly and show "100+" to the user. If you have no time I could do that.

Uploader name should be YouTube
Uploader avatar url should point to the YouTube logo

I think this is the way to go. The mix creator is Youtube, actually. So even though there is support for null/empty uploaders, it is better to return an hardcoded "YouTube" anyway. Also, for the uploader image, it is ok if you return a hardcoded youtube image url.

I noticed that there are quite a few duplicates if you continue scrolling down in a mix, especially the first video of the mix is repeated often.

Nothing to worry about, this is YouTube's fault ;-)

extractor/src/test/java/org/schabi/newpipe/extractor/services/youtube/YoutubeServiceTest.java

extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeService.java

...va/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeMixPlaylistExtractor.java

...rg/schabi/newpipe/extractor/services/youtube/extractors/YoutubeMixPlaylistExtractorTest.java

...va/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeMixPlaylistExtractor.java

XiangRongLin · 2020-03-17T13:30:34Z

@Stypox
I applied your suggestions and updated to apk.
I'll leave the changes in the frontend to you. I have no clue how to integrate the "+" in "100+", so that it works in all languages.

I just saw that my CI build failed because i got a reCaptcha challange. The "getPageMultipleTimes" test may need to be removed.

Stypox

Last few suggestions, thank you :)

I'll leave the changes in the frontend to you. I have no clue how to integrate the "+" in "100+", so that it works in all languages.

As soon as you are done implementing the renaming suggestion I will take this up. There is no need you upload another apk, since I will do it as soon as I implement this.

I just saw that my CI build failed because i got a reCaptcha challange. The "getPageMultipleTimes" test may need to be removed.

Don't worry, it happens sometimes ;-)

extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeService.java

...va/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeMixPlaylistExtractor.java

Stypox · 2020-03-17T17:53:23Z

I opened the corresponding client PR: TeamNewPipe/NewPipe#3243
Debug apk: app-debug.zip

Stypox

Again, thank you for your work :-D

Stypox

@wb9688 suggested in TeamNewPipe/NewPipe#3243 (comment) to separate "infinite" from "more than 100". So, sorry for asking you to change this again, but could you add another field named INFINITE_ITEMS = -2 and returning that instead? Also keep MORE_THAN_100_ITEMS = -3, since @wb9688 plans to use it. If you want you can just commit with the buttons below, or tell me to do that ;-)

extractor/src/main/java/org/schabi/newpipe/extractor/playlist/PlaylistExtractor.java

...va/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeMixPlaylistExtractor.java

XiangRongLin · 2020-03-17T19:52:27Z

@Stypox Feel free to just commit small stuff like that in the future.

...va/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeMixPlaylistExtractor.java

Stypox · 2020-03-20T19:47:27Z

@XiangRongLin I moved things around once again ;-)

...rg/schabi/newpipe/extractor/services/youtube/extractors/YoutubeMixPlaylistExtractorTest.java

wb9688 · 2020-03-21T16:20:55Z

Does this support channel mixes (e.g. https://www.youtube.com/watch?v=mnk6gnOBYIo&list=RDCMUCXuqSBlHAE6Xw-yeJA0Tunw&start_radio=1&t=0)? You don't have tests for them.

...ain/java/org/schabi/newpipe/extractor/services/youtube/linkHandler/YoutubeParsingHelper.java

...va/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeMixPlaylistExtractor.java

TobiGr

Code looks good to me.
Thank you for the patience!
Can you fix the conflicts, please?

XiangRongLin · 2020-12-03T15:07:40Z

@TobiGr It seems like that this is already so old, that YouTube has already changed their json format. So I'll have to look into it again.

On another note, NewPipe want to be very privacy friendly and (I think) avoid any way to allow tracking. Does this feature fit in there? Because it uses cookies to allow for continuations.

TheAssassin · 2020-12-03T15:28:55Z

On another note, NewPipe want to be very privacy friendly and (I think) avoid any way to allow tracking. Does this feature fit in there? Because it uses cookies to allow for continuations.

That's quite a good point. You'd have to make sure to limit tracking really well (ideally store different cookies for different mixes, etc.). It allows for tracking across very long periods of times, too. Sometimes a mix may run for hours.

We should discuss that in the team, I guess.

XiangRongLin · 2020-12-03T15:43:11Z

You'd have to make sure to limit tracking really well (ideally store different cookies for different mixes, etc.).

If i remember correctly, YouTube provides the cookie and i just send it back on continuations request. For a mix to not provide duplicate videos, it is needed for as long as the mix is running

Stypox · 2020-12-12T13:58:07Z

If i remember correctly, YouTube provides the cookie and i just send it back on continuations request. For a mix to not provide duplicate videos, it is needed for as long as the mix is running

In my opinion this is ok, as it only tells YouTube that a random user seems to be really listening the mix, which YouTube would already know with more precision by looking at the video details request.
@XiangRongLin sorry for not merging this for a long long time. If you fix youtube mixes again I promise I'm going to review and finally merge in a short time (and ping me multiple times if I don't ;-) )

-New YoutubeMixPlaylistExtractor, that extracts from a mix (auto-generated playlist). -The url has the format of "youtube.com/watch?v=videoID&playlistID", where playlistID always starts with "RD" and usually followed by the videoID. -Change YoutubePlaylistLinkHandlerFactory to create a linkhandler with the given url if it is a mix. -Change YoutubeService to return YoutubeMixPlaylistExtractor if the url is a mix.

…ID" format. This occurs when sharing a mix from the official youtube app.

Also extract continuation now

…of mix Also fix getThumbnailUrl for "My Mix"

Move thumbnail id exctraction code to getThumbnailUrlFromId Add test for "My mix" detection to service tests Use ITEM_COUNT_UNKNOWN everywhere instead of -1 and add some tests

Channel mix adjusments and test Don't accept youtube music mix urls as playlist Don't override playlistData to keep getInitialPage() Remove json constants Indentation

This way youtube wont return duplicates when getting more items of the mix (but youtube can also track us)

XiangRongLin · 2020-12-12T19:45:34Z

@Stypox Problem was smaller than i expected, see the last commit of the bunch

As for the rebase, only conflicting file was YoutubeParsingHelper, where i think i got the correct stuff merged together.

XiangRongLin · 2020-12-14T07:23:52Z

apk at the top is already updated

Stypox · 2020-12-14T17:10:58Z

I tested it with a couple mixes, and also with continuations, and it works as expected. Thank you! Sorry for taking so long ;-)

Regression introduced by YouTube Mix support (#280)

wb9688 mentioned this pull request Mar 7, 2020

Don't accept YouTube Mix playlists #276

Merged

3 tasks

PeterHindes mentioned this pull request Mar 12, 2020

YoutubePlaylistExtractor error when import Youtube Mix #214

Closed

Stypox linked an issue Mar 13, 2020 that may be closed by this pull request

YoutubePlaylistExtractor error when import Youtube Mix #214

Closed

Stypox requested changes Mar 17, 2020

View reviewed changes

B0pol reviewed Mar 17, 2020

View reviewed changes

...va/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeMixPlaylistExtractor.java Outdated Show resolved Hide resolved

...va/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeMixPlaylistExtractor.java Outdated Show resolved Hide resolved

Stypox requested changes Mar 17, 2020

View reviewed changes

extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeService.java Outdated Show resolved Hide resolved

...va/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeMixPlaylistExtractor.java Outdated Show resolved Hide resolved

Stypox mentioned this pull request Mar 17, 2020

Show radio icon as uploader avatar in Youtube mixes TeamNewPipe/NewPipe#3243

Merged

1 task

Stypox approved these changes Mar 17, 2020

View reviewed changes

Stypox requested changes Mar 17, 2020

View reviewed changes

extractor/src/main/java/org/schabi/newpipe/extractor/playlist/PlaylistExtractor.java Outdated Show resolved Hide resolved

...va/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeMixPlaylistExtractor.java Outdated Show resolved Hide resolved

Stypox approved these changes Mar 17, 2020

View reviewed changes

TobiGr reviewed Mar 18, 2020

View reviewed changes

...va/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeMixPlaylistExtractor.java Outdated Show resolved Hide resolved

TheAssassin previously requested changes Mar 18, 2020

View reviewed changes

...va/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeMixPlaylistExtractor.java Outdated Show resolved Hide resolved

wb9688 mentioned this pull request Mar 20, 2020

YouTube Music search stuff TeamNewPipe/NewPipe#3240

Merged

1 task

Stypox requested changes Mar 20, 2020

View reviewed changes

...rg/schabi/newpipe/extractor/services/youtube/extractors/YoutubeMixPlaylistExtractorTest.java Outdated Show resolved Hide resolved

wb9688 reviewed Mar 21, 2020

View reviewed changes

...ain/java/org/schabi/newpipe/extractor/services/youtube/linkHandler/YoutubeParsingHelper.java Outdated Show resolved Hide resolved

wb9688 reviewed Mar 21, 2020

View reviewed changes

...va/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeMixPlaylistExtractor.java Outdated Show resolved Hide resolved

wb9688 reviewed Mar 21, 2020

View reviewed changes

...va/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeMixPlaylistExtractor.java Outdated Show resolved Hide resolved

wb9688 reviewed Mar 21, 2020

View reviewed changes

...va/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeMixPlaylistExtractor.java Outdated Show resolved Hide resolved

TobiGr approved these changes Dec 3, 2020

View reviewed changes

XiangRongLin and others added 14 commits December 12, 2020 20:30

[Youtube] Extract getThumbnailUrl into method and change getUploaderName

d74265c

[Youtube] Add some comments to mix

327a573

[Youtube] Handle case where url is in "youtube.com/playlist?list=list…

a376792

…ID" format. This occurs when sharing a mix from the official youtube app.

[Youtube] Extract initial playlist info

0ff054a

[Youtube] Adjust mix extractor to new user agent

ec6b99c

Also extract continuation now

[Youtube] Fix get banner url

68a3948

[Youtube] Add tests and take thumbnail image always from first video …

df38b19

…of mix Also fix getThumbnailUrl for "My Mix"

[Youtube] Add _ITEMS constants and improve code style

822cf30

Move thumbnail id exctraction code to getThumbnailUrlFromId Add test for "My mix" detection to service tests Use ITEM_COUNT_UNKNOWN everywhere instead of -1 and add some tests

[Youtube] apply wb9688 suggestion (mix)

3ff8619

Channel mix adjusments and test Don't accept youtube music mix urls as playlist Don't override playlistData to keep getInitialPage() Remove json constants Indentation

[Youtube] Add subchannel functions to mix and fix imports

4219354

[Youtube] Add cookies to youtube mix request

22d2f7e

This way youtube wont return duplicates when getting more items of the mix (but youtube can also track us)

[Youtube] Apply review suggestions and avoid channel mix edge case

a338e4e

[YouTube] Don't escape & in getUrlFromNavigationEndpoint for playlists

f90f6fc

XiangRongLin force-pushed the mixPL branch from a2a1b88 to f90f6fc Compare December 12, 2020 19:40

Stypox merged commit 85fa006 into TeamNewPipe:dev Dec 14, 2020

XiangRongLin deleted the mixPL branch December 14, 2020 17:51

This was referenced Dec 22, 2020

Release 0.20.7 TeamNewPipe/NewPipe#5245

Closed

Release 0.20.7 TeamNewPipe/NewPipe#5246

Merged

TobiGr added a commit that referenced this pull request Dec 23, 2020

[YouTube] Music Mix: Fix playlist not being accepted

3033c0b

Regression introduced by YouTube Mix support (#280)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Extractor for youtube mix (auto-generated playlist) #280

Extractor for youtube mix (auto-generated playlist) #280

XiangRongLin commented Mar 7, 2020 •

edited

Loading

Iamdeadlyz commented Mar 7, 2020

PeterHindes commented Mar 12, 2020

XiangRongLin commented Mar 12, 2020

Stypox commented Mar 13, 2020

wb9688 commented Mar 13, 2020

Stypox commented Mar 16, 2020 •

edited

Loading

XiangRongLin commented Mar 17, 2020

Stypox left a comment •

edited

Loading

XiangRongLin commented Mar 17, 2020

Stypox left a comment •

edited

Loading

Stypox commented Mar 17, 2020

Stypox left a comment

Stypox left a comment •

edited

Loading

XiangRongLin commented Mar 17, 2020

Stypox commented Mar 20, 2020

wb9688 commented Mar 21, 2020

TobiGr left a comment

XiangRongLin commented Dec 3, 2020

TheAssassin commented Dec 3, 2020

XiangRongLin commented Dec 3, 2020

Stypox commented Dec 12, 2020 •

edited

Loading

XiangRongLin commented Dec 12, 2020 •

edited

Loading

XiangRongLin commented Dec 14, 2020

Stypox commented Dec 14, 2020

Extractor for youtube mix (auto-generated playlist) #280

Extractor for youtube mix (auto-generated playlist) #280

Conversation

XiangRongLin commented Mar 7, 2020 • edited Loading

Problem

Changes

Testing

What's missing

Notes

Iamdeadlyz commented Mar 7, 2020

PeterHindes commented Mar 12, 2020

XiangRongLin commented Mar 12, 2020

Stypox commented Mar 13, 2020

wb9688 commented Mar 13, 2020

Stypox commented Mar 16, 2020 • edited Loading

XiangRongLin commented Mar 17, 2020

Stypox left a comment • edited Loading

Choose a reason for hiding this comment

XiangRongLin commented Mar 17, 2020

Stypox left a comment • edited Loading

Choose a reason for hiding this comment

Stypox commented Mar 17, 2020

Stypox left a comment

Choose a reason for hiding this comment

Stypox left a comment • edited Loading

Choose a reason for hiding this comment

XiangRongLin commented Mar 17, 2020

Stypox commented Mar 20, 2020

wb9688 commented Mar 21, 2020

TobiGr left a comment

Choose a reason for hiding this comment

XiangRongLin commented Dec 3, 2020

TheAssassin commented Dec 3, 2020

XiangRongLin commented Dec 3, 2020

Stypox commented Dec 12, 2020 • edited Loading

XiangRongLin commented Dec 12, 2020 • edited Loading

XiangRongLin commented Dec 14, 2020

Stypox commented Dec 14, 2020

XiangRongLin commented Mar 7, 2020 •

edited

Loading

Stypox commented Mar 16, 2020 •

edited

Loading

Stypox left a comment •

edited

Loading

Stypox left a comment •

edited

Loading

Stypox left a comment •

edited

Loading

Stypox commented Dec 12, 2020 •

edited

Loading

XiangRongLin commented Dec 12, 2020 •

edited

Loading