Add a proper support of other delivery methods than progressive HTTP #663

AudricV · 2021-06-20T17:09:33Z

I carefully read the contribution guidelines and agree to them.
I have tested the API against NewPipe.
I agree to create a pull request for NewPipe as soon as possible to make it compatible with the changed API.

This is the work made by @wb9688 (thank you for your work, I wouldn't have been able to do it myself) in #367 but rebased and with some changes:

Changes in Stream classes

Rename url (now deprecated) to content and add isUrl to indicate whether content contains the URL or the file itself (which is useful when we have to split DASH/HLS manifests);
Add id to identify a specific stream (there could be multiple Streams with the same ID if the same stream is offered through multiple delivery methods; in YouTube's case the ID would be the itag);
Add a baseUrl field, required for some players (like ExoPlayer) to play manifests.

Other changes:

Add DeliveryFormat enum with PROGRESSIVE_HTTP, DASH, HLS and TORRENT values;
Don't support SoundCloud HLS only-streams with the HLS delivery method, but with the current method which is progressive HTTP (so they are downloadable by NewPipe);
Extract the downloadable file of SoundCloud tracks (on tracks on which download is available);
Fix parsing of YouTube DASH manifests (not used now by the extractor);
Generate DASH manifests for YouTube OTF streams and post live streams, by using the data returned in the first sequence;
Format the ItagItem class;
Update DefaultStreamExtractorTest test with the changes made in Stream classes;
Update SoundcloudStreamExtractor test to test if streams returned by the extractor are progressive HTTP (test will fail if that's not the case);
Return the HLS URL of PeerTube streams and their HLS variant, if available;
Return PeerTube audio streams as audio streams (and not as video streams);
Add POST_LIVE_STREAM and POST_LIVE_AUDIO_STREAM to StreamType enum;
Return captions for YouTube age-restrcited videos (because we are now able to extract them, probably since the InnerTube migration for age-restricted contents).

Fixed issues

Closes #461, fixes #648

Stypox

Looks mostly good to me, though this will require testing many videos

.../java/org/schabi/newpipe/extractor/services/peertube/extractors/PeertubeStreamExtractor.java

...in/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeStreamExtractor.java

extractor/src/main/java/org/schabi/newpipe/extractor/stream/SubtitlesStream.java

Stypox · 2021-07-20T13:37:50Z

Oh, a question: the id field would be used by the client to only show one of the two equal options to the user?

.../src/main/java/org/schabi/newpipe/extractor/services/youtube/ContentAndItagItemAndIsUrl.java

Stypox

Looking good, except for some (probably unavoidable) code duplication :-)

...in/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeStreamExtractor.java

.../java/org/schabi/newpipe/extractor/services/peertube/extractors/PeertubeStreamExtractor.java

TobiGr

Nice structure!
I reviewed the first half of the files. I'll try to take a look at the others on the weekend.

extractor/src/main/java/org/schabi/newpipe/extractor/stream/AudioStream.java

extractor/src/main/java/org/schabi/newpipe/extractor/stream/Stream.java

extractor/src/main/java/org/schabi/newpipe/extractor/stream/StreamInfo.java

.../java/org/schabi/newpipe/extractor/services/peertube/extractors/PeertubeStreamExtractor.java

AudricV · 2021-08-09T20:07:06Z

Note that the CI is failing only because of the WhatIsPeerTube test (the issue is that some values about the channel were changed by Framasoft).

TobiGr

Went through the remaining files and only found one minor thing.

extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/ItagItem.java

.../org/schabi/newpipe/extractor/services/media_ccc/extractors/MediaCCCLiveStreamExtractor.java

...in/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeStreamExtractor.java

extractor/src/main/java/org/schabi/newpipe/extractor/stream/StreamInfo.java

.../test/java/org/schabi/newpipe/extractor/services/youtube/YoutubeDashManifestCreatorTest.java

.../java/org/schabi/newpipe/extractor/services/peertube/extractors/PeertubeStreamExtractor.java

Stypox · 2021-08-25T08:25:32Z

Does it make a difference to use EMPTY_STRING instead of ""?

…improve YoutubeParsingHelper The hardcoded client version used was outdated and buffering was present on a lot of streams so it seems we need to always get the most recent WEB client version; otherwise streams will be throttled. The hardcoded clients version have been updated to their latest version. In YoutubeParsingHelper: - unused methods have been removed; - documentations have been improved; - a lot of warnings have been fixed; - the extraction of the client version has been improved/fixed in terms of result and performance (the shortener version was returned even the standard client version was found). A parameter has been added to streaming URLs from the WEB client: the cver param, which appends the client version to the streaming URL. This param is added by YouTube web players, so let's use it to spoof better the official web clients. Some duplicated code (in YoutubeParsingHelper and YoutubeMusicSearchExtractor) has been merged into one common method named getYoutubeMusicHeaders, located in YoutubeParsingHelper.

…DASH manifests Add the rn param to OTF and post live DVR streaming URLs (rn means request number). This commit also improves documentation of YoutubeDashManifestCreator. YoutubeDashManifestCreatorTest has been also updated with the changes made in YoutubeDashManifestCreator.

Trying to prevent more throttling when streaming ended livestreams or OTF streams by using first streams of the Android client.

…ive streams and massive improvements to YoutubeDashManifestCreator Generate DASH manifests for YouTube progressive streams like Invidious does, fix some bugs, add documentation everywhere in the class. Relevant test class has been updated. For more detailed changes, look at code changes.

… DASH URLs extraction Also improve a bit the code of the files changed with this PR.

… length The value expected is a long so an implicit conversion was done and on bigger videos, the content length value is more than the capacity of an integer, which was making the extractor crash when trying to get video/audio streams. This issue is now fixed with this commit.

Apply opusforlife2's suggestions for comments and JavaDocs of YoutubeDashManifestCreator.

The boolean keyAndVersionExtracted in YoutubeParsingHelper was not set to false when resetting the client version and the key, which makes the extractor uses null on the next getting of the client version or the key if the clientVersion and the key were extracted before.

…reams on long video streams Also apply suggested changes. Co-authored-by: TobiGr <TobiGr@users.noreply.github.com>

Also make parameters in setters final.

…ove some code In order to keep compability of the extractor for Android KitKat, we can't use StandardCharsets.UTF_8 but only the UTF_8 string. Also improve some code: remove some duplication and fix some bugs. For more detailed changes, check out code changes.

This field, made private, is recommended to be added by the Java serialization specification.

…hods and add documentation to the DeliveryMethod enum It is named SS in the DeliveryMethod enum, because SmoothStreaming is a bit long word and we are already using abbreviations with DASH and HLS delivery methods. Also add documentation of delivery methods of the enum constants and of the enum itself.

Documentation has been added for the enum constants and for the enum itself.

… resolution and isVideoOnly fields in VideoStream but deprecate their use Getters should be used instead of the fields, like for the others. Return a new constant (one for the AudioStream class, one for the VideoStream class, but they have the same value (because this method is not implemented in the Stream class)) for streams which don't have an itag instead of 0: ITAG_NOT_AVAILABLE_OR_NOT_APPLICABLE, which is equal to -1 (because an itag id should be not negative). Also fix some small issues in the Stream class.

AudricV · 2022-03-09T19:39:49Z

Closing in favor of #810.

AudricV added enhancement multiservice Issues related to multiple services labels Jun 20, 2021

AudricV mentioned this pull request Jun 20, 2021

Add support for delivery methods other than progressive HTTP (in the player only) TeamNewPipe/NewPipe#6537

Closed

9 tasks

AudricV force-pushed the delivery-methods branch from dd8cac4 to 7564042 Compare June 27, 2021 15:09

AudricV marked this pull request as draft July 5, 2021 15:30

AudricV force-pushed the delivery-methods branch from 7564042 to d8ccc7c Compare July 8, 2021 15:46

AudricV marked this pull request as ready for review July 14, 2021 14:19

AudricV force-pushed the delivery-methods branch from 07a2da9 to 1c763a7 Compare July 17, 2021 15:02

Stypox requested changes Jul 20, 2021

View reviewed changes

AudricV force-pushed the delivery-methods branch from 1c763a7 to 3650265 Compare July 21, 2021 16:41

Stypox mentioned this pull request Aug 4, 2021

Add support for different types of stream delivery by a service #358

Closed

AudricV force-pushed the delivery-methods branch 2 times, most recently from e39ce8e to f31a7b8 Compare August 5, 2021 17:41

AudricV commented Aug 7, 2021

View reviewed changes

.../src/main/java/org/schabi/newpipe/extractor/services/youtube/ContentAndItagItemAndIsUrl.java Outdated Show resolved Hide resolved

AudricV marked this pull request as draft August 7, 2021 15:47

Stypox requested changes Aug 9, 2021

View reviewed changes

TobiGr reviewed Aug 9, 2021

View reviewed changes

AudricV force-pushed the delivery-methods branch from 71ebbf6 to 43e0e6a Compare August 9, 2021 20:00

TobiGr reviewed Aug 12, 2021

View reviewed changes

extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/ItagItem.java Outdated Show resolved Hide resolved

AudricV force-pushed the delivery-methods branch from d60743d to eb03497 Compare August 14, 2021 16:06

opusforlife2 mentioned this pull request Aug 15, 2021

[HELP NEEDED] Major planned/missing features TeamNewPipe/NewPipe#6448

Open

34 tasks

AudricV force-pushed the delivery-methods branch from 7f19059 to a138daf Compare August 18, 2021 17:39

AudricV marked this pull request as ready for review August 21, 2021 17:34

AudricV force-pushed the delivery-methods branch from b29f53c to 7531650 Compare August 24, 2021 10:36

AudricV requested a review from Stypox August 24, 2021 10:36

Stypox requested changes Aug 24, 2021

View reviewed changes

TobiGr reviewed Aug 24, 2021

View reviewed changes

.../java/org/schabi/newpipe/extractor/services/peertube/extractors/PeertubeStreamExtractor.java Outdated Show resolved Hide resolved

AudricV and others added 22 commits February 26, 2022 18:22

[YouTube] Always use again Android streams first

8a1a6a1

Trying to prevent more throttling when streaming ended livestreams or OTF streams by using first streams of the Android client.

[YouTube] Extract the content length field of streams when available

91def33

Apply requested changes and cache audio and video streams and HLS and…

eaa70a1

… DASH URLs extraction Also improve a bit the code of the files changed with this PR.

[YouTube] Apply documentation suggestions to YoutubeDashManifestCreator

3a309a1

Apply opusforlife2's suggestions for comments and JavaDocs of YoutubeDashManifestCreator.

[YouTube] Fix StackOverflow error when generating manifests of OTF st…

50b74c6

…reams on long video streams Also apply suggested changes. Co-authored-by: TobiGr <TobiGr@users.noreply.github.com>

Apply suggested changes on StreamInfo

ee673f7

Also make parameters in setters final.

Override hashCode and equals methods on Stream classes

8357f6f

Add ability to set maximum size of caches

7d1cafa

Update YoutubeDashManifestCreatorTest to Junit 5

bdd9341

Add basic tests for ManifestCreatorCache in UtilsTest

f773190

Add serialVersionUID to Serializable classes changed

4e59ea8

This field, made private, is recommended to be added by the Java serialization specification.

Add documentation in the StreamType enum

77026d9

Documentation has been added for the enum constants and for the enum itself.

Apply some requested changes

3bec1f0

Adress new requested changes

bc33879

AudricV force-pushed the delivery-methods branch from 624ffdf to bc33879 Compare February 26, 2022 17:23

AudricV closed this Feb 27, 2022

TobiGr reopened this Feb 27, 2022

AudricV marked this pull request as draft March 1, 2022 19:20

AudricV mentioned this pull request Mar 9, 2022

Support delivery methods other than progressive HTTP #810

Merged

3 tasks

AudricV closed this Mar 9, 2022

AudricV deleted the delivery-methods branch March 9, 2022 19:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a proper support of other delivery methods than progressive HTTP #663

Add a proper support of other delivery methods than progressive HTTP #663

AudricV commented Jun 20, 2021 •

edited

Loading

Stypox left a comment

Stypox commented Jul 20, 2021

Stypox left a comment

TobiGr left a comment

AudricV commented Aug 9, 2021

TobiGr left a comment

Stypox commented Aug 25, 2021

AudricV commented Mar 9, 2022

Add a proper support of other delivery methods than progressive HTTP #663

Add a proper support of other delivery methods than progressive HTTP #663

Conversation

AudricV commented Jun 20, 2021 • edited Loading

Changes in Stream classes

Other changes:

Fixed issues

Stypox left a comment

Choose a reason for hiding this comment

Stypox commented Jul 20, 2021

Stypox left a comment

Choose a reason for hiding this comment

TobiGr left a comment

Choose a reason for hiding this comment

AudricV commented Aug 9, 2021

TobiGr left a comment

Choose a reason for hiding this comment

Stypox commented Aug 25, 2021

AudricV commented Mar 9, 2022

AudricV commented Jun 20, 2021 •

edited

Loading