Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support delivery methods other than progressive HTTP #810

Merged
merged 38 commits into from
Jun 2, 2022

Commits on May 28, 2022

  1. Add support of other delivery methods than progressive HTTP in Stream…

    … classes
    
    Stream constructors are now private and streams can be constructed with new Builder classes per stream class. This change has been made to prevent creating and using several constructors in stream classes.
    
    Some default cases have been also added in these Builder classes, so not everything has to be set, depending of the service and the content.
    AudricV committed May 28, 2022
    Configuration menu
    Copy the full SHA
    2f061b8 View commit details
    Browse the repository at this point in the history
  2. Remove fetching of the DASH manifest extracted when getting informati…

    …on of a content with StreamInfo
    
    DashMpdParser is only working with YouTube streams, as it uses the ItagItem class.
    
    Also improve code and comments of StreamInfo (especially final use where possible).
    AudricV committed May 28, 2022
    Configuration menu
    Copy the full SHA
    ad993b9 View commit details
    Browse the repository at this point in the history
  3. Move DashMpdParser to the YouTube package and fix extraction of streams

    DashMpdParser is only working with YouTube streams, as it uses the ItagItem class.
    Also update creation of AudioStreams and VideoStreams objects.
    AudricV committed May 28, 2022
    Configuration menu
    Copy the full SHA
    7c67d46 View commit details
    Browse the repository at this point in the history
  4. [YouTube] Return more values returned inside the ItagItems of the pla…

    …yer response and deprecate use of public audio and video fields
    
    These fields can be now replaced by a getter and a setter.
    
    New fields have been added and will allow the creation of DASH manifests for OTF and ended livestreams. There are:
    - contentLength;
    - approxDurationMs;
    - targetDurationSec;
    - sampleRate;
    - audioChannels.
    AudricV committed May 28, 2022
    Configuration menu
    Copy the full SHA
    d5f3637 View commit details
    Browse the repository at this point in the history
  5. Apply changes in all StreamExtractors except YouTube's one and fix ex…

    …traction of PeerTube audio streams as video streams
    
    Some code in these classes has been also refactored/improved/optimized.
    Also fix the extraction of PeerTube audio streams as video streams, which are now returned as audio streams.
    AudricV committed May 28, 2022
    Configuration menu
    Copy the full SHA
    881969f View commit details
    Browse the repository at this point in the history
  6. Add POST_LIVE_STREAM and POST_LIVE_AUDIO_STREAM stream types

    This allows the extractor to determine if a content is an ended audio or video livestream.
    AudricV committed May 28, 2022
    Configuration menu
    Copy the full SHA
    4330b5f View commit details
    Browse the repository at this point in the history
  7. Apply changes in YoutubeStreamExtractor

    Extract post live DVR streams as post live streams instead of live streams.
    
    A new class has been in order to improve code: ItagInfo, which stores an itag, the content (URL) extracted and if its an URL or not.
    A functional interface has been added in order to abstract the stream building: StreamBuilderHelper.
    Also add the cver parameter added by the desktop web client on the corresponding streams (a new method has been added in YoutubeParsingHelper to check this and another for Android streams).
    
    Some code in these classes has been also refactored/improved/optimized.
    AudricV committed May 28, 2022
    Configuration menu
    Copy the full SHA
    a857684 View commit details
    Browse the repository at this point in the history
  8. [YouTube] Add ability to generate manifests of progressive, OTF and p…

    …ost live streams
    
    A new class has been added to do so: YoutubeDashManifestCreator.
    It relies on a new class: ManifestCreatorCache, to cache the content, which relies on a new pair class named Pair.
    Results are cached and there is a cache per delivery type, on which cache limit, clear factor, clearing and resetting can be applied to each cache and to all caches.
    Look at code changes for more details.
    AudricV committed May 28, 2022
    Configuration menu
    Copy the full SHA
    7477ed0 View commit details
    Browse the repository at this point in the history
  9. Update DefaultStreamExtractorTest and SoundcloudStreamExtractorTest t…

    …o support changes made in Stream classes
    AudricV committed May 28, 2022
    Configuration menu
    Copy the full SHA
    f6ec7f9 View commit details
    Browse the repository at this point in the history
  10. Add tests for YoutubeDashManifestCreator and ManifestCreatorCache

    The test added in YoutubeDashManifestCreator uses a video of the Creative Commons channel, licenced under the Creative Commons Attribution licence (reuse allowed).
    Also remove public keywords of tests in UtilsTest, as suggested by SonarLint, because they are not needed with Junit 5.
    AudricV committed May 28, 2022
    Configuration menu
    Copy the full SHA
    6985167 View commit details
    Browse the repository at this point in the history
  11. Improve documentation and adress most of the requested changes

    Also fix some issues in several places, in the code and the documentation.
    AudricV committed May 28, 2022
    Configuration menu
    Copy the full SHA
    aa4c10e View commit details
    Browse the repository at this point in the history
  12. [YouTube] Return a copy of the hardcoded ItagItem instead of returnin…

    …g the reference to the hardcoded one in ItagItem.getItag
    
    To do so, a copy constructor has been added in the class.
    
    This fixes, for instance, an issue in NewPipe, in which the ItagItem values where not the ones corresponsing to a stream but to another, when generating DASH manifests.
    AudricV committed May 28, 2022
    Configuration menu
    Copy the full SHA
    f61e209 View commit details
    Browse the repository at this point in the history
  13. Fix Checkstyle issues, revert resolution string changes for YouTube v…

    …ideo streams and don't return the rn parameter in DASH manifests
    
    This parameter is still used to get the initialization sequence of OTF and POST-live streams, but is not returned anymore in the manifests.
    It has been removed in order to avoid fingerprinting based on the number sent (e.g. when starting to play a stream close to the end and using 123 as the request number where it should be 1) and should be added dynamically by clients in their requests.
    The relevant test has been also updated.
    
    Checkstyle issues in YoutubeDashManifestCreator have been fixed, and the changes in the resolution string returned for video streams in YoutubeStreamExtractor have been reverted, as they create issues on NewPipe right now.
    AudricV committed May 28, 2022
    Configuration menu
    Copy the full SHA
    2fb1a41 View commit details
    Browse the repository at this point in the history
  14. Move ManifestCreatorCache tests to a separate class and remove overri…

    …de of equals and hashCode methods in ManifestCreatorCache
    
    These methods don't need to be overriden, as they are not excepted to be used in collections.
    Also improve the toString method of this class, which contains also now clearFactor and maximumSize attributes and for each operations.
    AudricV committed May 28, 2022
    Configuration menu
    Copy the full SHA
    d64d7bb View commit details
    Browse the repository at this point in the history
  15. Configuration menu
    Copy the full SHA
    436ddde View commit details
    Browse the repository at this point in the history
  16. [YouTube] Support the iOS client in YoutubeDashManifestCreator and de…

    …crypt again the n parameter, if present, for all clients
    
    This commits reverts a new behavior introduced in this branch, which only applied the decryption if needed on streams from the WEB client.
    Also fix rebase issues and documentations style in YoutubeDashManifestCreator.
    AudricV committed May 28, 2022
    Configuration menu
    Copy the full SHA
    07b045f View commit details
    Browse the repository at this point in the history
  17. Configuration menu
    Copy the full SHA
    50272db View commit details
    Browse the repository at this point in the history
  18. Configuration menu
    Copy the full SHA
    159d05c View commit details
    Browse the repository at this point in the history
  19. [YouTube] Secure DashManifestCreator against XEE attack

    Also remove duplicate lines from javadoc comments, otherwise checkstyle complaints.
    XEE prevention is inspired from https://github.com/ChuckerTeam/chucker/pull/201/files
    For an explanation of XEE see https://rules.sonarsource.com/java/RSPEC-2755, though the solution reported there causes crashes on Android
    Stypox authored and AudricV committed May 28, 2022
    Configuration menu
    Copy the full SHA
    ba68b8c View commit details
    Browse the repository at this point in the history
  20. Configuration menu
    Copy the full SHA
    8226fd0 View commit details
    Browse the repository at this point in the history
  21. Configuration menu
    Copy the full SHA
    5c83409 View commit details
    Browse the repository at this point in the history
  22. [YouTube] Refactor YoutubeDashManifestCreator

    - Remove all of the methods used to access caches and replace them with three caches getters
    - Rename caches to shorter and more meaningful names
    - Remove redundant @throws tags that just say "if this method fails to do what it should do", which is obvious
    Stypox authored and AudricV committed May 28, 2022
    Configuration menu
    Copy the full SHA
    3708ab9 View commit details
    Browse the repository at this point in the history
  23. Configuration menu
    Copy the full SHA
    4da05af View commit details
    Browse the repository at this point in the history
  24. [YouTube] Make exception messages smaller

    Stypox authored and AudricV committed May 28, 2022
    Configuration menu
    Copy the full SHA
    00bbe5e View commit details
    Browse the repository at this point in the history
  25. [YouTube] Reduce exception generation code and move several attribute…

    …s of MPD documents into constants
    
    Rename YoutubeDashManifestCreationException to CreationException.
    Also use these constants in YoutubeDashManifestCreatorTest.
    Stypox authored and AudricV committed May 28, 2022
    Configuration menu
    Copy the full SHA
    cfc13f4 View commit details
    Browse the repository at this point in the history
  26. Rename Stream's baseUrl to manifestUrl

    Stypox authored and AudricV committed May 28, 2022
    Configuration menu
    Copy the full SHA
    2321822 View commit details
    Browse the repository at this point in the history
  27. Configuration menu
    Copy the full SHA
    54d323c View commit details
    Browse the repository at this point in the history
  28. [Bandcamp] Fix regression of Opus radio streams extraction

    When moving opus-lo into a constant, opus-lo was renamed to opus_lo and was only used if no MP3 stream was available (which was not the case before the changes in BandcampRadioStreamExtractor related to the addition of the support of all delivery methods), so these changes removed the ability to get Opus streams of Bandcamp radios.
    
    This commit reverts this unwanted change.
    AudricV committed May 28, 2022
    Configuration menu
    Copy the full SHA
    4158fc4 View commit details
    Browse the repository at this point in the history
  29. [YouTube] Return approxDurationMs value from YouTube's player respons…

    …e in ItagItems generated
    
    This change allows to build DASH manifests using YoutubeDashManifestCreator with the real duration of streams and prevent potential cuts of the end of progressive streams, because the duration in YouTube's player response is in seconds and not milliseconds.
    AudricV committed May 28, 2022
    Configuration menu
    Copy the full SHA
    2f3920c View commit details
    Browse the repository at this point in the history
  30. Configuration menu
    Copy the full SHA
    301b9fa View commit details
    Browse the repository at this point in the history
  31. Configuration menu
    Copy the full SHA
    f17f7b9 View commit details
    Browse the repository at this point in the history
  32. [YouTube] Refactor DASH manifests creation

    Move DASH manifests creation into a new subpackage of the YouTube package, dashmanifestcreators.
    This subpackage contains:
    
    - CreationException, exception extending Java's RuntimeException, thrown by manifest creators when something goes wrong;
    - YoutubeDashManifestCreatorsUtils, class which contains all common methods and constants of all or a part of the manifest creators;
    - a manifest creator has been added per delivery type of YouTube streams:
      - YoutubeProgressiveDashManifestCreator, for progressive streams;
      - YoutubeOtfDashManifestCreator, for OTF streams;
      - YoutubePostLiveStreamDvrDashManifestCreator, for post-live DVR streams (which use the live delivery method).
    
    Every DASH manifest creator has a getCache() static method, which returns the ManifestCreatorCache instance used to cache results.
    
    DeliveryType has been also extracted from the YouTube DASH manifest creators part of the extractor and moved to the YouTube package.
    
    YoutubeDashManifestCreatorTest has been updated and renamed to YoutubeDashManifestCreatorsTest, and YoutubeDashManifestCreator has been removed.
    
    Finally, several documentation and exception messages fixes and improvements have been made.
    AudricV committed May 28, 2022
    Configuration menu
    Copy the full SHA
    f7b1515 View commit details
    Browse the repository at this point in the history
  33. [YouTube] Add missing Nonnull annotations in getCache method of YouTu…

    …be DASH manifest creators
    AudricV committed May 28, 2022
    Configuration menu
    Copy the full SHA
    fffbbee View commit details
    Browse the repository at this point in the history
  34. Configuration menu
    Copy the full SHA
    c33d392 View commit details
    Browse the repository at this point in the history
  35. Solve some review comments

    Stypox authored and AudricV committed May 28, 2022
    Configuration menu
    Copy the full SHA
    044639c View commit details
    Browse the repository at this point in the history
  36. Configuration menu
    Copy the full SHA
    d652e05 View commit details
    Browse the repository at this point in the history
  37. Apply code review and Streams rework

    Stypox authored and AudricV committed May 28, 2022
    Configuration menu
    Copy the full SHA
    b3c620f View commit details
    Browse the repository at this point in the history

Commits on May 29, 2022

  1. [SoundCloud] Use the HLS delivery method for all streams and extract …

    …only a single stream URL from HLS manifest for MP3 streams
    
    SoundCloud broke the workaround used to get a single file from HLS manifests for Opus manifests, but it still works for MP3 ones.
    
    The code has been adapted to prevent an unneeded request (the one to the Opus HLS manifest) and the HLS delivery method is now used for SoundCloud MP3 and Opus streams, plus the progressive one (for tracks which have a progressive stream (MP3) and for the ones which doesn't have one, it is still used by trying to get a progressive stream, using the workaround).
    
    Streams extraction has been also moved to Java 8 Stream's API and the relevant test has been also updated.
    AudricV committed May 29, 2022
    Configuration menu
    Copy the full SHA
    287d1df View commit details
    Browse the repository at this point in the history