Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support delivery methods other than progressive HTTP #810

Merged
merged 38 commits into from
Jun 2, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
2f061b8
Add support of other delivery methods than progressive HTTP in Stream…
AudricV Mar 3, 2022
ad993b9
Remove fetching of the DASH manifest extracted when getting informati…
AudricV Mar 3, 2022
7c67d46
Move DashMpdParser to the YouTube package and fix extraction of streams
AudricV Mar 3, 2022
d5f3637
[YouTube] Return more values returned inside the ItagItems of the pla…
AudricV Mar 3, 2022
881969f
Apply changes in all StreamExtractors except YouTube's one and fix ex…
AudricV Mar 3, 2022
4330b5f
Add POST_LIVE_STREAM and POST_LIVE_AUDIO_STREAM stream types
AudricV Mar 6, 2022
a857684
Apply changes in YoutubeStreamExtractor
AudricV Mar 6, 2022
7477ed0
[YouTube] Add ability to generate manifests of progressive, OTF and p…
AudricV Mar 6, 2022
f6ec7f9
Update DefaultStreamExtractorTest and SoundcloudStreamExtractorTest t…
AudricV Mar 8, 2022
6985167
Add tests for YoutubeDashManifestCreator and ManifestCreatorCache
AudricV Mar 8, 2022
aa4c10e
Improve documentation and adress most of the requested changes
AudricV Mar 15, 2022
f61e209
[YouTube] Return a copy of the hardcoded ItagItem instead of returnin…
AudricV Mar 28, 2022
2fb1a41
Fix Checkstyle issues, revert resolution string changes for YouTube v…
AudricV Apr 3, 2022
d64d7bb
Move ManifestCreatorCache tests to a separate class and remove overri…
AudricV Apr 3, 2022
436ddde
Use assertThrows in YoutubeDashManifestCreatorTest
AudricV Apr 3, 2022
07b045f
[YouTube] Support the iOS client in YoutubeDashManifestCreator and de…
AudricV Apr 11, 2022
50272db
Apply reviews: improve comments, remove FILE, remove Stream#equals(St…
Stypox Apr 30, 2022
159d05c
Remove unused DashMpdParser (but kept in git history)
Stypox Apr 30, 2022
ba68b8c
[YouTube] Secure DashManifestCreator against XEE attack
Stypox Apr 30, 2022
8226fd0
[YouTube] Suppress Sonar security warning for XEE
Stypox Apr 30, 2022
5c83409
[YouTube] Rewrite manifest test and rename long methods
Stypox May 1, 2022
3708ab9
[YouTube] Refactor YoutubeDashManifestCreator
Stypox May 1, 2022
4da05af
[YouTube] Inline collectSegmentsData in YoutubeDashManifestCreator
Stypox May 1, 2022
00bbe5e
[YouTube] Make exception messages smaller
Stypox May 1, 2022
cfc13f4
[YouTube] Reduce exception generation code and move several attribute…
Stypox May 1, 2022
2321822
Rename Stream's baseUrl to manifestUrl
Stypox May 1, 2022
54d323c
Fix Checkstyle issue in YoutubeDashManifestCreator
AudricV May 2, 2022
4158fc4
[Bandcamp] Fix regression of Opus radio streams extraction
AudricV May 7, 2022
2f3920c
[YouTube] Return approxDurationMs value from YouTube's player respons…
AudricV May 7, 2022
301b9fa
Remove hashCode and equals methods overrides of Stream classes
AudricV May 7, 2022
f17f7b9
Apply requested changes in YoutubeParsingHelper
AudricV May 10, 2022
f7b1515
[YouTube] Refactor DASH manifests creation
AudricV May 11, 2022
fffbbee
[YouTube] Add missing Nonnull annotations in getCache method of YouTu…
AudricV May 14, 2022
c33d392
Fixed typo XEE → XXE (Xml eXternal Entity attack)
litetex May 16, 2022
044639c
Solve some review comments
Stypox May 18, 2022
d652e05
[MediaCCC] Fix comments about containsSimilarStream
Stypox May 20, 2022
b3c620f
Apply code review and Streams rework
Stypox May 27, 2022
287d1df
[SoundCloud] Use the HLS delivery method for all streams and extract …
AudricV May 29, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -30,9 +30,12 @@
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.BASE_API_URL;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.BASE_URL;
import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImageUrl;
import static org.schabi.newpipe.extractor.utils.Utils.EMPTY_STRING;

public class BandcampRadioStreamExtractor extends BandcampStreamExtractor {

private static final String OPUS_LO = "opus-lo";
private static final String MP3_128 = "mp3-128";
private JsonObject showInfo;

public BandcampRadioStreamExtractor(final StreamingService service,
Expand Down Expand Up @@ -116,23 +119,27 @@ public long getLength() {

@Override
public List<AudioStream> getAudioStreams() {
final ArrayList<AudioStream> list = new ArrayList<>();
final List<AudioStream> audioStreams = new ArrayList<>();
final JsonObject streams = showInfo.getObject("audio_stream");

if (streams.has("opus-lo")) {
list.add(new AudioStream(
streams.getString("opus-lo"),
MediaFormat.OPUS, 100
));
if (streams.has(MP3_128)) {
audioStreams.add(new AudioStream.Builder()
.setId(MP3_128)
.setContent(streams.getString(MP3_128), true)
.setMediaFormat(MediaFormat.MP3)
.setAverageBitrate(128)
.build());
}
if (streams.has("mp3-128")) {
list.add(new AudioStream(
streams.getString("mp3-128"),
MediaFormat.MP3, 128
));

if (streams.has(OPUS_LO)) {
audioStreams.add(new AudioStream.Builder()
.setId(OPUS_LO)
.setContent(streams.getString(OPUS_LO), true)
.setMediaFormat(MediaFormat.OPUS)
.setAverageBitrate(100).build());
}

return list;
return audioStreams;
}

@Nonnull
Expand All @@ -156,14 +163,14 @@ public List<StreamSegment> getStreamSegments() throws ParsingException {
@Override
public String getLicence() {
// Contrary to other Bandcamp streams, radio streams don't have a license
return "";
return EMPTY_STRING;
}

@Nonnull
@Override
public String getCategory() {
// Contrary to other Bandcamp streams, radio streams don't have categories
return "";
return EMPTY_STRING;
}

@Nonnull
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,15 @@
package org.schabi.newpipe.extractor.services.bandcamp.extractors;

import static org.schabi.newpipe.extractor.services.bandcamp.extractors.BandcampExtractorHelper.getImageUrl;
import static org.schabi.newpipe.extractor.utils.Utils.EMPTY_STRING;
import static org.schabi.newpipe.extractor.utils.Utils.HTTPS;

import com.grack.nanojson.JsonObject;
import com.grack.nanojson.JsonParserException;

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import org.schabi.newpipe.extractor.MediaFormat;
import org.schabi.newpipe.extractor.StreamingService;
import org.schabi.newpipe.extractor.downloader.Downloader;
Expand All @@ -27,16 +28,15 @@
import org.schabi.newpipe.extractor.utils.JsonUtils;
import org.schabi.newpipe.extractor.utils.Utils;

import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;

import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.util.stream.Collectors;

public class BandcampStreamExtractor extends StreamExtractor {

private JsonObject albumJson;
private JsonObject current;
private Document document;
Expand Down Expand Up @@ -88,7 +88,7 @@ public String getName() throws ParsingException {
public String getUploaderUrl() throws ParsingException {
final String[] parts = getUrl().split("/");
// https: (/) (/) * .bandcamp.com (/) and leave out the rest
return "https://" + parts[2] + "/";
return HTTPS + parts[2] + "/";
}

@Nonnull
Expand Down Expand Up @@ -119,10 +119,10 @@ public DateWrapper getUploadDate() throws ParsingException {
@Override
public String getThumbnailUrl() throws ParsingException {
if (albumJson.isNull("art_id")) {
return Utils.EMPTY_STRING;
} else {
return getImageUrl(albumJson.getLong("art_id"), true);
return EMPTY_STRING;
}

return getImageUrl(albumJson.getLong("art_id"), true);
}

@Nonnull
Expand All @@ -139,24 +139,26 @@ public String getUploaderAvatarUrl() {
public Description getDescription() {
final String s = Utils.nonEmptyAndNullJoin(
"\n\n",
new String[]{
new String[] {
current.getString("about"),
current.getString("lyrics"),
current.getString("credits")
}
);
});
return new Description(s, Description.PLAIN_TEXT);
}

@Override
public List<AudioStream> getAudioStreams() {
final List<AudioStream> audioStreams = new ArrayList<>();

audioStreams.add(new AudioStream(
albumJson.getArray("trackinfo").getObject(0)
.getObject("file").getString("mp3-128"),
MediaFormat.MP3, 128
));
audioStreams.add(new AudioStream.Builder()
.setId("mp3-128")
.setContent(albumJson.getArray("trackinfo")
.getObject(0)
.getObject("file")
.getString("mp3-128"), true)
.setMediaFormat(MediaFormat.MP3)
.setAverageBitrate(128)
.build());
return audioStreams;
}

Expand Down Expand Up @@ -184,11 +186,11 @@ public StreamType getStreamType() {
@Override
public PlaylistInfoItemsCollector getRelatedItems() {
final PlaylistInfoItemsCollector collector = new PlaylistInfoItemsCollector(getServiceId());
final Elements recommendedAlbums = document.getElementsByClass("recommended-album");
document.getElementsByClass("recommended-album")
.stream()
.map(BandcampRelatedPlaylistInfoItemExtractor::new)
.forEach(collector::commit);

for (final Element album : recommendedAlbums) {
collector.commit(new BandcampRelatedPlaylistInfoItemExtractor(album));
}
return collector;
}

Expand All @@ -200,15 +202,17 @@ public String getCategory() {
.flatMap(element -> element.getElementsByClass("tag").stream())
.map(Element::text)
.findFirst()
.orElse("");
.orElse(EMPTY_STRING);
}

@Nonnull
@Override
public String getLicence() {
/* Tests resulted in this mapping of ints to licence:
/*
Tests resulted in this mapping of ints to licence:
https://cloud.disroot.org/s/ZTWBxbQ9fKRmRWJ/preview (screenshot from a Bandcamp artist's
account) */
account)
*/

switch (current.getInt("license_type")) {
case 1:
Expand All @@ -233,14 +237,9 @@ public String getLicence() {
@Nonnull
@Override
public List<String> getTags() {
final Elements tagElements = document.getElementsByAttributeValue("itemprop", "keywords");

final List<String> tags = new ArrayList<>();

for (final Element e : tagElements) {
tags.add(e.text());
}

return tags;
return document.getElementsByAttributeValue("itemprop", "keywords")
.stream()
.map(Element::text)
.collect(Collectors.toList());
}
}
Loading