Download files from own playlist #2193

davidhedlund · 2014-01-22T09:13:18Z

Please implement these features in youtube-dl:

Example of playlist file "Antares - Exodus.txt":
Antares - The Exodus
Antares - Mysticism
Antares - Aurum Solis
Antares - The 4th Dimension
Antares - Dreaming Universe
Antares - Mount Meru
Antares - Astral Plane
Antares - Sun Sanctuary

#!/bin/bash

# youtube-dl-pl-sh is a youtube-dl Bash extension to download Youtube files from text files.

# This file is released to the public domain.
# Created by David Englund

rm *.html

for x in $(find . -name "*.txt"); do


  file_full=$(echo $x | awk -F/ '{print $NF}');
  file=$(echo $x | sed -e "s|.txt||");


################################################
# Find links, Youtube links are outdated regualry so it is pointless to collect them

mkdir $file

  while read line; do

    # At the moment M4A (MP4) "audio only" are only supported on Youtube
    # Download youtube files, primarily "audio only" to not damage SSD drives, and to save time by not having to convert video to audio. Convert proprietary formats to free ones.
    # This might run faster but on the other hand, youtube handles different formats (eg .webm, .m4a) which make this inconvenient.
    # for i in $(cat list-id); do youtube-dl $i.ogg --title $i; ffmpeg -i $i -vn -acodec libvorbis; done

    # context-data-item clearfix = Ads
    # __title__ = Dummy

    # &search_sort=video_view_count (that is https://www.youtube.com/results?search_query=$line&search_sort=video_view_count) will repeat videos with many views and not show it for the unique track.

    # Youtube file facts: M4A files have unknown bitrate.
    # find . -name '*.m4a' -exec sh -c 'ffmpeg -i "$0" -sameq "${0%%.mp4}.ogg"' {} \; # Converting M4A files to OGG like this makes them about 10 times larger in file size

    # Sound quality log:

    # Use: SAVEIFS=$IFS; IFS=$(echo -en "\n\b"); for i in $(find . -type f); do tid=$(mplayer -vo null -ao null -identify -frames 0 "$i" | grep --max-count="1" "BITRATE" | sed "s|ID_AUDIO_BITRATE=||"); if [ $tid -lt 128000  ]; then echo $i >> foo.txt; fi; done; IFS=$SAVEIFS
    #2014-01-21: About 50% of OGG files extracted from WEBM had lower bitrate than 128 kb/s.

# Run detox later
artist="${line%% -*}";
artist_url=$(echo $artist | sed "s|\&|%26|g; s|\/|%26|g; s|!||g; s|?||g;");
artist_html=$(echo $artist | sed "s|\&|\&amp;|g; s|\/|\&amp;|g; s|'|\&#39;|g; s|!||g; s|?||g;");
artist_unix=$(echo $artist | sed "s|\/|\&|g;");
track="${line##*- }";
track_url=$(echo $track | sed "s|\&|%26|g; s|\/|%26|g; s|!||g; s|?||g;");
track_html=$(echo $track | sed "s|\&|\&amp;|g; s|\/|\&amp;|g; s|'|\&#39;|g; s|!||g; s|?||g;");
track_unix=$(echo $track | sed "s|\/|\&|g;");



wget --output-document=1.html "https://www.youtube.com/results?search_query=$artist_url+-+$track_url"; cp -a 1.html 2.html; sed -i '/__title__/d; /full album/Id; /playlist/d; /context-data-item clearfix/d;' 2.html; grep "data-context-item-id=" 2.html | grep -i "$track_html" >> 3.html; cat 3.html >> metadata.log;
id=$(cp -a 3.html 4.html; sed -i "s|data-context-item-id=\"|\ndata-context-item-id=|g; s|\" |\n|g; s|\"|\n|g;" 4.html; grep data-context-item-id 4.html > 5.html; sed -i "s|data-context-item-id=||g; s|\"||" 5.html);


downloaded="no";

while read linex; do

  if [ $downloaded == "no" ]; then


    cd $file
     # Bug: --extract-audio does not work with -o: youtube-dl $linex -o "$line - %(id)s.ogg" -f 43 --extract-audio --no-overwrites
    youtube-dl $linex --id -f 43 --extract-audio --no-overwrites --abort-on-error && mv $linex.ogg "$artist_unix - $track_unix - $linex.ogg" && downloaded="yes"; # --no-overwrites only works with default filenames so we have to include the Youtube ID in the filename make it unique (as done by default with youtubes own filename)
    #   youtube-dl $linex -f 43 --abort-on-error && exit
    cd ..

echo $linex > x

  fi

done <5.html



#detox $x; # Could not be used directly after "mv" above

rm *.html

  done <"$x"

done >> sessions.log

jaimeMF · 2014-01-22T10:04:39Z

It seems that you can do that with youtube-dl 'ytsearch1:Antares The Exodus' (you can use all the normal options to select format, extract audio ...).
I don't really understand if you are requesting something else, could you explain what do you exactly want?

sahutd · 2014-01-22T11:58:11Z

are you suggesting that we run queries on youtube based on the search text in the file?
or
the urls are present in the txt(which does not seem the case) ?

davidhedlund · 2014-01-22T12:40:25Z

On 2014-01-22 12:58, Saimadhav wrote:

are you suggesting that we run queries on youtube based on the search
text in the file?

Yes.

jaimeMF · 2014-01-22T13:20:03Z

Maybe the request is too specific, you can use the --batch-file for that. You just need to write the "urls" in urls.txt (for example):

ytsearch1:Antares The Exodus
ytsearch1:Antares - Mysticism

And then you can just use youtube-dl --batch-file 'urls.txt' (with all the options you need). You could write a simple script that writes urls.txt from the contents of Antares - Exodus.txt.

davidhedlund · 2014-01-22T13:28:49Z

On 2014-01-22 14:20, Jaime Marquínez Ferrándiz wrote:

Maybe the request is too specific, you can use the |--batch-file| for
that. You just need to write the "urls" in |urls.txt| (for example):

|ytsearch1:Antares The Exodus
ytsearch1:Antares - Mysticism
|

And then you can just use |youtube-dl --batch-file 'urls.txt'| (with
all the options you need). You could write a simple script that writes
|urls.txt| from the contents of |Antares - Exodus.txt|.

—
Reply to this email directly or view it on GitHub
#2193 (comment).

How do I quit youtube-dl once it have started to download?

davidhedlund · 2014-01-22T13:30:59Z

On 2014-01-22 14:20, Jaime Marquínez Ferrándiz wrote:

Maybe the request is too specific, you can use the |--batch-file| for
that. You just need to write the "urls" in |urls.txt| (for example):

|ytsearch1:Antares The Exodus
ytsearch1:Antares - Mysticism
|

And then you can just use |youtube-dl --batch-file 'urls.txt'| (with
all the options you need). You could write a simple script that writes
|urls.txt| from the contents of |Antares - Exodus.txt|.

—
Reply to this email directly or view it on GitHub
#2193 (comment).

How do I exclude "full album" results in:

while read line; do youtube-dl "ytsearch1:$line" -o "$line - %(id)s.ogg"
-f 43 --extract-audio --no-overwrites --abort-on-error; done <
Top_tracks.txt

davidhedlund · 2014-01-22T13:52:36Z

On 2014-01-22 14:20, Jaime Marquínez Ferrándiz wrote:

Maybe the request is too specific, you can use the |--batch-file| for
that. You just need to write the "urls" in |urls.txt| (for example):

|ytsearch1:Antares The Exodus
ytsearch1:Antares - Mysticism
|

And then you can just use |youtube-dl --batch-file 'urls.txt'| (with
all the options you need). You could write a simple script that writes
|urls.txt| from the contents of |Antares - Exodus.txt|.

—
Reply to this email directly or view it on GitHub
#2193 (comment).

My personal opinion is that a nifty one-line like this should be
included as an example in the man page:

while read line; do youtube-dl "ytsearch1:$line" -o "$line - %(id)s.ogg"
-f 43 --extract-audio --no-overwrites --abort-on-error --max-filesize
30m; done < Top_tracks.txt

Can you forward that?

phihag · 2014-01-22T14:03:01Z

As @jaimeMF said, your batch file is a really complicated way of writing

#!/bin/sh

for x in $(find . -name "*.txt"); do
    youtube-dl --batch-file "$x" --default-search ytsearch: -o "$(echo $x | sed -e s,.txt,,)/%(playlist)s.%(ext)s" -f 43 --extract-audio --no-overwrites

with the newest version of youtube-dl (2014.01.22.3).

I'm also puzzled of the choice of 43 as the quality, as that is a video format. Although the wearability of modern SSDs is so good that you're unlikely to ever have to worry about it, why download a video format when audio formats are available? A sensible value would be -f bestaudio/43/best. For most YouTube videos, that would mean that recoding is not necessary.

Also, this comment is almost certainly dead wrong:

# Bug: --extract-audio does not work with -o: youtube-dl $linex -o "$line - %(id)s.ogg" -f 43 --extract-audio --no-overwrites

Your output format is obviously defective (you want %(ext)s in there), but I see no reason why it shouldn't work, and no bug entry here. It works just fine for me:

$ youtube-dl --extract-audio  -o '%(id)s.ogg' BaW_jenozKc
[youtube] Setting language
[youtube] BaW_jenozKc: Downloading webpage
[youtube] BaW_jenozKc: Downloading video info webpage
[youtube] BaW_jenozKc: Extracting video information
[download] Destination: BaW_jenozKc.ogg
[download] 100% of 154.06KiB in 00:00
[ffmpeg] Destination: BaW_jenozKc.m4a
Deleting original file BaW_jenozKc.ogg (pass -k to keep)

davidhedlund · 2014-01-22T14:06:11Z

Webm contains OGG which respect freedom contrary to MP*

You are right in everything else in this email.

On 2014-01-22 15:03, Philipp Hagemeister wrote:

As @jaimeMF https://github.com/jaimeMF said, your batch file is a
really complicated way of writing

|#!/bin/sh

for x in $(find . -name "*.txt"); do
youtube-dl --batch-file "$x" --default-search ytsearch: -o "$(echo $x | sed -e s,.txt,,)/%(playlist)s.%(ext)s" -f 43 --extract-audio --no-overwrites
|

with the newest version of youtube-dl (2014.01.22.3).

I'm also puzzled of the choice of 43 as the quality, as that is a
video format. Although the wearability of modern SSDs is so good that
you're unlikely to ever have to worry about it, why download a video
format when audio formats are available? A sensible value would be |-f
bestaudio/43/best|. For most YouTube videos, that would mean that
recoding is not necessary.

Also, the comment is almost certainly ddead wrong

|# Bug: --extract-audio does not work with -o: youtube-dl $linex -o "$line - %(id)s.ogg" -f 43 --extract-audio --no-overwrites
|

Your output format is obviously defective (you want |%(ext)s| in
there), but I see no reason why it shouldn't work, and no bug entry
here. It works just fine for me:

|$ youtube-dl --extract-audio -o '%(id)s.ogg' BaW_jenozKc
[youtube] Setting language
[youtube] BaW_jenozKc: Downloading webpage
[youtube] BaW_jenozKc: Downloading video info webpage
[youtube] BaW_jenozKc: Extracting video information
[download] Destination: BaW_jenozKc.ogg
[download] 100% of 154.06KiB in 00:00
[ffmpeg] Destination: BaW_jenozKc.m4a
Deleting original file BaW_jenozKc.ogg (pass -k to keep)
|

—
Reply to this email directly or view it on GitHub
#2193 (comment).

jaimeMF · 2014-01-22T14:10:53Z

Some videos have an audio only version in webmwith itags 171 and 172. you could use -f 172/171/43 to request them.

davidhedlund · 2014-01-22T14:12:25Z

On 2014-01-22 14:20, Jaime Marquínez Ferrándiz wrote:

Maybe the request is too specific, you can use the |--batch-file| for
that. You just need to write the "urls" in |urls.txt| (for example):

|ytsearch1:Antares The Exodus
ytsearch1:Antares - Mysticism
|

And then you can just use |youtube-dl --batch-file 'urls.txt'| (with
all the options you need). You could write a simple script that writes
|urls.txt| from the contents of |Antares - Exodus.txt|.

—
Reply to this email directly or view it on GitHub
#2193 (comment).

Can I "ytsearch1:Antares - Mysticism" at the end of line to ensure that
no remix is included in the search (eg Antares - Mysticism (SOME ARTIST
REMIX))?

phihag · 2014-01-22T14:15:00Z

If you want webm and other free formats, simply pass in --prefer-free-formats. There is really no need to pass in any other format specification then.

You can use --reject-title REMIX as well as simply include -remix in the search query.

davidhedlund · 2014-01-22T14:25:14Z

On 2014-01-22 15:11, Jaime Marquínez Ferrándiz wrote:

Some videos have an audio only version in |webm|with itags |171| and
|172|. you could use |-f 172/171/43| to request them.

—
Reply to this email directly or view it on GitHub
#2193 (comment).

"from Philipp Hagemeister

If you want webm and other free formats, simply pass in
|--prefer-free-formats|. There is really no need to pass in any other
format specification then.

You can use |--reject-title REMIX| as well as simply include |-remix| in
the search query.

—
Reply to this email directly or view it on GitHub
#2193 (comment).

"

Would a mixture of your suggestions be ideal for saving disk space,
time, and freedom, or should I stick with one of the solutions (if yes,
which one?)?

davidhedlund · 2014-01-22T14:42:33Z

How do I do this with --prefer-free-formats?

for x in $(find . -name "*.txt"); do
youtube-dl --batch-file "$x" --default-search ytsearch: -o "$(echo
$x | sed -e s,.txt,,)/%(playlist)s.%(ext)s" --prefer-free-formats
--extract-audio --no-overwrites --ignore-errors --max-filesize 30m
done

That downloads M4A instead of WEBM. What am I doing wrong?

On 2014-01-22 15:03, Philipp Hagemeister wrote:

As @jaimeMF https://github.com/jaimeMF said, your batch file is a
really complicated way of writing

|#!/bin/sh

for x in $(find . -name "*.txt"); do
youtube-dl --batch-file "$x" --default-search ytsearch: -o "$(echo $x | sed -e s,.txt,,)/%(playlist)s.%(ext)s" -f 43 --extract-audio --no-overwrites
|

with the newest version of youtube-dl (2014.01.22.3).

I'm also puzzled of the choice of 43 as the quality, as that is a
video format. Although the wearability of modern SSDs is so good that
you're unlikely to ever have to worry about it, why download a video
format when audio formats are available? A sensible value would be |-f
bestaudio/43/best|. For most YouTube videos, that would mean that
recoding is not necessary.

Also, the comment is almost certainly ddead wrong

|# Bug: --extract-audio does not work with -o: youtube-dl $linex -o "$line - %(id)s.ogg" -f 43 --extract-audio --no-overwrites
|

Your output format is obviously defective (you want |%(ext)s| in
there), but I see no reason why it shouldn't work, and no bug entry
here. It works just fine for me:

|$ youtube-dl --extract-audio -o '%(id)s.ogg' BaW_jenozKc
[youtube] Setting language
[youtube] BaW_jenozKc: Downloading webpage
[youtube] BaW_jenozKc: Downloading video info webpage
[youtube] BaW_jenozKc: Extracting video information
[download] Destination: BaW_jenozKc.ogg
[download] 100% of 154.06KiB in 00:00
[ffmpeg] Destination: BaW_jenozKc.m4a
Deleting original file BaW_jenozKc.ogg (pass -k to keep)
|

—
Reply to this email directly or view it on GitHub
#2193 (comment).

davidhedlund · 2014-01-22T14:58:44Z

On 2014-01-22 15:15, Philipp Hagemeister wrote:

If you want webm and other free formats, simply pass in
|--prefer-free-formats|. There is really no need to pass in any other
format specification then.

You can use |--reject-title REMIX| as well as simply include |-remix|
in the search query.

—
Reply to this email directly or view it on GitHub
#2193 (comment).

youtube-dl XCHrWyMpPhM --prefer-free-formats # Downloads M4A instead of WEBM

youtube-dl XCHrWyMpPhM -F # Shows WEBM that contains OGG (free)

Is this a bug?

phihag · 2014-01-22T16:30:10Z

Everything is working as intended. The default preference for audio extracting (bestaudio/best) with --prefer-free-formats is (simplified)

webm audio
audio
webm HQ video + audio
HQ video + audio
webm video + audio
video + audio

Therefore, if there's no webm audio format, you'll get mp4 audio. If you pass in -f 172/171/43/webm/bestaudio/best, you're changing that to

webm audio
webm HQ video + audio
webm video + audio
audio
HQ video + audio
video + audio

Which one you prefer is up to you - would you rather avoid non-free formats at all cost or load a far smaller audio-only file in a non-free format?

davidhedlund · 2014-01-22T18:44:42Z

FSF would certainly say that |-f 172/171/43/webm/ is the only way to go
as the rest is non-free. Can you please make a new option
"--require-free-formats" that run ||-f 172/171/43/webm/|

By the way, what is the last "webm" in |-f 172/171/43/webm/ for? Isn't
43 for webm?|

On 2014-01-22 17:30, Philipp Hagemeister wrote:

Everything is working as intended. The default preference for audio
extracting (|bestaudio/best|) with |--prefer-free-formats| is (simplified)

|webm audio
audio
webm HQ video + audio
HQ video + audio
webm video + audio
video + audio
|

Therefore, if there's no webm audio format, you'll get mp4 audio. If
you pass in |-f 172/171/43/webm/bestaudio/best|, you're changing that to

|webm audio
webm HQ video + audio
webm video + audio
audio
HQ video + audio
video + audio
|

Which one you prefer is up to you - would you rather avoid non-free
formats at all cost or load a far smaller audio-only file in a
non-free format?

—
Reply to this email directly or view it on GitHub
#2193 (comment).

phihag · 2014-01-22T18:53:03Z

See #2196 for --require-free-formats. In short, I do think that virtually all users would prefer to get a video (which can be played by open-source software like mplayer and vlc just fine) than get no video.

43 is just one of the webm formats on YouTube, and will match webm on other sites.

davidhedlund · 2014-01-22T19:14:14Z

On 2014-01-22 19:53, Philipp Hagemeister wrote:

See #2196 #2196 for
|--require-free-formats|. In short, I do think that virtually all
users would prefer to get a video (which can be played by open-source
software like mplayer and vlc just fine) than get no video.

43 is just one of the |webm| formats on YouTube, and will match webm
on other sites.

—
Reply to this email directly or view it on GitHub
#2193 (comment).

Thanks for everything. I guess this is what I wanted after all.

#!/bin/sh

for x in $(find . -name "*.txt"); do
youtube-dl --batch-file "$x" --default-search ytsearch: -o "$(echo $x |
sed -e s,.txt,,)/%(playlist)s - %(id)s.%(ext)s" -f webm/ogg
--extract-audio --no-overwrites --max-filesize 30m -i
done

Please share =)

davidhedlund · 2014-01-22T19:29:54Z

youtube-dl --batch-file "Afgin - Old Is Gold" --default-search ytsearch:
-o "$(echo "Afgin - Old Is Gold" | sed -e s,.txt,,)/%(playlist)s -
%(id)s.%(ext)s" -f webm/ogg --extract-audio --no-overwrites
--max-filesize 30m -i

This will download "Afgin - Old Is Gold (Part 2)", not "Afgin - Old Is
Gold". How do I search exact phrase and case insensetive?

phihag · 2014-01-22T20:07:20Z

youtube-dl doesn't implement any special kind of search, it's just relaying the YouTube search. Therefore, you may be able to use tricks from the YouTube search (maybe -part or so). I believe the YouTube search to be case-insensitive.

davidhedlund · 2014-01-22T20:34:18Z

On 2014-01-22 21:07, Philipp Hagemeister wrote:

youtube-dl doesn't implement any special kind of search, it's just
relaying the YouTube search. Therefore, you may be able to use tricks
from the YouTube search (maybe |-part| or so). I believe the YouTube
search to be case-insensitive.

—
Reply to this email directly or view it on GitHub
#2193 (comment).

How do I define to search only type "video" (eg not "playlist") are
included in youtube-dl?

davidhedlund · 2014-01-22T20:55:27Z

On 2014-01-22 21:07, Philipp Hagemeister wrote:

youtube-dl doesn't implement any special kind of search, it's just
relaying the YouTube search. Therefore, you may be able to use tricks
from the YouTube search (maybe |-part| or so). I believe the YouTube
search to be case-insensitive.

—
Reply to this email directly or view it on GitHub
#2193 (comment).

How can I make this try next youtube hit as I receive "ERROR: requested
format not available":

youtube-dl https://www.youtube.com/watch?v=-sN9ATB7juU -f webm/ogg

phihag · 2014-01-22T21:19:47Z

See #2203 for an answer to that.

phihag added a commit that referenced this issue Jan 22, 2014

Add new --default-search option (#2193)

04b4d39

phihag closed this as completed Jan 22, 2014

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Download files from own playlist #2193

Download files from own playlist #2193

davidhedlund commented Jan 22, 2014

jaimeMF commented Jan 22, 2014

sahutd commented Jan 22, 2014

davidhedlund commented Jan 22, 2014

jaimeMF commented Jan 22, 2014

davidhedlund commented Jan 22, 2014

davidhedlund commented Jan 22, 2014

davidhedlund commented Jan 22, 2014

phihag commented Jan 22, 2014

davidhedlund commented Jan 22, 2014

jaimeMF commented Jan 22, 2014

davidhedlund commented Jan 22, 2014

phihag commented Jan 22, 2014

davidhedlund commented Jan 22, 2014

davidhedlund commented Jan 22, 2014

davidhedlund commented Jan 22, 2014

phihag commented Jan 22, 2014

davidhedlund commented Jan 22, 2014

phihag commented Jan 22, 2014

davidhedlund commented Jan 22, 2014

davidhedlund commented Jan 22, 2014

phihag commented Jan 22, 2014

davidhedlund commented Jan 22, 2014

davidhedlund commented Jan 22, 2014

phihag commented Jan 22, 2014

Download files from own playlist #2193

Download files from own playlist #2193

Comments

davidhedlund commented Jan 22, 2014

jaimeMF commented Jan 22, 2014

sahutd commented Jan 22, 2014

davidhedlund commented Jan 22, 2014

jaimeMF commented Jan 22, 2014

davidhedlund commented Jan 22, 2014

davidhedlund commented Jan 22, 2014

davidhedlund commented Jan 22, 2014

phihag commented Jan 22, 2014

davidhedlund commented Jan 22, 2014

jaimeMF commented Jan 22, 2014

davidhedlund commented Jan 22, 2014

phihag commented Jan 22, 2014

davidhedlund commented Jan 22, 2014

davidhedlund commented Jan 22, 2014

davidhedlund commented Jan 22, 2014

phihag commented Jan 22, 2014

davidhedlund commented Jan 22, 2014

phihag commented Jan 22, 2014

davidhedlund commented Jan 22, 2014

davidhedlund commented Jan 22, 2014

phihag commented Jan 22, 2014

davidhedlund commented Jan 22, 2014

davidhedlund commented Jan 22, 2014

phihag commented Jan 22, 2014