Dlib's equivalent to cv::VideoCapture, but better

[pfeatherstone]

A fresh PR has landed in dlib which includes:

• dlib::ffmpeg::decoder
• dlib::ffmpeg::demuxer

which you can find by adding

#include <dlib/media.h>

to your code. These are lightweight wrappers of libavformat and libavcodec, the underlying C libraries used by ffmpeg.
You will need to build dlib with -DDLIB_USE_FFMPEG=ON. By default, if an installation off ffmpeg (libavformat, libavcodec) is found on your system, then this will be done automatically.

dlib::ffmpeg::demuxer is similar to OpenCV's cv::VideoCapture except it can do far more! It can read:

• video files
• audio files
• RTSP streams
• webcam devices
• screen grabs
• microphones
• etc
• basically anything ffmpeg can open via ffmpeg -i <>

dlib::ffmpeg::decoder is an object used to decode and extract frames (images or audio) from RAW data.

If you are interested, please read the documentation in dlib/media/ffmpeg_abstract.h and look at the new examples which start with ffmpeg_ in the filename.

I would be very keen for people to try it out. If they have any issues, please submit github issues using the template and a reproducible example, ideally with a main.cpp file and CMakeLists.txt file.
The more people use it, the more quickly potential issues get ironed out, the more robust it gets. Though I already expect it to be fairly mature, you never know. Eventually, hopefully some time soon, it will be part of the next dlib release.

[pfeatherstone]

Future PRs will include

• dlib::ffmpeg::encoder
• dlib::ffmpeg::muxer
• support for ffmpeg 5
• hardware acceleration

[pfeatherstone]

I think wait for encoder+muxer PR (coming shortly. Basically done but needs docs and stuff). Hopefully people will have played with it by then. Then release

[pfeatherstone]

I was also going to PR some more advanced examples.

[pfeatherstone]

Like setting advanced codec, format, protocol options if required.

[davisking]

Sweet. That all sounds great.

[pfeatherstone]

I would also wait for ffmpeg 5 support. That shouldn't take long at all (though I need to find time to do it). But it may require a slight API change. Would be good to have a fairly fixed API before the release.
Basically, i either need to define a custom channel_layout object which wraps both the old and new channel layout customization in ffmpeg. Or i write some adaptors to convert between old and new formats if they don't exist already. The former is an API break. The latter maybe not. Either way, that should be in dlib before the release. Then there should be little for users to complain about. Things like support for hardware acceleration don't affect the API.

[rossfg]

currently experiencing build errors with examples in ffmpeg code.

This is on ubuntu 22.04.2 LTS
using gcc 11.3 and ffmpeg 4.4.2

seems to be some misplaced const declarations in media/ffmpeg*.h

here is an error output during compile:

In file included from /home/rfg/w/dlib/dlib/../dlib/media.h:13,
from /home/rfg/w/dlib/examples/ffmpeg_webcam_face_pose_ex.cpp:35:
/home/rfg/w/dlib/dlib/../dlib/media/ffmpeg_demuxer.h: In member function ‘bool dlib::ffmpeg::demuxer::open(const dlib::ffmpeg::demuxer::args&)’:
/home/rfg/w/dlib/dlib/../dlib/media/ffmpeg_demuxer.h:657:73: error: invalid conversion from ‘const AVInputFormat*’ to ‘AVInputFormat*’ [-fpermissive]
657 | AVInputFormat* input_format = st.args_.input_format.empty() ? nullptr : av_find_input_format(st.args_.input_format.c_str());
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| |
| const AVInputFormat*

/home/rfg/w/dlib/dlib/../dlib/media/ffmpeg_demuxer.h: In lambda function:
/home/rfg/w/dlib/dlib/../dlib/media/ffmpeg_demuxer.h:695:100: error: invalid conversion from ‘AVCodec**’ to ‘const AVCodec**’ [-fpermissive]
695 | v_find_best_stream(st.pFormatCtx.get(), media_type, -1, -1, &pCodec, 0);
| ^~~~~~~
| |
| AVCodec**

Ross

[pfeatherstone]

I have a PR in at the moment that fixes this :)

[pfeatherstone]

Can you file an issue? That's more likely to get the PR in quicker.

[pfeatherstone]

Also, you have a weird installation of ffmpeg. ffmpeg 4.4.3 comes with libavformat 58.76.100. Somehow your installation of ffmpeg 4.4.2 has libavformat >= 59.0.100. So I would also check your version of ffmpeg

[pfeatherstone]

@rossfg can you close this thread as it's in a GitHub issue now?

[davisking]

Ha, I'm paying attention :)

I thought you were still working on the PR or I would have merged it already :p

[pfeatherstone]

FFmpeg 6.0 is out. So more compatibility layer PRs coming

[arrufat]

You're doing a great and appreciated job by maintaining this, I will port all my stuff that depends on OpenCV for the camera to dlib and report any issues soon :)

[davisking]

Indeed :D

[pfeatherstone]

FFmpeg 6.0 should be supported in master. If not, issues please

[arrufat]

I've built all the examples with FFmpeg 6 and it works great! Thank you so much!

[rossfg]

Just tested with ffmpeg 5.1.2 and 5.1.3

...
[ 96%] Building CXX object CMakeFiles/ffmpeg_webcam_face_pose_ex.dir/ffmpeg_webcam_face_pose_ex.cpp.o
[ 96%] Linking CXX executable ffmpeg_webcam_face_pose_ex
[ 96%] Built target ffmpeg_webcam_face_pose_ex
[ 96%] Building CXX object CMakeFiles/ffmpeg_video_demuxing_ex.dir/ffmpeg_video_demuxing_ex.cpp.o
[ 97%] Linking CXX executable ffmpeg_video_demuxing_ex
[ 97%] Built target ffmpeg_video_demuxing_ex
[ 98%] Building CXX object CMakeFiles/ffmpeg_video_decoding_ex.dir/ffmpeg_video_decoding_ex.cpp.o
[ 98%] Linking CXX executable ffmpeg_video_decoding_ex
[ 98%] Built target ffmpeg_video_decoding_ex
[ 98%] Building CXX object CMakeFiles/ffmpeg_info_ex.dir/ffmpeg_info_ex.cpp.o
[ 99%] Linking CXX executable ffmpeg_info_ex
[ 99%] Built target ffmpeg_info_ex
[ 99%] Building CXX object CMakeFiles/ffmpeg_screen_grab_ex.dir/ffmpeg_screen_grab_ex.cpp.o
[ 99%] Linking CXX executable ffmpeg_screen_grab_ex
[ 99%] Built target ffmpeg_screen_grab_ex
[ 99%] Building CXX object CMakeFiles/webcam_face_pose_ex.dir/webcam_face_pose_ex.cpp.o
[100%] Linking CXX executable webcam_face_pose_ex
[100%] Built target webcam_face_pose_ex
rfg@r-infinity:~/w/dlib/examples/build$

looks much better!

I'll let you know how it goes with ffmpeg 6.0.

Thanks all,

Ross

[pfeatherstone]

Does anybody care about python bindings for this ? I don't personally but I can oblige.

[arrufat]

So far, I don't, but Ican definitelysee myself needing them in the future.
I think we can add them as we need them.

[davisking]

I don't. There aren't already some python bindings for ffmpeg that are satisfactory in any event?

[pfeatherstone]

Some benchmarks against OpenCV:

#include <chrono>
#include <dlib/config.h>
#include <dlib/media.h>
#include <dlib/array2d.h>
#include <dlib/cmd_line_parser.h>
#include <opencv2/opencv.hpp>

using namespace std::chrono;

void bench_dlib(const std::string& video_path)
{
    using namespace dlib;
    using namespace dlib::ffmpeg;

    demuxer cap(video_path);

    if (!cap.is_open())
    {
        printf("Failed to open %s\n", video_path.c_str());
        return;
    }

    auto start = high_resolution_clock::now();
    frame frame;
    array2d<rgb_pixel> img;
    int counter{0};

    while (cap.read(frame))
    {
        convert(frame, img);
        ++counter;
    }
        
    auto stop = high_resolution_clock::now();
    printf("Dlib read %i frames in %f seconds\n", counter, duration_cast<milliseconds>(stop-start).count() * 1e-3);
}

void bench_opencv(const std::string& video_path)
{
    cv::VideoCapture cap(video_path);

    if (!cap.isOpened())
    {
        printf("Failed to open %s\n", video_path.c_str());
        return;
    }

    auto start = high_resolution_clock::now();
    cv::Mat frame;
    int counter{0};

    while (cap.read(frame))
        ++counter;

    auto stop = high_resolution_clock::now();
    printf("Opencv read %i frames in %f seconds\n", counter, duration_cast<milliseconds>(stop-start).count() * 1e-3);
}

int main(int argc, const char** argv)
{
    dlib::command_line_parser parser;
    parser.add_option("i",      "input video", 1);
    parser.set_group_name("Help Options");
    parser.add_option("h",      "alias of --help");
    parser.add_option("help",   "display this message and exit");

    parser.parse(argc, argv);
    const char* one_time_opts[] = {"i"};
    parser.check_one_time_options(one_time_opts);

    if (parser.option("h") || parser.option("help"))
    {
        parser.print_options();
        return 0;
    }

    bench_opencv(parser.option("i").argument());
    bench_dlib(parser.option("i").argument());
    return EXIT_SUCCESS;
}

using

cmake_minimum_required(VERSION 3.13)

project(Test)

find_package( OpenCV REQUIRED )
find_package( dlib REQUIRED )

add_executable(Test
    src/main.cpp)

target_link_libraries(Test 
    PRIVATE dlib::dlib
    PRIVATE ${OpenCV_LIBS})

target_include_directories(Test
    PRIVATE ${OpenCV_INCLUDE_DIRS})

On my machine, I get:

Opencv read 752 frames in 0.317000 seconds
Dlib read 752 frames in 1.189000 seconds

[pfeatherstone]

Not sure if this is the whole story. But opencv definitely uses multiple cores. So why shouldn't dlib too.

[pfeatherstone]

It also depends on the container type and codec. OpenCV has multiple backends: ffmpeg, GStreamer, V4L2 and native. It's possible that for some of the native backends, it's faster than ffmpeg. I've just done some tests on other video files, and in a lot of cases, without fiddling with AVCodecContext::thread_count dlib's ffmpeg wrapper was faster than OpenCV. So that's not to say "job done", but it's not that bad.

[pfeatherstone]

Ultimately, we (I) need to run a profiler on all of this.

[pfeatherstone]

By the way, you can now set:

• demuxer::args::args_codec_image.thread_count
• demuxer::args::args_codec_audio.thread_count

I did another test where I set these to std::thread::hardware_concurrency() and it beat OpenCV. So that's job done for me.

[UlrichVonRekkenin]

By the way, you can now set:

* `demuxer::args::args_codec_image.thread_count`

* `demuxer::args::args_codec_audio.thread_count`

I did another test where I set these to std::thread::hardware_concurrency() and it beat OpenCV. So that's job done for me.

Opencv read 128 frames in 13.061000 seconds
Dlib read 128 frames in 12.815000 seconds

with this one:

    demuxer cap([&]
                {
                    ffmpeg::demuxer::args args;
                    args.filepath   = video_path;
                    args.args_codec_image.thread_count = std::thread::hardware_concurrency();
                    return args;
                }());

and release.

[arrufat]

I am still reading through the examples and they're really nice!
I think my most common use case will be to generate a video out of some frames that I got.
Is there an example for that? I will try to see if I can do it :)
Thank you!

[pfeatherstone]

decoder and encoder only concern themselves with encoded frames/audio using codecs, e.g. H264, MPEG4. demuxer and muxer strip/interleave encoded packets from/to containers as well as using either decoder or encoder under the hood for extracting/ writing encoded packets and removing/adding header information relevant to the container.

[pfeatherstone]

All this feedback is good. Hopefully it will lead to better docs, and if required, better APIs

[arrufat]

Thank you, I'll keep playing with this and let you know :)

[arrufat]

I would like to change the quality of the video, I was trying by doing something like this:

            dlib::ffmpeg::muxer::args args;
            args.filepath = "output.webm";
            args.format_options["crf"] = "4";
            args.enable_image = true;
            args.enable_audio = false;
            args.args_image.codec_name = "libvpx-vp9";
            args.args_image.h = source.nr();
            args.args_image.w = source.nc();
            args.args_image.framerate = 30;

But it still says: Neither bitrate nor constrained quality specified, using default CRF of 32.
I know I need to read more the FFmpeg docs, but that's maybe one thing that can be provided in the examples :)

[arrufat]

Solved, I needed to use the codec_options, inside the args_image:

    dlib::ffmpeg::muxer writer(
        [&]
        {
            dlib::ffmpeg::muxer::args args;
            args.filepath = "output.webm";
            args.enable_image = true;
            args.enable_audio = false;
            args.args_image.codec_name = "libvpx-vp9";
            args.args_image.h = source.nr();
            args.args_image.w = source.nc();
            args.args_image.framerate = 10;
            args.args_image.codec_options["crf"] = "4";
            return args;
        }());

I am used to using the command line, where all options are mixed, between encoder and muxer.

[arrufat]

I was thinking that maybe we should add a global function to dlib that does something like this:

template <typename image_type>
void load_image(image_type& image, const std::string& file_name)
{
    ffmpeg::demuxer reader(file_name);
    ffmpeg::frame frame;
    if (!(reader.is_open() && reader.read(frame)))
        throw image_load_error("Error reading from " + file_name);
    ffmpeg::convert(frame, image);
}

Then, we would never have to add extra image codecs support in dlib.
We might need to name it differently or change its signature to avoid conflicts.

I have a half-baked branch with JPEG XL, but I don't think it's meaningful unless we need JPEG XL specific features.

What do you think?

[arrufat]

Also, is there a way to make FFmpeg not output stuff to the terminal when it's being used from dlib?

[pfeatherstone]

There is probably a way to configure the logging level in libavformat and co. There is also probably a way to customize the logging such that we could forward it to dlib::logger appropriately. That might be nice.

[pfeatherstone]

Regarding your suggestion for load_image() à la ffmpeg: maybe. I think for most image types (jpeg, png, etc), ffmpeg has native decoders, so it's not like you would be depending on libjpeg or libpng further down the line. However, I'm interested in benchmarking this vs what dlib already has.

On a side note, I think dlib should have some benchmarking code using https://github.com/google/benchmark or similar. Every release (or on every commit even), we could run the benchmarks in Github Actions, and add the results somewhere, possibly update some graph.

[pfeatherstone]

libavutil has av_log_set_callback(). So we can easily redirect all libav logging to dlib::logger setting the appropriate logging level as well. We could call av_log_set_callback() in register_ffmpeg::register_library() in ffmpeg_utils.h. Nice and easy. PR welcome.

[pfeatherstone]

This is all done

[pfeatherstone]

FYI, all of:

• dlib::ffmpeg::demuxer
• dlib::ffmpeg::decoder
• dlib::ffmpeg::muxer
• dlib::ffmpeg::encoder
• ffmpeg 5 support
• ffmpeg 6 support
• various examples

are in master!
Would be great to get some user feedback. Thank you @arrufat for starting already.
The more feedback we get, the quicker things get fixed (if anything needs fixing), optimized or generally improved.

[pfeatherstone]

FYI, demuxer and decoder have API changes. I believe it's a lot better and more flexible. There are additional examples as well. Feedback would be great. If all goes well, we can ship a release with all the new ffmpeg goodies.

[davisking]

Yeah it's really nice :)

[pfeatherstone]

A few more things to do before release:

• Add save_frame()
• Add muxer::push(image_type img) where image_type is a dlib image object
• Add encoder::push(image_type img)

Most importantly, this could do with some trials. So users, please stress test and submit issues if there are any.

[pfeatherstone]

I think this is all done now. Just needs a bit of field experience.

[pfeatherstone]

At some point we can consider adding FFmpeg as an ExternalProject to cmake so users can optionally build FFmpeg from source if not found on the system. I've mentioned this before I think. This would be great if you don't want the full FFmpeg installation, but just want to enable precisely what you need. It also means you can use the absolute latest release of FFmpeg. This is not easily done with OpenCV.

[davisking]

Yeah that would be cool so long as it was configured off in cmake by default.

[pfeatherstone]

Quick example of how one might use the new decoding/demuxing API. This reads frames from a video, encodes them, then directly decodes them using callbacks.

    using namespace dlib;
    using namespace dlib::ffmpeg;

    demuxer cap({video_path, video_enabled, audio_disabled});

    if (!cap.is_open())
    {
        printf("Failed to open %s\n", video_path.c_str());
        return;
    }

    encoder enc([&] {
        encoder::args args;
        args.args_codec.codec           = AV_CODEC_ID_MPEG4;
        args.args_codec.codec_options   = {{"q:v", "1"}};
        args.args_image.h               = cap.height();
        args.args_image.w               = cap.width();
        args.args_image.fmt             = cap.pixel_fmt();
        args.args_image.framerate       = cap.fps();
        return args;
    }());

    if (!enc.is_open())
    {
        printf("Failed to create encoder\n");
        return;
    }

    decoder dec([&] {
        decoder::args args;
        args.codec = AV_CODEC_ID_MPEG4;
        return args;
    }());

    if (!dec.is_open())
    {
        printf("Failed to create decoder\n");
        return;
    }

    int counter1{0};
    int counter2{0};

    const auto clb_dec = [&](frame&) {
        ++counter2;
    };

    const auto clb_enc = [&](size_t ndata, const char* data) {
        dec.push((const uint8_t*)data, ndata, wrap(clb_dec));
        return true;
    };

    frame f;
    while (cap.read(f))
    {
        ++counter1;
        enc.push(std::move(f), clb_enc);
    }

    enc.flush(clb_enc);
    dec.flush(wrap(clb_dec));

[pfeatherstone]

Or, if you don't like callbacks/asynchrony, and you want to use buffers instead, then the main loop can be replaced with:

    frame f;
    std::vector<uint8_t> buf;
    std::queue<frame> frames;

    while (cap.read(f))
    {
        ++counter1;
        enc.push(std::move(f), sink(buf));
        dec.push(buf.data(), buf.size(), wrap(frames));
        buf.clear();
    }

    enc.flush(sink(buf));
    dec.push(buf.data(), buf.size(), wrap(frames));
    dec.flush(wrap(frames));
    counter2 = frames.size();

[pfeatherstone]

Maybe at some point we can add coroutine support which wraps the callbacks and allows you to co_await either encoded data or frames. This could be enabled if __cpp_impl_coroutine and __cpp_lib_coroutine were enabled. Not now though...

[pfeatherstone]

Just did some more benchmarking against OpenCV. I was confused why Opencv was nearly 4X faster than vanilla ffmpeg. It turns out, by default, OpenCV will use half your cores. So on an 8-core CPU, it will use 4 threads. By default, ffmpeg, and therefore dlib, will use 1 thread.
You can set demuxer::args::args_codec_image.thread_count = ... or decoder::args::thread_count = ... to whatever you want. When i set this value to be the same as what OpenCV uses, I actually get better performance. I don't have hard concrete evidence with a ton of unit tests on different machines with different versions of ffmpeg, but I think dlib has thinner and better wrappers than OpenCV. Could be wrong. If someone has a different experience, please submit a reproducible example, including cmake script and I will endeavour to fix it.

[davisking]

I'm totally willing to believe the wrapper you made is just faster 😁

[facug91]

I've finally had the time to try this out, and it works like a charm. I tested most of the examples, and in particular, made a comparison with OpenCV using ffmpeg_video_muxing_ex.cpp.
I modified it to work as cv::VideoCapture/cv::VideoWriter would, reading with a BGR frame and using that to write the video file. I also hardcoded the same codec I was using with OpenCV (mpeg4) with a video of 40 seconds in Full HD (1920x1080) and 25 fps (1000 frames total). OpenCV took ~19 seconds to read the video and write it in a new file. This implementation took ~13 seconds. In both cases I used only one thread for the decoding process. It was weird that with RGB frames it took around ~14 seconds. Don't know why, but it must be something with ffmpeg, not dlib. And it's awesome to be able to access the YUV frame directly (in that case it took ~8 seconds only!).
I can say that in the following months I'll start using this instead of OpenCV for video decoding on CPU, thank you @pfeatherstone for the great job 😄.

There are still a couple of scenarios that I'm not sure if it's possible to use this, maybe it is with some modifications.
cv::VideoCapture can be configured with CAP_PROP_FORMAT in -1, so that it works as a demuxer only (no decoding), which can be used to retrieve raw data directly. I find this useful for two different scenarios:

• Using a different decoder (NVDEC for example).
• In the latest version, OpenCV added a new property to cv::VideoWriter, which is cv::VideoWriterProperties::VIDEOWRITER_PROP_RAW_VIDEO, that allows you to write encoded video streams to video containers (like mp4). In combination with a cv::VideoCapture reading raw data, you can write files directly, without decoding/enconding at all. With the video from the example above, it took ~110 ms to rewrite the video in a new file.

[pfeatherstone]

I also want to add filter support, i.e, have a nice functional wrapper for AVFilter. So you can equalize your audio, fancy image filtering etc

[pfeatherstone]

The audio filtering could be a nice way to augment audio for ML stuff

[pfeatherstone]

Make sure you test all this stuff well before you put it into production code!

[facug91]

I can add the feature you want at some point. Should be pretty easy

I'd like to help if possible. I'm not a decoding/enconding expert, I usually just use it for computer vision applications, but I'd gladly learn whatever necessary.
I've been reading about ffmpeg, and from what I see we already have AVPacket objects in this implementation, but we pass them directly to the decoder inside the demuxer. As I understand, we could add a class similar to frame, which wraps AVFrame, but for AVPacket, is that correct? And that would be the output of the demuxer in case it is configured to return compressed data.

Also, ffmpeg supports hardware decoding. I should add that at some point

The problem I see here is that in order to use hardware acceleration with ffmpeg, it needs to be compiled with --non-free flag, which could cause some problems with commercial applications, is that correct?

Make sure you test all this stuff well before you put it into production code!

Of course, I plan to test this with many different cameras, but so far the results are really good. If I find any problems I'll comment here or in a new issue.

[pfeatherstone]

Yeah, I would wrap AVPacket sensibly. You can have a go at a PR. I know the implementation details inside out, so splitting demuxer into demuxer_base and decoder wouldn't be hard at all. And similarly for muxer. Hardware acceleration can be detected at run time through the ffmpeg API. So it would be up to the user to install a version of ffmpeg which had it enabled. So no biggy. In general, giving this more field experience would be great.