Enable hwaccel h264 encoding on raspberry pi

[jasaw]

Enable hardware accelerated H264 and MPEG4 video encoding on raspberry
pi via libopenmax.

Description:

• Added a new config item to FFMPEG package to enable raspberry pi HW acceleration. This option handles dependencies, configure options, and include path automatically.
• Modified motion to look for "h264_omx" and "mpeg4_omx" encoders first. If the encoders are not found, fall back to software encoder.
• If using h264_omx encoder, force best video quality. Any lower quality seems to produce very blocky video.
• ffmpeg omx seems to work with /usr/lib libraries only, so I symlinked /opt/vc/lib to /usr/lib.

This is my first attempt at enabling hwaccel properly. Any feedback welcomed.

[ccrisan]

I'll look into this but we really need to test this thoroughly before merging it. We need to make sure things don't break when using USB cameras or netcams or anything else.

[jasaw]

Yes, if someone can help test this, that would be great. I have only tested this on a Pi, so very limited testing done on my end as I don't have other hardware.

[FilippoChiappo]

Can this method be used at motioneye (in a normal pc)?

[jasaw]

I don't see why it can't be used on a normal PC, as long as the PC has these libraries installed: libOMX_Core.so, libOmxCore.so

[jasaw]

The new version of motion has slightly cleaner code, attempts to apply quality settings to encoders, so I bumped ours up to the newer version and applied patches to support omx encoder.
With the latest tweaks for h264_omx encoder, I managed to improve movie quality without sacrificing frame rate. On a Pi Zero W, I'm getting roughly 3.5Mbps, 25 fps, at 800 x 600 resolution.

[ccrisan]

All this looks amazing but before we do the merge (with master) and release it, we should wait for the motion project to come up with a release (be it beta or something). Otherwise it may start raining with bug reports here on motionEyeOS and I can't possibly deal with them.

[jasaw]

@ccrisan Yes, wait for the latest motion release. This feature requires a lot more testing before merging, and I only have raspberry pis, only very limited testing is done. We should test this against all supported hardware, with various setup. Any volunteers?

I've compiled these patches into binaries if anyone is interested in testing.

[owenthewizard]

Using your binary ffmpeg -hwaccels doesn't list any hardware acceleration methods, is this intended?

[jasaw]

@owenthewizard Use ffmpeg -encoders | grep omx instead and if hwaccel is available, you should get V..... h264_omx OpenMAX IL H.264 video encoder (codec h264).
You can also try recording a movie, see what frame rate you're getting. Without hwaccel 2 fps, anything higher means you have hwaccel.
What hardware are you using? Please provide feedback so we know people have actually tested this. :-)

[DruidKin]

I'm able to test with my Pine64 and AXIS IP cameras, and a PELCO IP camera. Would also like to see remote storage to a NAS if possible. Most of these single board's won't have the thruput to HDD to support more than 2 cameras.

[chappy1978]

I have the PiZeroW working with this hwaccel distro but even with the settings (jasaw) suggested. I don't get the kind of high performance I had expected. With the 20fps I only get at best 15fps. That is only when I have the overclocking set to turbo. Is it a wifi issue on my network or am I missing something?

[DruidKin]

It's the encoders they are using. Very low frame rate and recording rates. It's a fairly young endeavor so they will get better. I'm trying the AXIS companion now as it's more the level I'm used to being at.

…

________________________________ From: chappy1978 <notifications@github.com> Sent: Sunday, July 9, 2017 4:27:41 PM To: ccrisan/motioneyeos Cc: DruidKin; Comment Subject: Re: [ccrisan/motioneyeos] Enable hwaccel h264 encoding on raspberry pi (#998) I have the PiZeroW working with this hwaccel distro but even with the settings (jasaw) suggested. I don't get the kind of high performance I had expected. With the 20fps I only get at best 15fps. That is only when I have the overclocking set to turbo. Is it a wifi issue on my network or am I missing something? — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#998 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AcouOIYcMUIsEG1EH3Zyuse5vNZJ7S0Aks5sMTe9gaJpZM4N6xDP>.

[jasaw]

@chappy1978 What tool did you use to determine the frame rate? I'm not sure if there's a standard tool that everyone uses, but I'm using ffprobe to get the average frame rate.

ffprobe -v 0 -show_entries stream=avg_frame_rate 09-59-58.mp4
[STREAM]
avg_frame_rate=5386240/245883
[/STREAM]

The above is one sample of my recorded movie, recorded at 20 fps. The average frame rate turns out to be 21.906 fps. I noticed when there's gap in the movie (i.e. during motion gap), the average frame rate becomes very high and it's noticeable during playback (lots of movements within one second). I don't know what can be done to fix that.

Also, with Pi Zero W, you shouldn't need to overclock it because it's already clocked at 1GHz by default. cat /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_cur_freq.

[chappy1978]

@jasaw Ive been working with the Motioneye web interface and am using a battery system so overclocking is a factor. I been testing lower Ghz settings to mAh low.

Ive also been getting my frame rates from MotionEye web interface (i.e. click on the video stream and a fps pops up on the bottom left of the screen). What I have noted was that when Im streaming from port 8081 there is a marked increase in the frame rate but still not close 20fps more like 3-12 fps.

[jasaw]

@chappy1978 If you're going to underclock the hardware, you should watch the CPU usage, make sure you're not pegged at 100% in all scenarios. Streaming video via MotionEye web interface, plus motion detection, plus recording a movie, will load up your CPU.
I don't know where the web interface got its frame rate, but what really matters is the frame rate in the recorded movie, which is why I used ffprobe to check.

[chappy1978]

OK Ill test it out with ffprobe

[jasaw]

@ccrisan What are you thoughts on motion using extpipe to call ffmpeg for encoding movie using h264_omx encoder? This means when H264 output format is selected from web UI, motioneye will have to change the generated motion config file to use extpipe.

Reasons for using extpipe to ffmpeg:

• Too much changes to motion to support talking to OpenMAX directly, and it needs to go into motion master to avoid maintenance nightmare.
• OpenMAX on Raspberry Pi is difficult to debug because GPU firmware is not open source.
• ffmpeg has a huge user base, more exposure, more testing, and we should leverage that.
• Running ffmpeg as a separate process achieves more parallelism on multi-CPU machines, better performance.
• So far it's stable at various resolutions and frame rates that I've tested, but my testing is obviously very limited.

Should we create a new branch for testing this stuff and I can point my pull-request to this new branch?

[jasaw]

@ccrisan I've committed changes to motioneye in this branch so you can see what the intent is when I say use extpipe. Thoughts?

Just a note: recording movie may fail with certain configuration. The fix should be merged into motion master any minute now.

[jasaw]

@ccrisan Keep in mind that the new motion config patch hasn't been accepted into motion master yet.

Summary of motion config file change:

• extend the ffmpeg_video_codec config option to take the optional preferred_codec in this format: ffmpeg_video_codec video_format[:preferred_codec]
• To enable hardware acceleration, we choose the h264_omx encoder. Example: ffmpeg_video_codec mp4:h264_omx
• To disable hardware acceleration, we just let ffmpeg choose the default encoder by not specifying the h264_omx encoder. Example: ffmpeg_video_codec mp4, which is exactly the same as the existing format.

MotionEye changes needed:

• Run ffmpeg -codecs -hide_banner and look for h264_omx. This will tell us if the device supports hardware acceleration.
• Only allow hardware acceleration if motion binary is certain version and above. (this needs the patch to be merged into motion master).
• If hardware acceleration is supported, motionEye can show the Hardware Acceleration option on the web UI.
• Only allow hardware acceleration if the movie format is mp4 or mkv (both are H264 format, but in different container).
• If hardware acceleration is disabled, no change to motionEye behaviour.
• If hardware acceleration is enabled, then write something like this to motion config file: ffmpeg_video_codec mp4:h264_omx.
• When reading from motion config to UI dict, motionEye also needs to deal with the ":h264_omx" optional string in the "ffmpeg_video_codec" config item.
• See package/motioneye/add-hwaccel-video-encoding-option.patch in this pull-request as an example. I'll revert this commit when you're ready.

[ccrisan]

@jasaw this is exactly what I needed. I'll add required support to motionEye will keep you posted.

[ccrisan]

@jasaw I added support for h264_omx in motionEye: motioneye-project/motioneye@1744cfa. It's on a separate branch (rpi-omx).

I have also created a rpi-omx branch in motionEyeOS and updated this PR accordingly. I'll make sure to fix any conflicts and no-longer-necessary changes before merging it.

[ccrisan]

@jasaw I hope you don't mind that I went my way with implementing the preferred codec support in motionEye. Instead of adding a separate UI setting for this purpose, I thought it would be best to have separate movie format options for both accelerated and normal methods. If the accel one causes troubles to a user, they can simply change it to the old one.

The codec is also used by the timelapse command and is chosen by default on platforms that support it. There's a new find_ffmpeg() function that will also tell which codecs are supported, so that we can add support for other such optimizations in the future.

[ccrisan]

@jasaw it seems my find_ffmpeg() method doesn't do the correct job in detecting the omx support. It doesn't look on the right column in ffmpeg -codecs. I'll try to fix it asap.

[jasaw]

@ccrisan No worries regarding the motionEye preferred_codec implementation. My motionEye patch was more of an example of how it may be implemented. You're the best person to implement motionEye side of things. :-)

Let me know when find_ffmpeg() is fixed and I'll give it a test.

[jasaw]

@ccrisan Please delete package/motion/temp-fix-for-extpipe-subdir-creation.patch (provided you bump the motion version to latest master).
The patch has just been merged into motion master: Motion-Project/motion#462

[ccrisan]

@jasaw the obsolete patch has been removed, motion has been updated to latest master commit and motionEye ffmpeg codec detection has been fixed. The rpi-omx branch can be compiled and tested.

I compiled myself the OS and ran some tests. While the CPU usage was significantly lower than in software encoding mode, the video quality was horrible. Is there some issue with quality passing from motion to libav? I tested using the defaults provided by motionEye (75%).

[jasaw]

@ccrisan To control video quality, h264_omx takes bit_rate rather than CRF, which is covered by this patch file: package/motion/tune-h264-encode-quality.patch. You'll get horrible blocky video without this patch. Can you please check the buildroot output/raspberrypi/build/motion-d23e263490a54329329b64883f1860ada5e5920b/.applied_patches_list file to make sure it's already patched?

[ccrisan]

Yes, the patch is there:

cat output/raspberrypi3/build/motion-ab9e800d5984f2907f00bebabc794d1dba9682ad/.applied_patches_list

/home/ccrisan/motioneyeos/package/motion/0001-version.patch
/home/ccrisan/motioneyeos/package/motion/preferred-encoder.patch
/home/ccrisan/motioneyeos/package/motion/temp-fix-for-extpipe-subdir-creation.patch
/home/ccrisan/motioneyeos/package/motion/tune-h264-encode-quality.patch

[jasaw]

What is your generated thread-1.conf file? It should have this:

ffmpeg_variable_bitrate 75
ffmpeg_video_codec mp4:h264_omx

I'll try to compile and test the latest rpi-omx branch tomorrow when I get some free time, see whether I can reproduce the problem.

Just FYI, h264_omx bit_rate works like this:
max bit_rate = ffmpeg->width * ffmpeg->height * ffmpeg->fps, which gives lossless encoding, but it's not feasible to do real-time encoding on this bit rate, so we multiply it with a quality factor that is set by the motionEye quality percentage, vbr in this case.
bit_rate = ffmpeg->width * ffmpeg->height * ffmpeg->fps * quality_factor
That is what the tune-h264-encode-quality.patch does. We also clip the maximum bit_rate to a more sensible value to maintain real-time encoding.

[ccrisan]

I read your patch and it makes sense. However I'm puzzled by the fact that the same codec (h264) has different quality config requirements for different encoders.

P.S. yes, that's the way my thread-1.conf looks like.

[jasaw]

@ccrisan I've reworked the ffmpeg_set_quality function. It should also fix the bad video quality problem. I've also removed the max bit rate limit.
Pull-request #1105 for your review.

[jasaw]

@ccrisan The latest rpi-omx appears to suffer from performance issue where frame rate is lower than what I've tested. I haven't worked out which software introduced this.
I've been testing with motion-ab9e800d5984f2907f00bebabc794d1dba9682ad, with motionEyeOS version 20170710, which gave me higher performance than the rpi-omx branch.

Another thing, rpi-omx branch motionEye UI is greyed out after I log in as admin. I've been modifying thread-1.conf file by hand to test.

[ccrisan]

1. I also noticed the performance issues when using latest motion git instead of the 4.0.1. I have no idea what happened.
2. The latest motionEyeOS version changes the way the admin password is handled. There are also some UI changes and I'd recommend clearing the browser's cache if you experiment weird UI behaviour. If the problem still persists, please open a separate issue with some screenshots.
3. I'll test your ffmpeg quality changes and report back.

[jasaw]

I dont know motion-ab9e800d5984f2907f00bebabc794d1dba9682ad is which release version of motion, 4.0.1 or not.

We can fallback to motion-ab9e800d5984f2907f00bebabc794d1dba9682ad since that's the version that I know works. You'll also need to restore back the extpipe patch.

I'll test rpi-omx branch with motion-ab9e800d5984f2907f00bebabc794d1dba9682ad tomorrow to verify that it's the latest motion issue. I'll also try clearing my browser cache.

[jasaw]

@ccrisan It's not motion software that is causing the performance issue. I ran motion-ab9e800d5984f2907f00bebabc794d1dba9682ad on rpi-omx branch and get the same performance issue.

[jasaw]

Rolling rpi-userland version back to cdb5da59f939eb4078e90ed0e3c231c498ba9957 seems to fix the performance issue. Next is to find out the offending commit from rpi-userland.

[jasaw]

I've rolled rpi-userland version back to 9aab1498b531b50585b206232d6baea64c0789f7, and it's still OK. Haven't tried later commits.

[ccrisan]

Is the performance issue present on 20170827? It uses the most recent rpi-userland commit as of the time of writing.

[jasaw]

I haven't tested 20170827. I'll investigate a bit more first before I try 20170827. Will keep you updated.

[jasaw]

@ccrisan I can't reproduce the performance issue anymore. I'm happy with it in its current state. Have you done any testing?

[ccrisan]

@jasaw by "anymore" do you mean that 20170827 doesn't suffer from the performance issue? Or did you find a fix for the rpi-omx branch?

[jasaw]

@ccrisan I couldn't find the problem, and repeated my steps to reproduced the problem but couldn't reproduce the problem. It may very well be my fault when I edit the thread-1.conf by hand.
To answer your question, 20170827 does not suffer from performance issue, even after I applied all the h264_omx patches on top of it.

[ccrisan]

@jasaw I'll be busy the next week with other stuff but I'll run more thorough tests afterwards and get back to you. I think we're getting close to a merge.

[ccrisan]

@jasaw I have run some more detailed tests on the current rpi-omx and I have good news:

• the "performance" issue is due to the fact that somehow the streaming framerate motion setting is limiting the actual framerate (solution: motionEye will set the streaming framerate to the capture framerate + 5 fps, by default)
• the h264 video quality with the current version and the default 75% setting is perfect
• when testing on a RPi3 with CSI camera module at 1024x768@25fps, while encoding h264 during continuous motion, here are my findings:
  • without hardware acceleration: sysload ~ 1.8, framerate ~ 10
  • with hardware acceleration: sysload ~ 0.9, framerate ~ 16..20

In conclusion, things are looking great and I must say your work here is amazing.

I have yet to run some tests with a regular USB camera and a network camera. I'd also like to run these tests on a 1st generation RPi.