Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature request: enable/integrate rockchip hwaccel support #34

Closed
dev-zero opened this issue Apr 12, 2020 · 109 comments · Fixed by #318
Closed

Feature request: enable/integrate rockchip hwaccel support #34

dev-zero opened this issue Apr 12, 2020 · 109 comments · Fixed by #318
Labels
enhancement New feature or request

Comments

@dev-zero
Copy link

It would be really nice if the jellyfin-ffmpeg would support the rockchip hardware acceleration.

For 4.1 there's a maintained ffmpeg-fork here with additional patches it seems: https://github.com/rockchip-linux/ffmpeg/tree/rockchip/release/4.1 but basic rockchip acceleration support was already included via the RK-MPP in ffmpeg-3.

This library is needed to enable support: https://github.com/rockchip-linux/mpp

Communication with the kernel is then through a /dev/videoN device if I understood it correctly.

@JustAMan
Copy link
Contributor

We'd like to keep in line with upstream ffmpeg as much as possible. Do you know if ffmpeg 4.2 or newer has this enabled?

@dev-zero
Copy link
Author

It seems that support for rockchip-acceleration was added in ffmpeg-3 (and continues to be there). It is by default not enabled since it needs the mpp library:

 ~/Downloads/ffmpeg-4.2.2 $ ./configure --help | grep mpp
  --enable-rkmpp           enable Rockchip Media Process Platform code [no]

@JustAMan
Copy link
Contributor

Looking at their repo I don't see a license. We cannot redistribute builds without compatible licenses... please investigate the licensing of their code and report back :)

@dev-zero
Copy link
Author

dev-zero commented Apr 30, 2020

The debian control file and the headers say Apache 2.0 (which is probably why ffmpeg requires to be built with --enable-version3):

Just did a succesful buidl test with the MPP library from their devel branch (the latest release is quiet old) and ffmpeg with ./configure --enable-rkmpp --enable-version3 --enable-libdrm.

@JustAMan
Copy link
Contributor

I think you can try and make a PR yourself, this is probably the file you need to modify: https://github.com/jellyfin/jellyfin-ffmpeg/blob/jellyfin/docker-build.sh

@RlndVt
Copy link

RlndVt commented Oct 29, 2020

I am also interested in having rockchip hwaccel, specifically for the Helios64. I've noticed that the mpp repo https://github.com/rockchip-linux/mpp doesn't exist anymore. There is a alternative here https://github.com/HermanChen/mpp. Don't know how official it is.

Is this a problem for redistribution?

@nyanmisaka
Copy link
Member

I am also interested in having rockchip hwaccel, specifically for the Helios64. I've noticed that the mpp repo https://github.com/rockchip-linux/mpp doesn't exist anymore. There is a alternative here https://github.com/HermanChen/mpp. Don't know how official it is.

Is this a problem for redistribution?

I think it's still official. It has been moved to a personal repo of a Rockchip employee.
There are only decoders for rkmpp in ffmpeg, encoders and scaler have not been implemented.

@NablaMetal
Copy link

Any news on this one ? I am also very insterested in having hwaccel enabled for jellyfin upstream.

@dev-zero
Copy link
Author

dev-zero commented Feb 7, 2022

Not from me unfortunatley: I moved from a Rockchip back to Intel

@Shadowghost Shadowghost added the enhancement New feature or request label Jun 7, 2022
@tandy-1000
Copy link

tandy-1000 commented Jun 21, 2022

Does anyone know what actually needs to be done to enable this?

If I understand correctly:

  • Package rockchip-mpp in the Jellyfin repo
  • Add it to the requirements arm64 build and enable the build flag

@nyanmisaka
Copy link
Member

You are free to do it. But note that ffmpeg only has rockchip’s hardware decoder, which means transcoding speed will still bottlenecked by software scaling and software encoding.

@NablaMetal
Copy link

@nyanmisaka absolutely. That is why I personnaly moved to x86 platform whith much much better performance.

@tandy-1000
Copy link

You are free to do it. But note that ffmpeg only has rockchip’s hardware decoder, which means transcoding speed will still bottlenecked by software scaling and software encoding.

That makes sense, I think this will have a reasonable benefit on rockchip servers.

Does my todo list make sense? If so, how would I get started with packaging rockchip-mpp in the jellyfin repo?

@nyanmisaka
Copy link
Member

@tandy-1000 tandy-1000 mentioned this issue Jun 22, 2022
2 tasks
@prahal
Copy link

prahal commented Nov 25, 2022

At least, the Armbian edge kernel for helios64 has hantro-vpu encoder/decoder and rockchip_vdec support.
FFmpeg support is not official but active upstream seems to currently be https://github.com/jernejsk/FFmpeg (devel in vc1-v2 branch) shipped by LibreELEC in https://github.com/LibreELEC/LibreELEC.tv/tree/master/packages/multimedia/ffmpeg/patches/v4l2-request.
Maybe https://github.com/LibreELEC/LibreELEC.tv/blob/master/packages/multimedia/ffmpeg/patches/v4l2-drmprime/ffmpeg-001-v4l2-drmprime.patch is of use too, but I do not know if required.
I have not yet tested the above, but at first glance, a few patches are not required for non-Kodi FFmpeg.
I have an helios64 with the Jellyfin docker container so I will be eager to test.

@prahal
Copy link

prahal commented Nov 25, 2022

Another option might be the vaapi v4l2-request backend https://github.com/bootlin/libva-v4l2-request (this repository is inactive but there are newer low activity forks)

@RicardoCst
Copy link

I think this is a good idea because it moves away from the typical: "Oh you need an intel/amd processor with integrated graphics to support transcoding" statement. And its another thing which makes jellyfin different from the rest (emby, plex).

@IngwiePhoenix
Copy link

The --enable-rkmpp flag exists in the configure script, but is defaulted to off.

I mainly use the Jellyfin Docker container. Any good way that I can replace that with a self-built jellyfin-ffmpeg that has rkmpp enabled - and then actually use it in Jellyfin itself?

@nyanmisaka
Copy link
Member

nyanmisaka commented Apr 26, 2023

The --enable-rkmpp flag exists in the configure script, but is defaulted to off.

I mainly use the Jellyfin Docker container. Any good way that I can replace that with a self-built jellyfin-ffmpeg that has rkmpp enabled - and then actually use it in Jellyfin itself?

The fact is that rkmpp hwaccel in ffmpeg is outdated to support rk35xx chips, and it doesn’t support encoding. Nothing would work unless someone rewrite it from scratch.

JeffyCN/FFmpeg#4 (comment)

@jjm2473
Copy link

jjm2473 commented Apr 27, 2023

If you just want to use HW accel in jellyfin, my project should be enough https://github.com/jjm2473/ffmpeg-rk/wiki

Note: HW decodeing/encoding works, RGA scale works. No tone mapping supported. Tested on rk3568/rk3588.

Update 2023-11-24: now supports HW mjpeg decoding; rga_scale filter supports setting output pix format, supports yuvj42xp/yuyv422/rgb as input .

@prahal
Copy link

prahal commented May 13, 2023

@jjm2473 I think mpp kernel driver is a rockchip vendor kernel only thing.
Armbian for one has moved to mainline thus as far as I understand I cannot use ffmpeg-rk on an Armbian install.

My understanding is that rockchip has no plan to port mpp drivers upstream.
And hantro h264 encoding has been added to the upstream hantro driver.
So I believe v4l-request is the way to support upstream kernels.

@jjm2473
Copy link

jjm2473 commented May 13, 2023

@prahal No, the MPP drivers not upstreamed. But mainline linux can be patch to supports MPP. Not only MPP drivers, but also RGA drivers shoubld be supported, because some SoC has not enough performance to encoding 4K/30FPS video (for example RK3568, up to 1080p/30FPS encoding), must use RGA to scale video.

I have no experience with v4l. The ffmpeg-rk project is only implemented by referring to RK's old ffmpeg and MPP/RGA development documents, so it depends on related kernel functions.

@prahal
Copy link

prahal commented Jun 23, 2023

@jjm2473 https://aur.archlinux.org/packages/rockchip-mpp?all_reqs=1 tells tells MPP only works with rockchip BSP kernel ( latest 5.10 in a develop branch).

Where could I get these MPP and RGA drivers against linux 6.4-rcN?

To me, the issue is not that MPP and RGA are not upstream yet. But that rockchip loos like it has no plan to devote developer time to upstream it. This looks like an internal project for their custom kernel that they develop on a public github to help their customers customize products based on their kernel.

@IngwiePhoenix
Copy link

linux 6.4-rcN

I hate to quote a meme but... That's the neat part, you don't!

There has not been a single commit to their kernel repo in forever - let alone a new branch to support anything newer than 5.10.x

In fact, here's my current one on a RK3588 (FriendlyElec NanoPi R6s)

# uname -a
Linux FriendlyWrt 5.10.110 #1 SMP Sat Dec 3 01:25:15 CST 2022 aarch64 GNU/Linux

This corresponds to https://github.com/rockchip-linux/kernel/tree/develop-5.10

@prahal
Copy link

prahal commented Jun 27, 2023

linux 6.4-rcN

I hate to quote a meme but... That's the neat part, you don't!

Then should never be part of a linux distribution as not security supported. I wonder why Archlinux ships userspace mpp-git https://aur.archlinux.org/packages/mpp-git.

Sadly v4lrequests development is somewhat stalled especially on the vaapi side (there is a ffmpeg v4lrequests specific patch which is more up to date but not upstreamed either). I do not know about scaling with v4lrequests.

@prahal
Copy link

prahal commented Jul 7, 2023

@jjm2473 if it was that easy I hope rockchip would have paid their developers to do the upstreaming. No definitely MPP is for a rockchip fork of the linux kernel. This is not a Linux kernel, it is only based on it.
A little like the Windows TCP/IP stack based on BSD code does not make Windows a BSD kernel, except here the fork is only 3 years old.
Like you can add hw video acceleration for a BSD stack to jellyfin ffmpeg you can add the Rockchip kernel video acceleration stack support to jellyfin ffmpeg. This will not support for Linux on Rockchip hardware.

I am after an hw acceleration stack for Armbian current and edge, that is Linux per see. So MPP is of no help. Sadly v4l-request upstreaming seems to have stalled. I will try to contribute to it if it is not dead yet. I am especially fond of the v4l-request to vaapi bridge that could enable to share code with the intel vaapi support that is already in (sadly this bridge is the part that looks completely dead development-wise).
It looks to me like v4l-request does the decode/encode job (hantro vpu and rockchip hw decoder.
But I know little about rockchip RGA code. Might be this part cannot be supported by v4l-request only.

@HeyMeco
Copy link

HeyMeco commented Oct 29, 2023

linux 6.4-rcN

I hate to quote a meme but... That's the neat part, you don't!

There has not been a single commit to their kernel repo in forever - let alone a new branch to support anything newer than 5.10.x

In fact, here's my current one on a RK3588 (FriendlyElec NanoPi R6s)


# uname -a

Linux FriendlyWrt 5.10.110 #1 SMP Sat Dec 3 01:25:15 CST 2022 aarch64 GNU/Linux

This corresponds to https://github.com/rockchip-linux/kernel/tree/develop-5.10

Well Friendlyelec just updated their Rockchip branch to 6.1

https://github.com/friendlyarm/kernel-rockchip

@mrodriguez3313
Copy link

mrodriguez3313 commented Jan 9, 2024

Im kind of a novice with Linux. I was excited to see that HW accel on RK3588 boards was becoming a thing. I was referred here to check out how to run this. Im open to test things out if needed.

When I run uname -a on my system I get

Linux fulatower 5.10.160-legacy-rk35xx #1 SMP Tue Nov 28 02:45:16 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

This told me, this could be compatible with my device. I ran the docker command with sudo. and I believe this is working properly. I made sure to select "Rockchip MPP (RKMPP)" under Dashboard -> Playback -> Transcoding. I kept "h264, HEVC 10bit, vp9 10bit" enabled by default. I made sure to turn on "Enable HW encoding, allow encoding in HEVC format, allow encoding in AV1 format, and Enable Tone Mapping" I left everything else the same.

after I run htop, I can see that cpu usage is lower than it was before, but other than that idk what else I should be testing to confirm it is working.

@HeyMeco
Copy link

HeyMeco commented Jan 10, 2024

@mrodriguez3313 allow encoding in AV1 format

RK3588 can only decode AV1 and encode H264 & HEVC

@mrodriguez3313
Copy link

Got it. So even though the options are there, some options aren't supported. What else can I do to test that HWA is working on my machine?

@nyanmisaka
Copy link
Member

As long as you can find *_rkmpp from the stream mapping in FFmpeg logs, it works.

Stream mapping:
  Stream #0:0 (hevc_rkmpp) -> setparams:default
  overlay_rkrga:default -> Stream #0:0 (hevc_rkmpp)
  Stream #0:1 -> #0:1 (copy)

@HeyMeco
Copy link

HeyMeco commented Jan 21, 2024

@nyanmisaka I finally got my hands on the RK3588 (Rock 5B) and I got everything working besides Tone-Mapping. For some reason I get 1-2 fps (if I even get it to play a few secs). I was using armbian legacy with this Mali OpenCL blob libmali-valhall-g610-g13p0-gbm_1.9-1_arm64.deb. Do you have a recommened image / kernel I can reference?

EDIT: This was a result of using a power supply with only 9v 2a, switching to 20v 3a fixed it.

@jepperc
Copy link

jepperc commented Jan 29, 2024 via email

@RicardoCst
Copy link

Good job! 👍 🥇

@HeyMeco
Copy link

HeyMeco commented Jan 29, 2024

From which Jellyfin version on can we expect to use the main docker image? As far as I can tell the WebUI branch hasn’t been updated yet to have the option available but I could be wrong

@nyanmisaka
Copy link
Member

From which Jellyfin version on can we expect to use the main docker image? As far as I can tell the WebUI branch hasn’t been updated yet to have the option available but I could be wrong

JF 10.9

Merge order:
ffmpeg -> server -> web

@Qwertylala1
Copy link

Transcoding HDR movies with tone mapping doesn't seem to work with the latest Rockchip 6.1.43 kernel. I have a test install of Armbian with the new kernel and transcoding with Jellyfin works with SDR movies. Everything else should be identical to my normal Armbian install with the old 5.10.160 kernel where HDR transcoding works great.

I have the latest version of the libmali drivers installed (v1.9-1-b9619b9).

Log.txt

@HeyMeco
Copy link

HeyMeco commented Mar 9, 2024

@Qwertylala1 maybe attach clinfo output too.

@Qwertylala1
Copy link

Of course, attached here.

clinfo.txt

@HeyMeco
Copy link

HeyMeco commented Mar 9, 2024

@Qwertylala1 latest driver requires DDK g21p0-01eac0 and 6.1 currently is reverted back to g18p0-01eac0 because g21 is broken with RK6.1 today. Maybe try with an older driver build before the g21 DDK requirement

@Qwertylala1
Copy link

Qwertylala1 commented Mar 9, 2024

Yes, that did it! I tried v1.9-1-2d267b0 from November since I know that I have used that version before without any problems and now transcoding works fine with HDR movies. Thank you!

@jwsky
Copy link

jwsky commented May 25, 2024

Yes, that did it! I tried v1.9-1-2d267b0 from November since I know that I have used that version before without any problems and now transcoding works fine with HDR movies. Thank you!

the same issue like yours,i'm using orange pi 5 plus,rk3588 @HeyMeco @Qwertylala1
,i've already installed my lib to the correct version v1.9-1-2d267b0 and such HWA settings by the official tutorial,and the hwa works,but the tone mapping does not work.it's a dovi movie file ,the color seems a little like to purple and green.

i‘ve’ research this for days,but it still didn't work.
It'll be so appreciate if you can have some time looking up this issue,thank you

i'm using the OS:
1,Official Images of ubuntu focal ,downloaded from the official website. not work
2,ubuntu 24 from https://github.com/Joshua-Riek/ubuntu-rockchip . not work

uname -a
Linux ubuntu 6.1.0-1013-rockchip #13-Ubuntu SMP Thu May 9 21:20:56 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux

log11.txt
clinfo_output.txt

video file info:
Video Title: 4K HEVC HDR Codec: HEVC AVC: No Profile: Main 10 Level: 150 Resolution: 3840x2160 Aspect ratio: 16:9 Anamorphic: No Interlaced: No Framerate: 23.976025 Bitrate: 24998 kbps Bit depth: 10 bit Video range: HDR Video range type: DOVI DV title: DV Profile 5 DV version major: 1 DV version minor: 0 DV profile: 5 DV level: 6 DV rpu preset flag: 1 DV el preset flag: 0 DV bl preset flag: 1 DV bl signal compatibility id: 0 Pixel format: yuv420p10le Ref frames: 1

@nyanmisaka
Copy link
Member

https://jellyfin.org/docs/general/administration/hardware-acceleration/#hardware-accelerated-tone-mapping

Rockchip RKMPP currently only support HDR10 and HLG tone-mapping.

@jwsky Dolby Vision metadata is not properly supported by RKMPP. Please open an issue there to let them know.

https://github.com/rockchip-linux/mpp/issues

@jwsky
Copy link

jwsky commented May 25, 2024

https://jellyfin.org/docs/general/administration/hardware-acceleration/#hardware-accelerated-tone-mapping

Rockchip RKMPP currently only support HDR10 and HLG tone-mapping.

@jwsky Dolby Vision metadata is not properly supported by RKMPP. Please open an issue there to let them know.

https://github.com/rockchip-linux/mpp/issues

Thank you so much for your help @nyanmisaka ,i've opened the issue here in rockchip-linux/mpp/issues
Seems like NVIDIA might be the best hardware for the hwa and tone mapping,i'm using nvdia 3060 in Windows PC,and it works fine for me. the issue is, leaving the computer on all the time is too wasteful.~:)

@HeyMeco
Copy link

HeyMeco commented May 25, 2024

@nyanmisaka Dolby Vision Tonemapping in general isn't an issue. In this case it's the DoVi Profile 5 based source that's causing problems. It uses a proprietary color space:

As defined in SMPTE ST 2128:2023, IPT-PQ-C2, previously a Dolby Vision proprietary color space, is similar to BT.2100 ICtCp, where I is similar to I, P similar to Cp, and T similar to Ct. As of the date of this specification, the MPEG Joint Video Experts Team (JVET) working group has standardized the code point for this previously proprietary color space, also known as IPT-PQ-C2, using code point 15.

@jwsky It would make more sense to use dovi_tool to convert the source file from 5 to 8.1 for maximum compatibility.

@nyanmisaka
Copy link
Member

Jellyfin actually supports DV P5 to SDR tone-mapping on platforms other than Rockchip. Just because the metadata cannot be properly imported from MPP into FFmpeg.

@stevenlafl
Copy link

@nyanmisaka Whatever Jellyfin maintainers did (or did not do) in their main repository results in RK3588 not using the VPU anymore. I had to revert to using your image. Could you share how you built it: commands, sources? I'm thinking of poking around it and pulling just the pieces relevant to VPU support from your version.

@nyanmisaka
Copy link
Member

@nyanmisaka Whatever Jellyfin maintainers did (or did not do) in their main repository results in RK3588 not using the VPU anymore. I had to revert to using your image. Could you share how you built it: commands, sources? I'm thinking of poking around it and pulling just the pieces relevant to VPU support from your version.

@stevenlafl The RKMPP support has been added to Jellyfin 10.9.x, so my test image is no longer needed. I don't quite understand what you are saying. If there are any problems please provide detailed error logs.

@stevenlafl
Copy link

stevenlafl commented Jun 6, 2024

@nyanmisaka

If there are any problems please provide detailed error logs.

Testing method:

  1. Snapshot volumes
  2. Played the first 20 seconds of sample media (Starting with image nyanmisaka/jellyfin:latest-rockchip)
  3. Downloaded ffmpeg log and screenshot cpu usage
  4. Restore snapshots
  5. Change image to jellyfin/jellyfin:latest
  6. Played the first 20 seconds of sample media
  7. Downloaded ffmpeg log and screenshot cpu usage

Here's with your image nyanmisaka/jellyfin:latest-rockchip, with sample media:

FFmpeg Logs: https://gist.github.com/stevenlafl/2d6380a30aa64d18c0e555b144270058
Rate: 22.5x

CPU Usage: image

Here's the new jellyfin image jellyfin/jellyfin:latest, with the exact same media:

FFmpeg Logs: https://gist.github.com/stevenlafl/729562c22c3c4a370afefc5ae59af3f7
Rate: 1.26x

CPU Usage: image

Is there anything else you need me to look at?

@stevenlafl
Copy link

stevenlafl commented Jun 6, 2024

I see the problem now looking at the logs again. Your image seemed to use h264 as the output no matter what setting for the "Encoding format options", and in the production version it actually did try to use AV1/HEVC which were non-accelerated.

Apologies, the behavior wasn't enough to indicate a problem and I missed it in the logs.

image

@nyanmisaka
Copy link
Member

nyanmisaka commented Jun 7, 2024

@stevenlafl You just need to disable the AV1 encoder Allow encoding in AV1 format. Enabling this option means using the SVT-AV1 software encoder. Currently Rockchip only supports H264 and H265/HEVC hardware encoders.

In the future we will make this transcoding setup automatic and eliminate these hassle.

@HeyMeco
Copy link

HeyMeco commented Aug 11, 2024

Jellyfin actually supports DV P5 to SDR tone-mapping on platforms other than Rockchip. Just because the metadata cannot be properly imported from MPP into FFmpeg.

Is there a way to make it automatically fallback to cpu decoding instead of hw accel with profile 5 media?
When I manually tick off 10bit HEVC then CPU decoding works fine with 1080P and just enough with 4k high bitrate media @nyanmisaka

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet