opus 5.1 and 7.1 channel order is incorrect

[Tupsi]

Describe how the issue can be reproduced, ideally using the ExoPlayer demo app
or a small sample app that you’re able to share as source code on GitHub. To
increase the chance of your issue getting attention, please also include:

• Clear reproduction steps including observed and expected behavior

1. Create a mkv with video and opus 5.1 or 7.1 audio
   (I use ffmpeg, with the releveant audio part as -c:a libopus -b:a 256000 -mapping_family 1 -af aformat=channel_layout=5.1 (or 7.1 respectively))
2. Playback on FireTV Stick
   (I am using emby app with my emby server, nerd stats inside showing native opus playback (no transcoding)

Observed behavior:
Playback works but channels are somehow mixed up. The dialogs which should be on center are now ONLY on the right channel and the center only plays back some surround channels ambient music/sound.

Expected behavior:
Channels should be where there are suppose to be. I can force that by enabling transcoding in the emby app (server side), so the server transcodes the audio from opus to dd+, resulting the dialog gets played at the center speaker).

You will NOT hear the problem with 2.0 stereo sets as the downmixing seems to work in the sense, that the original center gets mixed into both channels.

I also isolated only the center channel on my computer from the audio to verify that this is indeed the original center audio.

• ExoPlayer version number: 2.12.2
• Android version: API Level 25 (Android 7.1) (according to amazon dev specs)
• Android device: Amazon Fire TV 4k Stick

For reference, I already talked about this problem in detail over at the emby forum over a year ago and provided a sample file (in case you need one)
https://emby.media/community/index.php?/topic/71360-playing-back-opus-5171-does-not-work-center-channel-is-missing/

[marcbaechinger]

Thanks! Can you provide us with a sample stream? I know you said it's in the thread you are referencing, but besides the thread being huge, I seem to not have access to the sample.zip that might be what you say is the sample stream.

[Tupsi]

Yeah sorry, just now realized I password protected that file because of the overzealous german laws when it comes to copyright stuff. Does github offer any form of private upload?

[marcbaechinger]

Please send an email to dev.exoplayer@gmail.com using a subject in the format "Issue #8396". Please also update this issue to indicate you’ve done this.

Many thanks!

[Tupsi]

Please send an email to dev.exoplayer@gmail.com using a subject in the format "Issue #8396". Please also update this issue to indicate you’ve done this.

Many thanks!

done.

[krocard]

I though at first that your channels where in a different order than what Android was expected. It does not seem to be the case as ffmpeg and android have the same channel order. I also looked at the sample and it has the expected channel order.

I then thought that maybe the format was played in passthrough and the tv/av receiver was decoding this incorrectly, but it unlikely that your setup supports Opus passthrough.

The channels also seems to be correctly mapped post decoding as dd+ is correctly played.

So the audio is most probably decoded on your FireTV Stick, and the channel incorrectly mapped there.

Unfortunately, I don't have access to a FireTV, nor a 5.1 setup.

In order to isolate variables and confirm the issue, could you play your sample with our DemoApplication?
If our demo app also exhibits the issue, then it is most probably a platform decoder issue, and you could block this decoder, if there is an other one on the platform supporting Opus or use the ffmpeg extension.
If our demo app does not exhibit the issue, then something else in your setup is causing the problem and I don't think we can help you figuring out what it is.

[Tupsi]

Sorry, but I am not enough of a programmer to get that DemoApplication running. While I could try to install Android Studio there is talk about "connecting a physical device" and I have no clue how to do that and what is actually ment by that.

At least I can give some more information to convince you that the bug is actually in the exoplayer code and not somewhere else.
I installed the Kodi app (which to my knowledge does not use any of your code) on FireTV app and it plays my files nicely, meaning opus audio always has the correct channel order.
Over the holidays, I switched to Nvidia Shield and installed Emby there with the same result, channels are not where they should be.
The emby people tell me that for playback of stuff like opus, they use your code and do not have anything else for it, hence my cry for help here.

You mentioning to "block the decoder" would not really help me, as emby rely on your code to decode it, but maybe I do not understand really what you are trying to say with that.

[Tupsi]

ah wth, I will give it a try, but its to late for today, so I will try tomorrow. Can you give me some pointers please in how to connect that compiled app to my shield and how I would have to edit that playlist I seem to need?

[krocard]

Sorry, but I am not enough of a programmer to get that DemoApplication running. While I could try to install Android Studio there is talk about "connecting a physical device" and I have no clue how to do that and what is actually ment by that.

I will take a look if we have a prebuilt DemoApp for you to install direcly instead of building your own.

I installed the Kodi app (which to my knowledge does not use any of your code) on FireTV app and it plays my files nicely, meaning opus audio always has the correct channel order.

I had a quick look to the Kodi and it seems to be using ffmpeg, not the platform decoders.

The emby people tell me that for playback of stuff like opus, they use your code and do not have anything else for it, hence my cry for help here.

ExoPlayer does not decode itself media, it uses the platform decoders (aka the Opus decoder bundled by Amazon in the FireTV os), and optionally ffmpeg if bundled.

If that platform opus decoder has a channel mapping bug, your best way to avoid the issue would be to ask the emby app to use the ExoPlayer extension to offer the option to use ffmpeg instead of the platform decoder.

But first we need to make sure that the platform Opus decoder is indeed the problem and not some other code. Let me search for an app that offers both ffmpeg and platform decoding to compare both decoding.

[krocard]

Can you try playing your sample with VLC for Android and compare that to playing the sample with the galerie app of your FireTV?
If the galerie app reproduces the issue, it will definitely be a platform issue.

[Tupsi]

Can you try playing your sample with VLC for Android and compare that to playing the sample with the galerie app of your FireTV?
If the galerie app reproduces the issue, it will definitely be a platform issue.

Just installed VLC on the Shield and it plays all files I have in 5.1 or 7.1 opus as it should be (aka fine), not just the sample I send you.
I could plug in the FireTV again as well, but as both devices show the same outcome, I see no point atm.

What exactly do you mean by "galerie app of your FireTV"?

From what you last replied it seemed that in both Amazon FireTV and Nvidia Shield your mentioned platform decoder gets used, which in turn means, you are unable to fix it in ExoPlayer itself because you do not handle the actual decoding. Which means I knocked on the wrong door to fix my problem. At least you offered a way out of this mess for me, IF I can convince the emby programmer to add that ExoPlayer extension.

In a general sense you guys might think about not using that plattform stuff if its that crappy and your have that ffmpeg option anyway in your product.

[krocard]

In a general sense you guys might think about not using that plattform stuff if its that crappy and your have that ffmpeg option anyway in your product.

Platform decoders are more power efficient, can handle higher bitrate/resolution and do not impact app size.

I'm surprised Nvidia Shield has the same issue.
Could you try to play your file with the built-in media player? I'm not sure how to do it with the shield, but you should be able to download the file and then play it from the android file manager at least.

[Tupsi]

As far as workaround, we already have a class that remaps channels, so we could imagine remaping the channels when this decoder is used with a multichannel output. Though it will depend if this channel issue is always present or just for some specifically constructed Opus file.

well I do not know if my way of getting myself opus into my mkv is specific or rather general, but I will just post you my usual ffmpeg audio part here for reference:

5.1
-c:a libopus -b:a 192000 -mapping_family 1 -af aformat=channel_layouts=5.1
7.1
-c:a libopus -b:a 450000 -mapping_family 1 -af aformat=channel_layouts=7.1

so I do not think, that is anything special.

[Tupsi]

@Tupsi I was able to reproduce on my phone tip of tree (dumping the 6 channels before they were downmixed to stereo). No need for you to provide a bug report. So it is indeed an AOSP issue, all android devices are probably affected. I will report it to the Google internal team in charge. Thanks for your persistence in reporting this issue.
Internal issue: b/177524746.

nice to see that we are getting somewhere and thanks for helping out!

Will you update this when there are changes to the base or is there some other place I could try to keep track, so I can revisit this once there is progress?

[andrewlewis]

With the ExoPlayer opus extension (not depending on the platform opus decoder) the same issue is reproducible.

I noticed in the ffmpeg sources that a different channel mapping is shown for 'vorbis' and 'libopus': https://github.com/FFmpeg/FFmpeg/blob/f4bdedd/libavcodec/libopusenc.c#L62-L84 If I step through the ExoPlayer opus decoder, we read a mapping for this file of 0 4 1 2 3 5, which appears in ffmpeg's vorbis mapping, and if I hardcode this to 0 1 4 2 3 5, which appears in ffmpeg's libopus mapping, the audio sounds correct.

[krocard]

well I do not know if my way of getting myself opus into my mkv is specific or rather general, but I will just post you my usual ffmpeg audio part here for reference.

I will update this issue when significant progress is made internally (b/177524746). In case I forget, you may request an update in a couple of month if nothing is posted here.

[Tupsi]

Thanks!

I found a DTS 7.1 Testaudio file today in my lib which nicely shows which speaker is currently suppose todo something in the test (in the video).
So as expected this works as it should, my reciever gets the DTS Signal and decodes all channels properly, Then I put a 2nd audio stream in the mkv as 7.1 opus and now I get a better idea which channels are switched and to where, so I though I would share that with you guys. Maybe that helps hunting it down faster.

correct (DTS)       | wrong (opus)
----------------------------------------------------------
Left                | Left
Center              | Right
Right               | Center
Right Side Surround | Left Rear Surround
Right Rear Surround | Left Side Surround
Left Rear Surround  | Right Rear Surround
Left Side Surround  | LFE
LFE                 | Right Side Surround

with the left side "correct (DTS)" meaning that it works as intended and the right side "wrong(opus)" meaning that this is where I can hear the channels went.

Getting media info of the files show clearly that opus put the channels in a different order then DTS.

Audio #1
ID                                       : 2
Format                                   : DTS XLL
Format/Info                              : Digital Theater Systems
Commercial name                          : DTS-HD Master Audio
Codec ID                                 : A_DTS
Duration                                 : 1 min 17 s
Bit rate mode                            : Variable
Bit rate                                 : 5 002 kb/s
Channel(s)                               : 8 channels
Channel layout                           : C L R LFE Lb Rb Lss Rss
Sampling rate                            : 48.0 kHz
Frame rate                               : 93.750 FPS (512 SPF)
Bit depth                                : 24 bits
Compression mode                         : Lossless
Stream size                              : 46.5 MiB (15%)
Title                                    : DTS-HD Master (7.1)
Language                                 : English
Default                                  : Yes
Forced                                   : No

Audio #2
ID                                       : 3
Format                                   : Opus
Codec ID                                 : A_OPUS
Duration                                 : 1 min 17 s
Bit rate                                 : 408 kb/s
Channel(s)                               : 8 channels
Channel layout                           : L R C Ls Rs Lb Rb LFE
Sampling rate                            : 48.0 kHz
Frame rate                               : 50.007 FPS (960 SPF)
Compression mode                         : Lossy
Stream size                              : 3.80 MiB (1%)
Title                                    : OPUS 7.1
Writing library                          : Lavc58.115.102 libopus
Language                                 : English
Default                                  : No
Forced                                   : No

[Sray69]

I was wondering if there is any update on this issue? I can confirm that this is still an issue with the new Fire TV Stick 4K Max and ExoPlayer 2.17.1.

There was mention above of a possible workaround. Using ffmpeg? Re-encoding/Re-mapping channels? ExoPlayer plays AV1 files better than any other player on the Fire Stick. I really hope it gets worked out.

[tonihei]

There is no progress on the underlying platform bug I'm afraid.

The workaround mentioned above is to use a ChannelMappingAudioProcessor (that can be injected into the DefaultAudioSink returned from DefaultRenderersFactory.buildAudioSink). This processor can reorder channels as required. However, you'd need to know exactly when the problem will occur in advance to be able to add the processor to the player only when needed.

[Sray69]

Thanks for the response. I ended up finding a player that actually works.

[Tupsi]

My problem occured in the software emby. After convincing the guys finally with this that there is actually a problem in the ExePlayer code (which they use underneath) they just switched to software decode and let them handle it (guess with ffmpeg) and now everything is fine. Hope you guys get it fixed someday, I have my fingers crossed! After all, it was a nightmare convincing the guys that I am not a total idiot in encoding stuff, but that there was a problem with the decoder. :-)

[Sray69]

Thanks for that info. I don't have much confidence it will get fixed any time soon. Which is odd to me because I would think it affects a lot of people. There are a lot of players/apps that are built off of ExoPlayer. And some big ones like Emby, Plex, Jellyfin, etc. I know Opus has not been a common codec for 5.1 content but it seems to be becoming one of the most popular for its quality and size. Especially when combined with AV1.

Thanks again.

[channeladam]

I just came across this issue on a new Fire TV Stick 4K Max.
Am very disappointed that it hasn't been fixed for almost 3 years.

@tonihei , is there any reason why there has been no progress on this?
Will there ever be progress? Is it on a roadmap? Or does no one care?

[channeladam]

And also... the YouTube app uses Opus... (you know, YouTube, by GOOGLE)... also has this problem... so anyone who buys an Android platform media player / Google TV and tries to play 5.1 audio through YouTube won't hear the audio as it is meant to be.
Is that not incentive enough for Google to fix it?

[johnmolina]

@tonihei Any update on this? I just bought a brand new Samsung Galaxy Tab S9 with Android 13 and guess what? Multiple apps and even the integrated Samsung video player are all still affected by this bug!

How will there ever be any successful free alternative to DTS and Dolby for multichannel audio if Google can't fix such a fundamental bug.

[tonihei]

We are aware this is still a problem :)

Our hope was that it can be fixed in the actual codec implementation itself, and once it's fixed, we can target a workaround to the Android versions that don't have and need the fix. It seems we won't get a proper fix in the codec anytime soon though.

Another interesting element is that it could be argued that the opus decoders follow the spec too strictly.

• The Opus spec defines that the output channel layout must follow the VORBIS layout . So OMX.google.opus.decoder, c2.android.opus.decoder and ExoPlayer's libopus based extension all generate the output in the VORBIS layout (Left, Center, Right, Back left, Back right, Low freq for 5.1)
• Android's audio framework expects channels in the bitmask order defined by AudioFormat. This is Left, Right, Center, Low freq, Back left, back right for 5.1.

The conclusion is that all 3 decoder implementations as a standalone component follow the OPUS spec correctly, but the conversion to Android channel layout still needs to happen.

Why does ffmpeg's opus decoding work then? It only works because ffmpeg defines its own channel layout, which is actually the same as Android's layout. And consequently ffmpeg's opus decoder internally converts the VORBIS layout to its own layout.

Given the connection to VORBIS layouts, I also checked a 5.1 vorbis file and it has the exact same issue with c2.android.vorbis.decoder and OMX.google.vorbis.decoder.

Since this behavior is coupled to the codec implementation and we know it won't be fixed anytime soon (or maybe never given how the actual codec is strictly speaking spec-compliant), I'll see how we can integrate the channel layout mapping into ExoPlayer directly.

[channeladam]

I'll see how we can integrate the channel layout mapping into ExoPlayer directly.

@tonihei thank you, thank you, thank you, thank you, thank you for responding and investigating!

It would be a wonderful end of year gift to all users of Google things (e.g. Android, YouTube app and watchers of any modern media via other players) to have the channel layout mapping in place so we can hear the audio coming from the intended speakers.

Every end-user will agree that it is much more important for the end result to be correct asap than waiting for arguable technical codec fixes that may never happen!

thank you, thank you, thank you!

Do you think the mapping might happen this year?

thank you, thank you, thank you!

[tonihei]

This should be fixed now by the commit above (and will be released as part of the next bugfix and minor release = Media3 1.2.1 and/or 1.3.0)