HLS muxed audio missing format label

[szaboa]

We have the following HLS (live) master playlist:

#EXTM3U
#EXT-X-VERSION:6
#EXT-X-INDEPENDENT-SEGMENTS
#EXT-X-STREAM-INF:BANDWIDTH=130780,CODECS="mp4a.40.2",AUDIO="aac-primary"
../../<path>/chunklist_ao.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=130780,CODECS="mp4a.40.2",AUDIO="aac-backup"
../../<path>/chunklist_ao.m3u8
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="aac-primary",NAME="Home",LANGUAGE="eng",DEFAULT=YES,AUTOSELECT=YES,CHANNELS="2"
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="aac-backup",NAME="Home",LANGUAGE="eng",DEFAULT=YES,AUTOSELECT=YES,CHANNELS="2"
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="aac-primary",NAME="Away",LANGUAGE="eng",URI="../../<path>/chunklist_ao.m3u8",DEFAULT=NO,AUTOSELECT=YES,CHANNELS="2"
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="aac-backup",NAME="Away",LANGUAGE="eng",URI="../../<path>/chunklist_ao.m3u8",DEFAULT=NO,AUTOSELECT=YES,CHANNELS="2"

Based on the HlsPlaylistParser (line 514), if an audio track has no URI defined, then it will be stored in the muxedAudioFormat. However in this case, whenever we query the audio tracks, the muxed one has no id/label in the Format.

See the event logger's output:

MediaCodecAudioRenderer [
  Group:0, adaptive_supported=YES_NOT_SEAMLESS [
    [ ] Track:0, id=0, mimeType=audio/mp4a-latm, bitrate=130909, codecs=mp4a.40.2, channels=2, sample_rate=48000, supported=YES
    [ ] Track:1, id=1, mimeType=audio/mp4a-latm, bitrate=130909, codecs=mp4a.40.2, channels=2, sample_rate=48000, supported=YES
  ]
  Group:1, adaptive_supported=YES_NOT_SEAMLESS [
    [X] Track:0, id=aac-primary:Away, mimeType=audio/mp4a-latm, codecs=mp4a.40.2, channels=2, sample_rate=48000, language=en, label=Away, supported=YES
    [ ] Track:1, id=aac-backup:Away, mimeType=audio/mp4a-latm, codecs=mp4a.40.2, channels=2, sample_rate=48000, language=en, label=Away, supported=YES
  ]

This is causing a problem for us, as we would need to select the tracks based on the labels. The question is whether the missing labels of muxed audio tracks is intentional or if it's a bug? How could we workaround this issue on our side (without changing the manifest itself)?

Thanks in advance.

[ojw28]

Do you see this with 2.15.1 (i.e., the latest release)? I took a quick look at the code, and it seems to be written so that the label should be picked up. If that's not happening then I'm probably missing something, and it's probably a bug.

If you can reproduce with 2.15.1, could you provide a complete HLS sample that we can use to easily reproduce the issue, either as a link to an accessible test stream or as a zip file containing the necessary files? The master playlist alone isn't sufficient. Thanks!

[szaboa]

@ojw28 We can't really provide a clear sample stream easily, however we have found a solution that we can apply on our forked version. You can see the change in the linked pull request, feel free to give us a feedback on that.

[ojw28]

I'm not sure, but that commit doesn't seem to make much sense to me. I don't understand why you'd want to set primaryExtractorTrackType = C.TRACK_TYPE_VIDEO for an audio only stream.

We can't really provide a clear sample stream easily

Can you provide a DRM protected stream easily, even without the corresponding DRM server information? I'm not sure, but I suspect that the tracks are determined before the player needs the DRM keys, so that might be sufficient to be able to reproduce. Thanks!

[szaboa]

The primaryExtractorTrackType is set to C.TRACK_TYPE_VIDEO because otherwise the muxedAudioFormat won't be used at all, see the logic a few lines below the change in the PR (line 1371-1381).

[ojw28]

Ah, I see. Could you try applying this change locally, to see whether it also resolves the issue correctly: 6b78164

Thanks!

[guoen21]

I apply and test this commit locally, seems it also works for me, thanks very much!

Now we get two following muxed tracks, but the id is same, because we use one muxedAudioFormat, but from our master m3u8 it included two tracks which has no URI defined.

Group:0, adaptive_supported=YES_NOT_SEAMLESS [
  [X] Track:0, id=aac-backup:Home, mimeType=audio/mp4a-latm, bitrate=128206, codecs=mp4a.40.2, channels=2, sample_rate=48000, label=Home, supported=YES
  [X] Track:1, id=aac-backup:Home, mimeType=audio/mp4a-latm, bitrate=128206, codecs=mp4a.40.2, channels=2, sample_rate=48000, label=Home, supported=YES
]

expected

Group:0, adaptive_supported=YES_NOT_SEAMLESS [
  [X] Track:0, id=aac-primary:Home, mimeType=audio/mp4a-latm, bitrate=128206, codecs=mp4a.40.2, channels=2, sample_rate=48000, label=Home, supported=YES
  [X] Track:1, id=aac-backup:Home, mimeType=audio/mp4a-latm, bitrate=128206, codecs=mp4a.40.2, channels=2, sample_rate=48000, label=Home, supported=YES
]

[guoen21]

I also add comment for commit 45857b5,
line 1398, playlistFormat = playlistFormat.withManifestFormatInfo(muxedAudioFormat);
In here the language attribute of muxedAudioFormat may be miss, because of the playlistFormat.sampleMimeType is null.

For above audio-only case, it will output the expected result

Group:0, adaptive_supported=YES_NOT_SEAMLESS [
  [X] Track:0, id=aac-primary:Home, mimeType=audio/mp4a-latm, bitrate=128206, codecs=mp4a.40.2, channels=2, sample_rate=48000, label=Home, supported=YES
  [ ] Track:1, id=aac-backup:Home, mimeType=audio/mp4a-latm, bitrate=128206, codecs=mp4a.40.2, channels=2, sample_rate=48000, label=Home, supported=YES
]

But for normal case which has video tracks, muxed audio tracks, and additional audio tracks, the output has only one muxed audio track as following:

Group:0, adaptive_supported=YES_NOT_SEAMLESS [
  [X] Track:1, id=aac-backup:Home, mimeType=audio/mp4a-latm, bitrate=128206, codecs=mp4a.40.2, channels=2, sample_rate=48000, label=Home, supported=YES
]

In my commit I enhance it to support multiple muxed audio tracks - change the single Format muxedAudioFormat to List<Format> muxedAudioFormats. I also merge this 45857b5 commit. Now I can get the expected output

Group:0, adaptive_supported=YES_NOT_SEAMLESS [
  [X] Track:0, id=aac-primary:Home, mimeType=audio/mp4a-latm, bitrate=128206, codecs=mp4a.40.2, channels=2, sample_rate=48000, label=Home, supported=YES
  [ ] Track:1, id=aac-backup:Home, mimeType=audio/mp4a-latm, bitrate=128206, codecs=mp4a.40.2, channels=2, sample_rate=48000, label=Home, supported=YES
]

[ojw28]

In here the language attribute of muxedAudioFormat may be miss, because of the playlistFormat.sampleMimeType is null.

This is a good observation. We'll take a look; thanks!

In my commit I enhance it to support multiple muxed audio tracks

There is a TODO about this here. If you've implemented it, please feel free to send us a pull request. If you've implemented it in a different way then that might also be interesting, or you could try and adapt your approach to match what's written in the TODO.