Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flac in mp4 using DASH does not seem to work #6397

Closed
kgrevehagen opened this issue Sep 5, 2019 · 8 comments
Closed

Flac in mp4 using DASH does not seem to work #6397

kgrevehagen opened this issue Sep 5, 2019 · 8 comments
Assignees
@ojw28
Labels
bug content not playing

Comments

@kgrevehagen
Copy link

[REQUIRED] Content description

FLAC in mp4 using DASH does not seem to work.
The same track in regular flac container works as it should.
I'm of course using the Mp4Extractor, and playing AAC in the same setup works flawlessly. I have tested with various configuration in regards to renderer, explained in the bug report section below.

[REQUIRED] Link to test content

I can probably provide if needed. Do you have any test content that should work? I tried our .mpd file in VLC on my computer, and that played as it should.

[REQUIRED] Version of ExoPlayer being used

2.10.4 forked with this change included.

[REQUIRED] Device(s) and version(s) of Android being used

Pixel 3 Android 9 and 10.

[REQUIRED] A full bug report captured from the device

Full bug report can be provided if needed.
Here are some logs from my different tries:

Using LibflacAudioRenderer: It is not fetching the dash segments, but plays without sound(time is passing, can pause, play etc) and gives me no error message.

Using default MediaCodecAudioRenderer(afaik understood from #6392 this should work because of platform decoder on Android 8.1 and above) it is fetching dash segments, but does not play like above, and I'm getting the following error:

D/CCodec: allocate(c2.android.flac.decoder)
I/Codec2Client: Available Codec2 services: "default" "software"
I/Codec2Client: Creating a Codec2 client to service "default"
I/Codec2Client: Client to Codec2 service "default" created
I/CCodec: setting up 'default' as default (vendor) store
I/Codec2Client: Creating a Codec2 client to service "default"
I/Codec2Client: Client to Codec2 service "default" created
E/Codec2Client: createComponent(c2.android.flac.decoder) -- call failed: NOT_FOUND.
I/Codec2Client: Creating a Codec2 client to service "software"
I/Codec2Client: Client to Codec2 service "software" created
I/CCodec: Created component [c2.android.flac.decoder]
D/CCodecConfig: read media type: audio/flac
D/ReflectedParamUpdater: extent() != 1 for single value type: algo.buffers.max-count.values
D/ReflectedParamUpdater: extent() != 1 for single value type: output.subscribed-indices.values
D/ReflectedParamUpdater: extent() != 1 for single value type: input.buffers.allocator-ids.values
D/ReflectedParamUpdater: extent() != 1 for single value type: output.buffers.allocator-ids.values
D/ReflectedParamUpdater: extent() != 1 for single value type: algo.buffers.allocator-ids.values
D/ReflectedParamUpdater: extent() != 1 for single value type: output.buffers.pool-ids.values
D/ReflectedParamUpdater: extent() != 1 for single value type: algo.buffers.pool-ids.values
I/CCodecConfig: query failed after returning 8 values (BAD_INDEX)
D/CCodecConfig: c2 config is Dict {
      c2::u32 coded.bitrate.value = 768000
      c2::u32 input.buffers.max-size.value = 32768
      c2::u32 input.delay.value = 0
      string input.media-type.value = "audio/flac"
      string output.media-type.value = "audio/raw"
      c2::u32 raw.channel-count.value = 1
      c2::u32 raw.pcm-encoding.value = 0
      c2::u32 raw.sample-rate.value = 44100
    }
D/CCodecConfig: c2 config is Dict {
      c2::u32 coded.bitrate.value = 768000
      c2::u32 input.buffers.max-size.value = 32768
      c2::u32 input.delay.value = 0
      string input.media-type.value = "audio/flac"
      string output.media-type.value = "audio/raw"
      c2::u32 raw.channel-count.value = 2
      c2::u32 raw.pcm-encoding.value = 0
      c2::u32 raw.sample-rate.value = 44100
    }
W/Codec2Client: query -- param skipped: index = 1107298332.
D/CCodec: setup formats input: AMessage(what = 0x00000000) = {
      int32_t channel-count = 2
      int32_t max-input-size = 32768
      string mime = "audio/flac"
      int32_t pcm-encoding = 2
      int32_t sample-rate = 44100
    } and output: AMessage(what = 0x00000000) = {
      int32_t channel-count = 2
      string mime = "audio/raw"
      int32_t pcm-encoding = 2
      int32_t sample-rate = 44100
    }
W/Codec2Client: query -- param skipped: index = 1342179345.
W/Codec2Client: query -- param skipped: index = 2415921170.
D/CCodecBufferChannel: [c2.android.flac.decoder#18] Created input block pool with allocatorID 16 => poolID 17 - OK (0)
I/CCodecBufferChannel: [c2.android.flac.decoder#18] Created output block pool with allocatorID 16 => poolID 116 - OK
D/CCodecBufferChannel: [c2.android.flac.decoder#18] Configured output block pool ids 116 => OK
E/ion: ioctl c0044901 failed with code -1: Invalid argument
D/CCodecBufferChannel: [c2.android.flac.decoder#18] work failed to complete: 14
E/MediaCodec: Codec reported err 0xe, actionCode 0, while in state 6
D/CCodecBufferChannel: [c2.android.flac.decoder#18] work failed to complete: 22
E/MediaCodec: Codec reported err 0x16, actionCode 0, while in state 0
E/ExoPlayerImplInternal: Internal runtime error.
    java.lang.IllegalStateException
        at android.media.MediaCodec.native_dequeueOutputBuffer(Native Method)
        at android.media.MediaCodec.dequeueOutputBuffer(MediaCodec.java:2789)
        at com.google.android.exoplayer2.mediacodec.MediaCodecRenderer.drainOutputBuffer(MediaCodecRenderer.java:1437)
        at com.google.android.exoplayer2.mediacodec.MediaCodecRenderer.render(MediaCodecRenderer.java:653)
        at com.google.android.exoplayer2.ExoPlayerImplInternal.doSomeWork(ExoPlayerImplInternal.java:575)
        at com.google.android.exoplayer2.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:326)
        at android.os.Handler.dispatchMessage(Handler.java:103)
        at android.os.Looper.loop(Looper.java:214)
        at android.os.HandlerThread.run(HandlerThread.java:67)
W/System.err: com.google.android.exoplayer2.ExoPlaybackException: java.lang.IllegalStateException
W/System.err:     at com.google.android.exoplayer2.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:397)
W/System.err:     at android.os.Handler.dispatchMessage(Handler.java:103)
W/System.err:     at android.os.Looper.loop(Looper.java:214)
W/System.err:     at android.os.HandlerThread.run(HandlerThread.java:67)
W/System.err: Caused by: java.lang.IllegalStateException
E/ExoPlayerImplInternal: Disable failed.
    java.lang.IllegalStateException
        at android.media.MediaCodec.native_flush(Native Method)
        at android.media.MediaCodec.flush(MediaCodec.java:2194)
        at com.google.android.exoplayer2.mediacodec.MediaCodecRenderer.flushOrReleaseCodec(MediaCodecRenderer.java:702)
        at com.google.android.exoplayer2.mediacodec.MediaCodecRenderer.onDisabled(MediaCodecRenderer.java:580)
        at com.google.android.exoplayer2.audio.MediaCodecAudioRenderer.onDisabled(MediaCodecAudioRenderer.java:625)
        at com.google.android.exoplayer2.BaseRenderer.disable(BaseRenderer.java:158)
        at com.google.android.exoplayer2.ExoPlayerImplInternal.disableRenderer(ExoPlayerImplInternal.java:1088)
        at com.google.android.exoplayer2.ExoPlayerImplInternal.resetInternal(ExoPlayerImplInternal.java:851)
        at com.google.android.exoplayer2.ExoPlayerImplInternal.stopInternal(ExoPlayerImplInternal.java:813)
        at com.google.android.exoplayer2.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:399)
        at android.os.Handler.dispatchMessage(Handler.java:103)
        at android.os.Looper.loop(Looper.java:214)
        at android.os.HandlerThread.run(HandlerThread.java:67)
W/System.err:     at android.media.MediaCodec.native_dequeueOutputBuffer(Native Method)
W/System.err:     at android.media.MediaCodec.dequeueOutputBuffer(MediaCodec.java:2789)
W/System.err:     at com.google.android.exoplayer2.mediacodec.MediaCodecRenderer.drainOutputBuffer(MediaCodecRenderer.java:1437)
W/System.err:     at com.google.android.exoplayer2.mediacodec.MediaCodecRenderer.render(MediaCodecRenderer.java:653)
W/System.err:     at com.google.android.exoplayer2.ExoPlayerImplInternal.doSomeWork(ExoPlayerImplInternal.java:575)
W/System.err:     at com.google.android.exoplayer2.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:326)
W/System.err: 	... 3 more
E/ExoPlayerImplInternal: Reset failed.
    java.lang.IllegalStateException
        at android.media.MediaCodec.native_stop(Native Method)
        at android.media.MediaCodec.stop(MediaCodec.java:2147)
        at com.google.android.exoplayer2.mediacodec.MediaCodecRenderer.releaseCodec(MediaCodecRenderer.java:609)
        at com.google.android.exoplayer2.mediacodec.MediaCodecRenderer.onReset(MediaCodecRenderer.java:587)
        at com.google.android.exoplayer2.audio.MediaCodecAudioRenderer.onReset(MediaCodecAudioRenderer.java:635)
        at com.google.android.exoplayer2.BaseRenderer.reset(BaseRenderer.java:164)
        at com.google.android.exoplayer2.ExoPlayerImplInternal.resetInternal(ExoPlayerImplInternal.java:860)
        at com.google.android.exoplayer2.ExoPlayerImplInternal.stopInternal(ExoPlayerImplInternal.java:813)
        at com.google.android.exoplayer2.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:399)
        at android.os.Handler.dispatchMessage(Handler.java:103)
        at android.os.Looper.loop(Looper.java:214)
        at android.os.HandlerThread.run(HandlerThread.java:67)
I/ExoPlayerImpl: Release 5e5c6b5 [ExoPlayerLib/2.10.4] [blueline, Pixel 3, Google, 29] [goog.exo.core, goog.exo.dash, goog.exo.okhttp]
@ojw28
Copy link
Contributor

ojw28 commented Sep 5, 2019 •
edited
Loading

We were previously provided with some FLAC in DASH content in #4883, and I'm pretty sure we had it working at the time. Unfortunately it appears the samples are no longer available.

@ojw28
Copy link
Contributor

ojw28 commented Sep 5, 2019

@ronak2121 - Do you have any DASH/FLAC test streams that are still available?
@kgrevehagen - We probably need test content. It seems at least possible that your segmentation of FLAC into samples to put inside of the MP4 container is non-spec compliant.

As a general point, you're making life really hard for yourself by using FLAC. Do you really need to use it? Doing what nearly everyone else does (i.e. AAC) is definitely the path of least resistance, unless that's a non-started from a product perspective.

@ojw28 ojw28 self-assigned this Sep 5, 2019
@kgrevehagen
Copy link
Author

Here is an example of a FLAC in mp4/DASH.

@kgrevehagen
Copy link
Author

Tested this in the ExoPlayer Sample app, and when I build the flac extension and set DefaultRenderersFactory.EXTENSION_RENDERER_MODE_PREFER on the DefaultExtractorsFactory, it gives me the following error:

V/FLACParser: FLACParser::FLACParser
E/FLACParser: FLACParser::errorCallback status=0
E/FLACParser: metadata decoding failed
E/ExoPlayerImplInternal: Playback error.
    com.google.android.exoplayer2.ExoPlaybackException: com.google.android.exoplayer2.ext.flac.FlacDecoderException: Failed to decode StreamInfo
        at com.google.android.exoplayer2.audio.SimpleDecoderAudioRenderer.maybeInitDecoder(SimpleDecoderAudioRenderer.java:631)
        at com.google.android.exoplayer2.audio.SimpleDecoderAudioRenderer.onInputFormatChanged(SimpleDecoderAudioRenderer.java:697)
        at com.google.android.exoplayer2.audio.SimpleDecoderAudioRenderer.render(SimpleDecoderAudioRenderer.java:277)
        at com.google.android.exoplayer2.ExoPlayerImplInternal.doSomeWork(ExoPlayerImplInternal.java:575)
        at com.google.android.exoplayer2.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:326)
        at android.os.Handler.dispatchMessage(Handler.java:103)
        at android.os.Looper.loop(Looper.java:214)
        at android.os.HandlerThread.run(HandlerThread.java:67)
     Caused by: com.google.android.exoplayer2.ext.flac.FlacDecoderException: Failed to decode StreamInfo
        at com.google.android.exoplayer2.ext.flac.FlacDecoder.<init>(FlacDecoder.java:65)
        at com.google.android.exoplayer2.ext.flac.LibflacAudioRenderer.createDecoder(LibflacAudioRenderer.java:70)
        at com.google.android.exoplayer2.ext.flac.LibflacAudioRenderer.createDecoder(LibflacAudioRenderer.java:31)
        at com.google.android.exoplayer2.audio.SimpleDecoderAudioRenderer.maybeInitDecoder(SimpleDecoderAudioRenderer.java:624)
        at com.google.android.exoplayer2.audio.SimpleDecoderAudioRenderer.onInputFormatChanged(SimpleDecoderAudioRenderer.java:697) 
        at com.google.android.exoplayer2.audio.SimpleDecoderAudioRenderer.render(SimpleDecoderAudioRenderer.java:277) 
        at com.google.android.exoplayer2.ExoPlayerImplInternal.doSomeWork(ExoPlayerImplInternal.java:575) 
        at com.google.android.exoplayer2.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:326) 
        at android.os.Handler.dispatchMessage(Handler.java:103) 
        at android.os.Looper.loop(Looper.java:214) 
        at android.os.HandlerThread.run(HandlerThread.java:67) 
     Caused by: com.google.android.exoplayer2.ParserException: Failed to decode stream metadata
        at com.google.android.exoplayer2.ext.flac.FlacDecoderJni.decodeStreamMetadata(FlacDecoderJni.java:149)
        at com.google.android.exoplayer2.ext.flac.FlacDecoder.<init>(FlacDecoder.java:63)
        at com.google.android.exoplayer2.ext.flac.LibflacAudioRenderer.createDecoder(LibflacAudioRenderer.java:70) 
        at com.google.android.exoplayer2.ext.flac.LibflacAudioRenderer.createDecoder(LibflacAudioRenderer.java:31) 
        at com.google.android.exoplayer2.audio.SimpleDecoderAudioRenderer.maybeInitDecoder(SimpleDecoderAudioRenderer.java:624) 
        at com.google.android.exoplayer2.audio.SimpleDecoderAudioRenderer.onInputFormatChanged(SimpleDecoderAudioRenderer.java:697) 
        at com.google.android.exoplayer2.audio.SimpleDecoderAudioRenderer.render(SimpleDecoderAudioRenderer.java:277) 
        at com.google.android.exoplayer2.ExoPlayerImplInternal.doSomeWork(ExoPlayerImplInternal.java:575) 
        at com.google.android.exoplayer2.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:326) 
        at android.os.Handler.dispatchMessage(Handler.java:103) 
        at android.os.Looper.loop(Looper.java:214) 
        at android.os.HandlerThread.run(HandlerThread.java:67)

which tells me that decodeStreamMetadata() fails in FlacDecoder.java.

    FlacStreamMetadata streamMetadata;
    try {
      streamMetadata = decoderJni.decodeStreamMetadata();
    } catch (ParserException e) {
      throw new FlacDecoderException("Failed to decode StreamInfo", e);
    } catch (IOException | InterruptedException e) {
      // Never happens.
      throw new IllegalStateException(e);
    }

I tried to set correct FlacStreamMetaData by doing this instead(don't know if this is correct or okay to do, but it gives me correct FlacStreamMetaData, at least):

FlacStreamMetadata streamMetadata = new FlacStreamMetadata(initializationData.get(0), 0);

but that gives me the following error:

V/FLACParser: FLACParser::FLACParser
D/EventLogger: decoderInitialized [2.51, 0.00, window=0, period=0, audio, libflac]
D/EventLogger: decoderInputFormatChanged [2.51, 0.00, window=0, period=0, audio, id=0, mimeType=audio/flac, bitrate=1046930, codecs=flac, channels=2, sample_rate=44100, language=und]
E/FLACParser: FLACParser::readBuffer write invalid blocksize 4096
E/ExoPlayerImplInternal: Playback error.
    com.google.android.exoplayer2.ExoPlaybackException: com.google.android.exoplayer2.ext.flac.FlacDecoderException: Frame decoding failed
        at com.google.android.exoplayer2.audio.SimpleDecoderAudioRenderer.render(SimpleDecoderAudioRenderer.java:302)
        at com.google.android.exoplayer2.ExoPlayerImplInternal.doSomeWork(ExoPlayerImplInternal.java:575)
        at com.google.android.exoplayer2.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:326)
        at android.os.Handler.dispatchMessage(Handler.java:103)
        at android.os.Looper.loop(Looper.java:214)
        at android.os.HandlerThread.run(HandlerThread.java:67)
     Caused by: com.google.android.exoplayer2.ext.flac.FlacDecoderException: Frame decoding failed
        at com.google.android.exoplayer2.ext.flac.FlacDecoder.decode(FlacDecoder.java:101)
        at com.google.android.exoplayer2.ext.flac.FlacDecoder.decode(FlacDecoder.java:32)
        at com.google.android.exoplayer2.decoder.SimpleDecoder.decode(SimpleDecoder.java:224)
        at com.google.android.exoplayer2.decoder.SimpleDecoder.run(SimpleDecoder.java:189)
        at com.google.android.exoplayer2.decoder.SimpleDecoder.access$000(SimpleDecoder.java:25)
        at com.google.android.exoplayer2.decoder.SimpleDecoder$1.run(SimpleDecoder.java:67)
     Caused by: com.google.android.exoplayer2.ext.flac.FlacDecoderJni$FlacFrameDecodeException: Cannot decode FLAC frame
        at com.google.android.exoplayer2.ext.flac.FlacDecoderJni.decodeSample(FlacDecoderJni.java:187)
        at com.google.android.exoplayer2.ext.flac.FlacDecoder.decode(FlacDecoder.java:99)
        at com.google.android.exoplayer2.ext.flac.FlacDecoder.decode(FlacDecoder.java:32) 
        at com.google.android.exoplayer2.decoder.SimpleDecoder.decode(SimpleDecoder.java:224) 
        at com.google.android.exoplayer2.decoder.SimpleDecoder.run(SimpleDecoder.java:189) 
        at com.google.android.exoplayer2.decoder.SimpleDecoder.access$000(SimpleDecoder.java:25) 
        at com.google.android.exoplayer2.decoder.SimpleDecoder$1.run(SimpleDecoder.java:67) 

Hopefully you can test the provided url and see if you can get some more info on what is happening here.

@ojw28
Copy link
Contributor

ojw28 commented Sep 6, 2019 •
edited
Loading

I've figured out what's going on here. It's related to Format.initializationData, which is used to configure the decoder:

  • Samsung's FLAC decoder (OMX.SEC.flac.dec) works fine with the provided stream. I.e. playback works on a Galaxy S8, without using the FLAC extension.
  • Google's FLAC decoder (OMX.google.flac.decoder) requires the FLAC stream marker to be prefixed onto the initialization data, which is how FLAC codec private data is defined in the Matroska container here: "The private data contains all the header/metadata packets before the first data packet. These include the first header packet containing only the word fLaC as well as all metadata packets.". In MP4 the FLAC stream marker is not prefixed into the initialization data, which is why playback fails.
  • The FLAC extension also seems to require the FLAC stream marker to be present.

I was able to get all cases working by changing this code block to add the FLAC stream marker:

int childAtomBodySize = childAtomSize - Atom.FULL_HEADER_SIZE;
initializationData = new byte[childAtomBodySize + 4];
initializationData[0] = 0x66;
initializationData[1] = 0x4C;
initializationData[2] = 0x61;
initializationData[3] = 0x43;
parent.setPosition(childPosition + Atom.FULL_HEADER_SIZE);
parent.readBytes(initializationData, /* offset= */ 4, childAtomBodySize);

I'd like to double check that the approach above is the right thing to do before submitting a fix (and, obviously, this should be for TYPE_dfLa only).

@ojw28 ojw28 added bug and removed needs triage labels Sep 6, 2019
@ronak2121
Copy link

ronak2121 commented Sep 6, 2019 •
edited
Loading

Hey sorry I’m not in the office and can’t get my test streams back up. I had previously generated the above FLAC streams using FFmpeg.

I’ll be in the office on Monday and can help then.

How are you generating them? @kgrevehagen

@ojw28
Copy link
Contributor

ojw28 commented Sep 6, 2019

Thanks for responding. I think we've figured it out now as described above, so no worries!

ojw28 added a commit that referenced this issue Sep 10, 2019
@ojw28
Add fLaC prefix to FLAC initialization data
0706625 
The fLaC prefix is included in the initialization data output
from the MKV extractor, so this is highly likely ot be the
right thing to do.

Issue: #6397
PiperOrigin-RevId: 268244365
@ojw28
Copy link
Contributor

ojw28 commented Sep 10, 2019

This should be fixed by the commit above. Please give it a try.

@ojw28 ojw28 closed this as completed Sep 10, 2019
ojw28 added a commit that referenced this issue Sep 17, 2019
@ojw28
Add fLaC prefix to FLAC initialization data
23ddfaa 
The fLaC prefix is included in the initialization data output
from the MKV extractor, so this is highly likely ot be the
right thing to do.

Issue: #6397
PiperOrigin-RevId: 268244365
@google google locked and limited conversation to collaborators Nov 10, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@ojw28 ojw28

Labels
bug content not playing
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@kgrevehagen @ronak2121 @ojw28 @google-oss-bot