VideoDecoder Stalls

[seflless]

Hey everyone,

Does anyone else have an issue where a VideoDecoder you have enqueued for decoding enough EncodedVideoChunks (let's say 3), that you'd expect a VideoFrame to be outputted relatively quickly, but instead don't?

Either it sort of stalls and eventually you start to get VideoFrames again, sometimes it can be seconds or effectively never.

I'm doing all my work in a worker, I do periodic timeouts in my loops (where I wait for frames to be available) to leave time for decoding.

Francois

[sandersdan]

This behavior would be expected if VideoFrame objects are not getting closed quickly (output can stall waiting for output buffers), or if the stream is using reordering and the reorder window is large (it can take more than three chunks to output one frame).

[seflless]

Hmmmm, interesting hints. Thanks for the quick response!

Feels like the VideoFrames not getting freed fast enough just aggravates them not being freed enough. :)

Your point about the reorder window being sometimes >3. Do you have material I could reference to understand that aspect more? I can up from 3 max queued to something higher, but would rather have some reasoning as to what number I settle on.

[sandersdan]

For the first option, I would recommend testing by setting hardwareAcceleration: "prefer-software", since (at least in Chrome) software decoders do not have a limit on output buffers.

For the second option, the reorder window is typically a configurable property of encoding. It can be a complication with any codec, but is rarely an issue with VP9/AV1 as these codecs can bundle multiple logical frames into one "super frame" so that one input chunk always results in one output chunk immediately. It's more common with H.264 when B-frames are in use, in which case a reorder window of 4 is not uncommon. The actual limit is bound by the resolution and profile, or restricted further by sps.num_reorder_frames/sps.max_dec_frame_buffering. It can never exceed 16.

I would test by increasing the number of input buffers to at least 16. It's an unlikely situation, but can come up when the profile is set high and there is no VUI present to restrict the reorder window size.

Based on your description I would think the first problem to be more likely, since time passing won't affect reordering but can affect GC of frames that have been dropped but not close()d.

Edit: Note though that using software decoding can work around both issues; ffmpeg is often more clever about guessing the real reorder window if it's not specified. The tests I've described are not conclusive.

[seflless]

This is awesome detail.

What's the deal with the GCing of VideoFrames that you have a reference to?! It's not clear how to handle it or prevent it.

[sandersdan]

That won't happen. But dropping your reference without calling close() is the most common way to stall decoding.