Gear 360: OOM with poorly interleaved videos

[MateusZitelli]

Issue description

Experiencing a OOM crash in multiple devices when combining multiple ClippingMediaSource in a ConcatenatingMediaSource or DynamicConcatenatingMediaSource. I was only able to reproduce the issue in videos recorded by the camera Gear 360 (2016).

I was not able to reproduce the crash using ClippingMediaSource or ConcatenatingMediaSource independently.

Reproduction steps

I was able to reproduce the issue with the following changes in the demo app:

diff --git a/demo/src/main/assets/media.exolist.json b/demo/src/main/assets/media.exolist.json
index 59d8259d3..3a9b7b297 100644
--- a/demo/src/main/assets/media.exolist.json
+++ b/demo/src/main/assets/media.exolist.json
@@ -437,6 +437,18 @@
       }
     ]
   },
+  {
+    "name": "Concatenated Clipped Videos",
+    "samples": [
+      {
+        "name": "Crash",
+        "playlist": [
+          {"uri": "/storage/emulated/0/Movies/360/SAM_100_0024.mp4"},
+          {"uri": "/storage/emulated/0/Movies/360/SAM_100_0022.mp4"}
+        ]
+      }
+    ]
+  },
   {
     "name": "Playlists",
     "samples": [
diff --git a/demo/src/main/java/com/google/android/exoplayer2/demo/PlayerActivity.java b/demo/src/main/java/com/google/android/exoplayer2/demo/PlayerActivity.java
index 6416cd5aa..c1d2992a1 100644
--- a/demo/src/main/java/com/google/android/exoplayer2/demo/PlayerActivity.java
+++ b/demo/src/main/java/com/google/android/exoplayer2/demo/PlayerActivity.java
@@ -52,6 +53,7 @@ import com.google.android.exoplayer2.extractor.DefaultExtractorsFactory;
 import com.google.android.exoplayer2.mediacodec.MediaCodecRenderer.DecoderInitializationException;
 import com.google.android.exoplayer2.mediacodec.MediaCodecUtil.DecoderQueryException;
 import com.google.android.exoplayer2.source.BehindLiveWindowException;
+import com.google.android.exoplayer2.source.ClippingMediaSource;
 import com.google.android.exoplayer2.source.ConcatenatingMediaSource;
 import com.google.android.exoplayer2.source.ExtractorMediaSource;
 import com.google.android.exoplayer2.source.MediaSource;
@@ -370,8 +373,9 @@ public class PlayerActivity extends Activity implements OnClickListener, EventLi
       case C.TYPE_HLS:
         return new HlsMediaSource(uri, mediaDataSourceFactory, mainHandler, eventLogger);
       case C.TYPE_OTHER:
-        return new ExtractorMediaSource(uri, mediaDataSourceFactory, new DefaultExtractorsFactory(),
-            mainHandler, eventLogger);
+          ExtractorMediaSource mSource = new ExtractorMediaSource(uri, mediaDataSourceFactory,
+                  new DefaultExtractorsFactory(), mainHandler, eventLogger);
+          return new ClippingMediaSource(mSource, 0, 1000000);
       default: {
         throw new IllegalStateException("Unsupported type: " + type);
       }

Link to test content

https://drive.google.com/file/d/1_UGZiwwpm_8TEd1ycIw5HbFyS2lk3Qi4/view?usp=sharing
https://drive.google.com/file/d/1yh4GgIoahZqShtnjntGN19qEfhand6rA/view?usp=sharing

Version of ExoPlayer being used

r2.5.4

Device(s) and version(s) of Android being used

Galaxy Note 8 - Android 7.1.1
Galaxy S7 Edge - Android 7.1.1
Xiaomi Redmi Note 4 - Android 7.0

A full bug report captured from the device

OutOfMemory error loading stream
java.lang.OutOfMemoryError: Failed to allocate a 65548 byte allocation with 58912 free bytes and 57KB until OOM, max allowed footprint 536870912, growth limit 536870912
    at com.google.android.exoplayer2.upstream.DefaultAllocator.allocate(DefaultAllocator.java:102)
    at com.google.android.exoplayer2.source.SampleQueue.preAppend(SampleQueue.java:592)
    at com.google.android.exoplayer2.source.SampleQueue.sampleData(SampleQueue.java:516)
    at com.google.android.exoplayer2.extractor.mp4.Mp4Extractor.readSample(Mp4Extractor.java:455)
    at com.google.android.exoplayer2.extractor.mp4.Mp4Extractor.read(Mp4Extractor.java:152)
    at com.google.android.exoplayer2.source.ExtractorMediaPeriod$ExtractingLoadable.load(ExtractorMediaPeriod.java:704)
    at com.google.android.exoplayer2.upstream.Loader$LoadTask.run(Loader.java:315)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1133)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:607)
    at java.lang.Thread.run(Thread.java:762)

[tonihei]

It seems your mp4 files are very poorly interleaved which is causing the issue you see.
The first file for example (SAM_100_0024.mp4) has all the video data in the range [32, 400275315] followed by the audio data in [400275316, 401991423]. This means ExoPlayer needs to load all 400MB of video data in memory before finding the first audio sample. Your devices have enough memory to read these 400MB such that playback starts eventually without OOM errors.

When you use ConcatenatingMediaSource with a very short ClippingMediaMediaSource, the player tries to preload the next video after loading the first one. That's why you're seeing the OOM only when concatenating both videos.

To fix your issue, please make sure to interleave audio and video data in some sane way. See also #1405 and #1386 where similar issues occurred.

[MateusZitelli]

Do you have any recommendations about how to handle this type of error, or any method for identifying badly interleaved files? I ask that because this is the most common crash faced by my user base.

[ojw28]

Are these media files being produced entirely by Samsung's own hardware/software, or is your own software involved? It's really not great that the audio is just being shoved onto the end of the file like that, rather than being interleaved properly. If it's entirely Samsung's own hardware/software, we should have a discussion with them about why they're doing that.

[MateusZitelli]

Those files came directly from the Samsung Gear 360 (2016) cameras, however only some files present the problem.

Edit: Got some samples from Samsung Gear 360 (2017) and they also are bad interleaved.

[ojw28]

Lovely :(.

Just to clarify, I don't think you're actually seeing a crash are you? You should just be seeing playback fail, which we currently consider working as intended when media is this poorly interleaved. If you are seeing a crash, you haven't provided any logs that show it, so please do so (note the error shown in the snippet above is something our code catches and handles gracefully as playback failure).

[MateusZitelli]

In the demo app there is no crash happening but ExoPlayer keeps using the memory trying to load the video over and over again. Then in my app this over usage of memory generates OOM Exceptions in other threads.

[ojw28]

Got it. We could impose a hard limit on the maximum memory we'll consider using, but since different devices will allow us to use different amounts of memory, it's very unclear what that value should really be set to.

I don't think we retry indefinitely. I think we retry three times before giving up (and releasing all memory) by default. #2844 tracks not retrying at all for certain error types, which should probably be the case here. Either way if we're consuming nearly all available memory at some point in time, it's going to be possible that some other thread might OOM if it tries to allocate something whilst ExoPlayer is in this state.

[MateusZitelli]

You are right, in the demo app ExoPlayer only try to load the video three times before stopping. However, my application hangs with the garbage collector on the first try and some seconds later it crashes.

Seein that my application is a 360 video editor, videos from Gear 360 are very common and by far this is the most common issue that our users face. So even if my application didn't crash, just showing an error message would not suffice.

Therefore, I thought about two possible solutions:

1. Seeing that the issue just happens when I use ConcatenatingMediaSources to merge short ClippedMediaSources of badly interleaved video, maybe disabling the preload of videos that are not being played could avoid the excessive use of memory.
2. Identify the badly interleaved videos (need to think how) and fix them using FFmpeg.

Do you have any recommendations on how to achieve that or other solutions?

[ojw28]

• I think these files will cause excessive memory consumption whether you concatenate and clip or not. Although concatenation and clipping will exacerbate the issue.
• Identifying and fixing the poorly interleaved videos would be a good thing to do. It's quite likely that just re-packaging the file is sufficient to restore good interleaving, and this should be pretty fast to do. I'm not sure exactly how to do that, however.
• We're following up with Samsung to establish why these poorly interleaved videos are being generated [Internal ref: 69598050].

[MateusZitelli]

Great, I will follow that approach. On any news please let me know. Thanks!

[tonihei]

Closing the issue for now as the underlying cause is tracked with Samsung.

[ojw28]

The change reference above should allow for playback of these streams without hitting an OOM. Samsung may also issue a fix for the poor interleaving, which will make it easier for these files to be streamed.