60 FPS Widevine DASH content stutters on some devices

[matamegger]

Content description

60 FPS Widevine encrypted DASH streams stutter and/or have many dropped frames.
The (per email) provided content shows a live ticker at the bottom, where this stuttering and frame drops can be observed very well.

There are players like the NexPlayer (https://www.nexplayersdk.com/), which shows better performance on the same content and devices, therefore we rule out a weak DRM decryption module or hardware decoders.

We also tried to find the bottleneck in the code base, tried to tweak several parameters, and implemented a decoder version using the asynchronous callback Android provides. Unfortunately without major improvements.

Additionally, on the more powerful/recent devices we observed that frames are provided to the Surface in time (ExoPlayer business logic did not drop the frames) but the stuttering was still visible. When we changed the code so that the renderTimeStamp passed to releaseOutputBuffer is not "reasonably close" to the current system time (which causes the Surface to ignore the timestamp and display the buffer at the earliest feasible time. In this mode the Surface will not drop frames), we did not see any stuttering/frame drops anymore as expected. However, over time video and audio were drifting apart as if video would not be playing at 60 FPS anymore but rather at a lower frame rate. This observation lead us to the assumption that the surface itself maybe cannot keep up with the high frame rate.

Link to test content

Provided per email.

Version of ExoPlayer being used

2.10.4 and at least back to 2.9.6

Device(s) and version(s) of Android being used

Seems unrelated to the Android version.
Seen issues with:

• OnePlus 5T
• OnePlus 5
• Galaxy Tab E (SM-T337W) | 7.1.1, 6.0.1
• Galaxy Tab 3 Lite (SM-T113) | 4.4.4
• Galaxy S8 (Canadian and "regular" version) | 9.0.0
• OnePlus 5 | 9.0.0
• Google Pixel 3 | 9.0.0
• Samsung S6 (SM-G928V) | 7.0.0
• Samsung S7 (SM-G930T1) | 8.0.0
• Galaxy Tab 3 10.1 (GT-P5210)
• Huawei P30 Lite (MAR-LX3A)
• Galaxy Tab 3 (SM-T310)
• Galaxy Tab S3 (SM-T820)
• Galaxy Tab S2 (SM-T810) | 6.0.1,7.0,7.0
• Galaxy Core (SM-G386) | 4.4.2
• Galaxy Tab 4 (SM-T230) | 4.4.2
• MiBox3 (MIBOX3) | 8.0.0
• Galaxy A8 (SM-A530W) | 7.1.1,8.0.0,9

A full bug report captured from the device

Provided per email.

[matamegger]

@ojw28 Have you had already time to check this? If not, can you maybe provide a time frame?

We are happy to support you in further investigation, just let me know.

[ojw28]

@christosts - We should probably treat this as a duplicate of #5796 (or at least investigate both at the same time). The test content and information above will be useful for helping to look at that issue.

[christosts]

Ack

[christosts]

Hi @matamegger

Checked the DASH links provided over email yesterday and they returned 404*. Can you please check the health status of the links, and if they are not alive anymore, maybe provide new ones?

Also, regarding:

When we changed the code so that the renderTimeStamp passed to releaseOutputBuffer is not "reasonably close" to the current system time

Can you please share the change you made so that we can attempt to reproduce the behavior?

Thanks.

[edit]
*I tried to access the links from a browser and got the 404.

[gvidda]

We are experiencing - what probably is - the same issue with widevine drm, but with video tracks that are 720p50fps. They work fine unencrypted.

Iv'e done several tests with one problematic device, Xiamo Mi A3 (with Android One). Changing between L1/L3 does nothing. The only thing that helps is disableing the audio renderer (for some reason)

There's also an old issue from last year about this, but it's closed with no resolution.
#4496

[matamegger]

@christosts
I will check back on the assets, and provide you the renderTimeStamp change asap.

[matamegger]

Regarding the "renderTimeStamp". It's basically something like the following patch shows.
With this change in the MediaCodecVideoRenderer the Surface will (according to the doc) not drop any frames and renders them as fast as possible.

.../library/core/src/main/java/com/google/android/exoplayer2/video/MediaCodecVideoRenderer.java
@@ -912,8 +912,8 @@ public class MediaCodecVideoRenderer extends MediaCodecRenderer {
  protected void renderOutputBufferV21(
       MediaCodec codec, int index, long presentationTimeUs, long releaseTimeNs) {
     maybeNotifyVideoSizeChanged();
     TraceUtil.beginSection("releaseOutputBuffer");
-    codec.releaseOutputBuffer(index, releaseTimeNs);
+    codec.releaseOutputBuffer(index, 0);
     TraceUtil.endSection();
     lastRenderTimeUs = SystemClock.elapsedRealtime() * 1000;
     decoderCounters.renderedOutputBufferCount++;

Note: This is only executed on devices API >= 21

[christosts]

@matamegger

thanks. If you can check the content links (they are returning 404) and provide working links, I will try to repro and come back with findings.

[matamegger]

Mailed you a new set of assets.

[christosts]

Hi Matthias.

An update:

When tested with a Galaxy S7 (SM-G930F), I can see frames being dropped by the player at 60fps at the highest bitrate track. I will work on that to see why the player drops frames.

Apart from that, there is the issue for frames being dropped by the surface and not the player.
May I ask how observable is the problem in this case?

For example, in the test content I can see there's a news reel at the bottom which sometimes has glitches, and also there are glitches on the bottom right window. Is this the problem you observed? On my tests, I can see these glitches but not with the same severity across different devices (Samsung Galaxy S7, Pixel 2/3). In addition, I cannot observe glitches on the main window of the content. Therefore I am not 100% sure if these glitches are in content itself (from encoding) or happening at the device, therefore what you describe as frames being dropped by the surface.

I did not see a difference in the glitches by changing the releaseOutputBuffer() param to 0.

[matamegger]

Yes, depending on the device the "main content" is not glitching, on older or weaker phones the main content also has glitches, but in this case the glitches might be expected due to the processing power of the devices.
The glitches can be observed the best in the news ticker. Which should run smoothly (The glitches are not an effect of encoding or the content).

Depending on the device either exoplayer or the surface (maybe even both) drop frames.
For your tests, you could try to disable the frame dropping of exoplayer, to see the pure surface drops. (Disabling by commenting/deleting the drop code).

[jtreveset]

Hey there. We've also been facing exactly this issue. I agree with @ojw28 in that in order to really fix this problem, multithreaded rendering is required. I noticed that changing the time interval between rendering loops helps a bit. The delay is hardcoded to be 10ms, changing it to 0 definitely helped us.

.../library/core/src/main/java/com/google/android/exoplayer2/ExoPlayerImplInternal.java
@@ -83,7 +83,7 @@ import java.util.concurrent.atomic.AtomicBoolean;
   private static final int MSG_PLAYBACK_PARAMETERS_CHANGED_INTERNAL = 17;

   private static final int PREPARING_SOURCE_INTERVAL_MS = 10;
-  private static final int RENDERING_INTERVAL_MS = 10;
+  private static final int RENDERING_INTERVAL_MS = 0;
   private static final int IDLE_INTERVAL_MS = 1000;

   private final Renderer[] renderers;

With this change, some frames are still dropped depending on the device used though. I'm pretty sure this is not enough to be a production-ready patch, but maybe an improvement could be done by dynamically adjusting this value until proper multithreaded rendering is implemented.

[christosts]

Just an update:

As mentioned in my previous comment, when tested the 60fps 5.7Mbits widevine stream on a Galaxy S7, I can see ExoPlayer (not the surface) is dropping frames. I am working to address that. The fix will probably be inline with adding threads to handle rendering.

With regards to the 'glitches' in the news ticker. I am not sure it is the surface or I am not seeing the same symptoms. In detail, I played the 60fps track offline on multiple devices and multiple players (and asked more people to view!) and saw glitching on the news ticker on all setups.

• Samsung S7: ExoPlayer and VLC
• Linux: Media player and VLC
• Mac: Quicktime Player and VLC

In all devices/players, the ticker did not run 100% smoothly, which I interpret there is some glitching encoded. But I can't tell of course if the glitching that I observe is the same as the glitch reported originally.

@matamegger would you mind trying something similar, that is try to play offline the 60fps/5.7mbit content and see if you can see similar glitch along devices/players?

[matamegger]

There are no glitches in the stream itself. We have validated this many times.
Chrome reports a decoder underflow Quick chrome://media-internals

Also by observing the glitches closer and multiple times, one can see that they do not happen on the same position in the content.

[fredrikeneroth]

Hi. We are facing similar problems. Is there any updates to this issue?

In our case it's 720p50fps which is causing stuttering and a lot of dropped frames on some devices. Most problematic on Google Pixel (Gen1) running Android 10, but also on a few other devices.

We found a (to us) weird thing though which I would love to get help understanding. If we keep the main thread busy by either tapping quickly on the screen to show/hide player controls or by pressing the volume rocker to show the volume slider video becomes smooth with no dropped frames. You can see the behavior in this screen capture:
https://www.dropbox.com/s/039slthc6gbzzyo/sailfishQP1A.191005.007.A1eneroth01302020120714.mp4?dl=0

Video is blacked out, but you can see the db count increasing quickly until I start toggling the player controls during which db count is not increasing and video is playing fine.

We could also reduce the dropped frames by starting an endless Runnable no-op loop on the main thread while video was playing. But this still causes a few frame drops, so not a production stable workaround.

Reproduced this issue on the ExoPlayer demo app, running version 2.11.1.

[christosts]

Hi. We are facing similar problems. Is there any updates to this issue?

We have been working on it. It is still in progress but let me share an update.

Note: the following comment points to experimental code that is available on the dev-2 branch but it is still under testing/evaluation. Therefore we cannot guarantee at the moment the feature's stability or that the API will not change in a future release.

As a first step, we are experimenting with operating Android's MediaCodec in asynchronous mode. We have seen (in some devices) this alone improves the MediaCodec's performance. In the dev2 branch, the DefaultRenderersFactory currently has the method experimental_setMediaCodecOperationMode which sets ExoPlayer's MediaCodec operation mode. At the moment, there are four modes available: OPERATION_MODE_SYNCHRONOUS is the existing behavior and there are 3 additional modes which basically set the MediaCodec in asynchronous mode but differ in the threading model or implementation details (documented here).

For high frame rate video, we expect OPERATION_MODE_ASYNCHRONOUS_DEDICATED_THREAD and OPERATION_MODE_ASYNCHRONOUS_DEDICATED_THREAD_MULTI_LOCK to provide better performance because they introduce additional threads for the MediaCodec callbacks. That said, we are evaluating their impact on other aspects such as battery use and performance on devices on the lower end w.r.t. CPU.

We will be happy if you can experiment with the feature and provide feedback, but let me repeat that this feature is still under internal evaluation and the API is not final. An easy way would be to checkout the dev2 branch, set the MediaCdec operation mode in the demo app and play your test video.

On top of this work, we plan to add additional parallelization to offload the player's internal thread. When this feature is available on the dev2 branch, we will announce the experimental features (possibly in our blog) and invite the community to experiment and provide feedback.

[christosts]

@fredrikeneroth can share a link to the 720p50fps video to test it on a Pixel? If you cannot share it in public, will you be able to email to dev.exoplayer@gmail.com with subject "Issue #6348"?

Thanks

[fredrikeneroth]

Okay, that sounds promising @christosts. Will try it out. I've sent the test content to your email. Let me know if it doesn't work or if you need any additional content.

[christosts]

Nice to to see improvements! We are still evaluating those modes internally, and we haven't had the opportunity (and backing data) to plan additional steps yet.

Since you have a device, may I ask you to try the following? Can you please try increasing the hardcoded earlyUs threshold in the MediaCodecVideoRender? Try increasing it to 100000 or even 200000?

What does it do? This threshold controls how early ExoPlayer sends a decoded video frame to the MediaCodec to be rendered later on screen, in microseconds. Right now, ExoPlayer submits video frames up to 50ms ahead of time. Increasing the value to 100ms or 200ms will allow sending the decoded video frames sooner to the MediaCodec (they will still be displayed on time, MediaCodec will ensure that). By increasing the threshold, ExoPlayer will be placing a few more video frames in the MediaCodec's rendering queue waiting to be displayed, hence making the player somewhat more robust to sudden delays that might occur in other parts of the player's processing loop.

The caveat: decoded video frames that are passed to the MediaCodec for release, will definitely be displayed on screen. That is, if the user pauses, but the player has released video frames up to 200ms ahead, these will be rendered on screen first, and then playback will pause.

If you try this, please let us know if it mitigated the remaining dropped frames on the Sony device, but also consider pausing playback to see if UX is impacted.

Note that this is an early thought and not an agreed idea that we have planned to implement. As said above, we are still collecting data on the existing experimental modes.

[Stonos]

I've noticed an improvement with OPERATION_MODE_ASYNCHRONOUS_DEDICATED_THREAD_MULTI_LOCK_ASYNCHRONOUS_QUEUEING on 1080p 50fps stream with a Xiaomi Redmi Note 8 Pro. However, there is still noticeable stuttering.

I also tried increasing the earlyUs threshold mentioned above to 200000, but unfortunately I didn't notice any improvement.

One interesting thing that I've noticed is that if I repeatedly put my fingers on the display then the issue mostly goes away! That's also visible from the vfpo stat: I managed to increase it from -6000 to +20000 just by "tickling" my phone's display for 2-3 minutes. As soon as I stop doing that, the video starts to stutter again.

My guess is that this happens because Android by default boosts the CPU frequency as soon as you touch the display in order to make interaction smoother. That boost seems to be enough to get the video to play smoothly.

Here is a chart (obtained by running while true; do cat "/sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq"; sleep 1; done) which shows that the CPU frequency never drops below 1.5 GHz while the display is being tickled:

Unfortunately, my device does not support setSustainedPerformanceMode() to test with a fixed CPU frequency.

[Stonos]

Update: CPU (or maybe GPU?) frequency definitely seems to play a role here.

Since it's known that MediaTek cheats on benchmarks, and my device has a MediaTek SoC, I decided to take advantage of this to increase the CPU frequency (since my device is not rooted).

I changed my applicationId to one of a benchmark app, and lo and behold, the system pinned the CPU to 2.0 GHz while my app was running!

With mediaCodecOperation set to ASYNCHRONOUS_DEDICATED_THREAD_MULTI_LOCK_ASYNCHRONOUS_QUEUEING, and the CPU running @ 2.0 GHz, I didn't notice any stuttering.

I did notice stuttering with OPERATION_MODE_SYNCHRONOUS, which surprised me a bit, but that might had been because I had the earlyUs modification active.

So what does this tell us? Honestly, I don't know if there is much that ExoPlayer can do here. It seems to me that the device's CPU governor is over-conservative, and doesn't scale up enough to meet the demands of 1080p50 video playback.

[tibor-leo-safar-accedo]

Hi ExoPlayer team,
A question more to @christosts about the roadmap and future of the dev-2 branch and the experimental_setMediaCodecOperationMode() functionality.

The issue we have with Live Dash HD streams with 50fps, is kind a burning one.
We've found with the experimental feature the payback to be much more stable with acceptable amount of dropped frames.

Can we have a feedback if the experimental feature is stable enough to be added to any of the next stable versions? If so, is there a timeline for it?
In case, in you opinion, this is a no go for releasing to stable branch is there any initiative or idea what we could pick in order to have a smooth playback of these kind of streams?
Really appreciate your feedback.

[ojw28]

We're planning to do a release pretty soon (i.e., within the next couple of weeks) to pick up changes related to #6429, but it will be based on 2.11.X rather than tip-of-tree dev-v2, and so by default the changes here will not be picked up.

It seems reasonable that we would try and also pick up the changes for this issue at the same time, but whether that's practical will depend on how nicely they cherry-pick onto 2.11.X, which we wont know until we try. So, we'll see if it's possible, but we can't promise anything. If it's not possible then the changes will need to wait until 2.12.0, for which we don't have a good time estimate yet.

[perholgersson]

@ojw28 any update on this?

[paceraudio]

@RichardDapice I found forcing certain devices to use L3 worked as well, however this used to be easy by using drmSessionManager.setPropertyString("securityLevel", "L3"). I am in the process of updating Exo to 2.11.3, and this method is gone, and I don't see any straightforward way to force L3. @ojw28 It might be worth making this available again in a straightforward way?

[christosts]

@ojw28 any update on this?

We plan to include this API as experimental (therefore subject to change) in 2.12. We cannot promise a date yet though.

[christianfrank84]

Are there any new updates to this topic?

I've been experimenting now with the dev-v2 branch for a while and as the Modes 2 and 4 certainly help with performance. It for sure doesn't make the stream run flawlessly.

I've been testing Dash/Widevine Streams mainly with a Samsung Galaxy Tab 2 and with either Mode 2 or 4, I'd still need to cancel out the highest abr profile to achieve a somewhat acceptable result.

The best result i achieved on the Tab S2 is by filtering the max Bitrate to 2048kbps with Mode 2. This gave me an average vfpo of 40ms with occasional dropped frames every now and then.

I've also been testing with a Pixel 3a, and tho this device is not as problematic with stuttering playback, Mode 4 always result in a DecoderFailure Message.

I'd appreciate if someone could give me an update on this topic, if it is even worked on currently and i there is anything new to expect anytime soon.

Thank you very much.

[christosts]

We are targeting to release v2.12 soon (within September) and these APIs will be available for use, but as experimental. The API will allow enabling modes 2 and 4 separately for audio and video. If you upgrade to v2.12, you can opt-in to these modes in your app. Our internal evaluation shows that the overall best performance is achieved with mode 4 enabled on both audio and video.

Our future plan is to have mode 4 enabled by default for audio and video in a future release. However, we do see some sporadic errors that we need to analyse first before we enable this by default.

The best result i achieved on the Tab S2 is by filtering the max Bitrate to 2048kbps with Mode 2. This gave me an average vfpo of 40ms with occasional dropped frames every now and then.

It's likely that is hitting the device limits and there's not much room for improvement inside ExoPlayer.

I've also been testing with a Pixel 3a, and tho this device is not as problematic with stuttering playback, Mode 4 always result in a DecoderFailure Message.

Can you please help us to investigate this further? A bugreport obtained from the device when the error happens and a link to test content would be great. Can you email them to dev.exoplayer@gmail.com with subject "Issue #6348"?

[mahdi24rajabi]

Hi @christosts. I'm trying to work on this issue, I've found out if I increase earlyUs more than 500ms up to 1s, the issue on Samsung TAB 3 T310 will get resolved. But I think it may negatively affect the streaming sometimes with 500ms latency, based on assigned value. Since I'm new in Exoplayer codebase, my question is that what sort of side-effects I may supposed to face if I update earlyUs in this way?

[christosts]

@mahdi24rajabi what is the Android version of your device? The experimental APIs introduced are available for Android API 23+ and modifying the earlyUs should be effective on API 21+.

[mahdi24rajabi]

@christosts It's pretty old device and API version is 19.

[christosts]

@mahdi24rajabi the earlyUs here should only be affected on API 21+.

Can you please point me the exact changes you made? If it's just one line, a link to the source code will do explaining what values you are setting, otherwise can you attach a patch?

[mahdi24rajabi]

@christosts Just changed following lines in MediaCodecVideoRenderer.java:

private static boolean isBufferLate(long earlyUs) {
return earlyUs < -1000000;
}

private static boolean isBufferVeryLate(long earlyUs) {
return earlyUs < -1500000;
}

I've prevented frame dropping in short time in this way. It's smooth now, but not sure whether does it have any side effects or not.

[christosts]

There can be some interesting timing interactions between audio and video if the video renderer keeps trying to present frames that should have been dropped otherwise. For example, I've seen the side effect where a bunch of delayed video frames are displayed in higher rate, so the user sees video playing faster for a short duration (1 second). However, I have not investigated these changes further, so I cannot offer more information.

I would be interesting to hear your findings, in case you notice a certain kind of side effects.

[mahdi24rajabi]

@christosts I've found something that improves the situation:

Instead of sending rendererPositionUs and rendererPositionElapsedRealtimeUs directly from doSomeWork() to render() method, I tried to send mediaClock directly and get the timeline position when I need it. I've found that the process from doSomeWork until the moment we check earlyUs value takes between 10 to 20 ms. Moreover, by doing so we can remove lots of adjustment that we do currently on the time value to find appropriate earlyUs.

By doing so, I reduced worst value for earlyUs from more than 1 second to less that 600ms. By monitoring earlyUs value of dropped frames, I found out that the largest segment, after code update, belongs to interval from 30ms to 250ms. So I decided to change the checking from -30ms to -250ms.

Previously I got smooth rendering at 1s. Now I get it at 250ms.

I have one question. The code inside the doSomeWork tries to launch all renders in sequential way. Why don't we launch those in parallel way?

[christosts]

I have one question. The code inside the doSomeWork tries to launch all renders in sequential way. Why don't we launch those in parallel way?

The various improvements mentioned in this thread are actually offloading some of the work done inside renderers in other threads so that the actual time that each Renderer.render() takes is minimal. Unfortunately, we are seeing some errors raised from the device when multiple threads are interacting with MediaCodec instances, especially on older Android versions. Therefore, accessing media components from multiple threads is not 100% safe and it takes us time until we are confident we can enable any form of parallelisation by default.

Triggering renderers fully in parallel is in our radar and tracked in #5796, but our plans for the immediate future is to stabilize the performance improvements mentioned in this thread and enabling them by default.

[Rafayelpoghosyan-wmg]

Hi All, I have one issue, In my case, I need to show the video player rotated for example 45 degrees, for rotation I found that needed to use app:surface_type="texture_view so for using TextureView there is a low performance of the video. Can you please say how can I solve the performance issue? I checked the comments and understood that the MediaCodecOperation configuration will solve my issue. How can I configure ExoPlayer or this is not released yet?

[christosts]

@Rafayel93 the optimizations described in this thread were made with respect to the performance of Android's MediaCodec and I'm not sure if they can address your use case. Nevertheless, if you want to try them, they are still in experimental mode in v2.12.2.

If you use the player's default renderers, you need to create a DefaultRenderersFactory instance and call experimentalSetAudioMediaCodecOperationMode() and experimentalSetVideoMediaCodecOperationMode() passing MediaCodecRenderer.OPERATION_MODE_ASYNCHRONOUS_DEDICATED_THREAD_ASYNCHRONOUS_QUEUEING on both methods, and then pass the DefaultRenderersFactory on the player builder you are using.

[zahmedFDL]

Hi All,

I tried the experimental mode on Galaxy S6 (European version) and it worked well. We got many issues related to the Exynos chip here.

Firstly, I have used the branch v2.12.2

Then I have modified MediaCodecVideoRenderer as below as it was recommended:

//codec.releaseOutputBuffer(index, releaseTimeNs);
codec.releaseOutputBuffer(index, 0);

And the MainActivity as below:

return new DefaultRenderersFactory(context.getApplicationContext()) .experimentalSetAudioMediaCodecOperationMode(MediaCodecRenderer.OPERATION_MODE_ASYNCHRONOUS_DEDICATED_THREAD_ASYNCHRONOUS_QUEUEING) .experimentalSetVideoMediaCodecOperationMode(MediaCodecRenderer.OPERATION_MODE_ASYNCHRONOUS_DEDICATED_THREAD_ASYNCHRONOUS_QUEUEING) .setExtensionRendererMode(extensionRendererMode);

Previously we were not able to play a LIVE DASH widevine content smoothly on this layer :
width="854" height="480" sar="1:1" frameRate="25/1" bandwidth="1499968" codecs="avc1.42C015

And we were limiting the track selector to max bitrate 1500000 because it got worse for the layers up.

Now we are able to play this level smoothly but also the higher levels:
width="1280" height="720" sar="1:1" frameRate="25/1" bandwidth="9499968" codecs="avc1.64001F"

Finally, the MediaCodecOperation configuration solves my issue.

Let me know when you will release the change.

Zaki

[christosts]

We pushed commit 7a2c7c3 that promotes the experimental asynchronous queueing feature to be used by default. This will be included on the next release (2.16.0) which is expected to be out during November 2021.

The player will use this feature by default on devices with Android 12+, but it will not enable it automatically for devices running previous Android versions. Asynchronous queueing largely addressed stuttering and dropped frames observed on some devices when playing high frame Widevine-encrypted content. However, we observed some devices can be fragile to the additional threads introduced, especially on older platforms (e.g., prior to Android 8). The number of issues was very low, yet is prevented us from widely enabling the feature on all platforms.

The commit adds APIs to enable/disable the feature, therefore you can manually enable it on a device running an earlier Android version.

I will close this issue. If you observe any issues with the new feature, please file a new ticket in order to focus the conversation on the specific feature. Thanks.