Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SurfaceView inside Compose AndroidView often shown stretched/cropped on API 34 #1237

Closed
1 task
MykolaKosianchuk opened this issue Apr 2, 2024 · 40 comments
Closed
1 task
Assignees

Comments

@MykolaKosianchuk
Copy link

MykolaKosianchuk commented Apr 2, 2024

Version

Media3 main branch

More version details

1.3.0, 1.2.1, 1.1.1

Devices that reproduce the issue

Pixel running on Android 14 with updated to 5 March 2024 (important)

Devices that do not reproduce the issue

Pixel running on Android 14 without updated to 5 March 2024

Reproducible in the demo app?

Not tested

Reproduction steps

Here is code that could help to reproduce.

class MainActivity : ComponentActivity() {
    @OptIn(UnstableApi::class)
    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContent {
            val context = LocalContext.current

            val exoPlayer = remember { ExoPlayer.Builder(context).build() }

            DisposableEffect(Unit) {
                onDispose {
                    exoPlayer.release()
                }
            }

            Box(
                modifier = Modifier
                    .fillMaxWidth()
                    .fillMaxHeight(0.5f) //important to have
//                    .height(500.dp) - also reproducible
                    .background(Color.Black),
                contentAlignment = Alignment.Center
            ) {
                AndroidView(
                    factory = { ctx ->
                        PlayerView(ctx).apply {
                            useController = false
                            resizeMode = AspectRatioFrameLayout.RESIZE_MODE_ZOOM
                        }
                    },
                    update = {
                        it.player = exoPlayer
                        exoPlayer.setMediaItem(
                            MediaItem.fromUri("https://commondatastorage.googleapis.com/gtv-videos-bucket/sample/BigBuckBunny.mp4")
                        )
                        exoPlayer.playWhenReady = true
                        exoPlayer.prepare()
                    },
                )
            }
        }
    }
}

Expected result

Media should be visible correctly

Actual result

Media shows with a wrong aspect ratio

Media

Here is how it looks

Screenshot 2024-04-02 at 15 52 20

Bug Report

@phcannesson
Copy link
Contributor

+1 this
We've spent days trying to understand issues we have with Android 14 devices recently.
Going back to our previous app versions doesn't fix anything, so we came up with the same conclusion as @MykolaKosianchuk , it's probably related to a recent Android patch.

@MykolaKosianchuk
Copy link
Author

@phcannesson Yes, we found this issue and tried to reproduce it on the device without the last patch but we can't. However, once we updated the phone, the issue appeared.

@harry248
Copy link

harry248 commented Apr 3, 2024

We have the same problem. We had already observed this in the QPR but unfortunately did not create an issue. Interestingly, we "only" have the problem if we use the AndroidView (which creates the PlayerView) in an if block, i.e. the composable is not called immediately.

@icbaker
Copy link
Collaborator

icbaker commented Apr 8, 2024

Reproducible in the demo app?

Not tested

Please can you test this? It would help us to debug if we know whether it can be reproduced in the demo app.


Interestingly, we "only" have the problem if we use the AndroidView (which creates the PlayerView) in an if block, i.e. the composable is not called immediately.

Is this true for everyone else that has commented on this bug? Is everyone else experiencing this only in Compose contexts, or also in conventional views? This looks like it might be a duplicate of #1123 but it's interesting to hear that the behaviour changes between Android 14 before the March 5th update and after it.

@phcannesson
Copy link
Contributor

Reproducible in the demo app?

Not tested

Please can you test this? It would help us to debug if we know whether it can be reproduced in the demo app.

Interestingly, we "only" have the problem if we use the AndroidView (which creates the PlayerView) in an if block, i.e. the composable is not called immediately.

Is this true for everyone else that has commented on this bug? Is everyone else experiencing this only in Compose contexts, or also in conventional views? This looks like it might be a duplicate of #1123 but it's interesting to hear that the behaviour changes between Android 14 before the March 5th update and after it.

On our side we're using Compose.
We did not test with conventional views.

@promanowicz
Copy link

I have faced exact same issue, except I was using resizeMode with fixed height.
I addition I have noticed that the video gets right aspect ratio after I switched orientation to landscape and then back to portrait.

@MykolaKosianchuk
Copy link
Author

@icbaker It is not reproducible in the demo app
Looks like it is related to the compose

@speedclaud
Copy link

speedclaud commented Apr 11, 2024

Hi @icbaker , I was also not able to reproduce the issue on your demo app on Pixel 7, Android 14, march 5th update, but we have the issue in our compose app using AndroidView. The issue is present in this sample app that uses compose, (press on the "Exo PlayerView" button):

compose-media.zip

This is what it looks like:

Screenshot_20240411_121959

@phcannesson
Copy link
Contributor

phcannesson commented Apr 11, 2024

On our side, after some testing, we've detected that it seems to be related to the way the frame layout gets notified of changes in the player.
We've successfully "fixed" the issue by adding some delay before displaying the AndroidView.
That apparently allows the player to be fully ready when the view is loaded.

@speedclaud
Copy link

On our side, after some testing, we've detected that it seems to be related to the way the frame layout gets notified of changes in the player. We've successfully "fixed" the issue by adding some delay before displaying the AndroidView. That apparently allows the player to be fully ready when the view is loaded.

I've tried adding a delay to the sample app above (with a boolean mutable state and a LaunchedEffect), but the stretch is still present when It shows.

@speedclaud
Copy link

speedclaud commented Apr 11, 2024

I found a "fix", the stretch is gone when using app:surface_type="texture_view". So I assume there's something wrong with app:surface_type="surface_view" inside AndroidView on the latest android?

Is it true that texture_view drains more battery?

@icbaker
Copy link
Collaborator

icbaker commented Apr 11, 2024

Thanks for the repro app in #1237 (comment). I ran this on a Pixel 7a running lynx-userdebug AP1A.240405.002 and observed the problematic behaviour shown in your screenshot nearly every time (the first time i launched the app it seemed to work OK, but the subsequent ~10 times it shows the problem - even after swiping the app away from recents, and clearing app data).

I added an EventLogger (https://developer.android.com/media/media3/exoplayer/debug-logging) and grabbed the logcat filtered to EventLogger of a problematic repro.

I then flashed my device back to before the March feature release, to lynx-userdebug UQ1A.240205.002 and installed the app. The first time I launched the video it played fine. I grabbed the same filtered logcat, and diffed it against the problematic behaviour. Then subsequent launches of the video (without dismissing the app) showed the problem (and I grabbed the logcat). Swiping the app away and re-launching it seems to allow the playback to work correctly once, then subsequent launches are problematic.

I observed the following in logcat right at the start of playback in the problematic case on AP1A.240405.002:

timeline [eventTime=0.02, mediaPos=0.00, window=0, periodCount=1, windowCount=1, reason=PLAYLIST_CHANGED
  period [?]
  window [?, seekable=false, dynamic=true]
]
mediaItem [eventTime=0.02, mediaPos=0.00, window=0, reason=PLAYLIST_CHANGED]
state [eventTime=0.02, mediaPos=0.00, window=0, BUFFERING]
surfaceSize [eventTime=0.06, mediaPos=0.00, window=0, 0, 0]
surfaceSize [eventTime=0.08, mediaPos=0.00, window=0, 1080, 1080]
timeline [eventTime=0.09, mediaPos=0.00, window=0, periodCount=1, windowCount=1, reason=PLAYLIST_CHANGED
  period [?]
  window [?, seekable=false, dynamic=true]
]
positionDiscontinuity [eventTime=0.10, mediaPos=0.00, window=0, reason=REMOVE, PositionInfo:old [mediaItem=0, period=0, pos=0], PositionInfo:new [mediaItem=0, period=0, pos=0]]
mediaItem [eventTime=0.10, mediaPos=0.00, window=0, reason=PLAYLIST_CHANGED]
loading [eventTime=0.11, mediaPos=0.00, window=0, true]
timeline [eventTime=0.11, mediaPos=0.00, window=0, period=0, periodCount=1, windowCount=1, reason=SOURCE_UPDATE
  period [?]
  window [?, seekable=false, dynamic=false]
]
timeline [eventTime=1.15, mediaPos=0.00, window=0, period=0, periodCount=1, windowCount=1, reason=SOURCE_UPDATE
  period [634.40]
  window [634.40, seekable=true, dynamic=false]
]
videoEnabled [eventTime=1.16, mediaPos=0.00, window=0, period=0]
tracks [eventTime=1.16, mediaPos=0.00, window=0, period=0
  group [
    [X] Track:0, id=1, mimeType=video/av01, res=854x480, fps=29.999998, supported=YES
  ]
]
downstreamFormat [eventTime=1.16, mediaPos=0.00, window=0, period=0, id=1, mimeType=video/av01, res=854x480, fps=29.999998]
videoDecoderInitialized [eventTime=1.20, mediaPos=0.00, window=0, period=0, c2.google.av1.decoder]
videoInputFormat [eventTime=1.21, mediaPos=0.00, window=0, period=0, id=1, mimeType=video/av01, res=854x480, fps=29.999998]
videoSize [eventTime=1.33, mediaPos=0.00, window=0, period=0, 854, 480]
renderedFirstFrame [eventTime=1.34, mediaPos=0.00, window=0, period=0, Surface(name=null)/@0xe4d0ed2]
surfaceSize [eventTime=1.35, mediaPos=0.00, window=0, period=0, 1080, 607]
state [eventTime=1.36, mediaPos=0.02, window=0, period=0, READY]
isPlaying [eventTime=1.36, mediaPos=0.02, window=0, period=0, true]
loading [eventTime=1.91, mediaPos=0.56, window=0, period=0, false]
loading [eventTime=3.42, mediaPos=2.08, window=0, period=0, true]
loading [eventTime=3.48, mediaPos=2.14, window=0, period=0, false]

The same part of logcat in the working case on UQ1A.240205.002:

timeline [eventTime=0.03, mediaPos=0.00, window=0, periodCount=1, windowCount=1, reason=PLAYLIST_CHANGED
  period [?]
  window [?, seekable=false, dynamic=true]
]
mediaItem [eventTime=0.03, mediaPos=0.00, window=0, reason=PLAYLIST_CHANGED]
state [eventTime=0.03, mediaPos=0.00, window=0, BUFFERING]
surfaceSize [eventTime=0.07, mediaPos=0.00, window=0, 1080, 1080]
timeline [eventTime=0.09, mediaPos=0.00, window=0, periodCount=1, windowCount=1, reason=PLAYLIST_CHANGED
  period [?]
  window [?, seekable=false, dynamic=true]
]
positionDiscontinuity [eventTime=0.09, mediaPos=0.00, window=0, reason=REMOVE, PositionInfo:old [mediaItem=0, period=0, pos=0], PositionInfo:new [mediaItem=0, period=0, pos=0]]
mediaItem [eventTime=0.09, mediaPos=0.00, window=0, reason=PLAYLIST_CHANGED]
loading [eventTime=0.11, mediaPos=0.00, window=0, true]
timeline [eventTime=0.11, mediaPos=0.00, window=0, period=0, periodCount=1, windowCount=1, reason=SOURCE_UPDATE
  period [?]
  window [?, seekable=false, dynamic=false]
]
timeline [eventTime=1.26, mediaPos=0.00, window=0, period=0, periodCount=1, windowCount=1, reason=SOURCE_UPDATE
  period [634.40]
  window [634.40, seekable=true, dynamic=false]
]
videoEnabled [eventTime=1.45, mediaPos=0.00, window=0, period=0]
tracks [eventTime=1.46, mediaPos=0.00, window=0, period=0
  group [
    [X] Track:0, id=1, mimeType=video/av01, res=854x480, fps=29.999998, supported=YES
  ]
]
downstreamFormat [eventTime=1.46, mediaPos=0.00, window=0, period=0, id=1, mimeType=video/av01, res=854x480, fps=29.999998]
videoDecoderInitialized [eventTime=1.49, mediaPos=0.00, window=0, period=0, c2.google.av1.decoder]
videoInputFormat [eventTime=1.49, mediaPos=0.00, window=0, period=0, id=1, mimeType=video/av01, res=854x480, fps=29.999998]
videoSize [eventTime=1.52, mediaPos=0.00, window=0, period=0, 854, 480]
surfaceSize [eventTime=1.52, mediaPos=0.00, window=0, period=0, 1080, 607]
renderedFirstFrame [eventTime=1.53, mediaPos=0.00, window=0, period=0, Surface(name=null)/@0xb940abd]
state [eventTime=1.53, mediaPos=0.00, window=0, period=0, READY]
isPlaying [eventTime=1.53, mediaPos=0.00, window=0, period=0, true]
loading [eventTime=2.29, mediaPos=0.76, window=0, period=0, false]
loading [eventTime=3.61, mediaPos=2.08, window=0, period=0, true]
loading [eventTime=3.67, mediaPos=2.15, window=0, period=0, false]

And finally, the problematic case on UQ1A.240205.002:

timeline [eventTime=0.02, mediaPos=0.00, window=0, periodCount=1, windowCount=1, reason=PLAYLIST_CHANGED
  period [?]
  window [?, seekable=false, dynamic=true]
]
mediaItem [eventTime=0.02, mediaPos=0.00, window=0, reason=PLAYLIST_CHANGED]
state [eventTime=0.02, mediaPos=0.00, window=0, BUFFERING]
surfaceSize [eventTime=0.06, mediaPos=0.00, window=0, 0, 0]
surfaceSize [eventTime=0.07, mediaPos=0.00, window=0, 1080, 1080]
loading [eventTime=0.08, mediaPos=0.00, window=0, period=0, true]
timeline [eventTime=0.08, mediaPos=0.00, window=0, period=0, periodCount=1, windowCount=1, reason=SOURCE_UPDATE
  period [?]
  window [?, seekable=false, dynamic=false]
]
timeline [eventTime=0.08, mediaPos=0.00, window=0, periodCount=1, windowCount=1, reason=PLAYLIST_CHANGED
  period [?]
  window [?, seekable=false, dynamic=true]
]
positionDiscontinuity [eventTime=0.08, mediaPos=0.00, window=0, reason=REMOVE, PositionInfo:old [mediaItem=0, period=0, pos=0], PositionInfo:new [mediaItem=0, period=0, pos=0]]
mediaItem [eventTime=0.08, mediaPos=0.00, window=0, reason=PLAYLIST_CHANGED]
timeline [eventTime=0.09, mediaPos=0.00, window=0, period=0, periodCount=1, windowCount=1, reason=SOURCE_UPDATE
  period [?]
  window [?, seekable=false, dynamic=false]
]
timeline [eventTime=1.30, mediaPos=0.00, window=0, period=0, periodCount=1, windowCount=1, reason=SOURCE_UPDATE
  period [634.40]
  window [634.40, seekable=true, dynamic=false]
]
videoEnabled [eventTime=1.31, mediaPos=0.00, window=0, period=0]
tracks [eventTime=1.31, mediaPos=0.00, window=0, period=0
  group [
    [X] Track:0, id=1, mimeType=video/av01, res=854x480, fps=29.999998, supported=YES
  ]
]
downstreamFormat [eventTime=1.31, mediaPos=0.00, window=0, period=0, id=1, mimeType=video/av01, res=854x480, fps=29.999998]
videoDecoderInitialized [eventTime=1.36, mediaPos=0.00, window=0, period=0, c2.google.av1.decoder]
videoInputFormat [eventTime=1.36, mediaPos=0.00, window=0, period=0, id=1, mimeType=video/av01, res=854x480, fps=29.999998]
videoSize [eventTime=1.51, mediaPos=0.00, window=0, period=0, 854, 480]
renderedFirstFrame [eventTime=1.51, mediaPos=0.00, window=0, period=0, Surface(name=null)/@0x4a7b0d3]
surfaceSize [eventTime=1.53, mediaPos=0.00, window=0, period=0, 1080, 607]
state [eventTime=1.53, mediaPos=0.02, window=0, period=0, READY]
isPlaying [eventTime=1.54, mediaPos=0.02, window=0, period=0, true]
loading [eventTime=2.21, mediaPos=0.70, window=0, period=0, false]
loading [eventTime=3.60, mediaPos=2.08, window=0, period=0, true]
loading [eventTime=3.67, mediaPos=2.15, window=0, period=0, false]

There are two key differences between the working and problematic cases:

  1. The problematic cases both have an additional surfaceSize [eventTime=0.06, mediaPos=0.00, window=0, 0, 0] before surfaceSize [eventTime=0.07, mediaPos=0.00, window=0, 1080, 1080]
  2. The problematic cases both see renderedFirstFrame before surfaceSize [eventTime=1.52, mediaPos=0.00, window=0, period=0, 1080, 607], whereas in the working case the surfaceSize arrives before renderedFirstFrame.

@phcannesson
Copy link
Contributor

I found a "fix", the stretch is gone when using app:surface_type="texture_view". So I assume there's something wrong with app:surface_type="surface_view" inside AndroidView on the latest android?

Is it true that texture_view drains more battery?

I can confirm that it seems to fix.

I've created a public repo to reproduce and play with the configuration :
https://github.com/phcannesson/AspectFrameLayoutDemo

The player with app:surface_type="texture_view" works well.

@phcannesson
Copy link
Contributor

While digging in AOSP code, I've noticed this change :
https://cs.android.com/android/_/android/platform/frameworks/base/+/1b152e712159c0b82831a7567ce2f3c49cdc11bf
It apparently modified how SurfaceView is clipped.

It seemed to have been released in Android 14 QPR2.

It's a wild guess, but I thought it was worth mentioning.

@icbaker
Copy link
Collaborator

icbaker commented Apr 12, 2024

Trying the same video in the main ExoPlayer demo app on the same device + build (lynx-userdebug UQ1A.240205.002), I see the 'problematic sequence' in logcat (surfaceSize(0,0) near the start, and then renderedFirstFrame before surfaceSize), but not the problematic symptoms.

This is perhaps obvious, but it adds more weight to the theory that the problem is related to how AspectRatioFrameLayout (or perhaps SurfaceView) behaves specifically in a Compose context.

Full logcat:

playWhenReady [eventTime=0.00, mediaPos=0.00, window=0, true, USER_REQUEST]
surfaceSize [eventTime=0.00, mediaPos=0.00, window=0, 0, 0]
timeline [eventTime=0.00, mediaPos=0.00, window=0, periodCount=1, windowCount=1, reason=PLAYLIST_CHANGED
  period [?]
  window [?, seekable=false, dynamic=true]
]
mediaItem [eventTime=0.01, mediaPos=0.00, window=0, reason=PLAYLIST_CHANGED]
state [eventTime=0.01, mediaPos=0.00, window=0, BUFFERING]
surfaceSize [eventTime=0.04, mediaPos=0.00, window=0, 1080, 2219]
loading [eventTime=0.04, mediaPos=0.00, window=0, period=0, true]
timeline [eventTime=0.04, mediaPos=0.00, window=0, period=0, periodCount=1, windowCount=1, reason=SOURCE_UPDATE
  period [?]
  window [?, seekable=false, dynamic=false]
]
timeline [eventTime=0.77, mediaPos=0.00, window=0, period=0, periodCount=1, windowCount=1, reason=SOURCE_UPDATE
  period [634.40]
  window [634.40, seekable=true, dynamic=false]
]
videoEnabled [eventTime=0.79, mediaPos=0.00, window=0, period=0]
tracks [eventTime=0.79, mediaPos=0.00, window=0, period=0
  group [
    [X] Track:0, id=1, mimeType=video/av01, res=854x480, color=BT709/Limited range/SDR SMPTE 170M/8/8, fps=29.999998, supported=YES
  ]
  Metadata [
    TSSE: description=null: values=[Google]
    Mp4Timestamp: creation time=0, modification time=0, timescale=1000
  ]
]
downstreamFormat [eventTime=0.81, mediaPos=0.00, window=0, period=0, id=1, mimeType=video/av01, res=854x480, color=BT709/Limited range/SDR SMPTE 170M/8/8, fps=29.999998]
videoDecoderInitialized [eventTime=0.84, mediaPos=0.00, window=0, period=0, c2.google.av1.decoder]
videoInputFormat [eventTime=0.84, mediaPos=0.00, window=0, period=0, id=1, mimeType=video/av01, res=854x480, color=BT709/Limited range/SDR SMPTE 170M/8/8, fps=29.999998]
videoSize [eventTime=1.07, mediaPos=0.00, window=0, period=0, 854, 480]
renderedFirstFrame [eventTime=1.07, mediaPos=0.00, window=0, period=0, Surface(name=null)/@0x12990bc]
surfaceSize [eventTime=1.09, mediaPos=0.00, window=0, period=0, 1080, 607]
state [eventTime=1.09, mediaPos=0.01, window=0, period=0, READY]
isPlaying [eventTime=1.10, mediaPos=0.02, window=0, period=0, true]
loading [eventTime=1.78, mediaPos=0.70, window=0, period=0, false]
loading [eventTime=3.17, mediaPos=2.09, window=0, period=0, true]

@harry248
Copy link

While digging in AOSP code, I've noticed this change : https://cs.android.com/android/_/android/platform/frameworks/base/+/1b152e712159c0b82831a7567ce2f3c49cdc11bf It apparently modified how SurfaceView is clipped.

It seemed to have been released in Android 14 QPR2.

It's a wild guess, but I thought it was worth mentioning.

With QPR2 our QA also started to report the issue, so very likely related.

@fpitterstv2
Copy link

We are seeing this now when migrating to compose :(
Using RESIZE_MODE_FIT inside a fillMaxSize composable

@jdelga
Copy link

jdelga commented Apr 17, 2024

This seems a duplicate of #1123

@fpitterstv2
Copy link

Also reproducible on an emulator using VanillaIceCream

@icbaker
Copy link
Collaborator

icbaker commented Apr 19, 2024

I have filed an issue internally to ask for assistance from the Window Manager team (who own SurfaceView) and Compose folks: b/334901521

@icbaker
Copy link
Collaborator

icbaker commented Jun 14, 2024

I've done a bit more testing on different Android versions. This is on Pixel 7a and Pixel Fold (folded). My observations:

  • I haven't managed to repro on Android 13
  • On Android 14 before the March 2024 update (specifically I tested UP1A.231005.007 (the first stable Android 14 build) and UQ1A.240205.002.B1 (just before the March release))
    • I observe the issue at a high repro rate (~90%)
    • As soon as you tap on the playback to make the controls appear, the issue is resolved
  • On Android 14 after the March 2024 update (specifically I tested AP1A.240305.019.A1)
    • The repro rate appears to be about the same
    • Tapping the playback to make the controls appear doesn't resolve the issue

I suspect this may be why the issue has become more obvious since the March 2024 release. Before this point it was quickly resolved as soon as something else changed (in this case the controls appearing), but after this the problem persists.

@icbaker icbaker changed the title AspectRatioFrameLayout problem SurfaceView inside Compose AndroidView often shown stretched/cropped on API 34 Jun 19, 2024
copybara-service bot pushed a commit that referenced this issue Jun 19, 2024
This workaround is only applied on API 34, because the problem isn't
present on API 33 and it is fixed in the platform for API 35 onwards.

Issue: #1237

#cherrypick

PiperOrigin-RevId: 644729909
@icbaker
Copy link
Collaborator

icbaker commented Jun 19, 2024

I've updated the title of this issue to reflect that this isn't specific to AspectRatioFrameLayout, and it's more related to SurfaceView inside a Compose AndroidView on Android 14.

With the support of the graphics team, I've also submitted a workaround (linked above) to PlayerView, which resolves this issue for users of this class. This will be included in 1.4.0-rc01. I'm going to close this issue, since that should resolve the issue for users of media3's UI components. Those who have written their own UI components may be able to adopt a similar workaround.

@icbaker icbaker closed this as completed Jun 19, 2024
copybara-service bot pushed a commit that referenced this issue Jun 20, 2024
Previously we assumed that `surfaceSyncGroupV34` was always non-null on
API 34, but this isn't true in edit mode. Instead we add an explicit
null-check before accessing it. We don't need to null-check it at the
other usage site because we are already null-checking `surfaceView` (via
`instanceof` check).

Issue: #1237

#cherrypick

PiperOrigin-RevId: 645008049
tianyif pushed a commit that referenced this issue Jul 2, 2024
This workaround is only applied on API 34, because the problem isn't
present on API 33 and it is fixed in the platform for API 35 onwards.

Issue: #1237

#cherrypick

PiperOrigin-RevId: 644729909
(cherry picked from commit 968f72f)
tianyif pushed a commit that referenced this issue Jul 2, 2024
Previously we assumed that `surfaceSyncGroupV34` was always non-null on
API 34, but this isn't true in edit mode. Instead we add an explicit
null-check before accessing it. We don't need to null-check it at the
other usage site because we are already null-checking `surfaceView` (via
`instanceof` check).

Issue: #1237

#cherrypick

PiperOrigin-RevId: 645008049
(cherry picked from commit 9980306)
@kotya341
Copy link

Hello folks,
just want to share with you some updates.
after updating to the stable version of 1.4.0 I can still reproduce my problem from this ticket

@devno44
Copy link

devno44 commented Jul 30, 2024

Is there any workaround for older versions ?
We can not change the ExoPlayer version right now, and can not use TextureView instead of SurfaceView.

@icbaker
Copy link
Collaborator

icbaker commented Jul 30, 2024

@devno44 In order to backport the code change to an older version of the library you will need to build your chosen version locally (media3 instructions, ExoPlayer 2.19.0 instructions), and then apply the changes in 968f72f and 9980306 to your local copy of the code (note that in the exoplayer2, the equivalent of media3's PlayerView is StyledPlayerView.

@kotya341
Copy link

kotya341 commented Jul 30, 2024

@oceanjules
could you please reopen our thread, I can't reply on it
This conversation has been locked and limited to collaborators.

@icbaker
Copy link
Collaborator

icbaker commented Jul 30, 2024

@kotya341 Looks like @tonihei has unlocked the thread and you should be able to reply - apologies for missing that when I re-opened it.

@phcannesson
Copy link
Contributor

Should we also reopen this one or does 1.4.0 contain a valid fix ?

@icbaker
Copy link
Collaborator

icbaker commented Jul 30, 2024

@phcannesson This issue is fixed in 1.4.0-rc01. If you're experiencing similar problems on that version or later, please open a new issue with details & repro steps.

@Omkar-Amberkar
Copy link

Omkar-Amberkar commented Aug 2, 2024

@icbaker how would we be able to replicate your fix when we aren't using the PlayerView directly but instead use the AspectRatioFrameLayout and SurfaceView inside the AndroidView. I have created something similar in compose for our use case as below. let me know if this looks good

const val RESIZE_MODE_FIT = 0
const val RESIZE_MODE_FIXED_WIDTH = 1
const val RESIZE_MODE_FIXED_HEIGHT = 2
const val RESIZE_MODE_FILL = 3
const val RESIZE_MODE_ZOOM = 4

private const val MAX_ASPECT_RATIO_DEFORMATION_FRACTION = 0.01f

@Composable
fun Surface(
    modifier: Modifier = Modifier,
    aspectRatio: Float = 16f / 9f,
    resizeMode: Int = RESIZE_MODE_FIT,
    onSurfaceCreated: (SurfaceView) -> Unit = {},
    onSurfaceDestroyed: (SurfaceView) -> Unit = {},
) {
    val mainLooperHandler = remember { Handler(Looper.getMainLooper()) }
    val surfaceSyncGroupV34 = remember {
        if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.UPSIDE_DOWN_CAKE) {
            SurfaceSyncGroupCompatV34()
        } else {
            null
        }
    }
    AspectRatioLayout(
        aspectRatio = aspectRatio,
        resizeMode = resizeMode,
        modifier = modifier
            .drawWithContent {
                // Draw the existing content
                drawContent()

                if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.UPSIDE_DOWN_CAKE) {
                    surfaceSyncGroupV34?.maybeMarkSyncReadyAndClear()
                }
            },
    ) {
        Box {
            AndroidExternalSurface(
                content = { context, state ->
                    SurfaceView(context).apply {
                        state.onSurface { surface, width, height ->
                            onSurfaceCreated(this@apply)
                            surface.onChanged { width, height ->
                                if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.UPSIDE_DOWN_CAKE) {
                                    surfaceSyncGroupV34?.postRegister(
                                        mainLooperHandler,
                                        this@apply,
                                        this@apply::invalidate
                                    )
                                }
                            }
                            surface.onDestroyed {
                                onSurfaceDestroyed(this@apply)
                            }
                        }
                        holder.addCallback(state)
                    }
                }
            )
        }
    }
}

@Composable
private fun AspectRatioLayout(
    modifier: Modifier,
    aspectRatio: Float,
    resizeMode: Int,
    content: @Composable () -> Unit,
) {
    if (aspectRatio <= 0) return
    Layout(
        content = content,
        modifier = modifier
    ) { measurables, constraints ->
        var width = constraints.maxWidth
        var height = constraints.maxHeight
        val viewAspectRatio = width.toFloat() / height
        val aspectDeformation = aspectRatio / viewAspectRatio - 1
        if (abs(aspectDeformation) > MAX_ASPECT_RATIO_DEFORMATION_FRACTION) {
            when (resizeMode) {
                RESIZE_MODE_FIXED_WIDTH -> height = (width / aspectRatio).toInt()
                RESIZE_MODE_FIXED_HEIGHT -> width = (height * aspectRatio).toInt()
                RESIZE_MODE_ZOOM -> if (aspectDeformation > 0) {
                    width = (height * aspectRatio).toInt()
                } else {
                    height = (width / aspectRatio).toInt()
                }

                RESIZE_MODE_FIT -> if (aspectDeformation > 0) {
                    height = (width / aspectRatio).toInt()
                } else {
                    width = (height * aspectRatio).toInt()
                }
            }
        }

        layout(width, height) {
            val childConstraints = constraints.copy(
                minWidth = 0,
                minHeight = 0,
                maxWidth = width,
                maxHeight = height
            )
            measurables.forEach { measurable ->
                val placeable = measurable.measure(childConstraints)
                placeable.placeRelative(0, 0)
            }
        }
    }
}


@RequiresApi(34)
private class SurfaceSyncGroupCompatV34 {
    private var surfaceSyncGroup: SurfaceSyncGroup? = null

    fun postRegister(
        mainLooperHandler: Handler,
        surfaceView: SurfaceView,
        invalidate: Runnable
    ) {
        mainLooperHandler.post {
            val rootSurfaceControl = surfaceView.rootSurfaceControl ?: return@post
            surfaceSyncGroup = SurfaceSyncGroup("exo-sync-b-334901521")
                .apply { add(rootSurfaceControl) {} }
            invalidate.run()
            rootSurfaceControl.applyTransactionOnDraw(SurfaceControl.Transaction())
        }
    }

    fun maybeMarkSyncReadyAndClear() {
        surfaceSyncGroup?.markSyncReady()
    }
}

@icbaker
Copy link
Collaborator

icbaker commented Aug 5, 2024

Heads up for those subscribed to this thread: We have received a report (#1594) that the workaround we introduced in 30cb762 has introduced a regression in the behaviour of Shared Element Transitions. We are currently investigating this, and will update #1594 when we have more info.


@Omkar-Amberkar Does it work? That's probably the best/first test. In general we don't have the capacity for 1:1 code review, and we are also not the experts in these graphics APIs - so I'm afraid not best placed to debug your integration with them, if you're unable to adapt our published workaround to your needs.

@Omkar-Amberkar
Copy link

@icbaker yes it does but wanted to check in with you guys to see if that seems correct when using compose.

@efemoney
Copy link

efemoney commented Aug 9, 2024

@Omkar-Amberkar it is correct. It does the exact same thing. I have the same workaround in my code.

l1068 added a commit to l1068org/media that referenced this issue Aug 16, 2024
* Remove `@UnstableApi` from demo apps

This annotation is only needed on public classes that are part of a
distributed library.

Switch to `@OptIn` for the one file where `@UnstableApi` was
suppressing lint errors.

PiperOrigin-RevId: 633858516

* Remove `@UnstableApi` from package-private files

This annotation is only needed on public classes.

This change also removes the `/* package */` comment from some `public`
classes.

PiperOrigin-RevId: 633864544

* Remove `/* package */` comment from `public` classes

Also make one class truly package-private and keep the comment instead.

This comment should only appear on elements with default (package-private) visibility.

PiperOrigin-RevId: 633911914

* Add unit test for seeking in clipped MediaItem

PiperOrigin-RevId: 633912487

* Clear the PreloadMediaSource when deprioritized by the preload manager

PiperOrigin-RevId: 633917110

* Fix test flakiness

The test currently resets the time too far in the past and then has
to run through ~30000 additional iterations of doSomeWork to reach
the end, sometimes triggering the test timeout.

Fix it by resetting the time to the intended start position when
transitioning items.

PiperOrigin-RevId: 633918706

* Publish `CompositionPlayer` for playing compositions

This class is not ready for production app usage yet, so it is still
marked `@RestrictTo(LIBRARY_GROUP)` for now. Apps can experiment with it
in a non-prod context by suppressing the associated lint error.

* Issue: androidx/media#1014
* Issue: androidx/media#1185
* Issue: androidx/media#816

PiperOrigin-RevId: 633921353

* Support AVIF in exoplayer

https://developer.android.com/media/platform/supported-formats#image-formats was updated to include AVIF support in API 34+, so <unknown commit> updated our associated Util's to reflect this. After that change, ExoPlayer's BitmapFactoryImageDecoder will be able to decode AVIF, but the player won't be able to detect or extract it. Add this support for completeness, so that ExoPlayer can continue to say it supports all formats in https://developer.android.com/media/platform/supported-formats#image-formats.

PiperOrigin-RevId: 633956245

* Add debug trace logs for AudioMixer & AudioGraph events.

PiperOrigin-RevId: 633973723

* Default to parse subtitles while extracting, instead of while rendering

To override this change, and go back to parsing during rendering,
apps must make two method calls:

1. `MediaSource.Factory.experimentalParseSubtitlesDuringExtraction(false)`
2. `TextRenderer.experimentalSetLegacyDecodingEnabled(true)`

PiperOrigin-RevId: 634262798

* Update PgsParser.java

Fix "subtitles rendered with border and no fill color" problem:
https://github.com/moneytoo/Player/issues/413

because '0' means it's an index to the palette table, not an RGBA value of 0

* Format with google-java-format and add release note

* Handle playToEndOfStream called before configuring the audio sink

ExoPlayer sometimes calls AudioSink.playToEndOfStream before configuring
the sink. Before this CL, the composition player was failing if this
happened.

PiperOrigin-RevId: 634306592

* Fix image not ignored for non-images in setImageDurationMs

PiperOrigin-RevId: 634345071

* Use List in `createRenderers` for better readability

Also Use `Iterables.toArray()` to void the confusing `List.toArray()` method call

PiperOrigin-RevId: 634351844

* Check for .heif extension in File types inference

PiperOrigin-RevId: 634409758

* Fix and/or bug in `XingSeeker`

This was accidentally introduced in https://github.com/androidx/media/commit/4fde35c9cc03947aecd3f8713f633f3d4dcd6aa0

PiperOrigin-RevId: 634465380

* Add debug trace log for AssetLoader Renderer format events.

PiperOrigin-RevId: 634474584

* Add toggle for DebugTraceUtil to Transformer demo.

PiperOrigin-RevId: 634495944

* Throw errors from `WebvttPlaybackTest.stallUntilPlayerCondition`

Before this change, if a playback error is thrown the test fails with a
timeout and no additional info:

```
java.util.concurrent.TimeoutException
	at androidx.media3.exoplayer.e2etest.WebvttPlaybackTest.stallPlayerUntilCondition(WebvttPlaybackTest.java:361)
```

After this change, the test failure includes a much more useful stack
trace, e.g. from https://github.com/androidx/media/commit/0352db9a375cff16b14e3e8a653e30dcfa589b74:

```
Caused by: java.lang.IllegalStateException: Legacy decoding is disabled, can't handle text/vtt samples (expected application/x-media3-cues).
	at androidx.media3.common.util.Assertions.checkState(Assertions.java:100)
	at androidx.media3.exoplayer.text.TextRenderer.assertLegacyDecodingEnabledIfRequired(TextRenderer.java:587)
	at androidx.media3.exoplayer.text.TextRenderer.onStreamChanged(TextRenderer.java:210)
```

PiperOrigin-RevId: 634672138

* Add missing check before calling discardBuffer

The method is only allowed to be called on prepared items.
This check was currently missing and also causing the
corresponding test to be flaky in ExoPlayerTest.

PiperOrigin-RevId: 634694077

* Fix flakiness in ExoPlayerTest

The two affected tests where playing until a specific
position to enable the player to read ahead. The method
pauses at exactly the target position, but then has
temporarily undetermined behavior because the playback
thread uses player.getClock().onThreadBlocked() that lets
the playback thread make progress in parallel to the test
thread. The tests were flaky because they sometimes made
so much progress that they ended playback before we could
query the updated renderer state.

This can be fixed by using
run(player).untilBackgroundThreadCondition instead, which
is guaranteed to be fully deterministic, but may not be able
to stop at exactly the desired position (which we don't
really need anyway for this test).

PiperOrigin-RevId: 634699752

* Fix AV sync for sequences with audio track shorter than video

For each item, AudioGraphInput now pads the input audio track with silence
to the duration given in onMediaItemChanged.

Possibly resolves Issue: androidx/media#921 .

PiperOrigin-RevId: 634753721

* Clean up unused members in `XingFrame`

PiperOrigin-RevId: 634817614

* Handle timeline updates where all periods in window have been replaced

This case is most likely to happen when re-preparing a multi-period
live stream after an error. The live timeline can easily move on to
new periods in the meantime, creating this type of update.

The behavior before this change has two bugs:
 - The player resolves the new start position to a subsequent period
   that existed in the old timeline, or ends playback if that cannot
   be found. The more useful behavior is to restart playback in the
   same live item if it still exists.
-  MaskingMediaSource creates a pending MaskingMediaPeriod using the
   old timeline and then attempts to create the real period from the
   updated source. This fails because MediaSource.createPeriod is
   called with a periodUid that does no longer exist at this point.
   We already have logic to not override the start position and need
   to extend this to also not prepare the real source.

Issue: androidx/media#1329
PiperOrigin-RevId: 634833030

* Refactor pending clock sync logic in DashMediaSource

This should be no-op overall and helps to disentangle the clock sync
update from the state of the manifest.

We currently check oldPeriodCount==0 to trigger the clock sync load,
which only works because the manifest happens to be null whenever
we need a new clock sync. We can decouple these concepts by directly
checking whether we have an existing elapsedRealtimeOffsetMs.

This also requires to set this value explicitly at the point where we
consider it set to the local device clock fallback when the timing
element load fails.

PiperOrigin-RevId: 634844921

* Keep manifest in DashMediaSource after release

This is a fix for the fix in https://github.com/androidx/media/commit/319854d624a95d3f71a26c96ecb4e5827d4a7f4e. The original fix did
not reset the firstPeriodId to avoid any id clashes with future
updates. This however only works under the assumption that the
next manifest load at the next call to prepare() is exactly the
same as the current manifest. This is not true unless the call
happens very quickly (and may fail even then). Instead we should
keep the existing manifest directly as a reference so we can use
it to find the number of removed periods when we get a new manifest
at the next call to prepare().

Issue: androidx/media#1329
PiperOrigin-RevId: 634853524

* Fix image seeking

Queue image again after position reset, and reset timestamp iterator.

PiperOrigin-RevId: 635049953

* Allow LoadControl.shouldContinueLoading accept playWhenReady as a parameter

* Move parameters inside LoadControl and use it for shouldStartPlayback

+additional formatting and Javadoc changes

* Add debug trace for Muxer completely ending.

PiperOrigin-RevId: 635459757

* Make flag for debug traces in logcat private and final.

PiperOrigin-RevId: 635465538

* Fix issue links from `google/ExoPlayer` to `androidx/media`

PiperOrigin-RevId: 635469173

* Add OptIn annotations to declarations in demo app files

#minor-release

PiperOrigin-RevId: 635469477

* Create equals method for gainmaps

Gainmaps don't currently have an equals override, only reference equality is checked by Objects.equals(gainmap1, gainmap2). Create an equals method for gainmaps with the fields we care about to ensure we don't incur false positives in our equality checks.

PiperOrigin-RevId: 635510451

* Work around SurfaceTexture implicit scale

If MediaCodec allocates passes an image buffer with a cropped region,
SurfaceTexture.getTransformMatrix will cut off 2 pixels from each dimensions.
The resulting videos will appear a little stretched.

This patch inspects the SurfaceTexture transform matrix, and guesses what the
unscaled transform matrix should be.
Behind experimentalAdjustSurfaceTextureTransformationMatrix flag

PiperOrigin-RevId: 635721267

* Move bitmap coordinate flip out of fragment shader

Fragment shaders in OpenGL ES shader language aren't guaranteed
to support highp, required to correctly represent pixel coordinates
inside large images (e.g. 1920x1080).
This change moves coordinate mirroring for images out of fragment shader.

Fixes http://Issue: androidx/media#1331

PiperOrigin-RevId: 635732208

* Fix javadoc formatting

PiperOrigin-RevId: 635737466

* Move license to top for consistency

PiperOrigin-RevId: 635742699

* Assert file exists before trying to re-decode for test assertions.

PiperOrigin-RevId: 635748820

* MP3: Add test CBR sample with 'too small' `PCUT` frame

This shows ExoPlayer currently wrongly reports the duration of this
sample, because it assumes every frame is 32kbps (104 bytes) due to the
`PCUT` frame immediately after the `Info` frame.

A follow-up change will modify `Info` frame handling to resolve this
issue.

This sample was crafted using a hex editor to insert the additional
`PCUT` frame (the pattern of `null` and `x` is taken from the sample
file in Issue: androidx/media#1376, the header is modified to set the channel count
to 1 to match the rest of the file), and then update the frame count
and data size of the `Info` header to match.

Issue: androidx/media#1376
PiperOrigin-RevId: 635772837

* Re-apply CEA-708 `rowLock/columnLock` fix

This change was originally made in https://github.com/androidx/media/commit/6f8249184bd69b126c93479f918768b0cf2d0889

It was then accidentally lost in when `Cea708Parser` was merged back
into `Cea708Decoder` in https://github.com/androidx/media/commit/51b4fa2cc83b60fcb313fd0e6afd2d45fe64e535.

This is the only change made to the actual 'decoding' logic in
`Cea708Parser` between it being split from `Cea708Decoder` and merged
back in again, all the other changes in this period relate to the
implementation of the `SubtitleParser` interface, so don't need to be
preserved in `Cea708Decoder`:
https://github.com/androidx/media/commits/51b4fa2cc83b60fcb313fd0e6afd2d45fe64e535/libraries/extractor/src/main/java/androidx/media3/extractor/text/cea/Cea708Parser.java

`Cea608Parser` was also merged back into `Cea608Decoder` in
https://github.com/androidx/media/commit/25498b151ba298ef359f245e2ed80718b4adf556
and so is vulnerable to the same risk of accidental loss of changes. To
be sure, I also checked the history of this file:
https://github.com/androidx/media/commits/25498b151ba298ef359f245e2ed80718b4adf556/libraries/extractor/src/main/java/androidx/media3/extractor/text/cea/Cea608Parser.java

The only 'decoding logic' change there is https://github.com/androidx/media/commit/379cb3ba540d7044785f8355b2df9220e7f5eb8f,
which was also lost in https://github.com/androidx/media/commit/25498b151ba298ef359f245e2ed80718b4adf556.
I will send a separate change to resolve this.

PiperOrigin-RevId: 635796696

* Re-apply CEA-608 `validDataChannelTimeoutMs` assertion

This change was originally made in https://github.com/androidx/media/commit/379cb3ba540d7044785f8355b2df9220e7f5eb8f.

It was then accidentally lost in when `Cea608Parser` was merged back
into `Cea608Decoder` in https://github.com/androidx/media/commit/25498b151ba298ef359f245e2ed80718b4adf556.

This was spotted when re-doing a similar lost change to `Cea708Decoder`,
reported in https://github.com/androidx/media/pull/1315.

See reasoning on https://github.com/androidx/media/commit/e2847b3b80112e7bce0e0706604bdee8126704e5
about why this is the only 'lost' CEA-608 change.

PiperOrigin-RevId: 635803536

* Add new line between printing glsl source code in error

PiperOrigin-RevId: 635812838

* Migrate debug trace logs to track generic events for Muxer.

Track information is added to the details string where relevant.

PiperOrigin-RevId: 635815866

* Remove VideoFrameReleaseControl setter from SinkProvider

Move the parameter to the constructor instead.

PiperOrigin-RevId: 636077477

* Set image duration on all media types

This was previously only set on images because it was not ignored on
other media types. This parameter was made no-op for non-images in
https://github.com/androidx/media/commit/7b2a1b444312953f9518868f1cfc3e0b6c400d7b.

PiperOrigin-RevId: 636078142

* Make getIconResIdForIconConstant public

This allows controller apps to map these constants to suitable
icons without creating a CommandButton instance first.

PiperOrigin-RevId: 636096841

* Work around 1080p export failures on certain devices

Fall back to using software decoder for 1920x1080 for certain
devices.

PiperOrigin-RevId: 636132298

* MP3: Derive duration and bitrate from frame count in `Info` header

`Info` header is used for CBR files, but in some cases not **every**
frame in these files is the same size. This change stops using the
single frame after the `Info` frame as the 'template' (and assuming all
subsequent frames are the same size/bitrate), and instead derives the
bitrate from fields in the `Info` header. This works for files which are
'almost' constant bitrate, like the one in Issue: androidx/media#1376 where every
frame is either 1044 or 1045 bytes except the one immediately after the
`Info` frame which is 104 bytes (32kbps), resulting in a wildly
incorrect duration calculation.

PiperOrigin-RevId: 636151605

* Parse dashif:Laurl license url in mpd

* Add test and formatting changes

* Trigger silence generation when end of stream is encountered

This change avoids a muxer deadlock when:
1. Sequence of items
2. First item has audio track that is shorter than video
3. Audio finishes, and muxer refuses to write more than 500ms of video
   consecutively.

SequenceAssetLoader fails to progress to the second item. A muxer
deadlock is possible when the audio of the first item finishes,
audio end-of-stream is not propagated through AudioGraph, and muxer blocks
video, preventing SequenceAssetLoader to move to the next item in sequence.

By triggering silence generation early as soon as audio EOS is
encountered, we ensure SequenceAssetLoader can progress to the next item.

PiperOrigin-RevId: 636179966

* MP3: Make a defensive copy of the header in `XingFrame` constructor

This is currently set from `Mp3Extractor.synchronizedHeader` which
gets overwritten every time we read a new frame. It seems safer to make
this defensive copy (and there will be at most one `XingFrame` instance
per-playback, so this is not prohibitively expensive).

PiperOrigin-RevId: 636181038

* Add DV profile 10 handling in getAlternativeCodecMimeType() method

* Fragmented Mp4Muxer: add support to B-frame Muxing

Add composition time offset parameter to TRUN box to
support muxing of videos containing B-frames by FragmentedMp4Muxer.
Update TRUN box version from 0 to 1 in order to manage signed
composition time offset.

PiperOrigin-RevId: 636426397

* Update session module registration

PiperOrigin-RevId: 636482934

* Assert file size in E2E android tests that claim to complete.

Remove redundant test logic to add file size to ExportResult because
the file size is already added to export result as part of an export
finishing.

PiperOrigin-RevId: 636499236

* Add missing module registrations

PiperOrigin-RevId: 636506860

* Ignore rowLock and numLock as define in CTA-708 spec.
Update current row value when new line is added.

* revert rowLock and colomnLock changes since it will be done separately

* remove test comment

* Add cea708Decoder test for setPenLocation command and newline handling

* Rename and reshuffle tests

* Add release note

* Remove jetifier config from gradle config

None of the AndroidX libs we depend on require jetification any more.

Issue: androidx/media#1362
PiperOrigin-RevId: 636544337

* Add more details to thread assertion in onAudioCapabilitiesChanged

This helps to debug issues reported in https://github.com/androidx/media/issues/1191

PiperOrigin-RevId: 636545970

* Migrate `buildConfig` from `properties.gradle` to `build.gradle`

This is generated in response to a deprecation warning in AS:

```
The option setting 'android.defaults.buildfeatures.buildconfig=true' is deprecated.
The current default is 'false'.
It will be removed in version 9.0 of the Android Gradle plugin.
You can resolve this warning in Android Studio via `Refactor` > `Migrate BuildConfig to Gradle Build Files`
```

PiperOrigin-RevId: 636546985

* Remove deprecated `setContentTypePredicate()` methods

Use the suggested alternative on the respective
`XXXDataSource.Factory` instead.

PiperOrigin-RevId: 636560182

* Remove `OkHttpDataSource` constructors & `OkHttDataSourceFactory`

Use `OkHttpDataSource.Factory` instead.

PiperOrigin-RevId: 636585523

* Print underlying extractor name in `UnrecognizedInputFormatException`

If subtitle-parsing-during-extraction is enabled (now defaults to on),
the 'outer' extractor class name is often
`SubtitleTranscodingExtractor`, leading to some slightly useless error
messages like:

`None of the available extractors (FragmentedMp4Extractor, Mp4Extractor, SubtitleTranscodingExtractor, SubtitleTranscodingExtractor, SubtitleTranscodingExtractor, SubtitleTranscodingExtractor, SubtitleTranscodingExtractor, SubtitleTranscodingExtractor, TsExtractor, MatroskaExtractor, SubtitleTranscodingExtractor, SubtitleTranscodingExtractor, SubtitleTranscodingExtractor, SubtitleTranscodingExtractor, AviExtractor, SubtitleTranscodingExtractor, SubtitleTranscodingExtractor, SubtitleTranscodingExtractor, SubtitleTranscodingExtractor, SubtitleTranscodingExtractor, SubtitleTranscodingExtractor)`

PiperOrigin-RevId: 636834354

* Remove `PlayerMessage.setHandler(Handler)`

Use `setLooper(Looper)` instead.

PiperOrigin-RevId: 636840566

* Maintain a consistent luminance range across HDR content in effects

PQ and HLG have different luminance ranges (max 10k nits and max 1k nits resp). In GL, colors work in a normalised 0 to 1 scale, so for PQ content, 1=10k nits and and for HLG content, 1=1k nits.

This cl scales and normalises PQ content appropriately so that all HDR content works in the HLG luminance range. This fixes two things

1. Conversions between HLG and PQ are "fixed" (before the output colors looked too bright or too dark depending on which way you are converting)
2. color-altering effects will be able to work consistently across HLG and PQ content

1 is tested in this cl. 2 will be tested when ultra HDR overlays are implemented, both cases have been manually tested to ensure the output looks correct on a screen.

PiperOrigin-RevId: 636851701

* Remove `ShadowLog` references from tests

These were accidentally submitted after being added for local debugging.

PiperOrigin-RevId: 636865825

* Remove deprecated `Timeline.Window.isLive` field

PiperOrigin-RevId: 636870982

* Remove deprecated `DefaultHttpDataSource` constructors

Use `DefaultHttpDataSource.Factory` instead.

PiperOrigin-RevId: 636875524

* Remove deprecated `DashMediaSource.DEFAULT_LIVE_PRESENTATION_DELAY_MS`

Use `DashMediaSource.DEFAULT_FALLBACK_TARGET_LIVE_OFFSET_MS` instead.

PiperOrigin-RevId: 636906922

* Remove deprecated `MediaCodecInfo.isSeamlessAdaptationSupported(...)`

Use `MediaCodecInfo.canReuseCodec(...)` instead.

PiperOrigin-RevId: 636918479

* Remove deprecated `DrmSessionManager.DUMMY` and getter method

Use `DRM_UNSUPPORTED` constant instead.

PiperOrigin-RevId: 636937592

* UltraHdr: Fix calculations to apply gainmap

PiperOrigin-RevId: 636964921

* Ensure single-frame videos are correctly exported

Add a wait in DefaultCodec.signalEndOfInputStream when no
video encoder output has been seen. This avoids a thread synchronization problem
between writing frames to video surface, and signaling end of stream,
which was hit for video input of only one frame on some devices.

PiperOrigin-RevId: 637844690

* Remove deprecated format changed methods

Use the overloads with an additional `@Nullable DecoderReuseEvaluation`
parameter instead.

PiperOrigin-RevId: 637851937

* Remove unused `Util.getCommaDelimitedSimpleClassNames` method

PiperOrigin-RevId: 637854422

* Add support for ultra HDR overlays

PiperOrigin-RevId: 637863706

* Reset input capacity when setting external shader program

Before this CL, externalShaderProgramInputCapacity was not reset when
the external shader program was reset (which occurs when  the
InputSwitcher switches to an input with a different ColorInfo). This is
due to a regression introduced in https://github.com/androidx/media/commit/bef3d518d2dd11155a759d4e62bd649e32752a2d.

PiperOrigin-RevId: 637869215

* Tighten exception handling in `TestPlayerRunHelper`

This removes `throws Exception` from public methods in favour of more
specific exception types (`TimeoutException` and `PlaybackException`).

PiperOrigin-RevId: 637880546

* Remove deprecated `RendererSupport.FormatSupport` IntDef & constants

Use `C.FormatSupport` and associated constants instead.

PiperOrigin-RevId: 637890304

* Speed up image to video Export

Only sample from input bitmap when the input image has changed.
Introduce GainmapShaderProgram.newImmutableBitmap API that signals
input bitmap changes to GainmapShaderProgram (DefaultShaderProgram).

PiperOrigin-RevId: 637920207

* Add a flag to control whether input bitmap resampling can be skipped

Add DefaultVideosFrameProcessor experimental flag that controls
whether input Bitmaps are sampled once for a repeating sequence of
output frames with the same contents, or once for each output frame.

PiperOrigin-RevId: 637921350

* Add an experimental analyzer mode to Transformer.

PiperOrigin-RevId: 637926059

* Import string translations for session module

#minor-release

PiperOrigin-RevId: 638224207

* Rename test so that it runs on MH

PiperOrigin-RevId: 638237657

* Use file inserts and string replacements in overlay shaders

PiperOrigin-RevId: 638251955

* Increase h.264 buffer size in `ShadowMediaCodecConfig`

Some test media has samples larger than 100kB

PiperOrigin-RevId: 638598553

* Rename `PreloadMediaSource.PreloadControl` methods

The IntDefs in `DefaultPreloadManager.Stage` are also renamed accordingly.

PiperOrigin-RevId: 638631357

* Suppress incorrect linter error in DebugRenderersFactory method

Method should use @RequiresApi annotation instead of @SdkSuppress

PiperOrigin-RevId: 638639225

* Regenerate TransformerHdrTest goldens after removing degammaing

Degammaing has been removed in https://github.com/androidx/media/commit/cb4b2ea55c70efa80adc3e664301b1990e89eaf3. The goldens for
TransformerHdrTest (previously TransformerSequenceEffectTestWithHdr)
were not regenerated because the test wasn't running due to its name
(fixed in https://github.com/androidx/media/commit/e41a966237528148590712048dc735dabfd344ca).

PiperOrigin-RevId: 638645635

* Remove unnecessary sortKey from Mp4Muxer.addTrack()

PiperOrigin-RevId: 638647112

* Use INDEX_UNSET instead of LENGTH_UNSET for next media sequence/part

Both constants have the same value, but the method returning initial values for the media sequence/part uses `INDEX_UNSET`, so it makes sense to use it.

PiperOrigin-RevId: 638673282

* Schedule exoplayer work task to when renderers can make progress

Currently ExoPlayer schedules its main work loop on a 10 ms interval. When renderers cannot make any more progress(ex: hardware buffers are fully written with audio data), ExoPlayer should be able to schedule the next work task further than 10Ms out.

Through `experimentalSetDynamicSchedulingEnabled`, ExoPlayer will dynamically schedule its work tasks based on when renderers are expected to be able to make progress.

PiperOrigin-RevId: 638676318

* Call PreloadControl.onSourcePrepared only once for each preload request

PiperOrigin-RevId: 638677090

* Schedule exoplayer work to when MediaCodecAudioRenderer can progress

Currently ExoPlayer schedules its main work loop on a 10 ms interval. When renderers cannot make any more progress (ex: hardware buffers are fully written with audio data), ExoPlayer should be able to schedule the next work task further than 10ms out into the future.

Through `experimentalSetDynamicSchedulingEnabled` and these changes to `MediaCodecAudioRenderer`, ExoPlayer can use the data provided by the audio renderer to dynamically schedule its work tasks based on when it is expected that progress can be made.

PiperOrigin-RevId: 638677454

* Add references in javadocs to relevant listeners for Player fields

PiperOrigin-RevId: 638688864

* Use `kotlinx-coroutines-guava` in session demo app

I originally tried switching to `Futures.addCallback` (as a follow-up
to Issue: androidx/media#890), but it seemed like a good chance to go further into
Kotlin-ification.

Before this change, if the connection to the session failed, the app
would hang at the 'waiting' screen with nothing logged (and the music
keeps playing). This behaviour is maintained with the `try/catch` around
the `.await()` call (with additional logging). Without this, the failed
connection causes the `PlayerActivity` to crash and the music in the
background stops. The `try/catch` is used to flag to developers who
might be using this app as an example that connecting to the session
may fail, and they may want to handle that.

This change also switches `this.controller` to be `lateinit` instead of
nullable.

Issue: androidx/media#890
PiperOrigin-RevId: 638948568

* Schedule doSomeWork when MediaCodec signals available buffers

When running in asynchronous mode, MediaCodec will be running the CPU to signal input and output buffers being made available for use by the player. With ExoPlayer.experimentalSetDynamicSchedulingEnabled set to true, ExoPlayer will wakeup to make rendering progress when MediaCodec raises these signals. In this way, ExoPlayer work will align more closely with CPU wake-cycles.

PiperOrigin-RevId: 638962108

* Allow `ByteArrayDataSource` to resolve the byte array when opened

This is a relatively small change, and massively simplifies the work
needed for an app to consume Kotlin Multiplatform resources (without a
full `KmpResourceDataSource` implementation, which poses some
dependency challenges for now).

Issue: androidx/media#1405
PiperOrigin-RevId: 638991375

* Fix a race condition in AudioGraphInput

AudioGraphInput.onMediaItemChanged is called on input thread. Pending
media item changes are processed on processing thread, inside calls to
getOutput().
This change allows multiple pending media item changes to be enqueued,
and processed in sequence.

PiperOrigin-RevId: 638995291

* Add `FOREGROUND_SERVICE_MEDIA_PLAYBACK` permission for test session app

Foreground service type `mediaPlayback` requires permission `android.permission.FOREGROUND_SERVICE_MEDIA_PLAYBACK`. The `MockMediaSessionService`, `LocalMockMediaSessionService`, `MockMediaLibraryService`
 and `LocalMockMediaLibraryService` declared in the manifest are in the `mediaPlayback` type.

PiperOrigin-RevId: 639013810

* Move effect final values and preset names and paths to arrays file.

PiperOrigin-RevId: 639034630

* Fix CMCD data assignment for init segment

The CMCD data was incorrectly added to the `dataSpec` of the media segment instead of the init segment.

Also relaxed the condition for playbackRate to be C.RATE_UNSET when creating an instance of CmcdData.Factory as there was nothing enforcing this check.

#minor-release

PiperOrigin-RevId: 639046080

* Drop API requirement from SeparableConvolution

Switch from 4-channel RGBA_16F lookup texture to 1-channel R_16F.
Do not use a bitmap when creating the lookup table texture.
Instead, fill the texture directly.
Do not manually convert 32-bit float to 16-bit. Instead, let OpenGL
libraries do this for us.

PiperOrigin-RevId: 639717235

* Make progress tests stricter.

These changes are possible because getProgress is no longer a blocking
operation on transformer.

* Tests call getProgress after every looper message executed.
* Use longer media assets for getProgress tests to give more progress
  intervals.
* Remove conditional assertions.

PiperOrigin-RevId: 639734368

* Remove module registration from HttpEngineDataSource

It used to be its own module in the first revision, but then moved
into the DataSource module as of https://github.com/androidx/media/commit/250fc80419b4b8ef1b753824e53ebdac176968f5.

PiperOrigin-RevId: 639745498

* Add an analyzer mode option to Transformer demo.

PiperOrigin-RevId: 639746452

* Fix the RELEASENOTES.md

This [commit](https://github.com/androidx/media/commit/d175223cc632f9a927a0cfe62bc808d336a8c83a) was introduced not long ago (after 1.4.0-alpha01), but has the release note added to the alpha01 release. Moving this piece of note to "unreleased changes" so that it will be included in the 1.4.0-alpha02 release note as a new change.

PiperOrigin-RevId: 639768495

* Move CEA-708 release note from 1.3.0 to 'unreleased' (1.4.0)

This was added in the wrong place as part of https://github.com/androidx/media/commit/0a58832d85a51a8ee8f50144c25d92ab8d8c0e78

PiperOrigin-RevId: 639783174

* Allow customisation of various icons in PlayerControlView

Before this change:
The only way to customize the icons was to override the drawables, e.g.
* `exo_styled_controls_play`
* `exo_styled_controls_pause`

However, that would set the drawables globally and prevent users from customizing the icons **per** PlayerView.

After the change, it is possible to provide drawable icons in the xml layout directly via `<androidx.media3.ui.PlayerView>` and
* `app:play_icon="@drawable/...`
* `app:pause_icon="@drawable/...`
* `app:vr_icon="@drawable/...`
* `app:fullscreen_exit_icon="@drawable/...`
* `app:next_icon="@drawable/...`

Note:
Two buttons that are left out of this change are fast-forward and rewind. They are more complicated due to layout insertion and customization with seek back/forward increments in the TextView.

Issue: androidx/media#1200
PiperOrigin-RevId: 639832741

* Support Separable Convolutions with negative coefficients

fragment_shader_separable_convolution_es2.glsl had optimizations that assumed
all convolution coefficients are positive. Support negative coefficients,
and add tests.

PiperOrigin-RevId: 640104741

* Add isReleased() to the Exoplayer

This avoids having to add AnalyticsListener and catching the `onPlayerReleased` callback.

The `Exoplayer.release()` method is blocking and one can be sure that the player is released if the call returned. However, the method is useful for UI testing and asserting that the player is released after a certain UI action, e.g. closing the activity or detaching a window.

PiperOrigin-RevId: 640114416

* Skip AV1 test on devices that don't support AV1 HDR encoding

Skip AV1 HDR test on some devices.

PiperOrigin-RevId: 640135455

* Update release notes for 1.4.0-alpha02

PiperOrigin-RevId: 640138307

* Version bump to media3:1.4.0-alpha02

#minor-release

PiperOrigin-RevId: 640196724

* Add icons for fastforward and rewind buttons

PiperOrigin-RevId: 640206040

* Decide whether to use the VideoSink when enabling renderer

This will simplify moving the release control inside the video sink

PiperOrigin-RevId: 640416128

* Add LanczosResample effect to scale videos in Transformer

Add SeparableConvolution.configure(inputSize) to allow effect configuration
depending on input dimensions.
Add LanczosResample.scaleToFit method to scale input images to fit inside
given dimensions.

PiperOrigin-RevId: 640498008

* MCVR: use shouldUseVideoSink where possible

Before this CL, we were checking whether the video sink was initialized
to determine whether it should be used. In the meantime,
shouldUseVideoSink has been introduced. Use this boolean to check
whether to use the video sink as it's a clearer signal.

PiperOrigin-RevId: 640499147

* Update FILE_UNKNOWN_DURATION to a longer fmp4 (~15s),

Removes the flakiness of
MediaItemExportTest.getProgress_unknownDuration_returnsConsistentStates
by using a longer input asset, such that ExoPlayer does not determine
the duration of the media.

PiperOrigin-RevId: 640502470

* Remove @UnknownInitialization DefaultCodec parameter

Using an uninitialised object seems unusual.

PiperOrigin-RevId: 640514501

* Move renderer state methods from release control to sink

PiperOrigin-RevId: 640515298

* Remove direct usages of release control when video sink is used

Usages removed in this CL are:
- onProcessedStreamChange, which was already called from the VideoSink
  (via VideoFrameRenderControl)
- setOutputSurface, which was also already called from the VideoSink
- setFrameRate, which this CL now sets in the VideoSink

PiperOrigin-RevId: 640530903

* Fix invalid state transitions for trim optimization getProgress.

Ensures valid progress state is returned. Should not return NOT_STARTED
once transformer.start has been called, until export ends.

PiperOrigin-RevId: 640533805

* Refactor trim optimization getProgress for improved readability.

This change is a no-op refactor to improve the readability of the
states returned in this method.

PiperOrigin-RevId: 640538374

* Remove unnecessary `throws Exception` from `HlsChunkSourceTest`

PiperOrigin-RevId: 640552488

* Add missing null check

PiperOrigin-RevId: 640555113

* Exit early if buffer becomes invalid

When the frame release control invalidates a buffer and returns that
the buffer must be ignored, we need to exit early before performing
additional checks that may result in method calls using the invalid
buffer.

PiperOrigin-RevId: 640555688

* MuxerWrapper: Fix spelling mistake in method name

PiperOrigin-RevId: 640838741

* Stabilize offload related error codes

PiperOrigin-RevId: 640839273

* Add CodecDetails to ExportException

This will replace the existing free-form string in the error message

PiperOrigin-RevId: 640954158

* Unify timestamp handling

Before this change, the timestamps output from composition playback is offset
with the renderer offset. After this change, the offset is removed and the
timestamp behaviour converges with Transformer, that is, the timestamps of
video/images frames will follow that of the composition. For example, with a
composition of two 10-s items, clipping the first with 2s at the start, the
timestamp of the first frame in the second item, will be 8s.

PiperOrigin-RevId: 641121358

* Fix a concurrency issue that shader capacity is set off GL thread

Also add test to cover transitions between BT709 and 601.

PiperOrigin-RevId: 641224971

* Fix timestamps in tests

https://github.com/androidx/media/commit/38a7229d9684970f3475eff203255b7d1c8cf747 changed only some of the timestamps, in fact all
of the timestamps should have the offset removed.

PiperOrigin-RevId: 641226102

* Fix test names

PiperOrigin-RevId: 641249128

* Mark `PlayerWrapper` as `final`

This package-private class isn't currently extended, so this change
makes it not possible to be extended in future.

PiperOrigin-RevId: 641270464

* Change externalShaderProgramInputCapacity to int

The field is only accessed on the GL thread.

PiperOrigin-RevId: 641273595

* Add `PlayerSurface` Compose component

Underneath, it delegates to either
* `AndroidExternalSurface` (equivalent of SurfaceView), which is generally better for power and latency
* `AndroidEmbeddedExternalSurface` (equivalent of TextureView) which is better for interactions with other widgets or applying visual effects through the graphicsLayer

Note: the resulting surface is stretched across the whole screen. Aspect ratio handling will be addressed in the follow-up changes.
PiperOrigin-RevId: 641285482

* Add support for setting and getting volume for composition preview.

PiperOrigin-RevId: 641822822

* Remember Player instance on recompositions

PiperOrigin-RevId: 641823733

* Remove incorrect 'bundleable' reference from `stringMapToBundle` param

This was accidentally copy-pasted in https://github.com/androidx/media/commit/5008417c8cd12680a3945945e00399cca05915c5

PiperOrigin-RevId: 641823991

* Set VideoSink listener when enabling MCVR

Before, the listener was set in onReadyToInitializeCodec, which means
that it was reset every time a new codec was used.

We need to set the listener every time MCVR is enabled (not only the
first time), because it might have been set by another renderer.

PiperOrigin-RevId: 641825717

* Add Transformer HDR-only assets sequence effect tests.

PiperOrigin-RevId: 641828107

* Rename "preset file" to "preset" in composition demo.
This change is made for consistency with transformer demo.

PiperOrigin-RevId: 641835714

* HDR-only assets sequence effect test: rename golden files

rename to match test name

PiperOrigin-RevId: 641836522

* Support upscaling with LanczosResample

Avoid scaling the lanczos windown function when scaling up.
When scaling up (output > input), we shouldn't scale the window
function or we risk not sampling any of the input pixels.

PiperOrigin-RevId: 641838149

* Call VideoFrameReleaseControl.setPlaybackSpeed from sink when enabled

Do not call VideoFrameReleaseControl.setPlaybackSpeed directly from
MCVR when the video sink is enabled.

PiperOrigin-RevId: 641840894

* Clarify docs on `ExoPlayer.setVideoEffects()` re calling `prepare()`

The previous wording suggested that `setVideoEffects()` may **only** be
called before `prepare()`, i.e. the effect cannot be changed during
playback. The intent is instead that `setVideoEffects()` must be called
once before playback in order to configure the effects pipeline, but
the effect can then be changed during playback by further calls to
`setVideoEffects()`.

Issue: androidx/media#1393
PiperOrigin-RevId: 641853629

* Remove most nullness marking from Transformer demos

The nullness checking is very verbose and redundant in some places, for
example, the ensures-non-null marking in the transformer activity, and it makes
changing features in the demo app more time consuming. The cost/benefit balance
seems to be in favor of removing this for demo code.

Note: the ExoPlayer main demo has nullness checks turned off.
PiperOrigin-RevId: 641890618

* Tidy passing views in transformer demos

PiperOrigin-RevId: 641891515

* Rename "preset file" to just "preset" in transformer demo

The new name means we can add streams and other sources that aren't files in
future, for example, screen recording input.

PiperOrigin-RevId: 641894319

* Fix an invalid javadoc link

PiperOrigin-RevId: 641895590

* Create explicit key for testing 'notification controller'

Previously these tests were setting the test-only `KEY_CONTROLLER`
bundle value to the prod value of
`MediaController.KEY_MEDIA_NOTIFICATION_CONTROLLER_FLAG`, which is a bit
confusing because this is intended to be used as a `Bundle` **key**
(for a boolean value), and not a value itself.

Instead this CL creates a test-only value that can be used by
`RemoteMediaSession` and related test-only friends to indicate that
something (e.g. error or extras) should only be set on the notification
controller.

This CL also changes the logic in `MediaSessionProviderService` to
avoid needing to special-case checking for this controller key.

PiperOrigin-RevId: 641939283

* Add helper function in Track.java to copy track without edit lists.

PiperOrigin-RevId: 642038117

* Add MediaSession.Callback.onPlayerInteractionFinished

This callback is useful for advanced use cases that care about
which Player calls are batched together. It can be implemented
by triggering this new callback every time a batch of Player
interactions from a controller finished executing.

PiperOrigin-RevId: 642189491

* Add stress test for image transform

PiperOrigin-RevId: 642213253

* Reduce flakiness of getProgress tests around percentage values.

There is no requirement for the first progress value to be 0, if
progress is made instantly.

PiperOrigin-RevId: 642245457

* Add SessionError and use it in service results

This change adds `SessionError` and uses it in `SessionResult`
and `LibraryResult` to report errors to callers.

Constructors and factory method that used a simple `errorCode` to
construct error variants of `SessionResult` and `LibraryResult`
have been overloaded with a variant that uses a `SessionError`
instead. While these methods and constructors are supposed to be
deprecated, they aren't yet deprecated until the newly added
alternative is stabilized.

PiperOrigin-RevId: 642254336

* Remove `Bundleable` type & `Bundleable.Creator<Foo> CREATOR` fields

This interface is not used in the library. Callers can use the
`Bundle toBundle()` and `static Foo fromBundle(Bundle)` methods
defined directly on each type instead.

PiperOrigin-RevId: 642271609

* Increase sleep duration when video encoder ends with no output frames

This duration was not sufficient on a Pixel 3a

PiperOrigin-RevId: 642276212

* Publish the media3 session controller test app

Issue: androidx/media#78
PiperOrigin-RevId: 642277900

* Add `@CallSuper` to `TrackSelectionParameters.toBundle`

PiperOrigin-RevId: 642292328

* Clean up unecessary zero-arg `toBundle()` methods

These are no longer needed now that the `Bundleable` interface has been
removed. Public methods are deprecated, package-private ones are
removed. Callers are migrated in both cases (except where tests
explicitly exist for the deprecated method).

PiperOrigin-RevId: 642294451

* Propagate media3 session extras to media1 `PlaybackStateCompat`

PiperOrigin-RevId: 642299178

* Update `@UnstableApi` docs with a link to DAC

The code snippets in javadoc don't render correctly due to
https://issuetracker.google.com/342557694

PiperOrigin-RevId: 642309632

* Make ExportResult.processedInputs package private

Rational for keeping field package private:
Since the overall export already very complex, by looking at the output file
its hard to know if the desired processing happened or not. Since we now
support sequences with "n" media items, its even more important to know if all
the media items were processed or not.
Although the field exposes implementation details, it seems ok as we get benefit of detailed testing.

PiperOrigin-RevId: 642337888

* Pass clock to release control from sink provider

In order to do that, make the VideoSink nullable in MCVR.

We want to avoid calling VideoFrameReleaseControl.setClock directly
from MCVR when the sink is enabled. The goal is to handle all the
communication with the release control from the sink/sink provider.

PiperOrigin-RevId: 642542063

* Remove useHdr from overlaySettings

useHdr is unused option and doesn't make sense is the dynamic range of the overlay and video must match

Also reorders and adds javadoc in line with coding conventions

PiperOrigin-RevId: 642555396

* Make ImageOutput clearable

It's currently not possible to remove a previously set image output
on ExoPlayer, although the underlying renderer already supports
receiving null to clear the output. Marking the parameter as
nullable allows apps to clear it as well.

PiperOrigin-RevId: 642569081

* Suppress warning about unused return value

The return value is intentionally unused because the check
is only done to see if the reflection operation succeeds.

PiperOrigin-RevId: 642579373

* Fix minor timestamp handling issue

- Video release should check for buffer timestamp (which is renderer-offsetted), rather than the frame timestamp
- ImageRenderer should report ended after all of it's outputs are released, rather than when finished consuming its input.

Add tests for timestamp handling

PiperOrigin-RevId: 642587290

* Sync and map fatal and non-fatal errors from and to the legacy session

A fatal `PlaybackException` is mapped to a legacy playback state
in state `STATE_ERROR` with error code, message and extras. A
non-fatal error sent to controllers with `MediaSession.sendError`
is synced to the legacy session by setting error code and message
and merging the extras while preserving the rest of the state in
sync with the session player.

Vice versa, a `MediaController` connected to a legacy session receives
fatal errors through `Player.onPlayerErrorChanged()` and non-fatal errors
through `MediaController.Listener.onError()`.

Error codes are mapped in `LegacyConversions`. Values of error codes
in `@SessionError.ErrorCode` come from `@PlaybackExceptino.ErrorCode`
with the exception of `@SessionError.ERROR_IO` and
`@SessionError.ERROR_UNKNOWN`. These already exist in
`@PlaybackException.ErrorCode` and are mapped accordingly to avoid
semantic duplicates.

PiperOrigin-RevId: 642595517

* Refactors the seek test and add more test cases

PiperOrigin-RevId: 642597298

* Make nit improvements in muxer code

PiperOrigin-RevId: 642599475

* Move usages of allowReleaseFirstFrameBeforeStarted inside sink

PiperOrigin-RevId: 642611061

* Add image samples and track selection to demo app

Also move the track selection header strings to the demo app
as they are only used there (except for audio, which stays in UI)

PiperOrigin-RevId: 642616037

* Clarify semantics of MSG_SET_IMAGE_OUTPUT

The value can be null, which isn't mentioned in the docs yet

PiperOrigin-RevId: 642622583

* Split muxer android tests between parameterized and non parameterized

This will be helpful for adding more tests which need not to
be parameterized.

PiperOrigin-RevId: 642638702

* Fix single sample handling ProgressiveMediaPeriod when disabling tracks

When deselecting the single sample track and later re-selecting this
track, the current shortcuts in ProgressiveMediaPeriod don't handle
this case correctly and cause assertion failures.

In particular, this change fixes 3 issues:
 1. When re-selecting the single sample track, we have cleared the
    SampleQueue and need to reload the sample. The existing shortcut
    should only be applied to avoid the reload when starting from a
    non-zero position.
 2. When de-selecting the track, ProgressiveMediaPeriod is left in
    an inconsistent state where the sample queues are empty but
    loadingFinished is still true. Fix this by resetting
    loadingFinished to false.
 3. When seeking, we avoid reloading the stream if we can keep
    inside the existing samples. This logic assumes that all
    remaining samples will continue to be loaded in the queue.
    This condition isn't true though for single sample tracks
    that have been de-selected. They appear to support the seek
    inside the queue (=no seek necessary, always supported), but
    still require a new load if there is no ongoing one to load
    the sample. Fix this by checking this implicit assumption
    (still loading, or loading finished).

PiperOrigin-RevId: 642650248

* Schedule refresh for all the playing playlists for HLS live stream

Issue: androidx/media#1240
PiperOrigin-RevId: 642927082

* Add a checkbox to use Media3 muxer in Transformer demo app

PiperOrigin-RevId: 642941439

* Fix bug where enabling CMCD for HLS live streams causes error

Determine `nextMediaSequence` and `nextPartIndex` based on the last `SegmentBaseHolder` instance, as it can update `mediaSequence` and `partIndex` depending on whether the HLS playlist has trailing parts or not.

Issue: androidx/media#1395
PiperOrigin-RevId: 642961141

* Fix AVI extractor WAVE format extraction

Allow `WAVEFORMAT` (in addition to `WAVEFORMATEX`), which omits initialization
data.

Fix reading of bits per sample to use little endian byte order like the other
reads. See also the wave format docs linked from
https://learn.microsoft.com/en-us/windows/win32/directshow/avi-riff-file-reference#avi-stream-headers.

PiperOrigin-RevId: 642962893

* Effect: Delay end of stream until all frames are rendered.

Previously, if renderFramesAutomatically = false, DefaultVideoFrameProcessor
may call onInputStreamProcessedListener after all frames have been
onOutputFrameAvailableForRendering, but before they had all been rendered and
freed.

Delay onInputStreamProcessedListener being called and subsequent DVFP
reconfiguration of effects, until all frames are rendered.

Tested using exoplayer setVideoEffects demo with playlist

PiperOrigin-RevId: 642963100

* Remove the code to restore current output position

Muxer never uses latest output position but always writes to
a specific location, so restoring its position does not add any
value.

PiperOrigin-RevId: 642996941

* Refactor mixedInput E2E test to use parameterized logic.

This CL is a pure refactor of the existing tests, iterative changes
will be done in follow-ups.

The only logic change is having all MP4 assets have the presentaton of
height=360. The old code had some with height 480.
PiperOrigin-RevId: 643020773

* Add a drop-in replacement for `MediaExtractor` in Media3

This change introduces a new class in Media3 `MediaExtractorCompat`, designed to be a drop-in replacement for platform `MediaExtractor`. While not all APIs are currently supported, the core functionality for the most common use cases of `MediaExtractor` is now available. Full API compatibility will be achieved in the future.

PiperOrigin-RevId: 643045429

* SpeedChangingAP: synchronize fields accessed from multiple threads

PiperOrigin-RevId: 643046385

* support hdr text overlays

adds luminance multiplier to allow the luminance (i.e. brightness) over a text overlay to be scaled

PiperOrigin-RevId: 643047928

* Parse 'max num reorder samples' values from h.264 and h.265 videos

This value is used in a follow-up change to re-order SEI messages
containing CEA-6/708 data from decode order to presentation order.

PiperOrigin-RevId: 643296338

* Remove PreloadConfiguration from public API

This feature isn't completed, so we should remove
the public facing API to avoid confusion.

#minor-release

PiperOrigin-RevId: 643318692

* Image support in PlayerView

Images are rendered into an ImageView (on top of the video shutter).
The image view is set to the images emitted by ExoPlayer's ImageOutput
and cleared when there is no longer a selected image track.

In order to keep the existing behavior of video tracks to only clear
the old output once the new first frame is rendered (avoiding short
periods of black between playlist items), we have to reorder this code
slightly to make it work for video and images. Both are treated in the
same way. If both are enabled, video takes precedence.

As the UI module only depends on the common module, we can't direcly
add the ImageOutput to ExoPlayer. This is done via reflection if
the provided Player is an ExoPlayer.

#cherrypick

PiperOrigin-RevId: 643320666

* Rollback of https://github.com/androidx/media/commit/cd9b914c423900edcfe1e64bfb775a601bc23844

PiperOrigin-RevId: 643329324

* Add `demo-composition` to `settings.gradle`

This was missed as part of https://github.com/androidx/media/commit/0e5a5e029455ea5e456cbb6028e9be1e87cdfe09

#cherrypick

PiperOrigin-RevId: 643331307

* Add more cases for parameterized video android tests.

PiperOrigin-RevId: 643346610

* Write moov box at the start of the file if possible

This is done by reserving some space for moov box at the start of the file and writing it there if it fits. If it doesn't fit, then it is written at the end of the file.

PiperOrigin-RevId: 643944698

* Reset release control from video sink when enabled

PiperOrigin-RevId: 643950097

* Merge pull request #1437 from MGaetan89:add_exoplayer_setMaxSeekToPreviousPosition

PiperOrigin-RevId: 643987403
(cherry picked from commit 67a7b41fa7453ac4fa06d4d9c7df68e72fbc58ea)

* Fix linter errors in `DemoMediaLibrarySessionCallback.kt`

PiperOrigin-RevId: 644002147
(cherry picked from commit ed07ac5d7de31a610b63d51242fc383d4bad9ffe)

* Update release notes for 1.4.0-beta01

#cherrypick

PiperOrigin-RevId: 644351958
(cherry picked from commit 794731607d2fd267d45f8f587812d11e655e59df)

* Version bump to media3:1.4.0-beta01

PiperOrigin-RevId: 644352776
(cherry picked from commit c07bbd333ca88ec93a9efdfcc037e406e1aa4099)

* Improve audio focus handling tests with ExoPlayer

There are a lot of tests for AudioFocusManager in isolation,
but almost none for the handling in ExoPlayer.

Add test coverage for all the common cases, including some
currently broken behavior that is indicated by TODOs.

PiperOrigin-RevId: 644319251
(cherry picked from commit e20e94fde277a8e9e9a09e502a5760772e9eb006)

* Audio focus player command clean up

The 'player commands' returned to ExoPlayerImpl instruct the
player on how to treat the current audio focus state.

The current return value when playWhenReady==false is misleading
because it implies we are definitely not allowed to play as if
we've lost focus. Instead, we should return the actual player
command corresponding to the focus state we are in.

This has no practical effect in ExoPlayerImpl as we already
ignore the 'player command' completely  when playWhenReady=false.

To facilitate this change, we also introduce a new internal
state for FOCUS_NOT_REQUESTED to distinguish it from the state
in which we lost focus.

#cherrypick

PiperOrigin-RevId: 644416586
(cherry picked from commit 66c19390e2b31106a2a467366f7945c37a273fc6)

* Allow session activity to be set per controller

PiperOrigin-RevId: 644693533
(cherry picked from commit 856d394c281cabc2b1932b003d752631549e7231)

* Use a localized fallback message for known error codes

In the case of a legacy session app providing an error
code different to `ERROR_UNKNOWN` without an error
message, a localized fallback message is provided
instead of ignoring the error.

#cherrypick

PiperOrigin-RevId: 644704680
(cherry picked from commit 6cc6444dd9be34ea3a40d2bb296247178ee86b1d)

* Use `SurfaceSyncGroup` to ensure resize transaction isn't dropped

This workaround is only applied on API 34, because the problem isn't
present on API 33 and it is fixed in the platform for API 35 onwards.

Issue: androidx/media#1237

#cherrypick

PiperOrigin-RevId: 644729909
(cherry picked from commit 968f72fec6d5452a34976824ac822782eeeb8f45)

* Move playWhenReadyChangeReason inside PlaybackInfo

This helps to keep the reason always together with the state it
is referring to, avoiding any side channels and making sure there
are no accidental inconsistencies.

#cherrypick

PiperOrigin-RevId: 644969317
(cherry picked from commit c0abd6f91ec62bc5347561c3f3174b3a660a7ba6)

* Clarify that onPlayWhenReadyChanged can be called again with new reason

Sometimes the reason for the current state may change. If we don't
report this again, users have no way of knowing that the reason
changed.

Also adjust ExoPlayerImpl and MediaControllerImplBase accordingly.
SimpleBasePlayer already adheres to this logic.

#cherrypick

PiperOrigin-RevId: 644970236
(cherry picked from commit 1d26d1891e6e3cd49a4b479723b8aba7be746c1c)

* Fix audio focus handling in ExoPlayerImpl

Some cases are not handled correctly at the moment:
 - Pausing during suppressed playback should not clear the
   suppression state.
 - Transient focus loss while paused should be reported as
   a playback suppression.

Issue: androidx/media#1436
#cherrypick
PiperOrigin-RevId: 644971218
(cherry picked from commit e84bb0d21cd114c5b726c6c3d799b194df068a14)

* Remove direct `AspectRatioFrameLayout` usage from session demo

This class is a lower-level UI component that isn't directly needed if
apps are using `PlayerView` to handle their video output (it is used as
an implementation detail of `PlayerView`).

Removing its unecessary usages from this demo avoids developers copying
this as an example when building their own apps.

#cherrypick

PiperOrigin-RevId: 644972454
(cherry picked from commit cb8f87e05e73d72ce7d2c405c4c5899dc31ade5e)

* Add null-check to `PlayerView` to avoid NPE in edit mode

Previously we assumed that `surfaceSyncGroupV34` was always non-null on
API 34, but this isn't true in edit mode. Instead we add an explicit
null-check before accessing it. We don't need to null-check it at the
other usage site because we are already null-checking `surfaceView` (via
`instanceof` check).

Issue: androidx/media#1237

#cherrypick

PiperOrigin-RevId: 645008049
(cherry picked from commit 99803066ea6d375f5527e0953aad4ecd91fbac3c)

* Use HttpEngineDataSource when supported in demo app.

HttpEngineDataSource is the recommended HttpDataSource when it is available. It uses the [HttpEngine](https://developer.android.com/reference/android/net/http/HttpEngine).

#cherrypick

PiperOrigin-RevId: 646051562
(cherry picked from commit e591c37b1e60d5138a612040efba752b7b7ab8e5)

* Fix flakiness in MediaCodecVideoRendererTest

The test is flaky because the decoding process in the renderer
depends on some timing from MediaCodec beyond our control and
the new keyframe added in the test is sometimes 'dropped' when
it arrives too late.

We can fix this by controlling the test progress a bit more
tightly: first rendering with the same current time until the
key frame is processed and then start increasing the time
until we've reached the end.

#cherrypick

PiperOrigin-RevId: 646064352
(cherry picked from commit ada4dc982fac32a57db069341f1f51c6a3bb6cc5)

* In DemoUtil, don't set cookie handler when using HttpEngineDataSource.

HttpEngine does not support cookie storage.

#cherrypick

PiperOrigin-RevId: 646084702
(cherry picked from commit bb568b5150781457fe22c3d4a877e8c99bddc4a2)

* Add fail-early checks for TrackSelectorResult correctness

The two arrays need to have the same length and the selection
must match in their nullness (unless for TYPE_NONE
renderers). Clarify this more clearly in the docs and add
new asssertions for it. This avoids that the player is failing
in obscure ways much later.

Issue: androidx/media#1473
#cherrypick
PiperOrigin-RevId: 646086833
(cherry picked from commit 71ef848ec3fa4409f3cdf797226c2a51ecd15da6)

* Merge pull request #1416 from khouzam:customFormat

PiperOrigin-RevId: 646121082
(cherry picked from commit b026271c8438f227f4e15e0eaedd7223b4492037)

* Merge pull request #1479 from dryganets:sdryanets/fix-handler-usage

PiperOrigin-RevId: 646402268
(cherry picked from commit 04667284975e9fc71c137b4a8c133ca5cc626c19)

* Add guard against additional tracks reported by Extractors

Extractors should not report additional tracks once they called
ExtractorOutput.endTracks. This causes thread safety issues in
ProgressiveMediaPeriod where the array of sample queues is
extended while the playback thread accesses the arrays.

Detecting this problem early is beneficial to avoid unexplained
exceptions later one. In most cases where this may happen (namely
TS extractors finding new tracks), it's better to ignore the new
tracks instead of failing completely. So this change adds a
warning log message and assigns a placeholder output.

Note: The same workaround already exists in HlsSampleStreamWrapper
and MediaExtractorCompat.

Issue: androidx/media#1476
#cherrypick
PiperOrigin-RevId: 646427213
(cherry picked from commit 18e631ff790e0e9b3630ea8e36efb468474d0877)

* Rename DummyTrackOutput and DummyExtractorOutput

#cherrypick

PiperOrigin-RevId: 646434450
(cherry picked from commit 867410fece006122820e26be4e0bea87309a4f2e)

* Use `MediaCodec.stop()` before `release()` for surface switching bug

ExoPlayer used to call `stop()` before `release()`. This was removed in
<unknown commit>.

A framework bug introduced in Android 11 (API 30) resulted in some
DRM -> clear transitions failing during `MediaCodec.configure()`. An
investigation in Issue: google/ExoPlayer#8696 and b/191966399 identified that this was
due to `release()` returning 'too early' and the subsequent
`configure()` call was then trying to re-use a `Surface` that hadn't
been fully detached from the previous codec. This was fixed in
Android 13 (API 33) with http://r.android.com/2094347.

ExoPlayer worked around the framework bug by adding an arbitrary 50ms
sleep after a failed codec initialization, followed by retrying. This
was enough to resolve the problem in the test scenario on a OnePlus
AC2003.

Issue: androidx/media#1497 points out that 50ms might not be the appropriate delay
for all devices, so it's an incomplete fix. They suggested re-adding the
`MediaCodec.stop()` call instead. This also reliably resolves the issue
on the OnePlus AC2003 (with neither workaround in place, the problem
repros almost immediately).
PiperOrigin-RevId: 646461943

(cherry picked from commit 5fcc7433a1f188c9644d4430a828a49d06c993cb)

* Use `removeKey` method instead of setting `null` for KEY_CODECS_STRING

Setting a `null` value doesn't remove the key as expected per the `MediaFormat` API documentation, using the `removeKey` method instead which is only available starting API level 29.

PiperOrigin-RevId: 646462402
(cherry picked from commit 12c42585d25d4c51bc5d022aa282d4d36b8e1ff2)

* Merge pull request #1487 from colinkho:main

PiperOrigin-RevId: 646917527
(cherry picked from commit 6244d8605f48d1b548201a8a964a9879961b3c68)

* Send pending updates before adding discontinuity for error

When handling a playback error that originates from a future item in
the playlist, we added support for jumping to that item first,
ensuring the errors 'happen' for the right 'current item'.
See https://github.com/androidx/media/commit/79b688ef30d4a3306d4d321ddf4464b428074ea2.

However, when we add this new position discontinuity to the
playback state, there may already be other position discontinuities
pending from other parts of the code that executed before the
error. As we can't control that in this case (because it's part
of a generic try/catch block), we need to send any pending
updates first before handling the new change.

Issue: androidx/media#1483
PiperOrigin-RevId: 646968309
(cherry picked from commit 727645179b3c26e495540f4445216fd035cc7654)

* Cache audio timestamp frame position across track transition reset

Upon track transition of offloaded playback of gapless tracks, the framework will reset the audiotrack frame position. The `AudioTrackPositionTracker`'s `AudioTimestampPoller` must be made to expect the reset and cache accumulated …
@androidx androidx locked and limited conversation to collaborators Aug 19, 2024
@androidx androidx unlocked this conversation Sep 4, 2024
@icbaker
Copy link
Collaborator

icbaker commented Sep 4, 2024

I'm afraid that due to the unintended side-effects of the workaround reported in #1594 (for non-Compose use-cases), I've had to make the workaround opt-in (rather than automatically enabled) in 87bd9ba.

This means that those of you using PlayerView inside AndroidView will need to call playerView.setEnableComposeSurfaceSyncWorkaround(true) to enable the workaround. This change will probably go live in 1.5.0-beta01.

v-nova-ci pushed a commit to v-novaltd/androidx-media that referenced this issue Sep 4, 2024
The workaround causes issues with XML-based shared transitions, so we
can't apply it unilaterally.

Issue: androidx#1594

Issue: androidx#1237
PiperOrigin-RevId: 670960693
@androidx androidx locked and limited conversation to collaborators Sep 5, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests