-
Notifications
You must be signed in to change notification settings - Fork 85
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support for extracting a series of video thumbnails #146
Support for extracting a series of video thumbnails #146
Conversation
.../main/java/com/linkedin/android/litr/thumbnails/behaviors/MediaMetadataExtractionBehavior.kt
Outdated
Show resolved
Hide resolved
litr/src/main/java/com/linkedin/android/litr/thumbnails/ThumbnailExtractParameters.kt
Outdated
Show resolved
Hide resolved
litr/src/main/java/com/linkedin/android/litr/thumbnails/behaviors/ExtractionBehavior.kt
Outdated
Show resolved
Hide resolved
litr/src/main/java/com/linkedin/android/litr/thumbnails/VideoThumbnailExtractor.kt
Outdated
Show resolved
Hide resolved
litr/src/main/java/com/linkedin/android/litr/render/GlThumbnailRenderer.kt
Outdated
Show resolved
Hide resolved
Add support for priority Simplify API Update demo to use a RecyclerView, and in-memory cache Behavior now only responsible for retrieving the frame at original resolution
* Optional size of the generated thumbnail Bitmap. The video frame will be resized and scaled to fill this [destSize], preserving the video frame's | ||
* original aspect ratio. | ||
*/ | ||
val destSize: Point? = null, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since we are preserving original aspect ratio, passing a resolution (smaller dimension) should be better. What happens when destSize
's aspect ratio is different from video's? Or should we just scale the video to fit destSize
like when we do in transcoding?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let me add this image just so we're using the same language around this
passing a resolution (smaller dimension) should be better
Can you clarify what you mean by resolution? Do you mean we would perform "aspect fit" scaling instead of "aspect fill"?
What happens when
destSize
's aspect ratio is different from video's?
Right now it will scale with "aspect fill": the video frame will be resized and cropped to fill the destSize
area, and the aspect ratio of the video frame is preserved.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually, do you mean we should do this? --
- the resulting bitmap will always have the same aspect ratio as the source video (i.e. we won't add any whitespace/padding)
- instead of "destination size", we pass in something like "maximum width" and "maximum height"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Originally, I was thinking of doing this:
- thumbnail bitmap will have the aspect ratio of video
- by default, thumbnail will also have a resolution of source video (true frame preview)
- optionally we can pass the "resolution" parameter, thumbnail bitmap's smaller dimension will be equal to resolution. This is closer to how video is defined on Android - resolution + rotation in 90 degree increments.
For thumbnail use case, it is probably reasonable to aspect fill (AKA center crop) the thumbnail, as long as it is clearly defined in the API.
litr/src/main/java/com/linkedin/android/litr/thumbnails/ThumbnailExtractParameters.kt
Outdated
Show resolved
Hide resolved
litr/src/main/java/com/linkedin/android/litr/thumbnails/VideoThumbnailExtractor.kt
Outdated
Show resolved
Hide resolved
litr/src/main/java/com/linkedin/android/litr/thumbnails/queue/PriorityThreadExecutor.kt
Outdated
Show resolved
Hide resolved
[PR] Fix newlines for arguments
litr/src/main/java/com/linkedin/android/litr/render/GlThumbnailRenderer.kt
Outdated
Show resolved
Hide resolved
litr/src/main/java/com/linkedin/android/litr/thumbnails/VideoThumbnailExtractor.kt
Outdated
Show resolved
Hide resolved
…escriptive about how this should be used Remove unused imports Add OptIn annotations to demo classes using the experimental frame extract feature
/** | ||
* Provides the request lifecycle for extracting video frames. The specifics of extraction work are delegated to [FrameExtractBehavior]s. | ||
*/ | ||
@ExperimentalFrameExtractorApi |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is exposed externally? We should make these internal
Note to self: In general, we should change our build config to enforce explicit access qualifiers in Kotlin classes, to prevent clients from using stuff that is not meant to be exposed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah good catch, done. I also made the executor/task related classes internal (they were originally being injected so they were open)
import com.linkedin.android.litr.frameextract.FrameExtractMode | ||
import com.linkedin.android.litr.frameextract.FrameExtractParameters | ||
|
||
@ExperimentalFrameExtractorApi |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we still need this here? Does this get inherited from parent interface?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah I believe we still need this -- basically this is propagating the opt-in requirement to whoever is using this concrete behavior. If I leave this out, we'll get warnings in this class that it's using an experimental API. So we have two choices: opt in, or propagate. If we opt in here, then callers won't know this class is using experimental APIs. So instead we propagate the requirement. https://kotlinlang.org/docs/opt-in-requirements.html#propagating-opt-in
litr/src/main/java/com/linkedin/android/litr/render/SingleFrameRenderer.kt
Show resolved
Hide resolved
[PR] Add experimental annotation for SingleFrameRenderer
This PR adds support for extracting thumbnails/frames from video as
Bitmap
s, while applying an optional set of filters.VideoThumbnailExtractor
serves as the entry point, and requiresThumbnailExtractParameters
which defines the extraction behavior and other parameters.One behavior,
MediaMetadataExtractionBehavior
(based onMediaMetadataRetriever
), is included. The intent is that other behaviors can be added by consumers -- for example, consumers might opt to provide a custom FFmpeg-based or MediaCodec-based extractor, which may yield performance or functional improvements overMediaMetadataRetriever
, while still retaining the ability to apply video filters to extracted frames.Demo app has an example of creating a "film strip":