-
Notifications
You must be signed in to change notification settings - Fork 3.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HTML: Add test for <video> dispatching resize event and displaying variable video track width and height #17821
Comments
Related w3c/csswg-drafts#4266 |
@foolip How to determine which specification owns resizing of |
@guest271314 that's defined in https://html.spec.whatwg.org/multipage/media.html, searcg for "named resize" to find the two places the event is dispatched. One way to find the relevant spec is by following spec links at the bottom of MDN pages, in this case https://developer.mozilla.org/en-US/docs/Web/API/HTMLVideoElement. |
Regarding the topic at hand, I didn't know it was even possible to construct a WebM video with variable dimensions, since the width and height is part of the metadata in the beginning of the file. It is of course possible to put video of a different size in the file and for the size to change at any keyframe, but is this a valid WebM file? Ogg is a format where the up-front metadata is optional and which I think would also trigger this problem. |
@foolip Yes, the WebM file is valid, in brief see https://gitlab.com/mbunkus/mkvtoolnix/issues/2582#note_189520756; https://mkvtoolnix.download/doc/NEWS.md
It is argued that H264 and AVC1 codecs are invalid for WebM files, though that has not stopped Chromium authors from implemented the codec for In fact, the only way that have been able to get Chromium, using a WebM file encoded at Chromium, to display variable pixel dimensions at Chromium is to use H264 or AVC1 codecs (openh264). |
Alright, I would have thought that producing a video stream like that using VP8 or VP9 would be straightforward, but maybe the decoder rejects it? |
@foolip The issue is that Chromium does not display the variable pixel dimension video at HTML This https://plnkr.co/edit/Axkb8s?p=info is the only code that have been able to compose which outputs variable pixel dimensions at HTML This demonstrates the issue with HTML If you follow the links in the issues you will see that it is not immediately clear which specification owns and is responsible for HTML The decoder at Chromium apparently only reads the initial frame dimensions when the codec is VP8 or VP9, though not for H264 or AVC1. Again, when trying to determine which specification owns and controls resizing of HTML |
@foolip Re
see https://github.com/guest271314/native-messaging-mkvmerge. |
@foolip For more details and to further demonstrate the issue see https://bugs.chromium.org/p/chromium/issues/detail?id=983777#c20, https://bugs.chromium.org/p/chromium/issues/detail?id=983777#c21, https://bugs.chromium.org/p/chromium/issues/detail?id=983777#c22. |
@foolip Interestingly Chromium does display variable pixel dimensions and dispatch The consequence is that It has been somewhat challenging to determine which specification and implemented API is responsible for the issue of resizing HTML Presently am not certain by reading the current iteration of the relevant specifications; even the new
|
It's the resize event in https://html.spec.whatwg.org/multipage/media.html that's relevant here. HTML together with container format specs like WebM are the specs that define what should happen. If both WebM and MPEG-4 support resizing video streams, then adding a test for this would be straightforward. |
Do we disregard or edit the language at MDN https://developer.mozilla.org/en-US/docs/Web/API/Window/resize_event?
Am currently banned (evidently indefinitely) from WHATWG. Am still willing to help or write the tests if a member of WPT files the PR. |
Yes, https://developer.mozilla.org/en-US/docs/Web/API/Window/resize_event should be disregarded for two reasons. First, its concerns the If you can produce media files for this, then getting a test into wpt should be straightforward. |
HTML: Add test for <video> dispatching resize event and displaying variable video track width and height #17821 web-platform-tests/wpt#17821 All files produced using the code at https://github.com/guest271314/MediaFragmentRecorder/blob/webrtc-replacetrack/MediaFragmentRecorder.html The codecs used are - "video/webm;codecs=vp8,opus" - "video/webm;codecs=vp9,opus" - "video/webm;codecs=h264" - "video/x-matroska;codecs=h264" (see https://bugs.chromium.org/p/chromium/issues/detail?id=999580; https://plnkr.co/edit/WUVbjz?p=info) Width and height of the encoded frames in the files in order WidthxHeight 768x576 480x240 640x360 400x300 1280x720
|
I guess MediaRecorder only supports WebM/Matroska, but is it possible to have an MPEG-4 container with video that changes its size mid-stream? The ideal test video files would be one that is red and 400x300 for 1 second, and then green and 300x150 for 1 second. The exact size isn't very important, but something smallish is good. The size of the videos in bytes should be as small as possible, it should be possible to get them below 100kb I think? Large test files makes the tests slower and makes it more likely that failures are due to something on the network side rather than the media decoding pipeline. |
Cannot fathom why that would not be possible. Would that be dissimilar from the Matroska container width H264 (openh264) codec at the previous link?
Created the various files that have previously tested to demonstrate the issue with interoperability. The variation of the media files and display of those files at HTML |
|
Compare the output of |
There are a lot of video files flying around now :) Is there a pair of small video files, one WebM and the other something that Safari supports, that could be used to test the resize event behavior? Please point directly to which ones you're proposing. |
The WebM with VP8 codec file is in the .zip archive at previous post. |
This was recorded at Firefox https://github.com/web-platform-tests/wpt/files/3588824/firefox_mediarecorder_1s_red_1s_green_variable_pixel_dimensions.webm.zip and should dispatch resize event at Chromium. The file recorded at Chromium (at this post #17821 (comment)) currently will not dispatch resize event at Chromium (TL;DR https://bugs.chromium.org/p/chromium/issues/detail?id=1001948). |
Fixes #17821. New media files are from that issue.
Cool, thanks! Putting that in a video element and playing in Chrome, I actually do see resize events. I don't think this is tested in wpt however, so turning it into a test is a good idea. I don't have time to finish it now, but #19030 is the structure for a test. |
Fixes #17821. New media files are from that issue.
Alright, I finished #19030, but it seems to work in Chrome, so not covering what you'd hoped for I think. |
Do not entertain "hope". The expected result is for the video produced by Chromium to resize at Chromium. You must have used the WebM file recorded at Firefox for the test, correct? |
The
Have you tried running the test with the WebM file generated by Chromium implementation of |
I'll have to double-check, I think I used the file in #17821 (comment) |
The file was generated by Firefox https://bugzilla.mozilla.org/show_bug.cgi?id=1177767, see
For a comprehensive test would include the file produced by Chrome as well. |
This demonstrates the issue at Chromium https://plnkr.co/edit/n1rGNe?p=info. |
@foolip FWIW Proposed a specification change to HTML |
From your reading of HTML Standard does the specification
mandate that Or, does the HTML specification (WHATWG or W3C) not mandate that the |
The resize event should be fired when the intrinsic size changes, but it comes down to the definition of intrinsic width and intrinsic height, which is ultimately a question for the container and video format. I don't know much about this, but there are probably video codecs that internally downscale to improve quality/bit, where a correct player should upscale the video frames. This is somewhat similar to the video having a different aspect ratio than the raw video frames. So, what should happen with the video streams you've produced depends on other specs that HTML, but my best guess is that resizing and firing a resize event is the right thing. |
@foolip The reason asked the question are the contents of this comment https://bugs.chromium.org/p/chromium/issues/detail?id=992235#c29
and specifically 3)
which implies that HTML does not consider or currently unambiguously define "intrinsic width" and "intrinsic height" as encompassing variable resolution frames encoded in the container which the developer is reasonably being expected to be decoded and displayed at HTML The containers Matroska and WebM and the codecs VP8 and VP9 do support encoding variable resolution frames. There is no dispute of facts there. The issue is the arbitrary decision by implementer of HTML The question is simple: where the specification currently states
do the terms "intrinsic width" and "intrinsic height" in the HTML specification apply to the underlying variable resolution encoded video frames? |
Fixes #17821. New media files are from that issue.
Fixes #17821. New media files are from @guest271314: #19030 (comment)
Fixes #17821. New media files are from @guest271314: #19030 (comment) #19030 (comment)
Fixes #17821. New media files are from @guest271314: #19030 (comment) #19030 (comment)
Fixes #17821. New media files are from @guest271314 in the review.
Fixes #17821. New media files are from @guest271314 in the review.
…estonly Automatic update from web-platform-tests Test video resizing during playback (#19030) Fixes web-platform-tests/wpt#17821. New media files are from @guest271314 in the review. -- wpt-commits: 3a65d13bb72841bfc61ed8365b09bf3ee7bde4fa wpt-pr: 19030
…estonly Automatic update from web-platform-tests Test video resizing during playback (#19030) Fixes web-platform-tests/wpt#17821. New media files are from @guest271314 in the review. -- wpt-commits: 3a65d13bb72841bfc61ed8365b09bf3ee7bde4fa wpt-pr: 19030
Consider a WebM video where the input video track contains variable width and height (differing pixel dimensions)
Firefox 70 dispatches
resize
event three times, once for the initial width and height and once each for the subsequent different pixel dimensions encoded into the WebM file and displays the correctvideoWidth
andvideoHeight
corresponding to the variable input pixel dimensions.Chromium 77 dispatches
resize
event one time, once for the initial width and height of the video track and does not display the correctvideoWidth
andvideoHeight
corresponding to the variable input pixel dimensions.Add tests for
HTMLVideoElement
to verify thatresize
event is dispatched for each change to width and height of video track of media file; and<video>
element displays the correctvideoWidth
andvideoHeight
corresponding the potentially variable input pixel dimensions encoded within the media file.The text was updated successfully, but these errors were encountered: