Spec optional conversion to RGB in VideoFrame.copyTo()

[Djuffin]

This change should address the issue of pixel format conversion in VideoFrame.copyTo().

---

Preview | Diff

[Djuffin]

There are probably some missing details, typos and omissions.
But I'd love to get some feedback on the general approach in this change.

[Djuffin]

@padenot Could you please take a look? I'd love to merge this PR some time soon. We've talked about it at TPAC last September.

[padenot]

This isn't something that we can do in spec, using other specs directly like that. All the objects you use can have been overridden in the global this algorithm is running in.

We should use internal stops for canvas instead.

[Djuffin]

I'm probably missing something here. Could you elaborate a bit?

Isn't it what we do in the Rendering section?

referring to CanvasDrawImage drawImage()

[padenot]

No, that's unrelated, we reference it in your link. What happens when drawImage is called with a VideoFrame is well-defined there. Here you're using another Web Platform object from the innards of our spec.

e.g. what happens if in the document we're doing this conversion, I've maliciously done:

window.OffscreenCanvas.prototype = function() { alert("what now"); };

before anything from our spec run?

[Djuffin]

This algorithm does not refer to any particular js code. It explicitly says what object needs to be created by the user agent. That's why I don't think that global overrides should concern us here. With a similar approach half of this spec can be put into question with code like

window.VideoFrame.prototype = function() { alert("what now"); };

[Djuffin]

I added a note clarifying that UAs don't have to do it via literally creating an OffscreenCanvas as long as the result is the same. I hope this eliminates possible confusion.

If you still think that the wording leaves room for misinterpretation, I'm very much open to suggestions how to rewrite it.

[Djuffin]

Thanks for the links!

I've noticed that drawImage() (step 6) does not specify any details about the rendering (sampling, color conversion etc). It's only saying that "The user agent may use any filtering algorithm".
So there isn't really an algorithm to extract and call.
I will have to rewrite it in a more abstract manner, similar to the text for drawImage()

[padenot]

We can add an export to something hand-wavy from canvas now, and merge what we want to do here, and then fix the canvas spec as well, your call.

[Djuffin]

I asked HTML spec owners about the best way to reuse canvas pixel format conversion algorithm.

[Djuffin]

I've looked at the rest of the WebCodecs spec. It's pretty light on the details when it comes to format and color spaces conversions. (for example Rendering and AudioData.copyTo )

So I rewrote "Convert to RGB frame" in the similar manner.
Please take another look.

[Djuffin]

@padenot gentle ping

[chrisn]

Looks good, but let's resolve the discussion on overridden globals (I don't mind if that's here or in a follow up PR).

[Djuffin]

@padenot PTAL

[padenot]

All good, thanks!

[Djuffin]

@aboba please take a look and merge if you don't have new comments.