Add an API for predecoding images

[domenic]

This thread is a spinoff of #1920. Around #1920 (comment) we came to the conclusion that there were two distinct use cases being addressed: the "deferring" use case, served by the content attribute discussed in #1920, and the "predecoding" use case, which I want to discuss here.

The tentative idea is something like:

var img = new Image();
img.src = '...';
// The decoded image is guaranteed to be kept in memory until the frame
// after the decode promise resolves.
img.decode().then(function() {
  // This is guaranteed to paint the image without flicker or decoding jank.
  document.body.appendChild(img);
});

In #1920 (comment) there was discussion as to whether scaling/cropping should be part of this, but it sounded like probably not.

Note that something similar can already be accomplished on the platform today, by using ImageBitmap and canvas:

createImageBitmap(img).then( image => {
     var canvas = document.createElement('canvas');
     canvas.getContext('bitmaprenderer').transferFromImageBitmap(image);
     document.body.appendChild(canvas);
});

This has a couple potential drawbacks:

• Less a11y/semantic support (e.g. requires adding aria-label inferred from the original img's alt; does not get access to right-click save image as)
• The decoded image is not evictable, or at least not as easily, as @junov noted in "decode" attribute on <img> #1920 (comment)

But in the end maybe it's enough to argue for not adding a new API for something that can already be done. What do people think?

/cc @vmpstr @smfr @ojanvafai @grorg

[smfr]

This has a couple potential drawbacks

More:

• Twice the memory use (you pay for the canvas backing store, and the decoded image).
• Maybe main thread decoding still? I'm not clear on whether createImageBitmap() is supposed to do non-main thread decoding.

[junov]

Twice the memory use (you pay for the canvas backing store, and the decoded image).

Actually, the image is "transferred", meaning that the ImageBitmap object is neutered, and the buffer that it once held became the canvas backing store.

Maybe main thread decoding still?

That is implementation dependent, the API is async and was deliberately spec'ed in a way that gives implementers the freedom (but not the obligation) to schedule the work on another thread.

[vmpstr]

To try and move this forward, I've put together a small demo that demonstrates how this API would be used and the results you might see:
https://drive.google.com/open?id=0B5cKk7MxQ2LsUUotR3RuWUlpSXc

In the video, an arrow keeps spinning using a raf animation. On the first full revolution, the image is "prepared" and appended into the DOM. In the first example ("sync"), the prepare function is as follows:

function prepareImage() {
var img = new Image();
img.src = "nebula.jpg";
img.onload = function() { document.body.appendChild(img); };
}

This waits until the image is loaded, and appends it. You can see a noticeable jank which is the result of an image decode.

In the second example ("async"), the prepare function is as follows:

function prepareImage() {
var img = new Image();
img.src = "nebula.jpg";
img.decode().then(function() { document.body.appendChild(img); });
}

This waits until the image is loaded and decoded, then appends it into the DOM.
(all of the code and images are here: https://drive.google.com/open?id=0B5cKk7MxQ2LsNVQzaVVoWHRnVHc)

The decode here uses similar code path to how images are typically decoded and locked (in the compositor). So the result is that when the decode promise resolves, the image is locked in the cache that is used at raster time. It is then unlocked after a few frames. Subsequent appends/raster of this image are hence not guaranteed to be jank-free.

One issue I found with this approach is that the image can be arbitrarily large. In Chromium, we have a limit to how much memory can be locked for an extended period of time. If the image is large enough to breach the limit, the promise resolves but the result is still jank. In other words, the effect is similar to simply waiting for the onload callback and appending the image.

I think it's reasonable to have a limit to how big of an image the user requests. Currently the promise still resolves, but it can also fail giving the user the opportunity to do something different. Does this seem reasonable?

Overall, should we move forward with formalizing this proposal?

[smfr]

That's a good demo. I think it's OK if UAs are allowed by break the contract for overly large images. I would like to move forward with the proposal.

[domenic]

Ah yay, glad people are still wanting to work on this. I felt that when we split into two threads things kind of fizzled. Glad to see that's not the case.

I'm happy to sign up for the spec work here, probably next week (this week is BlinkOn). If someone wants to get things started with some informal spec or algorithms or similar, even just in this thread, that would be a great headstart. E.g. what kind of phrasing you would use to enforce any normative requirements (of either the "must" or "should" variety).

[junov]

This seems very reasonable and it solves the problem in a way that createImageBitmap can't. I think the spec should be explicit about the fact that this API provides a "soft guarantee", especially if all implementations would otherwise be breaking the contract under RAM pressure.

[ojanvafai]

General +1 from me. I agree there are reasonable times to reject the promise (e.g. excessively large images). I think we probably want to experiment in the wild before we codify the cases under which this promise rejects, but in the meantime we should probably have a non-normative note in the spec that calls out the large image case.