Skip to content

Commit

Permalink
Add img.decode() API for "predecoding" images
Browse files Browse the repository at this point in the history 
This API allows the author to request that the user agent decode the
image, preferably off the main thread, so that when the returned promise
fulfills, the image is ready to be inserted into the document, without
causing dropped frames due to decoding delay.

Notable design choices made:

- Rejects for non-active documents (or documents that become non-active)
- Immediately fulfills for vector graphics or other formats that do not
  require decoding
- Fulfills for animated images after all of their frames are loaded, not
  just the first, ensuring jank-free animation

Closes #2037.

This also includes the editorial change of giving a <dfn> to the "update
the rendering" step, and linking to it where it is mentioned.
  • Loading branch information
@domenic
domenic committed Jun 30, 2017
1 parent c9f904e commit af6b0e9
Showing 1 changed file with 167 additions and 4 deletions.
171 changes: 167 additions & 4 deletions source
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14953,7 +14953,7 @@ c-end = "-->"</pre>
If the style sheet referenced no other resources (e.g. it was an internal style sheet given by a
<code>style</code> element with no <code data-x="">@import</code> rules), then the style rules must
be <span>immediately</span> made available to script; otherwise, the style rules must only be made available
to script once the <span>event loop</span> reaches its <i>update the rendering</i> step.</p>
to script once the <span>event loop</span> reaches its <span>update the rendering</span> step.</p>

<p>A style sheet in the context of the <code>Document</code> of an <span>HTML parser</span> or
<span>XML parser</span> is said to be <dfn>a style sheet that is blocking scripts</dfn> if the
Expand Down Expand Up @@ -25285,6 +25285,8 @@ interface <dfn>HTMLImageElement</dfn> : <span>HTMLElement</span> {
readonly attribute boolean <span data-x="dom-img-complete">complete</span>;
readonly attribute USVString <span data-x="dom-img-currentSrc">currentSrc</span>;
[<span>CEReactions</span>] attribute DOMString <span data-x="dom-img-referrerPolicy">referrerPolicy</span>;

Promise&lt;void&gt; <span data-x="dom-img-decode">decode</span>();
};</pre>
</dd>
<dd w-nohtml>Uses <code>HTMLImageElement</code>.</dd>
Expand Down Expand Up @@ -25582,6 +25584,23 @@ interface <dfn>HTMLImageElement</dfn> : <span>HTMLElement</span> {

</dd>

<dt><var>image</var> . <code data-x="dom-img-decode">decode</code>()</dt>

<dd>

<p>Images usually exist in some encoded form; user agents need to decode them into raw pixels
before displaying them. This process can be relatively expensive.</p>

<p>This method causes the user agent to decode the image <span>in parallel</span>, returning a
promise that fulfills when decoding is complete. The decoded image data will then be readily
available for at least one frame after the fulfillment, ensuring that attempting to display the
image will complete without decoding delay.</p>

<p>The promise will be rejected with an <span>"<code>EncodingError</code>"</span>
<code>DOMException</code> if the image cannot be decoded.</p>

</dd>

<dt><var>image</var> = new <code data-x="dom-image">Image</code>( [ <var>width</var> [, <var>height</var> ] ] )</dt>

<dd>
Expand Down Expand Up @@ -25650,6 +25669,148 @@ interface <dfn>HTMLImageElement</dfn> : <span>HTMLElement</span> {
<p>The <dfn data-x="dom-img-currentSrc"><code>currentSrc</code></dfn> IDL attribute
must return the <code>img</code> element's <span>current request</span>'s <span data-x="img-req-url">current URL</span>.</p>

<p>The <dfn data-x="dom-img-decode"><code>decode</code></dfn>() method, when invoked, must perform
the following steps:</p>

<ol>
<!-- TODO: much of this could be phrased entirely in terms of the current request if
https://github.com/whatwg/html/issues/1055 gets fixed. See
https://github.com/whatwg/html/pull/2332#issuecomment-278613713 -->
<li>
<p>If any of the following conditions are true about this <code>img</code> element:</p>

<ul class="brief">
<li><p>its <span>node document</span> is not an <span>active document</span>;</p></li>

<li><p>it has neither a <code data-x="attr-img-src">src</code> nor a <code
data-x="attr-img-srcset">srcset</code> attribute;</p></li>

<li><p>it has a <code data-x="attr-img-src">src</code> attribute but its value is the empty
string; or</p></li>

<li><p>its <span>current request</span>'s <span data-x="img-req-state">state</span> is <span
data-x="img-error">broken</span>,</p></li>
</ul>

<p>then return a promise rejected with an <span>"<code>EncodingError</code>"</span>
<code>DOMException</code>.</p>
</li>

<li><p>Let <var>promise</var> be a new promise.</p></li>

<li>
<p><span>In parallel</span>, wait for one of the following cases to occur, and perform the
corresponding actions:</p>

<dl class="switch">
<dt>This <code>img</code> element's <span>node document</span> stops being an <span>active
document</span></dt>
<dt>This <code>img</code> element's <span>current request</span> changes or is mutated</dt>
<dt>This <code>img</code> element's <span>current request</span>'s <span
data-x="img-req-state">state</span> becomes <span data-x="img-error">broken</span></dt>

<dd>
<p>Reject <var>promise</var> with an <span>"<code>EncodingError</code>"</span>
<code>DOMException</code>.</p>
</dd>

<dt>This <code>img</code> element becomes <span data-x="img-all">completely
available</span></dt>

<dd>
<p>Decode the image's media data entirely into its bitmap form, suitable for rapid painting to
the screen.</p>

<p>If decoding does not need to be performed for this image (for example because it is a
vector graphic), resolve <var>promise</var> with undefined.</p>

<p>If decoding fails (for example due to invalid image data), reject <var>promise</var> with
an <span>"<code>EncodingError</code>"</span> <code>DOMException</code>.</p>

<p>If the decoding process completes successfully, resolve <var>promise</var> with
undefined.</p>

<p>User agents should ensure that the decoded media data stays readily available until at
least the end of the next <span>update the rendering</span> step in the <span>event
loop</span>. This is an important part of the API contract, and should not be broken if at all
possible. (Typically, this would only be violated in low-memory situations that require
evicting decoded image data, or when the image is too large to keep in decoded form for this
period of time.)</p>
</dd>
</dl>

<p class="note">Animated images will become <span data-x="img-all">completely available</span>
only after all their frames are loaded. Thus, even though an implementation could decode the
first frame before that point, the above steps will not do so, instead waiting until all frames
are available.</p>
</li>

<li><p>Return <var>promise</var>.</p></li>
</ol>

</div>

<div class="example">
<p>Without the <code data-x="dom-img-decode">decode()</code> method, the process of loading an
<code>img</code> element and then displaying it might look like the following:</p>

<pre>const img = new Image();
img.src = "nebula.jpg";
img.onload = () => {
document.body.appendChild(img);
};
img.onerror = () => {
document.body.appendChild(new Text("Could not load the nebula :("));
};
</pre>

<p>However, this can cause notable dropped frames, as the paint that occurs after inserting the
image into the DOM causes a synchronous decode on the main thread.</p>

<p>This can instead be rewritten using the <code data-x="dom-img-decode">decode()</code>
method:</p>

<pre>const img = new Image();
img.src = "nebula.jpg";
img.decode().then(() => {
document.body.appendChild(img);
}).catch(() => {
document.body.appendChild(new Text("Could not load the nebula :("));
});</pre>

<p>This latter form avoids the dropped frames of the original, by allowing the user agent to
decode the image <span>in parallel</span>, and only inserting it into the DOM (and thus causing
it to be painted) once the decoding process is complete.</p>
</div>

<div class="example">
<p>Because the <code data-x="dom-img-decode">decode()</code> method attempts to ensure that the
decoded image data is available for at least one frame, it can be combined with the <code
data-x="dom-window-requestAnimationFrame">requestAnimationFrame()</code> API. This means it can
be used with coding styles or frameworks that ensure that all DOM modifications are batched
together as <span data-x="list of animation frame callbacks">animation frame
callbacks</span>:</p>

<pre>
const container = document.querySelector("#container");

const { containerWidth, containerHeight } = computeDesiredSize();
requestAnimationFrame(() => {
container.style.width = containerWidth;
container.style.height = containerHeight;
});

// ...

const img = new Image();
img.src = "supernova.jpg";
img.decode().then(() => {
requestAnimationFrame(() => container.appendChild(img));
});</pre>
</div>

<div w-nodev>

<p>A constructor is provided for creating <code>HTMLImageElement</code> objects (in addition to
the factory methods from DOM such as <code
data-x="dom-Document-createElement">createElement()</code>): <dfn
Expand Down Expand Up @@ -63364,7 +63525,7 @@ v6DVT (also check for '- -' bits in the part above) -->

<li><p><a href="#inform">Inform the user</a> that the focus is at the location given by the
intended path. User agents may wait until the next time the <span>event loop</span> reaches its
<i>update the rendering</i> step to optionally inform the user.</p></li>
<span>update the rendering</span> step to optionally inform the user.</p></li>

</ol>

Expand Down Expand Up @@ -63396,7 +63557,7 @@ v6DVT (also check for '- -' bits in the part above) -->

<li><p>Optionally, <a href="#inform">inform the user</a> that the caret or selection (or both)
cover <var>the specified rectangle</var> of the canvas. The user agent may wait until the next
time the <span>event loop</span> reaches its <i>update the rendering</i> step to
time the <span>event loop</span> reaches its <span>update the rendering</span> step to
optionally inform the user.</p></li>

</ol>
Expand Down Expand Up @@ -88411,7 +88572,7 @@ dictionary <dfn>PromiseRejectionEventInit</dfn> : <span>EventInit</span> {
<li><p><i>Microtasks</i>: <span>Perform a microtask checkpoint</span>.</p></li>

<li>
<p><i>Update the rendering</i>: If this <span>event loop</span> is a <span>browsing
<p><dfn>Update the rendering</dfn>: If this <span>event loop</span> is a <span>browsing
context</span> <span>event loop</span> (as opposed to a <a href="#workers">worker</a>
<span>event loop</span>), then run the following substeps.</p>

Expand Down Expand Up @@ -120909,6 +121070,7 @@ INSERT INTERFACES HERE
Silver Ghost, <!-- see bug 19614 -->
Silvia Pfeiffer,
&#x160;ime Vidas,
Simon Fraser,
Simon Montagu,
Simon Sapin,
Simon Spiegel,
Expand Down Expand Up @@ -120983,6 +121145,7 @@ INSERT INTERFACES HERE
Victor Costan,
Vipul Snehadeep Chawathe,
Vitya Muhachev,
Vlad Levin,
Vladimir Katardjiev,
Vladimir Vuki&#x0107;evi&#x0107;,
Vyacheslav Aristov,
Expand Down

0 comments on commit af6b0e9

Please sign in to comment.