The final decision of autoplay detection API name

[alastor0325]

Hi, I'm going to help on editing the spec of autoplay detection.

But I found that there are two proposed names for the API on HTMLMediaElement. One is autoplayPolicy from issue#11.

partial interface HTMLMediaElement {
  readonly attribute AutoplayPolicy autoplayPolicy;
};

Another one is the canAutoplay from the explainer, and this one returns a boolean.

partial interface HTMLMediaElement {
  readonly boolean canAutoplay; 
};

Should we decide a final naming and the form for this API before moving things forward? In addition, for the API naming on Document, does everyone agree with using autoplayPolicy like issue#11?

partial interface Document {
  readonly attribute AutoplayPolicy autoplayPolicy;
};

Firefox has implemented an exprimental feature called allowedToPlay based on the previous discussion threads and it's easy for us to change that name to the one that we will decide to use for the final decision.

[alastor0325]

@padenot would you mind to help me loop related people into this discussion? Thank you.

[chrisn]

Welcome @alastor0325, and thank you helping with editing the spec.

If you haven't done so already, please create a W3C account, associate it with your GitHub account, and join the Media WG here.

Looking at the people involved in the early discussion who are still active in the group, I recommend @beaufortfrancois and @jernoble as initial contacts.

[chrisn]

@alastor0325 We discussed briefly on the Media WG call today, @dalecurtis is the person to contact at Google for now.

[dalecurtis]

From the Google side, since we proposed it, we're happy with the existing names. We don't have a strong attachment though, so whatever gains consensus sgtm.

[dalecurtis]

canXXX() methods matches the existing HTMLMediaElement style FWIW.

[alastor0325]

My main concern of returning a boolean on the media element API is that returned information might not be enough for web developers to make the decision.

For instance, when video.canAutoplay() == false, does that mean no autoplay is allowed, or just not allow audible media to autoplay? Then the web developers have to use document.autoplayPolicy together in order to know the exact status for the media element.

if (video.canAutoplay()) {
   video.play();
} else if (!video.canAutoplay() && document.autoplayPolicy == "allowed-muted"){
   video.muted = true;
   video.play();
} else {
   // Not allow to autoplay, do some stuffs here
}

If we provide detailed information on the media element API as well, then we can achieve the same goal by only using one API.

switch (video.autoplayPolicy) {
  case "allowed":
     video.play();
     break;
  case "allowed-muted":
     video.muted = true;
     video.play();
     break;
  default:
     // Not allow to autoplay, do some stuffs here
}

WDYT @dalecurtis?

[dalecurtis]

autoplayPolicy only won't be misleading, but I'm unclear how the API is supposed to interact with user gestures. I.e., the 'do some stuffs here' section :) Are we solving the desired use cases if we only provide the policy value?

[gkatsev]

As a player developer, one of the main things I want is to know whether if I call play, with the current configuration of the media element, it will succeed, without needing to call play. This is what canAutoplay() provides. If it returns true, we can proceed with calling play; if it isn't, we can check the current autoplay policy, adjust the video element properties, and then call play() at that point (or try canAutoplay() again).
For example:

if (video.canAutoplay()) {
  video.play();
} else if (document.autoplayPolicy === 'allowed-muted') {
  video.muted = true;
  video.play();
} else {
 // fall back to click-to-play
}

[dalecurtis]

@gkatsev it sounds like you're expecting that canAutoplay() would include whether or not a user activation is currently present?

[gkatsev]

Yes, it would be a lot less useful to me if it didn't.

[alastor0325]

IMMO I think the user activation is no something we need to discuss in the API level, that is more like an implementation detail. User agents can freely determine what actual mechanism they want to use for blocking autoplay. If they want to use the user acitvation, then the result returned by the autoplay policy detection API should be already considered the user activation and to simply tell web developers whether they can play media or not. User agent can also choose other implementations, such like only allowing autoplay on the media element that has been clicked by users.

In addition, another benefit using video.autoplayPolicy is to allow web developers know the autoplay policy earlier without wating for loading the metadata. Because before loading metadata, we don't know if the media has audio track or not, which means not being able to determine if it's inaudible.

So if the page only allows inaudible media, calling video.canAutoplay before and after loading the metadata could lead to the different result. On Firefox, we regard a video without loading metadata as an audible media because we don't know its status yet.

// Assume that `document.autoplayPolicy === 'allowed-muted'`

// canAutoplay version
let video = document.createElement("video");
video.src = "video_without_audio_track.webm";
await new Promise(r => v.onloadedmetadata = r);
if (video.canAutoplay) {   
  video.play();
} else {
  // handle the case when autoplay is not allowed
  // eg. showing the click-to-play icon
}

// autoplayPolicy version
let video = document.createElement("video");
video.src = "video_without_audio_track.webm";
if (video.autoplayPolicy == "allow-muted") {  
  video.play();
} else {
  // handle the case when autoplay is not allowed
  // eg. showing the click-to-play icon
}

[gkatsev]

IMMO I think the user activation is no something we need to discuss in the API level, that is more like an implementation detail.

Yeah, I would agree with that. As long as if user activation is required, it's included in the decision for the response of canAutoplay.

So if the page only allows inaudible media, calling video.canAutoplay before and after loading the metadata could lead to the different result

That should be fine. canAutoplay should capture the current moment in time. If there isn't enough info, it should err on the side of it can't. Users could wait for loadedmetadata to be sure it won't change.

canAutoplay is simpler, since it tells me directly whether a call to play will succeed or not. video.autoplayPolicy will tell me "you can play muted", now I'll need to check whether my player is muted or there's no audio tracks before proceeding.

[dalecurtis]

@alastor0325 It sounds like you're saying that autoplayPolicy() alone solves all the use cases Firefox is concerned about?

If we do end up agreeing on canAutoplay(), leaving the user activation specifics out of the spec sgtm, but we may want to at least specify:

• That canAutoplay() is only valid within the current scope or something similar to set expectations.
• That canAutoplay() will return false in cases where it's unsure.

[alastor0325]

canAutoplay is simpler, since it tells me directly whether a call to play will succeed or not. video.autoplayPolicy will tell me "you can play muted", now I'll need to check whether my player is muted or there's no audio tracks before proceeding.

This sounds reasonable.

It sounds like you're saying that autoplayPolicy() alone solves all the use cases Firefox is concerned about?

If web developers care about the consistent result before and after loading metadata, then autoplayPolicy() can help to address that issue. But as gkatsev mentioned before, if web developers like to check that by manually loading the metadata, or via checking the error on the play promise, then I will happy to see using canAutoplay because it indeed simpler and straightforward.

That canAutoplay() is only valid within the current scope or something similar to set expectations.

@dalecurtis Does the current scope mean the moment while canAutoplay is being called?

That canAutoplay() will return false in cases where it's unsure.

Agree. Just for the confirmation, does it here mean that the user agent hasn't been able to make the decision because of still lacking some information (either media information, or some information related with the user agent's implementation)?

Thank you.

[dalecurtis]

[...] I will happy to see using canAutoplay because it indeed simpler and straightforward.

Sounds reasonable to me ass well.

That canAutoplay() is only valid within the current scope or something similar to set expectations.

@dalecurtis Does the current scope mean the moment while canAutoplay is being called?

I was using scope to refer to the current call stack, I expect there's some more proper webism/specism for that.

Agree. Just for the confirmation, does it here mean that the user agent hasn't been able to make the decision because of still lacking some information (either media information, or some information related with the user agent's implementation)?

Yes that's my read of the situation. I expect there's some risk that folks will unknowingly call it too early, so I wonder if returning false is actually okay and rather we should throw or return an enum value ("true", "false", "unknown").

[alastor0325]

Yes that's my read of the situation. I expect there's some risk that folks will unknowingly call it too early, so I wonder if returning false is actually okay and rather we should throw or return an enum value ("true", "false", "unknown").

What the difference would be for web developers when they receive "false" and "unknown"? If they will try to request the autoplay later on both situations, then it seems "unknown" would become useless.

In addition, as the returned value of canAutoplay might be chaged in the future (eg. if an user agent uses sticky user activation to determine the policy), I assume that web developers will also try to query that result multiple times?

@gkatsev I wonder if adding the "unknown" will help web developers, or simple boolean is enough?

Thank you.

[dalecurtis]

Yeah I couldn't really come up with any catastrophic failure modes if they call too early, but I did want to mention it in case anyone else here could think of something bad that might happen.

[alastor0325]

but I did want to mention it in case anyone else here could think of something bad that might happen.

Sure, thank for bring that to the table. So if no one has any other concern of returning false while calling canAutoplay too early, then we can use the boolean version if all people agree with that.

@jernoble As you proposed the AutoplayPolicy version in #11, may I ask your opinion here? What would Apple think about the boolean version canAutoplay?

Thank you.

[gkatsev]

I would say a boolean is fine. Though, I'm also slightly biased as I'm familiar with this work. A boolean is definitely simpler than an enum, especially if the enum looks like a boolean. We could put a note in the spec that recommends listening to loadedmetadata to make sure you have the most accurate response from canAutoplay.

I can ask the video-dev community for input if you think it would be valuable.

[eric-carlson]

So if no one has any other concern of returning false while calling canAutoplay too early, then we can use the boolean version if all people agree with that.

@jernoble As you proposed the AutoplayPolicy version in #11, may I ask your opinion here? What would Apple think about the boolean version canAutoplay?

The goal of the AutoplayPolicy version was to allow sites to know whether they could autoplay without fetching audio data, but a boolean canAutoPlay will work for WebKit too.

[gkatsev]

To clarify, are both document.autoplayPolicy and video.canAutoplay happening? Or is only one going to happen?
I think, ideally, I'd like both.

[alastor0325]

I can ask the video-dev community for input if you think it would be valuable.

That will be great, I'd like to know more about how web developers feel about current API proposal.

To clarify, are both document.autoplayPolicy and video.canAutoplay happening?

From my current understanding, yes. This issue is only aim to determine whether the API interface for HTMLMediaElement should be autoplayPolicy or canAutoplay. From the discussion in the past, seems all people agree with having document.autoplayPolicy.

The goal of the AutoplayPolicy version was to allow sites to know whether they could autoplay without fetching audio data, but a boolean canAutoPlay will work for WebKit too.

So far video.canAutoPlay has been agreed by Safari (@eric-carlson), Chrome (@dalecurtis) and Firefox (@alastor0325) and also get an positive feedback from video.js developer (@gkatsev). It seems that I can use canAutoPlay as a temporary result while editing the draft for the autoplay policy detection. And we can keep discuss this issue on this thread, if we find out any other new concern about using canAutoPlay.

[tjenkinson]

Firstly this is great to see that we're going to be able to determine if auto play will work synchronously without waiting for play() to reject 🎉

I'm trying to understand if we really need both video.canAutoPlay() and document.autoplayPolicy?

Wouldn't document.autoplayPolicy cover everything? If document.autoplayPolicy is allowed, or allowed-muted and the element is muted, then video.canAutoPlay() would always be true right?

e.g

if (document.autoplayPolicy === 'disallowed') {
  // fallback to click-to-play
} else {
  if (document.autoplayPolicy === 'allowed-muted') {
    vid.muted = true;
  }
  vid.play();
}

in a ‘loadedmetadata’ listener (edit: doesn’t have to be)

[dalecurtis]

canAutoPlay() could also return true in cases where a UA has some other mechanism (say user-activation) to allow autoplay. We could possibly work that into the document level autoplayPolicy though. I.e., the UA could return 'allowed' if the user activation (or whatever signal its using) is currently present.

[tjenkinson]

Ah right. So autoPlayPolicy would be disallowed in that case?

Should it change to allowed temporarily whilst the stack is running from a user action?

[tjenkinson]

I.e., the UA could return 'allowed' if the user activation (or whatever signal its using) is currently present.

Yeh I’d vote for this. I think if we have both canAutoplay() return true but the policy say ‘disallowed’ when asked in the same call stack it’s a bit strange.

And if that’s the case, I think canAutoPlay() might be redundant?

[gkatsev]

What about the following scenario:
UA only allows muted autoplay unless the user has clicked on a video, in which case, only that video element can autoplay with sound.
In that case, I'd expect document.autoplayPolicy to return allowed-muted but vid.canAutoplay could return true.

[tjenkinson]

If user agents could change their decision based on the specific element/audio context I think they should be separate rather than trying to have a global one, so yeh have one on AudioContext too if needed

[gkatsev]

Looking at the old explainer (#8), the document level property can help you decide which source to use. The element level property can decide whether you can continue playback as is.

[tjenkinson]

👍 We could still have ‘autoplayPolicy’ on the element itself for this though? The element doesn’t have to be initialised with a 'src' does it? So you could create the element, decide what source to load, and then set the ‘src’?

[alastor0325]

The element doesn’t have to be initialised with a 'src' does it? So you could create the element, decide what source to load, and then set the ‘src’?

I think that is what document.autoplayPolicy wants to address. By using that, web developer could stop loading the media resouce if the document.autoplayPolicy == disallowed. And use video.canAutoplay to know whether the media is allowed to play or not.

---

For video.autoplayPolicy, the pro is that web developers only need to check one API (HTMLMediaElement) and doesn't need to check document level API. The con is a more complicated returned result. (based on the feedback from gkatsev.)

if (vid.autoplayPolicy === 'disallowed') {
  // fallback to click-to-play and probably no need to load media resource
} else {
  vid.src = "video.webm";
  if (vid.autoplayPolicy === 'allowed-muted') {
    vid.muted = true; // or video.src = "video_no_audio.webm"
  }
  vid.play();
}

---

For video.canAutoplay, the pro is pretty straightforward and clearly allow web developers to know if they can start media or not. The con is needing to cooperate with document level API in terms of detemining whether they need to load the media source.

 // Determine the media resource
if (document.autoplayPolicy === 'disallowed') {
  // fallback to click-to-play and probably no need to load media resource for vid
} else if (document.autoplayPolicy === 'allowed-muted') {
  // either loading inaudible resource, or simply mute vid
}
await new Promise(r => v.onloadedmetadata = r);
if (vid.canAutoplay) {
  await vid.play();
}

---

But just looking the code above, I feel the vid.autoplayPolicy is actually simpler than vid.canAutoplay...? but I mixed the logic of selecting media resources and requesting for play together, which might be different from the real world situaion in the developers side.

[tjenkinson]

I think that is what document.autoplayPolicy wants to address. By using that, web developer could stop loading the media resouce if the document.autoplayPolicy == disallowed.

Yeh so this is still possible if it was video.autoplayPolicy, because you could create an element and add the src later.

For video.autoplayPolicy, the pro is that web developers only need to check one API (HTMLMediaElement) and doesn't need to check document level API. The con is a more complicated returned result. (based on the feedback from gkatsev.)

I think the result is only more complicated when it's on document, because on document it can't take context about specific elements into consideration? If it's on video and the call is from a user interaction and the user agent requires that, it would not be 'disallowed', wheras the document.autoplayPolicy could be 'disallowed'.

[alastor0325]

Sorry for my late reply. So one example of mixing video.canAutoplay and document.autoplayPolicy is proposed in this. But this example doesn't consider whether media resource finishs loading. If considering the resource, the example could look like that.

I think the result is only more complicated when it's on document, because on document it can't take context about specific elements into consideration?

Indeed, using video.autoplayPolicy itself can solve not only the resource selection problem, but also telling whether this media can continue playback or not. Based on this. It also doesn't conflict with the media resource status.

The original idea of introducing document level API is about solving the resource selection issue. If video.autoplayPolicy can already fullfill two goals together, should we really need to the document level API? Another reason for document level API is about telling the result for web audio, but creating API on AudioContext (Eg. ac.autoplayPolicy) seems more reliable?

@dalecurtis @eric-carlson @gkatsev May I get the opinion from y'all? Also, @padenot, WDYH about directly adding an attribute on AudioContext for autoplay detection?

Thank you.

[dalecurtis]

allowed-muted doesn't make a lot of sense on AudioContext, I think we'd just want a simple canAutoplay() there instead? I don't have a strong opinion on video.autoplayPolicy() versus video.canAutoplay() though. canAutoplay() seems easier to use, but the policy variant provides more power.

[padenot]

allowed-muted doesn't make a lot of sense on AudioContext

It does: https://github.com/WebAudio/web-audio-api-v2/issues/83#issuecomment-846127059

[padenot]

Also, @padenot, WDYH about directly adding an attribute on AudioContext for autoplay detection?

The document API is sufficient:

var ac = new AudioContext;
if (document.autoplayPolicy == "allowed") {
  ac.onstatechange = function() {
    if (ac.state == "running") {
      // start the audio app
    }
  } else {
    // display a bit of UI to ask the user to start the app
    // audio starts via calling `ac.resume()` from a handler,
    // `onstatechange` allows knowing when the audio stack
    // is ready. 
  }
}

I don't know of a reason to not do this, because the AudioContext has sticky activation.

[dalecurtis]

allowed-muted doesn't make a lot of sense on AudioContext

It does: WebAudio/web-audio-api-v2#83 (comment)

TIL. Thanks @padenot.

I think that takes us back to having document.autoplayPolicy and video.canAutoPlay ?

[tjenkinson]

Given web audio and the media element work differently and are different api’s I’m not sure they need to share the same document property? In the future will there always be a value that satisfies both apis?

I think the result is only more complicated when it's on document, because on document it can't take context about specific elements into consideration? If it's on video and the call is from a user interaction and the user agent requires that, it would not be 'disallowed', wheras the document.autoplayPolicy could be 'disallowed'.

I still think this is a reason to have an element.autoplayPolicy and audioContext.autoplayPolicy (or there could be a specific web audio one on document given it’s global, but wouldn’t it be better living with the api?)

It also means you only have one property to check for autoplay, not 2 (for the media element case)

[dalecurtis]

HTMLMediaElement.autoplayPolicy and (document.autoplayPolicy or audioContext.autoplayPolicy) are all fine with me, the policy enumeration is more verbose than canAutoplay(), but only requires checking one place.

In the code above is anyone actually doing:

// Determine the media resource
if (document.autoplayPolicy === 'disallowed') {
  // fallback to click-to-play and probably no need to load media resource for vid
} else if (document.autoplayPolicy === 'allowed-muted') {
  // either loading inaudible resource, or simply mute vid
}
await new Promise(r => v.onloadedmetadata = r);
if (vid.canAutoplay) {
  await vid.play();
}

Instead of

// Determine the media resource
if (document.autoplayPolicy === 'disallowed') {
  // fallback to click-to-play and probably no need to load media resource for vid
} else if (document.autoplayPolicy === 'allowed-muted') {
  // either loading inaudible resource, or simply mute vid
} else {
  vid.autoplay = true;
}

Basically the only benefit of the element level check is to incorporate metadata, but is that needed?

[tjenkinson]

Basically the only benefit of the element level check is to incorporate metadata

I don’t think that’s that’s the only purpose? An element api would take other context into consideration, like whether the click is from a user action if that’s required. A document api couldn’t do that.

[dalecurtis]

Why couldn't the document level API check the presence of a user-activation? Whether we want to do that is a different question.

[gkatsev]

If user-activation is per element rather than per document, the document policy couldn't reliably tell us whether a given element will be allowed or not allowed to play.

Personally, I find a document level attribute is useful in addition to a video element property. Using the document level property, we can check what the current state of the world is before deciding to go down a particular path, and it doesn't require us to create a potentially temporary video element to do this check (especially given the current limit on this in Chrome). Then, the video element property can tell us what this particular element can do. This is also why I favor canAutoplay, but I'd probably be fine with it being mediaElement.autoplayPolicy as well.

I'm not terribly familiar with the WebAudio API, but it seems like it'll be useful to have a single place where developers can query to know whether playing back something with sound will work or not.

[tjenkinson]

it doesn't require us to create a potentially temporary video element to do this check (especially given the current limit on this in Chrome)

Interesting. Does this throw when you create the element or at a later stage once hitting the limit?

If it’s after construction then this limit could also be factored in to the returned policy?

[gkatsev]

Looks like the limit has increased by quite a lot, but yes, it threw when the limit was hit. Previously it was 75, it'll be 1000 at some point soon. See https://bugs.chromium.org/p/chromium/issues/detail?id=1232649

[tjenkinson]

Just tried tweaking the code for the demo in that ticket slightly and it looks like it's not createElement that thows, but instead play() never resolves/rejects 😱 and you get the [Intervention] Blocked attempt to create a WebMediaPlayer as there are too many WebMediaPlayers already in existence. See crbug.com/1144736#c27 error in the console.

So it is an example of something else that could be part of the logic behind video.autoplayPolicy IMO

code to reproduce

for (let i = 0; i < 100; i++) {
  const v = document.createElement('video');
  v.muted = true;
  v.srcObject = localVideo.srcObject;
  Promise.race([v.play(), new Promise((resolve, reject) => setTimeout(() => reject('timeout'), 500))]).catch((e) => console.log(e))
}

on https://webrtc.github.io/samples/src/content/peerconnection/pc1/ after clicking start

gives 26 logs and 26 timeout's

[dalecurtis]

(I don't think that issue is relevant to this discussion, so we should probably avoid derailing -- that intervention affects loading prior to playback. FWIW: It was an intervention intended only to capture extremely rare cases of element leakage that crashed users systems, but the data used to set the limits was not normally distributed and unfortunately RTC ended up disproportionately affected)

[alastor0325]

Based on the discussions so far, the only argument for vid.autoplayPolicy was that the returned enum is more complicated than the boolean, but vid.canAutoplay got a strong disagreement from one of participants. Therefore, I would recommend to start with vid.autoplayPolicy for the further spec editing (I have a draft already, and will probably be able to make a first PR soon) and I also conclude the pros & cons from our discussion before in below. What do other people think? Thank y'all.

---

Proposal 1 : vid.canAutoplay
Positive feedback by : Chrome, Firefox, Safari, VideoJS
Negitive feedback by : BBC
Benefits : The returned value is a simple boolean to indicate if a specific media element can start playing
Arguments :
(1) Not powerful enough, it has to corporate with another document level API for resource selection
(2) Need to wait for metadata loading in order to know the right result

Proposal 2 : vid.autoplayPolicy
Positive feedback by : Chrome, Firefox, Safari, VideoJS, BBC
Negitive feedback by : None
Benefits :
(1) The returned enum can be used to determine not only if a specific media element can start playing, but also a resouce selection. So one API can solve all things without relying on another API.
(2) No need to wait for metadata loading
Arguments :
(1) Returned format is not as simple as the boolean, increasing some complexity

[dalecurtis]

sgtm, thanks @alastor0325!

[tjenkinson]

Lgtm too! By the way @alastor0325 I work for the BBC but my comments here were not on behalf of the BBC

[gkatsev]

Thanks, @alastor0325. SGTM.

Also, I speak on behalf of Brightcove as well as Video.js.

I think the main downside of video.autoplayPolicy is that if you want to know what a generic video element would do, you would need to create a new video element, and you can't re-use an existing one. Which, while it's a downside, it definitely a fairly minor issue. The use-case here is that specifically around ads, where if you loaded a video into the player, you want to know whether the ad will be able to autoplay or not, and you may not be able to rely on the player's video element to know this.

[tjenkinson]

Hmm I see your point.

Maybe we need to distinguish between:

1. whether play() would work if called right now in the current call stack (which is already conveyed in the play() promise but async so not super convenient)
2. the minimum conditions required for play() to work if it happened at any point (I.e non user action). I think this more fits the ‘could I play an ad somehow’ case?

?

What we have now just helps figure out how to make the play call work at a given point.

And after writing that I realise this is essentially what you had originally @gkatsev . Sorry for being so slow :|

So IMO:

• video.canAutoPlay() (taking element/call context into consideration) could solve (1)
• document.autoPlayPolicy (taking no element specific context into consideration) could solve (2)

I think both would be needed though for it to be most useful.

[alastor0325]

Hi, all,
I just created a new PR for the Autoplay Detection Spec draft. The content is based on our current discussion, but that haven't included the latest one yet posted by tjenkinson. Feel free to leave any comment there. In addition, I'm fairly a newbie in term of the spec editing, if there is any procedure which I've missed, please feel free to correct me without hesitatation.
Thank you so much.

[tjenkinson]

Hi @alastor0325,

The content is based on our current discussion, but that haven't included the latest one yet posted by tjenkinson

It looks like it does actually cover my points there given you’re proposing a property on document _and _ video

[alastor0325]

As the first version of PR has been merged, this issue could be closed. Because in the merged first version, we already chose to use autoplayPolicy as the final name for APIs, which was aslo decided based on the discussion here.