Implementing a Source Handlers interface for supporting adaptive formats

[heff]

The source handler pattern came from the need to support adaptive formats using the Media Source Extensions (or faux MSE with Flash) while keeping the code optional and external to core since it's very heavy. I'm hoping to get some feedback on the current implementation, especially from anyone working on the HLS plugin (e.g. @dmlap, @gkatsev, @seniorflexdeveloper)

TO-DO

HLS (to prove this will work)

• Implement duration in Flash-MSE (Implement a duration method videojs-contrib-media-sources#18)
• Combine Flash.currentTime and HLS.currentTime so they're the same (Combine the flash tech and HLS tech's currentTime methods videojs-contrib-hls#170)
• Move the HLS setCurrentTime code into a seek event handler (Seeking event never fires videojs-contrib-hls#175)
• Update the HLS plugin to be a source handler (Combine the flash tech and HLS tech's currentTime methods videojs-contrib-hls#170)

The source handler interface

// Define a source handler
var mySourceHandler = {
  // does the handler support source object type, return probably, maybe, or ''
  canHandleSource: function(sourceObject){},

  // receive the source and tech. Do whatever is needed to handle the source format.
  // through the tech you have direct access to the video element (or flash object), events, and the tech API
  handleSource: function(source, tech){}
};

// Register the source handler with the specific tech (source handlers are tech specific)
vjs.Html5.registerSourceHandler(mySourceHandler);
// or
vjs.Flash.registerSourceHandler(mySourceHandler);

After registering a source handler it will be included when the tech is asked if it can support a source (Html5.canPlaySource) and handed source types that it supports (Html5.prototype.setSource).

I've currently only implemented it on the HTML5 side, but Flash should look identical. (done)

Native format support is also set up as a source handler here, which allows it to be prioritized in case we want our own HLS support to come before native, for example.

HLS-HTML Source Handler Pseudo Code

videojs.Html5.registerSourceHandler({
  canHandleSource: function(source){
    if (MSEisSupported() && sourceIsM3u8(source)) {
      return 'maybe';
    } else {
      return '';
    }
  },

  handleSource: function(source, tech){
    return new Html5HLS(source, tech);
  }
});

function Html5HLS(source, tech){
  var mediaSource = createMediaSource(source);

  tech.setSrc(mediaSource.blobURL);

  tech.on('seeking', function(){
    var seekTarget = tech.currentTime();

    updateMediaSource(seekTarget);
  });
}

Html5HLS.prototype.dispose = function(){};

Overview

Native Source Handler

HLS Source Handler

Ideal Code Sharing

[gkatsev]

a lot of mobile devices don't support changing volume programmatically.

[heff]

Cool, I'll update that

[dmlap]

Does this mean we will have to standardize the tech API? Export/Extern it for gcc?

One thing that I worry about this design is tech-level state. Handing off the underlying tech element allows consumers to perform operations that can invalidate the assumptions of the tech. For instance, lastSeekTarget_ in the Flash tech. Am I misunderstanding how source handlers interact with the tech? Do you have any ideas how we can ensure things don't get out of sync?

[heff]

Does this mean we will have to standardize the tech API? Export/Extern it for gcc?

We already do that for the most part so the player can interact with external techs. I imagine we can improve that, but is there specific properties or whatever you're thinking of?

One thing that I worry about this design is tech-level state.

I at least don't mean for this to significantly change how the HLS logic interacts with the tech and tech element today, but I could be missing some details.

With HTML-MSE my understanding is that our code would respond to "need data" events, fetch the data, and feed it into the source buffer. Based on that (very unconfirmed) example I don't see it screwing with tech state, however the HLS (Flash) plugin does currently insert itself earlier in processes by modifying currentTime and duration. I don't have a story around how that would continue to work here yet. Do you have a better idea on the HTML-MSE needs?

Assuming MSE needs the same, we could consider other options for allowing a source handler to insert itself into currentTime and duration. Possibly firing events from those methods that allow preventDefault on them or modifying return values.

[heff]

Updated it to share the selection logic more, per @gkatsev's comment.

@dmlap I realized there was a place where only the element was passed to the source handler, not the tech. That was wrong in case that was confusing things. The source handler would be passed the source and the tech, not the el directly. handleSource: function(source, tech){},

[heff]

In the latest update I:

• Moved most of the source handler code into a mixin that can be applied to any tech
• Updated the Flash tech to use the source handler functions

In the process I was able to create an RTMP source handler, separating all the RTMP code from the main Flash code. So that's at least one nice thing that's come of this.

The next step is to approach the HLS plugin with these changes in mind and see what additional issue I run into. If there's any other feedback so far let me know.

@dmlap let me know if you have any follow up thoughts on my previous comments.

[heff]

Added a feature request on videojs-contrib-media-sources for MSE's duration method, videojs/videojs-contrib-media-sources#18. It would prevent source handlers having to be involved in the duration getting process.

[heff]

I added some pretty pictures to the issue description that probably don't do any better of a job explaining this, but hopefully. The pseudo code at the end of the description might help the most to illustrate the process.

[heff]

I added a pic on the ideal code sharing scenario to the description.

One question I'm trying to answer now is if we'll need Source Handlers to create instances. It'd be nice to keep them singletons, but then that leaves the question of where state should be stored. Possibly the tech itself, but that doesn't feel very clean.

[heff]

After chatting with @dmlap I changed the workflow to have the source handler return an instance, and the dispose method is (optionally) on that instance. The native source handlers don't need to create an instance or support a dispose method because they just pass the url to the video element.

[heff]

Rebased and updated with tests. I think this is ready to be pulled in.