MSE HLS?

[heff]

Just hoping to get an idea of how far we are from supporting HLS through MSE. That may be something I try to help out with here soon if it's needed.

We talked about this briefly before, but one thing I'm wondering is if we'll end up with two techs, hls-mse and hls-flash, since the current HLS extends vjs.Flash, and the easiest way to add MSE HLS would seem to be to extend vjs.Html5, like was done in vjs.Dashjs.

[gkatsev]

I think keeping it extending vjs.Flash for now is alright and once we're ready to make an html5 MSE HLS, we should probably just make it extend the MediaTechController directly and then have a Flash or Html tech inside of it depending on whether we're supporting native MSE or MSE via the flash fallback.

[heff]

So essentially having an html5 or Flash tech be a child component of the HLS tech that extends MediaTechController?

[gkatsev]

Yeah, I think that might be the way to go to so that we won't need to make two HLS techs.

[heff]

I'm a little worried about the amount of complexity that would add, but it's certainly worth trying. What do you see as the downsides of having two techs?

[gkatsev]

The downside would be that you'd need to maintain two techs and that you won't be able to switch between the two quite as easily. If there are two techs, you'd need to declare both techs in the tech ordering rather than just having one tech:

techs: ['html5', 'hls-mse', 'hls-flash', 'flash'],
//vs
techs: ['html5', 'hls', 'flash']

[heff]

you won't be able to switch between the two quite as easily

Do we know specific cases where we'd be switching between them? Like would we ever switch between them to play different renditions in the same manifest?

you'd need to declare both techs in the tech ordering

Yeah, that's not as clean. In the dashjs tech I added it by doing videojs.options.techOrder.push('dashjs');, so it could be that you don't have to deal with techOrder array manually and even see that. I remember there was some discussion as to if that should even be accessible, but that may have just been about accessing it in the player service UI specifically.

[gkatsev]

Don't really have a usecase for switching between them at run-time. More along the lines of switching between the two upon initialization when needed without needing to duplicate the tech.

[shacharz]

Any news here?

[dmlap]

@shacharz: this is more of a design issue. We've been focused on making the Flash HLS fallback robust and so there isn't any progress on native MSE for HLS.

@gkatsev @heff: I've been thinking recently that we may want to push the transmuxing logic into MSE itself. In this project we could focus on playlist and segment handling; and MSE would be responsible for abstracting out the particular implementation technology-- the video.js swf or native MSE. As an end developer, I think I'd prefer to have just one tech to have to worry about as well.

[gkatsev]

@dmlap what about DASH? Would MSE be incharge of deciding whether we're transmuxing to DASH/mp4/flv? Otherwise, I like this direction as well.

Perhaps we should have a separate module that is in charge of transmuxing and MSE is strictly into creating a cross-browser API for MSE.

[dmlap]

SourceBuffers require a MIME type to be constructed so I was thinking we could use that to determine whether or not to invoke transmuxing. As a user of vjs-contrib-mse, you'd throw DASH or HLS data at a source buffer, and we'd take care of choosing a display tech and converting it into a format that is usable on your platform.

We could look into breaking out the format conversion into a separate module as well. Depends on how much of a hassle it is to work with.

[heff]

@dmlap @gkatsev I’ve always liked the idea of contrib-mse working like an abstraction for both real MSE and Flash MSE, but I think the biggest issue we'll run into with that is the API for controlling the video object is different whether it’s using the Flash swf or the video element. So we couldn’t really provide an MSE fallback it in a way that works between browsers without also providing the video API translation (techs), right? I think we could still handle both types of MSE in the same project repo, but I’m not sure anymore if there’s much value in creating an abstraction between the two, since you still have to know what type of video object is being used underneath. Does that make sense, and am I missing any benefits there?

Jumping back up to the HLS tech, honestly the simplest way I can think to implement this is to create two techs, one that extends vjs.Html5 and one that extends vjs.Flash, and share code between them (e.g. manifest handling) through shared modules. They could still both be handled in the same project repo, but video.js would see them as two different techs. It’s not the prettiest to have techOrder:[‘hls-html5’, ‘hls-flash’], but the alternative seems pretty complicated to me -- have one tech that can handle loading and using the API methods from either the flash or html5 tech.

Anyway, the theme is, with both projects it sounds like we're about to make them smart enough to pick and load Flash or HTML5, which is like the one thing video.js core is good for. :) I just want to make sure if we're going to duplicate that functionality that it's worth it.

[heff]

Didn't mean to discourage forward progress on this if I did. Happy to jump in a hangout to discuss more if that'd be valuable.

[dmlap]

I'm not discouraged at all :) You're right, the way things are organized today would make it difficult to encapsulate the two implementations into MSE. I feel like there's a better solution out there than "double-tech" but it hasn't come to me yet.

[heff]

I think the design issue here is resolved by the source handlers (videojs/video.js#1560), so closing.

[sabberworm]

@gkatsev Regarding the “downside” you mentioned of having to declare hls three times in techOrder: I, for one, would love that. I have found hls-flash to be incredibly buggy and would love for there to be a way to excluding it from the available techs whilst keeping native and mse in. Also, how would that make “[switching] between the two” harder? It would make it easier, no?

[sabberworm]

@heff So if this issue was closed because the “design issue is resolved”, should I open a new issue for actually implementing HLS via MSE?

[dmlap]

@sabberworm we're right in the middle of it, so no need! If you'd like to check out our progress, you can keep an eye on the mse-magic branch in my fork. I'll be opening a PR on the main repo as soon as I get all the tests passing again.

[sabberworm]

@dmlap That’s great news!

[arikgilad]

I tried to test the mse-magic version but I was not able to make it work.
When running the example I found many issues:

1. in videojs-hls.js i got typerror on
   videojs.Hls = videojs.extends(Component, {...

I had to change videojs.extends to videojs.extend

1. The code is using request.response/request.responseText in videojs-hls.js and playlist-loader.js instead of request.body.

Even after fixing those issues, the video was still not playing.
It was downloading all the ts files but never played them because drainBuffer() was doing nothing because this.sourceBuffer.updating was set once and never unset.

[dmlap]

@arikgilad we've merged MSE into the development branch. The issue you mention should be fixed there.