Bug 57803 – Work-in-progress on rendering TextTracks on HTML5 Video

• Component: Media
• Status: RESOLVED INVALID

[Anna Cavender]

This bug is just to share code with dglazkov, will remove after discussion.

[Anna Cavender]

Created attachment 88159 [details]
Patch

[Anna Cavender]

dglazkov, sorry for such a huge patch, I hope you can skip to the relevant stuff!

thank you sooooooo much for taking a look and let me know if you have any questions.

[Roland Steiner]

is it possible to extract the VTT and Cue parsers into their own separate stand-alone patch(es) to reduce the overall patch size?.

[Dimitri Glazkov (Google)]

No worries. I realize this is a snapshot of your checkout :)

I read up on <track> element and now realize that this is the same use case as the <details>/<summary> element (http://www.whatwg.org/specs/web-apps/current-work/multipage/interactive-elements.html#the-details-element)

The approach you are currently taking (and the one I misled you toward) is workable, but ultimately is drenched with angel tears and droplets of unicorn blood. The problem here is that the <track> element itself is just data -- it's appearance is projecte onto the visual surface of the video element via MediaControlTextTrackElement. This means we have to manage lifetimes and lifecycles of both of these and synchronize them, and surely suffer through mysterious crashes and bugs all around.

Instead, we should take this approach:

1) Fix bug 56973, which would allow us to place <track> element's render object inside of the RenderVideo hierarchy
2) Implement <track> element as a box (probably a RenderBlock subclass), which displays just the right text inside of it.
3) Profit on a horse.

We are sprinting toward fixing bug 56973, so that we can get this done soon. Can you guys try to land other bits of the track stuff first and hang tight on implementing the visual parts of <track>? We promise to help with advice/code/reviews.

WDYT?

[Anna Cavender]

I think that sounds like a fine plan!  Thanks again for helping out.

I'm very relieved: I was starting to worry about all of those creates and removes going on in there :).

Believe me, we have plenty of other things to work on while you work on a new design.  Keep me in the loop.

And again, THANKS!

[Anna Cavender]

Quick question about creating a RenderTrack as a RenderBlock subclass and attaching to the RenderVideo hierarchy.  I think that will work fine for <track> as a RenderTrack can be associated with each HTMlTrackElement.  But, how would this work for in-band tracks that will be demuxed from MediaPlayer?  They will have no associated DOM element, so can HTMLMediaElement somehow manage them and attach them to the RenderVideo heirarchy?  Not quite clear how that will work?

[Dimitri Glazkov (Google)]

(In reply to comment #6)
> Quick question about creating a RenderTrack as a RenderBlock subclass and attaching to the RenderVideo hierarchy.  I think that will work fine for <track> as a RenderTrack can be associated with each HTMlTrackElement.  But, how would this work for in-band tracks that will be demuxed from MediaPlayer?  They will have no associated DOM element, so can HTMLMediaElement somehow manage them and attach them to the RenderVideo heirarchy?  Not quite clear how that will work?

Can we model this as adding a <track> element _in_ the shadow DOM, so that the script can't see it?

[Anna Cavender]

What do you think about creating a RenderCue instead of a RenderTrack?  That way it will have a guaranteed 1:1 relationship with a TextTrackCue.  I.e., a <track> could have multiple cues showing at a time and so it would have to manage an unknown number of RenderTracks, which might be fine but seems sort of complicated.

[Dimitri Glazkov (Google)]

(In reply to comment #8)
> What do you think about creating a RenderCue instead of a RenderTrack?  That way it will have a guaranteed 1:1 relationship with a TextTrackCue.  I.e., a <track> could have multiple cues showing at a time and so it would have to manage an unknown number of RenderTracks, which might be fine but seems sort of complicated.

Good question. Ideally, I want to avoid creating new custom RenderFoos altogether. <track> should just use RenderBlock, and the logic of putting the cue in it should be built into the track element.