Bug 194775 – HTMLMediaElement readyState unexpectedly changes to HAVE_FUTURE_DATA

• Component: Media
• Status: NEW

[Serban Ungureanu]

Created attachment 362296 [details]
variation of m_averageBufferRate throughout normal playout and playout containing a brief pause

Overview: 
    HTMLMediaElement readyState doesn't seem to accurately reflect the buffering state.
Steps to Reproduce: 
    Normal playout of MSE video(I tested with Hulu -> www.hulu.com).
Actual Results: 
    HTMLMediaElement readyState is frequently changing, for consistent periods of time(2-3 seconds) from HAVE_ENOUGH_DATA to HAVE_FUTURE_DATA
Expected Results: 
    HTMLMediaElement readyState is HAVE_ENOUGH_DATA as long as "data is being fetched at a rate where the current playback position, if it were to advance at the effective playback rate, would not overtake the available data before playback reaches the end of the media resource."(which is true throughout the whole playout)
Build Date & Hardware: 
    WebKitGTK 2.23.3 built on Centos 7.2

Additional information:
    The way webkit determines HTMLMediaElement readyState change from HAVE_FUTURE_DATA to HAVE_ENOUGH_DATA and back to HAVE_FUTURE_DATA doesn't seem to be correct. 

    The state change is determined in SourceBuffer.cpp(SourceBuffer::canPlayThroughRange) using a formula composed of the average buffering rate, the remaining unbuffered time and the remaining playback time. 
> unbufferedTime.toDouble() / m_averageBufferRate < timeRemaining.toDouble()

    I have been testing normal playout with Hulu Media player and what I see is that appending segments seems to happen in short bursts from time to time(about once every five seconds), where the application appends about five seconds of data. This should result in an average buffering rate of at least 1, but that doesn't happen. The average buffering rate seems to drop at around ~0.7 and is bumped up to ~1.2 after the segments are appended(It seems that SourceBuffer::canPlayThroughRange usually return true when m_averageBufferRate > 1 and false when 0 <= m_averageBufferRate < 1). This renders the readyState to frequently be HAVE_FUTURE_DATA for few seconds at a time, even though the playout smoothly continues(without any buffering) until the end of the video.

    Moreover, the formula for computing the average buffering rate doesn't take into account the paused state of the player or the playback rate. The readyState will incorrectly change to HAVE_FUTURE_DATA when resuming playout after a brief pause(few seconds) because of the average buffering rate. Also note that if the buffering rate is negative, the formula in SourceBuffer::canPlayThroughRange is incorrect as it will always return true.

    I have attached an archive containig two files with logging statements captured during playout of a video in hulu app(which will hopefully explain the issue better than I can with words): one during normal playout and another one containing a brief pause(of around 3 seconds).

[Jer Noble]

From your description, I don't understand what underlying problem you're interested in solving. Is Hulu stalling during playback waiting for the readyState to transition to HAVE_ENOUGH?

Frankly, the entire point of MSE is to allow pages to make sophisticated scheduling, routing, and caching decisions. The UA literally does not have enough information to correctly determine whether playback will be able to continue uninterrupted, as networking decisions are not made by the UA. Hulu (and other sophisticated MSE clients) likely do not even use readyState==HAVE_ENOUGH in their loading decision graph.

[Serban Ungureanu]

(In reply to Jer Noble from comment #1)
> From your description, I don't understand what underlying problem you're
> interested in solving. Is Hulu stalling during playback waiting for the
> readyState to transition to HAVE_ENOUGH?
> 
> Frankly, the entire point of MSE is to allow pages to make sophisticated
> scheduling, routing, and caching decisions. The UA literally does not have
> enough information to correctly determine whether playback will be able to
> continue uninterrupted, as networking decisions are not made by the UA. Hulu
> (and other sophisticated MSE clients) likely do not even use
> readyState==HAVE_ENOUGH in their loading decision graph.

During testing, it appeared that Hulu won't allow seeking or pausing(UI would become unresponsive) when HTMLMediaElement.readyState < HAVE_ENOUGH_DATA. 

The observed behavior is that the app appends one 5s segment(two actually, one audio and one video) every 5 seconds, but the UA keeps switching the readyState between HAVE_ENOUGH and HAVE_FUTURE, which, correct me if I am wrong, I believe shouldn't happen.

[Yacine Bandou]

(In reply to Serban Ungureanu from comment #2)
> (In reply to Jer Noble from comment #1)
> > From your description, I don't understand what underlying problem you're
> > interested in solving. Is Hulu stalling during playback waiting for the
> > readyState to transition to HAVE_ENOUGH?
> > 
> > Frankly, the entire point of MSE is to allow pages to make sophisticated
> > scheduling, routing, and caching decisions. The UA literally does not have
> > enough information to correctly determine whether playback will be able to
> > continue uninterrupted, as networking decisions are not made by the UA. Hulu
> > (and other sophisticated MSE clients) likely do not even use
> > readyState==HAVE_ENOUGH in their loading decision graph.
> 
> During testing, it appeared that Hulu won't allow seeking or pausing(UI
> would become unresponsive) when HTMLMediaElement.readyState <
> HAVE_ENOUGH_DATA. 
> 
> The observed behavior is that the app appends one 5s segment(two actually,
> one audio and one video) every 5 seconds, but the UA keeps switching the
> readyState between HAVE_ENOUGH and HAVE_FUTURE, which, correct me if I am
> wrong, I believe shouldn't happen.

Could you test this patch ? it should fixes your issue.
https://bug-197834-attachments.webkit.org/attachment.cgi?id=369955