Bug 164529 – Pages running in non-visible tabs use large amounts of CPU decoding video

• Status: NEW
• Component: New Bugs

[Jer Noble]

Pages running in non-visible tabs use large amounts of CPU decoding video

[Jer Noble]

After discussing this with rniwa, I have decided not to add a PerformanceTest test for this  change. Ryosuke explained that performance tests should not be used as regression tests. It would also be difficult to write a LayoutTest for this change to verify that decoders are not active during the running of such a test, as other video-decoder-using tests may be active concurrently with this hypothetical test.

The only remaining test to write would be one that returned the number of "active" tracks (rather than just enabled tracks) when the page is not visible. The only thing this test would verify is that we actually disable the tracks when pages are backgrounded, but not that decoders stop or that CPU resources are returned.

[Jer Noble]

Created attachment 294218 [details]
Patch

[Jer Noble]

<rdar://problem/28056151>

[Build Bot]

Comment on attachment 294218 [details]
Patch

Attachment 294218 [details] did not pass mac-ews (mac):
Output: http://webkit-queues.webkit.org/results/2481267

New failing tests:
media/video-muted-audible-tracks.html
media/video-background-tab-playback.html
media/video-background-tab-visible-tracks.html

[Build Bot]

Created attachment 294220 [details]
Archive of layout-test-results from ews103 for mac-yosemite

The attached test failures were seen while running run-webkit-tests on the mac-ews.
Bot: ews103  Port: mac-yosemite  Platform: Mac OS X 10.10.5

[Build Bot]

Comment on attachment 294218 [details]
Patch

Attachment 294218 [details] did not pass mac-wk2-ews (mac-wk2):
Output: http://webkit-queues.webkit.org/results/2481266

New failing tests:
media/video-muted-audible-tracks.html
media/video-background-tab-visible-tracks.html

[Build Bot]

Created attachment 294221 [details]
Archive of layout-test-results from ews105 for mac-yosemite-wk2

The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: ews105  Port: mac-yosemite-wk2  Platform: Mac OS X 10.10.5

[Build Bot]

Comment on attachment 294218 [details]
Patch

Attachment 294218 [details] did not pass ios-sim-ews (ios-simulator-wk2):
Output: http://webkit-queues.webkit.org/results/2481272

New failing tests:
media/video-muted-audible-tracks.html
media/video-background-tab-visible-tracks.html

[Build Bot]

Created attachment 294222 [details]
Archive of layout-test-results from ews123 for ios-simulator-wk2

The attached test failures were seen while running run-webkit-tests on the ios-sim-ews.
Bot: ews123  Port: ios-simulator-wk2  Platform: Mac OS X 10.11.6

[Jer Noble]

Created attachment 294232 [details]
Patch

[Build Bot]

Comment on attachment 294232 [details]
Patch

Attachment 294232 [details] did not pass mac-ews (mac):
Output: http://webkit-queues.webkit.org/results/2484214

New failing tests:
media/video-muted-audible-tracks.html
media/video-background-tab-visible-tracks.html

[Build Bot]

Created attachment 294238 [details]
Archive of layout-test-results from ews103 for mac-yosemite

The attached test failures were seen while running run-webkit-tests on the mac-ews.
Bot: ews103  Port: mac-yosemite  Platform: Mac OS X 10.10.5

[Build Bot]

Comment on attachment 294232 [details]
Patch

Attachment 294232 [details] did not pass mac-wk2-ews (mac-wk2):
Output: http://webkit-queues.webkit.org/results/2484235

New failing tests:
media/video-muted-audible-tracks.html
media/video-background-tab-visible-tracks.html

[Build Bot]

Created attachment 294241 [details]
Archive of layout-test-results from ews107 for mac-yosemite-wk2

The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: ews107  Port: mac-yosemite-wk2  Platform: Mac OS X 10.10.5

[Jer Noble]

Created attachment 294249 [details]
Patch

[Build Bot]

Comment on attachment 294249 [details]
Patch

Attachment 294249 [details] did not pass ios-sim-ews (ios-simulator-wk2):
Output: http://webkit-queues.webkit.org/results/2484535

New failing tests:
media/video-background-tab-visible-tracks.html

[Build Bot]

Created attachment 294263 [details]
Archive of layout-test-results from ews126 for ios-simulator-wk2

The attached test failures were seen while running run-webkit-tests on the ios-sim-ews.
Bot: ews126  Port: ios-simulator-wk2  Platform: Mac OS X 10.11.6

[Build Bot]

Comment on attachment 294249 [details]
Patch

Attachment 294249 [details] did not pass mac-ews (mac):
Output: http://webkit-queues.webkit.org/results/2484673

New failing tests:
media/video-muted-audible-tracks.html

[Build Bot]

Created attachment 294265 [details]
Archive of layout-test-results from ews100 for mac-yosemite

The attached test failures were seen while running run-webkit-tests on the mac-ews.
Bot: ews100  Port: mac-yosemite  Platform: Mac OS X 10.10.5

[Build Bot]

Comment on attachment 294249 [details]
Patch

Attachment 294249 [details] did not pass mac-debug-ews (mac):
Output: http://webkit-queues.webkit.org/results/2485328

New failing tests:
media/video-muted-audible-tracks.html

[Build Bot]

Created attachment 294291 [details]
Archive of layout-test-results from ews113 for mac-yosemite

The attached test failures were seen while running run-webkit-tests on the mac-debug-ews.
Bot: ews113  Port: mac-yosemite  Platform: Mac OS X 10.10.5

[Sam Weinig]

What happens to videos being drawn to canvases.  Do they still work?

[Jer Noble]

(In reply to comment #22)
> What happens to videos being drawn to canvases.  Do they still work?

No, they don't. If they're in the background they get black frames.

[Sam Weinig]

(In reply to comment #23)
> (In reply to comment #22)
> > What happens to videos being drawn to canvases.  Do they still work?
> 
> No, they don't. If they're in the background they get black frames.

Seems like should still work, no?

[Jer Noble]

Created attachment 294707 [details]
Patch

[Build Bot]

Comment on attachment 294707 [details]
Patch

Attachment 294707 [details] did not pass mac-ews (mac):
Output: http://webkit-queues.webkit.org/results/2514693

New failing tests:
media/video-muted-audible-tracks.html

[Build Bot]

Created attachment 294714 [details]
Archive of layout-test-results from ews101 for mac-yosemite

The attached test failures were seen while running run-webkit-tests on the mac-ews.
Bot: ews101  Port: mac-yosemite  Platform: Mac OS X 10.10.5

[Build Bot]

Comment on attachment 294707 [details]
Patch

Attachment 294707 [details] did not pass mac-debug-ews (mac):
Output: http://webkit-queues.webkit.org/results/2514657

New failing tests:
media/video-muted-audible-tracks.html

[Build Bot]

Created attachment 294715 [details]
Archive of layout-test-results from ews116 for mac-yosemite

The attached test failures were seen while running run-webkit-tests on the mac-debug-ews.
Bot: ews116  Port: mac-yosemite  Platform: Mac OS X 10.10.5

[Build Bot]

Comment on attachment 294707 [details]
Patch

Attachment 294707 [details] did not pass ios-sim-ews (ios-simulator-wk2):
Output: http://webkit-queues.webkit.org/results/2514665

New failing tests:
media/video-background-tab-visible-tracks.html

[Build Bot]

Created attachment 294717 [details]
Archive of layout-test-results from ews123 for ios-simulator-wk2

The attached test failures were seen while running run-webkit-tests on the ios-sim-ews.
Bot: ews123  Port: ios-simulator-wk2  Platform: Mac OS X 10.11.6

[Jer Noble]

(In reply to comment #24)
> (In reply to comment #23)
> > (In reply to comment #22)
> > > What happens to videos being drawn to canvases.  Do they still work?
> > 
> > No, they don't. If they're in the background they get black frames.
> 
> Seems like should still work, no?

No.

There's no way to synchronously enable a track and paint it.  We could hypothetically not disable tracks of videos that have been painted to a canvas, but we'd still not be able to synchronously paint a disabled track that has never before been painted, and since we paint videos when tab snapshots are taken, we'd likely never be able to disable videos in background tabs.

The way I see it, this is similar to disabling/coalescing timers in background tabs. Page authors can't depend on foreground tab behavior continuing when their tab is backgrounded.

[Jer Noble]

Created attachment 294734 [details]
Patch

[Build Bot]

Comment on attachment 294734 [details]
Patch

Attachment 294734 [details] did not pass mac-ews (mac):
Output: http://webkit-queues.webkit.org/results/2515699

New failing tests:
media/video-muted-audible-tracks.html
media/nodesFromRect-shadowContent.html
media/video-background-tab-visible-tracks.html
media/muted-video-is-playing-audio.html

[Build Bot]

Created attachment 294744 [details]
Archive of layout-test-results from ews103 for mac-yosemite

The attached test failures were seen while running run-webkit-tests on the mac-ews.
Bot: ews103  Port: mac-yosemite  Platform: Mac OS X 10.10.5

[Build Bot]

Comment on attachment 294734 [details]
Patch

Attachment 294734 [details] did not pass mac-wk2-ews (mac-wk2):
Output: http://webkit-queues.webkit.org/results/2515727

New failing tests:
media/video-muted-audible-tracks.html
media/nodesFromRect-shadowContent.html
media/video-background-tab-visible-tracks.html
media/muted-video-is-playing-audio.html

[Build Bot]

Created attachment 294750 [details]
Archive of layout-test-results from ews107 for mac-yosemite-wk2

The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: ews107  Port: mac-yosemite-wk2  Platform: Mac OS X 10.10.5

[Build Bot]

Comment on attachment 294734 [details]
Patch

Attachment 294734 [details] did not pass mac-debug-ews (mac):
Output: http://webkit-queues.webkit.org/results/2515689

New failing tests:
media/video-muted-audible-tracks.html
media/nodesFromRect-shadowContent.html
media/video-background-tab-visible-tracks.html
media/muted-video-is-playing-audio.html

[Build Bot]

Created attachment 294751 [details]
Archive of layout-test-results from ews113 for mac-yosemite

The attached test failures were seen while running run-webkit-tests on the mac-debug-ews.
Bot: ews113  Port: mac-yosemite  Platform: Mac OS X 10.10.5

[Build Bot]

Comment on attachment 294734 [details]
Patch

Attachment 294734 [details] did not pass ios-sim-ews (ios-simulator-wk2):
Output: http://webkit-queues.webkit.org/results/2515752

New failing tests:
media/video-muted-audible-tracks.html
media/video-background-tab-visible-tracks.html
media/muted-video-is-playing-audio.html

[Build Bot]

Created attachment 294753 [details]
Archive of layout-test-results from ews123 for ios-simulator-wk2

The attached test failures were seen while running run-webkit-tests on the ios-sim-ews.
Bot: ews123  Port: ios-simulator-wk2  Platform: Mac OS X 10.11.6

[Jer Noble]

Whoops. forgot to rebaseline.

[Jer Noble]

Created attachment 294891 [details]
Patch

[Build Bot]

Comment on attachment 294891 [details]
Patch

Attachment 294891 [details] did not pass mac-wk2-ews (mac-wk2):
Output: http://webkit-queues.webkit.org/results/2522558

New failing tests:
media/nodesFromRect-shadowContent.html
media/muted-video-is-playing-audio.html

[Build Bot]

Created attachment 294903 [details]
Archive of layout-test-results from ews107 for mac-yosemite-wk2

The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: ews107  Port: mac-yosemite-wk2  Platform: Mac OS X 10.10.5

[Build Bot]

Comment on attachment 294891 [details]
Patch

Attachment 294891 [details] did not pass mac-ews (mac):
Output: http://webkit-queues.webkit.org/results/2522672

New failing tests:
media/nodesFromRect-shadowContent.html
media/muted-video-is-playing-audio.html

[Build Bot]

Created attachment 294908 [details]
Archive of layout-test-results from ews102 for mac-yosemite

The attached test failures were seen while running run-webkit-tests on the mac-ews.
Bot: ews102  Port: mac-yosemite  Platform: Mac OS X 10.10.5

[Build Bot]

Comment on attachment 294891 [details]
Patch

Attachment 294891 [details] did not pass mac-debug-ews (mac):
Output: http://webkit-queues.webkit.org/results/2522661

New failing tests:
media/nodesFromRect-shadowContent.html
media/muted-video-is-playing-audio.html

[Build Bot]

Created attachment 294909 [details]
Archive of layout-test-results from ews115 for mac-yosemite

The attached test failures were seen while running run-webkit-tests on the mac-debug-ews.
Bot: ews115  Port: mac-yosemite  Platform: Mac OS X 10.10.5

[Build Bot]

Comment on attachment 294891 [details]
Patch

Attachment 294891 [details] did not pass ios-sim-ews (ios-simulator-wk2):
Output: http://webkit-queues.webkit.org/results/2522775

New failing tests:
media/video-background-tab-visible-tracks.html

[Build Bot]

Created attachment 294910 [details]
Archive of layout-test-results from ews124 for ios-simulator-wk2

The attached test failures were seen while running run-webkit-tests on the ios-sim-ews.
Bot: ews124  Port: ios-simulator-wk2  Platform: Mac OS X 10.11.6

[Gavin Barraclough]

(In reply to comment #32)
> (In reply to comment #24)
> > (In reply to comment #23)
> > > (In reply to comment #22)
> > > > What happens to videos being drawn to canvases.  Do they still work?
> > > 
> > > No, they don't. If they're in the background they get black frames.
> > 
> > Seems like should still work, no?
> 
> No.
> 
> There's no way to synchronously enable a track and paint it.  We could
> hypothetically not disable tracks of videos that have been painted to a
> canvas, but we'd still not be able to synchronously paint a disabled track
> that has never before been painted, and since we paint videos when tab
> snapshots are taken, we'd likely never be able to disable videos in
> background tabs.
> 
> The way I see it, this is similar to disabling/coalescing timers in
> background tabs. Page authors can't depend on foreground tab behavior
> continuing when their tab is backgrounded.

Agreed. If our choices are:

[1] Continue decoding video frames in background tabs.
[2] Stop decoding video frames in background tabs, continue everything else.
[3] Stop decoding video frames in background tabs, stop everything else.

We should test [2] for web compatibility. [1] is bad for CPU, memory, battery usage. [2] is better than [3] if we believe there is value in running background tabs at all.

[Sam Weinig]

(In reply to comment #32)
> (In reply to comment #24)
> > (In reply to comment #23)
> > > (In reply to comment #22)
> > > > What happens to videos being drawn to canvases.  Do they still work?
> > > 
> > > No, they don't. If they're in the background they get black frames.
> > 
> > Seems like should still work, no?
> 
> No.
> 
> There's no way to synchronously enable a track and paint it.  We could
> hypothetically not disable tracks of videos that have been painted to a
> canvas, but we'd still not be able to synchronously paint a disabled track
> that has never before been painted, and since we paint videos when tab
> snapshots are taken, we'd likely never be able to disable videos in
> background tabs.
> 
> The way I see it, this is similar to disabling/coalescing timers in
> background tabs. Page authors can't depend on foreground tab behavior
> continuing when their tab is backgrounded.

I don't think these are exactly the same. Imagine a page that is doing some kind of image analysis on a video. Going into a background tab should not break that functionality. Timer throttling might slow something down, but it won't change the result.

[Jer Noble]

(In reply to comment #53)
> I don't think these are exactly the same. Imagine a page that is doing some
> kind of image analysis on a video. Going into a background tab should not
> break that functionality. Timer throttling might slow something down, but it
> won't change the result.

That's not really true, especially with your image analysis example.  If the page is playing a video in a background tab and doing analysis on the output, it's going to set a timer to fire at the frame rate of the video, e.g. 24 times per second.  If timers are being throttled, the page is definitely going to miss decoded frames, which is functionally equivalent to those frames never being decoded.

[Brady Eidson]

Comment on attachment 294891 [details]
Patch

Patches that have been up for review since 2016 are almost certainly too stale to be relevant to trunk in their current form.

If this patch is still important please rebase it and post it for review again.