Implement mozilla's requestAnimationFrame API

• Component: New Bugs
• Status: RESOLVED FIXED

[James Robinson]

Mozilla has proposed an API to coordinate and rate limit imperative animations: http://weblogs.mozillazine.org/roc/archives/2010/08/mozrequestanima.html.  The basic idea is that instead of imperative animations driving themselves off of setTimeout()s, the author would inform the browser of elements that are animated and the browser would tell the page when to update the animation state.  I think we should implement something similar in WebKit.

Here are the goals that I think are important (I'll talk about possible implementations in comments):

1.) Make it easy for authors to write animation code that does not waste CPU updating animations that are not visible.  A WebGL game or Canvas 2D banner ad should not be consuming lots of resources when they are in a background tab or scrolled well outside the viewport.  Web developers are for the most part lazy so we should make the easy way efficient and not require a lot of work on the developer's part.

2.) Allow the browser to rate-limit animations to a reasonable target framerate and adapt to slow running JS.  On most displays it makes no sense to update animations at faster than 60Hz, but authors who write setTimeout(.., 0) may try to run faster today.  Additionally we should have some flexibility when dealing with pages that have some content that updates very slowly (say some highly complex WebGL-rendered scene) in order to not lock the entire page to the framerate of the slowest element.

The W3C FX task force has expressed interest in this proposal.  We should provide any feedback to them and be responsive to any input they might have.

[James Robinson]

Here's an initial proposal for webkitRequestAnimationFrame based off of the initial Mozilla proposal, this public-webapps thread: http://lists.w3.org/Archives/Public/public-webapps/2010OctDec/0644.html, and chatting with roc:

New function on Element:
void webkitRequestAnimationFrame(callback);

This indicates that the element is being animated and requests that the browser invoke the callback at some point in the future to update the element's animation state.  At some point after this function is called (for example just before the next repaint), the specified callback is invoked with no arguments and the page can update whatever animation state it needs to (update DOM positions, make WebGL or canvas 2d calls, etc).  The callback is only invoked in cases where the element is (or might soon be) visible - the callback won't be invoked if the tab is hidden or the element is scrolled well offscreen.

New property on DOMWindow:
readonly attribute DOMTimeStamp webkitAnimationTime;

This property always returns the "current time" to be used by animations.  This value does not change while javascript is running but is updated between calls.  Any programmatic queries or updates to declarative animations (like starting a new animation) take place logically at this time (rather than at the current time for the next recalcStyle) and any declarative animation updates in the next paint *should* use this time as well.

In the pure software rendering case, it's pretty easy to satisfy this API by calling all relevant webkitRequestAnimationFrame() callbacks before painting and then using the webkitAnimationTime value (if it was queried) to evaluate CSS/SVG animation state during the paint.  As an optimization, we could also monitor the amount of time that the webkitRequestAnimationFrame() callbacks are taking and decide to produce some frames without running the callbacks if they are super slow every time.

The h/w accelerated case is trickier but I still think the API is useful.  In particular, the composition can be happening in another thread and may be behind an opaque API.  However since WebKit is always aware of declarative animations we can at the very least 'guess' and invoke the webkitRequestAnimationFrame() callbacks for potentially visible content on a regular timer during times that the compositor is producing frames, with a best-guess approximation for window.webkitAnimationTime.  This would allow us to avoid invoking the callback for offscreen content, rate-limit animations to a reasonable framerate, and if the compositor provides sufficient information rate-limit the callbacks to the compositor's framerate if the compositor is not able to sustain 60FPS.

The main difference from the initial Mozilla proposal is dropping the beforepaint event and making requestAnimationFrame per-element rather than per-window so that we can bounds check the element with the viewport and avoid invoking the callback for offscreen elements.

I'm working on an initial implementation that will be fairly basic and not do many of the optimizations mentioned above.  I think it's important that we get something out there that developers can start experimenting with that has good semantics that we can then optimize later on, especially as people start experimenting more with WebGL and heavy canvas-based animations that can take considerable processing power to render.

[Simon Fraser (smfr)]

(In reply to comment #1)
> Here's an initial proposal for webkitRequestAnimationFrame based off of the initial Mozilla proposal, this public-webapps thread: http://lists.w3.org/Archives/Public/public-webapps/2010OctDec/0644.html, and chatting with roc:
> 
> New function on Element:
> void webkitRequestAnimationFrame(callback);
> 
> This indicates that the element is being animated and requests that the browser invoke the callback at some point in the future to update the element's animation state.  At some point after this function is called (for example just before the next repaint), the specified callback is invoked with no arguments and the page can update whatever animation state it needs to (update DOM positions, make WebGL or canvas 2d calls, etc).  The callback is only invoked in cases where the element is (or might soon be) visible - the callback won't be invoked if the tab is hidden or the element is scrolled well offscreen.

Simple callbacks like this have issues:
1. how to you unregister the callback?
2. what happens if you call webkitRequestAnimationFrame() twice on the same element? Does it replace the callback, or queue both?

Why is this better than firing an event?

It's not clear to me that bounds-checking the elements is really useful or do-able. How do you know that an element won't be visible for the next frame because of CSS animations, or scrolling?

[James Robinson]

(In reply to comment #2)
> (In reply to comment #1)
> > Here's an initial proposal for webkitRequestAnimationFrame based off of the initial Mozilla proposal, this public-webapps thread: http://lists.w3.org/Archives/Public/public-webapps/2010OctDec/0644.html, and chatting with roc:
> > 
> > New function on Element:
> > void webkitRequestAnimationFrame(callback);
> > 
> > This indicates that the element is being animated and requests that the browser invoke the callback at some point in the future to update the element's animation state.  At some point after this function is called (for example just before the next repaint), the specified callback is invoked with no arguments and the page can update whatever animation state it needs to (update DOM positions, make WebGL or canvas 2d calls, etc).  The callback is only invoked in cases where the element is (or might soon be) visible - the callback won't be invoked if the tab is hidden or the element is scrolled well offscreen.
> 
> Simple callbacks like this have issues:
> 1. how to you unregister the callback?

We could return an id from the register callback and provide a cancel() interface (like setTimeout/clearTimeout), if cancelling the callback is something we want to support.  I'm not sure of what the immediate need is for this.

> 2. what happens if you call webkitRequestAnimationFrame() twice on the same element? Does it replace the callback, or queue both?

Each call enqueues an additional callback.

> 
> Why is this better than firing an event?

The callback is a one-time thing so that the default behavior is better, so it doesn't really match the event semantics.  Also I find callbacks a little simpler to deal with (but that's more a matter of taste).

> 
> It's not clear to me that bounds-checking the elements is really useful or do-able. How do you know that an element won't be visible for the next frame because of CSS animations, or scrolling?

The important thing about this API is it is always safe to invoke the callback on an element that isn't really visible, so we can be conservative and fire the callback if in doubt.  In the case of CSS animations, WebCore knows what animations are in progress at any point in time so it can either estimate the animation state or just give up and fire the callback on any element that has an in-progress animations.  Fast scrolling is normally implemented by rendering content past the viewport in the scrollable direction by some margin, so we can fire the callback at any elements within this over-rendered region.

I think we'll want to have this flexibility down the road even if we don't take immediate advantage of it (and fire the callback on every element, for example).

[Simon Fraser (smfr)]

> > Simple callbacks like this have issues:
> > 1. how to you unregister the callback?
> 
> We could return an id from the register callback and provide a cancel() interface (like setTimeout/clearTimeout), if cancelling the callback is something we want to support.  I'm not sure of what the immediate need is for this.

I think it's always a good idea to allow cancellation for any API that allows registration.

> > 2. what happens if you call webkitRequestAnimationFrame() twice on the same element? Does it replace the callback, or queue both?
> 
> Each call enqueues an additional callback.

Even if the callback is the "same"?

Some similar issues came up in the discussion of styleMedia.matchMedia() (see bug 37205). Comparing callbacks is hard, which is why I'm hoping to avoid a callback-based solution here.

> > Why is this better than firing an event?
> 
> The callback is a one-time thing so that the default behavior is better, so it doesn't really match the event semantics.  Also I find callbacks a little simpler to deal with (but that's more a matter of taste).

I agree that repeating/not-repeating is the main difference, but I still think event semantics are a better fit. I does worry me that badly behaved content might leave a lot of 'beforePaint' event handlers registered though.

> > It's not clear to me that bounds-checking the elements is really useful or do-able. How do you know that an element won't be visible for the next frame because of CSS animations, or scrolling?
> 
> The important thing about this API is it is always safe to invoke the callback on an element that isn't really visible, so we can be conservative and fire the callback if in doubt.  In the case of CSS animations, WebCore knows what animations are in progress at any point in time so it can either estimate the animation state or just give up and fire the callback on any element that has an in-progress animations.  Fast scrolling is normally implemented by rendering content past the viewport in the scrollable direction by some margin, so we can fire the callback at any elements within this over-rendered region.
> 
> I think we'll want to have this flexibility down the road even if we don't take immediate advantage of it (and fire the callback on every element, for example).

Makes sense. Hopefully you can compute visibility when affected by scrolling & software animations before calling the callbacks.

[James Robinson]

(In reply to comment #4)
> > > Simple callbacks like this have issues:
> > > 1. how to you unregister the callback?
> > 
> > We could return an id from the register callback and provide a cancel() interface (like setTimeout/clearTimeout), if cancelling the callback is something we want to support.  I'm not sure of what the immediate need is for this.
> 
> I think it's always a good idea to allow cancellation for any API that allows registration.

Hard to argue with that :)

> 
> > > 2. what happens if you call webkitRequestAnimationFrame() twice on the same element? Does it replace the callback, or queue both?
> > 
> > Each call enqueues an additional callback.
> 
> Even if the callback is the "same"?

Why not?  If I call setTimeout() twice with the same callback I get two timers.

> 
> Some similar issues came up in the discussion of styleMedia.matchMedia() (see bug 37205). Comparing callbacks is hard, which is why I'm hoping to avoid a callback-based solution here.
> 
> > > Why is this better than firing an event?
> > 
> > The callback is a one-time thing so that the default behavior is better, so it doesn't really match the event semantics.  Also I find callbacks a little simpler to deal with (but that's more a matter of taste).
> 
> I agree that repeating/not-repeating is the main difference, but I still think event semantics are a better fit. I does worry me that badly behaved content might leave a lot of 'beforePaint' event handlers registered though.
> 

I think that people are used to using setTimeout() to update their animations today, so a one-shot callback based system is much closer to the model that developers are already using.  I think we'll probably want to hash this out in a broader list (public-fx or public-webapps) to get more feedback from other implementors, developers, etc.  There are valid points on both sides.

[James Robinson]

Created attachment 77944 [details]
initial patch for discussion

[Simon Fraser (smfr)]

Comment on attachment 77944 [details]
initial patch for discussion

View in context: https://bugs.webkit.org/attachment.cgi?id=77944&action=review

This seems like a reasonable start.

> WebCore/dom/Document.cpp:423
> +    , m_requestAnimationFrameCallbackId(false)

Don't you mean 0?

> WebCore/dom/Document.h:1390
> +    int m_requestAnimationFrameCallbackId;

I'd call this m_nextRequestAnimationFrameCallbackId

> WebCore/dom/RequestAnimationFrameCallback.h:47
> +    bool m_enabled;
> +    int m_id;
> +    RefPtr<Element> m_element;

Re-order these to optimize padding.

> WebCore/page/DOMWindow.cpp:1472
> +    // FIXME: Multiple queries of this value within the same script execution run
> +    // should return the same value, so we need to calculate this value lazily and
> +    // store it.

And use the same value for CSS animations etc. See AnimationControllerPrivate::beginAnimationUpdateTime() for something similar.

> WebCore/page/FrameView.h:101
> +    void requestAnimationFrames();

I don't really like the name (and "Frames" is really confusing in this context). serviceAnimations? processAnimations?

[Darin Fisher (:fishd, Google)]

Comment on attachment 77944 [details]
initial patch for discussion

View in context: https://bugs.webkit.org/attachment.cgi?id=77944&action=review

> WebKit/chromium/public/WebWidget.h:59
> +    virtual void animate() = 0;

nit: add a comment describing when this should be called?

> WebKit/chromium/public/WebWidgetClient.h:59
> +    virtual void scheduleAnimation() { }

nit: please add a comment similar to the one above scheduleComposite.

[James Robinson]

Thanks for the feedback.  I'll address it and work on implementing webkitAnimationTime as a separate patch as it seems fairly independent.

[James Robinson]

Created attachment 78468 [details]
Patch

[Darin Fisher (:fishd, Google)]

Comment on attachment 78468 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=78468&action=review

would be good to add tests for changing the display property (hiding/showing an element) from the animation callback of another element.

> Source/WebCore/dom/Document.cpp:4959
> +        m_requestAnimationFrameCallbacks = new Vector<RefPtr<RequestAnimationFrameCallback> >();

nit: a typedef might be nice?
typedef Vector<RefPtr<RequestAnimationFrameCallback> > RequestAnimationFrameCallbackList;

> Source/WebCore/dom/Document.cpp:4990
> +    // visibility so this code has to iterate carefully.

maybe just add a FIXME or something about the fact that we don't do this visibility test yet.

> Source/WebCore/dom/Document.cpp:5008
> +                callback->m_enabled = false;

m_enabled might be better if named m_firedOrCancelled (inverting the sense).

> Source/WebCore/dom/Document.cpp:5023
> +            m_requestAnimationFrameCallbacks->insert(callbacksAdded++, callbacks[i]);

any callback that is no longer enabled needs to be removed from this list.

maybe you even want to just loop over m_requestAnimationFrameCallbacks??

> Tools/DumpRenderTree/chromium/WebViewHost.cpp:636
> +    webkit_support::PostDelayedTask(invokeScheduleComposite, static_cast<void*>(this), 0);

isn't this cast unnecessary since 'this' is a pointer type?

[James Robinson]

(In reply to comment #11)
> (From update of attachment 78468 [details])
> View in context: https://bugs.webkit.org/attachment.cgi?id=78468&action=review
> 
> would be good to add tests for changing the display property (hiding/showing an element) from the animation callback of another element.

done

> 
> > Source/WebCore/dom/Document.cpp:4959
> > +        m_requestAnimationFrameCallbacks = new Vector<RefPtr<RequestAnimationFrameCallback> >();
> 
> nit: a typedef might be nice?
> typedef Vector<RefPtr<RequestAnimationFrameCallback> > RequestAnimationFrameCallbackList;

done (and it does help a lot)

> 
> > Source/WebCore/dom/Document.cpp:4990
> > +    // visibility so this code has to iterate carefully.
> 
> maybe just add a FIXME or something about the fact that we don't do this visibility test yet.

FIXME added

> 
> > Source/WebCore/dom/Document.cpp:5008
> > +                callback->m_enabled = false;
> 
> m_enabled might be better if named m_firedOrCancelled (inverting the sense).

done

> 
> > Source/WebCore/dom/Document.cpp:5023
> > +            m_requestAnimationFrameCallbacks->insert(callbacksAdded++, callbacks[i]);
> 
> any callback that is no longer enabled needs to be removed from this list.
> 
> maybe you even want to just loop over m_requestAnimationFrameCallbacks??

right, done

> 
> > Tools/DumpRenderTree/chromium/WebViewHost.cpp:636
> > +    webkit_support::PostDelayedTask(invokeScheduleComposite, static_cast<void*>(this), 0);
> 
> isn't this cast unnecessary since 'this' is a pointer type?

gcc says the cast is unnecessary.  I just copy-pasted that from other code - looks like the original reference to webkit_support::PostDelayedTask() had the static_cast<> (http://trac.webkit.org/changeset/66701) and I'm not sure why.  Removed

[James Robinson]

Created attachment 78604 [details]
Patch

[James Robinson]

Committed r76194: <http://trac.webkit.org/changeset/76194>

[WebKit Review Bot]

http://trac.webkit.org/changeset/76194 might have broken Chromium Win Release

[James Robinson]

Reverted r76194 for reason:

Caused mysterious compile failure on the chromium win build.webkit.org bots

Committed r76198: <http://trac.webkit.org/changeset/76198>

[James Robinson]

Reverted due to https://bugs.webkit.org/show_bug.cgi?id=52837.  Will re-land after the fix for that bug lands.

[James Robinson]

Committed r76278: <http://trac.webkit.org/changeset/76278>