Bug 84987 – [Qt] Accelerated 2D canvas

• Component: Canvas
• Status: RESOLVED INVALID

[Noam Rosenthal]

Implement a GL-backed accelerated canvas path for Qt.

[Noam Rosenthal]

I think that this can be done in two relatively separate phases:
1. use the current Qt GL paint engine
2. create an additional GL paint engine, tuned specifically for canvas cases.

There might be other ways to go about this (e.g. do this in trunk webkit instead of in Qt) but I believe this one is the path of least resistance.

[Zoltan Herczeg]

For me I have no preference. However, many ports could be benefit from a trunk accelerated 2D library, so perhaps we could share the work as well in that case.

[Jocelyn Turcotte]

Just my point of view after a quick investigation:

Having a shared implementation would be major work, we basically would need a pure OpenGL implementation of CanvasRenderingContext2D. And I'm not sure if it would be legally possible to fork the Qt OpenGL paint engine under WebKit. It could be worth to do it, but I think it's easy to underestimate the amount of work it would need.

We're trying to get the canvas working with the OpenGL paint engine first to see what kind of issues we'll get.
Creating another paint engine would still force us to keep it under QPainter, so we might as well maintain the current paint engine as long as our fixes/optimizations aren't interfering with more complex QPainter use cases.

A similar option to that would be to extract and access the OpenGL paint engine's routines directly from CanvasRenderingContext2D, and have different implementations for our canvas when it matters.

[Noam Rosenthal]

(In reply to comment #3)
> Just my point of view after a quick investigation:
> 
> I'm not sure if it would be legally possible to fork the Qt OpenGL paint engine under WebKit. It could be worth to do it, but I think it's easy to underestimate the amount of work it would need.

It doesn't have to be under WebKit, we can create a custom paint engine in Qt optimized for canvas.
But I think that in all cases the first step should be to try to run accelerated canvas with the current GL engine.

[Noam Rosenthal]

Created attachment 139925 [details]
WIP: not for review!

I'm trying something out wrt gluing this with the canvas code.
The idea is that CanvasRenderingContext2D would paint to a paint-command buffer instead of directly to an image buffer, and we'll render the results only when we need to (e.g. when we need to composite the canvas, or somebody called toDataURL/getImageData).
Before we do anything with GL, we can do something that would make life easier on us, by creating that paint-buffer abstraction.

This patch is not ready yet (still some tests fail etc.) but it's almost there, so I wanted some feedback!

[Igor Trindade Oliveira]

+1 for this patch. With this "journaling" approach we can store the canvas2d calls(when using opengl it will be opengl calls) and push all them in batches, probably we will reduce context switch.

[Zoltan Herczeg]

> I'm trying something out wrt gluing this with the canvas code.
> The idea is that CanvasRenderingContext2D would paint to a paint-command buffer instead of directly to an image buffer, and we'll render the results only when we need to (e.g. when we need to composite the canvas, or somebody called toDataURL/getImageData).
> Before we do anything with GL, we can do something that would make life easier on us, by creating that paint-buffer abstraction.

Just trying to understand the bigger picture. We introduce a new PaintBuffer. Do we want to keep it or it is just a temporary new class? I mean the long term plan is to switch the GraphicsContext to use GLES2 on mobile.

[Jocelyn Turcotte]

The idea looks cool, but is there any way to avoid yet another #if USE?

I'd also like to know what would be the benefits regarding canvas animations using constant timers and requestAnimationFrame. Buffering commands for 5 frames wouldn't be good.

The way I see it with my limited knowledge, requestAnimationFrame should get a response fired when we are ready to composit a new frame, and no command buffering would be needed.

[Noam Rosenthal]

(In reply to comment #8)
> The idea looks cool, but is there any way to avoid yet another #if USE?
Sure, we can go either way with that.

> I'd also like to know what would be the benefits regarding canvas animations using constant timers and requestAnimationFrame. Buffering commands for 5 frames wouldn't be good.
> 
No, we buffer only the commands that go in a single frame.
First of all, for each frame we'll probably get to the point where the frame needs resolving, which would clear the buffer.
Second, usually in the beginning of the frame there is some kind of command that clears the canvas, which would also clear the queue.

[Noam Rosenthal]

> Just trying to understand the bigger picture. We introduce a new PaintBuffer. Do we want to keep it or it is just a temporary new class? I mean the long term plan is to switch the GraphicsContext to use GLES2 on mobile.

Not sure if that's feasible. The idea is that because canvas is more limited, we have more information to accelerate it. PaintBuffer would accumulate the information, and at some point would choose how to render based on the information it has.
I'm not sure we can do that with standard painting or in the information we have under GraphicsContext.

[Jocelyn Turcotte]

I worked for a while on an implementation using the QOpenGLPaintEngine in Qt starting from No'am's patch.
It is still pretty unpolished but it works well, I'll have a look at pushing it further if you think it's worth it.

Currently most of the visual issues are related to layouting of the layers and the compositing. To see which issues are related to the paint engine I added a "fake" accelerated mode that use the raster engine and transfers it directly through GraphicsSurface instead of non-composited contents tiles (itself already improve performances a bit).

Issues with QOpenGLPaintEngine that would remain to solve:
1- When mixing non-accelerated (QPixmap based) and accelerated (FBO based) canvases, there is no way to efficiently know if the QPixmap should be re-uploaded in the texture cache. Right now a QPixmap is re-uploaded for every paint if a QPainter is still active on it, which is always the case for the canvas.
I (ab)used the ensureActiveTarget callback in QOpenGLPaintDevice to achieve detecting buffer dirtiness with accelerated-to-accelerated drawImage()s, but there is no such thing with the raster paint engine.
To solve this I think that we would need to remove the paintingActive check in the texture cache and add dirtiness detection in GraphicsContext to know when to trigger a re-upload (possibly by calling QPixmap::detach() and making sure there is only one implicitely shared instance).
One of the commits force to always use accelerated canvases to work around this.

2- There is still work to do to make it fast at least on desktop. Right now my Intel HD 3000 video card produce only slightly better performance than the raster engine with my dual i7 processor. I'd like to know what kind of improvements this would give on a device but I don't currently have access to one implementing GraphicsSurface, so I can't see myself.
I think that there just isn't anybody yet that tried hard to polish the paint engine for speed, I saw a couple of places where simple patches could improve the speed already.

Anyway here it is, the patches for WebKit are there:
https://github.com/jturcotte/webkit/tree/qpainterAcceleratedCanvas
And you also need to cherry-pick this texture tiling implementation in qtbase (that I wouldn't want to push before 5.0 is branched):
https://qt.gitorious.org/~jturcotte/qt/jturcottes-qtbase/commits/textureTiling

[Noam Rosenthal]

(In reply to comment #11)
> I worked for a while on an implementation using the QOpenGLPaintEngine in Qt starting from No'am's patch.
> It is still pretty unpolished but it works well, I'll have a look at pushing it further if you think it's worth it.
> 

cool! I plan to have a look next week since it's kind of a long git download :)

[Jocelyn Turcotte]

=== Bulk closing of Qt bugs ===

If you believe that this bug report is still relevant for a non-Qt port of webkit.org, please re-open it and remove [Qt] from the summary.

If you believe that this is still an important QtWebKit bug, please fill a new report at https://bugreports.qt-project.org and add a link to this issue. See http://qt-project.org/wiki/ReportingBugsInQt for additional guidelines.