Bug 241402 – Decoded image data is not reused by canvas causing long task blocking on the main thread

• Status: NEW
• Component: Canvas

[Xidorn Quan]

Created attachment 460082 [details]
testcase

STR:
1. download the attachment and serve it and open it
2. click the button in the page

Expected result:
The decoding takes sometime, but drawing is instant, and there is no long task detected on the main thread.

Actual result:
Both the decoding and the drawing takes roughly the same amount of time, and there is a long task indicates the main thread is blocked.


Seemingly that the drawing the image on canvas then `getImageData` forces a decoding of the image even if the image is already decoded through `img.decode()`. It makes it hard to avoid a long task on main thread when dealing with large images.

Blink has similar issue: https://bugs.chromium.org/p/chromium/issues/detail?id=1334448

On Gecko, I get the expected result.

[Radar WebKit Bug Importer]

<rdar://problem/95171712>

[Kimmo Kinnunen]

Thanks for the test-case!
For WebKit internals:
The long duration after initial decode() is due to the way WP transfers the data to GPUP the first time. The new NativeImage gets sent over to GPUP during this duration.

There are at least 3 different undesirable behaviors in WebKit:
1. initial draw after decode() is slow
2. Multiple .src assignments and decode()s are slow for same url. 
3. Big getImageData() is slow (10ms on WebKit, ~0ms on Firefox)

[Thomas Steiner]

See https://bugs.chromium.org/p/chromium/issues/detail?id=1334448#c9 for a hosted demo and https://bugs.chromium.org/p/chromium/issues/detail?id=1334448#c10 for a theory of why the seemingly double decoding might happen.

[Xidorn Quan]

Thomas suggested to us that for this testcase, we can set `image.decoding` to `async`, and that seems to help on Chrome (where `draw` call is now immediate most of time), but not on Safari.

[Xidorn Quan]

No, I was wrong. It didn't help on Chrome either.

Chrome was to some extent cheating by somehow reusing a decoded image from a previous image load. If I randomize the URL by adding `?` with `Math.random()` to the end of the URL, double decoding happens every time again.

[Said Abou-Hallawa]

This scenario shows multiple perf issues which are related to explicit image decoding and then drawing the same image.

Calling HTMLImageElement.decode() forces decoding the image asynchronously  with its natural size. In this test case, the size of ss_s3.webp is { 7200 x 4200 }. The size of the decoded image in bytes ~= 120 MB. When drawing this image to a size of { 2400, 1400 }, WebKit uses the last decoded image since its size is larger than the requested size. So it transfers these 120MB to GPUProcess. And to transfer this data it draws the image to GraphicsContext backed by a shared memory of this size. After GPUProcess receives this data it draws it by scaling it down to { 2400, 1400 } which is 1/9 the original size. So WebKit does the following to draw the image in the canvas:

a. Decode an image into 120MB buffer
b. Drawing an image into 120MB shared memory
c. Sending these 120MB to GPUProcess
d. Drawing these  these 120MB into 13MB canvas

We could have saved 8/9 of the sizes if we knew from the beginning that this image will be drawn to a much smaller buffer. I can see two solution to this problem.

i. Adding an optional size argument to the decode() method which I already suggested here: https://github.com/whatwg/html/issues/11935. In this case, the test case will decode the image like this `await image.decode(2400, 1400);`. This will make us decode the image to 13MB not 120MB.
ii. Currently we re-decode the image only if the decoded size is smaller than the destination size. I think we should re-decode the image if the decoded size is much larger than the destination size. We can use a heuristic like if the area(decoded size) / area(destination size) >= 3, then we re-decode the image. I think an extra decoding, sharing and drawing 13MB will be faster than sticking and dealing with 120MB.

[Said Abou-Hallawa]

I was wrong, the image is drawn to the canvas without scaling:

    ctx.drawImage(image, 0, 1400, 2400, 1400, 0, 0, 2400, 1400);

This draws a sub-rectangle from the image to the canvas without scaling down.