Bug 259676 – safari throw error "Unable to get image data from canvas. Requested size was 640 x 480"

• Status: RESOLVED
• Resolution: INVALID
• Component: Canvas

Created attachment 467162 [details]
the core code

In Ipad safari, Just draw a video frame to canvas and call getImageData, sometimes will throw the error occasionally. I found some same issue but no reply. 

https://discussions.apple.com/thread/254087950
https://developer.apple.com/forums/thread/711639

In my test case, if I have opened the browser for a long time, the probability of occurrence will become higher. Reload the page is not works, but restart it can be recovered.
It will be easier to trigger with devtools on, but it can be confirmed that even if devtools is never opened, the problem still occurs.

It seems that some of memory for canvas cannot be released. But I have no idea.

I can reproduce the issue in IpadOS15/safari15 and ipados16/safari16. I tried using OffscreenCanvas, it didn't work.

I want to know what factors would affect the normal execution of getImageData. Is it related to me creating too many canvas on the page, or if there is an issue with how I'm invoking it, is there another way I can collect the data?

[Radar WebKit Bug Importer]

<rdar://problem/113542310>

Hi, is there any update? We strongly rely on this feature. Hope to get a reply. Thanks.

[Said Abou-Hallawa]

I think your web page is causing WebKit to reach the memory limit on your device. If you are calling getImageData() very often without getting rid of the existing ones, this will cause this error to be thrown.

CanvasRenderingContext2DBase::getImageData() tries to avoid terminating the MobileSafari by not blindly allocates whatever you are asking for. It tries to allocate the ImageData if it can. If it can't it will return a nil ImageData.

This block of code in this function throws the error you see in your page:

    if (!is<ByteArrayPixelBuffer>(pixelBuffer)) {
        canvasBase().scriptExecutionContext()->addConsoleMessage(MessageSource::Rendering, MessageLevel::Error,
            makeString("Unable to get image data from canvas. Requested size was ", imageDataRect.width(), " x ", imageDataRect.height()));
        return Exception { InvalidStateError };
    }

An ImageData with size 640 x 480 requires 640 x 480 x 4 = 1.23MB which is not big unless your page calls getImageData() for every frame in the video.

The general rule is do not call getImageData() unless you really need it. And if you do, get rid of it as soon as you can. 

But remember JavaScript does not release the unreferenced objects immediately. There is no way to trigger JS garbage collector pragmatically and there is no way also to know when it is going to be triggered.

I am almost sure the garbage collector can't be triggered after drawing every frame from the video.

[Said Abou-Hallawa]

I am resolving this bug as "Invalid".

If you think my analysis and expectations about your page is incorrect, please reopen it. 

It will be great if you share more info about what your page is exactly doing over time.

It will be also helpful if you can create a reduced test case. A simple webpage which mimics what you do will be great as well.

(In reply to Said Abou-Hallawa from comment #4)
> I think your web page is causing WebKit to reach the memory limit on your
> device. If you are calling getImageData() very often without getting rid of
> the existing ones, this will cause this error to be thrown.
> 
> CanvasRenderingContext2DBase::getImageData() tries to avoid terminating the
> MobileSafari by not blindly allocates whatever you are asking for. It tries
> to allocate the ImageData if it can. If it can't it will return a nil
> ImageData.
> 
> This block of code in this function throws the error you see in your page:
> 
>     if (!is<ByteArrayPixelBuffer>(pixelBuffer)) {
>        
> canvasBase().scriptExecutionContext()->addConsoleMessage(MessageSource::
> Rendering, MessageLevel::Error,
>             makeString("Unable to get image data from canvas. Requested size
> was ", imageDataRect.width(), " x ", imageDataRect.height()));
>         return Exception { InvalidStateError };
>     }
> 
> An ImageData with size 640 x 480 requires 640 x 480 x 4 = 1.23MB which is
> not big unless your page calls getImageData() for every frame in the video.
> 
> The general rule is do not call getImageData() unless you really need it.
> And if you do, get rid of it as soon as you can. 
> 
> But remember JavaScript does not release the unreferenced objects
> immediately. There is no way to trigger JS garbage collector pragmatically
> and there is no way also to know when it is going to be triggered.
> 
> I am almost sure the garbage collector can't be triggered after drawing
> every frame from the video.

Thank you for the reply. In our business, I must use getImageData to capture video frames, so it is unavoidable to use this API. I also noticed this issue is related to some memory not being released, but I'm not sure where exactly the problem is.

As you mentioned, I need to get rid of existing ImageData objects immediately. I want to confirm whether this refers to releasing references to the ImageData objects that have been acquired, or releasing resources of the canvas element itself.

I tried some release logic previously but it didn't work. This should be an issue in my code. I just want to confirm what call I should use to ensure the resource release logic is correct.