Bug 70013 – [Chromium] Canvas tests crash on Windows GPU

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Kent Tamura]

The following layout test is failing on Windows:

fast/canvas/canvas-fillPath-gradient-shadow.html

Probable cause:
Unknown

[Kent Tamura]

This is a GPU-only issue.

Backtrace:
	_tnl_import_array [0x1015F744+564] (e:\b\build\slave\webkit_win__dbg__2_\build\src\third_party\mesa\mesalib\src\mesa\tnl\t_draw.c:156)
	bind_inputs [0x1015F481+321] (e:\b\build\slave\webkit_win__dbg__2_\build\src\third_party\mesa\mesalib\src\mesa\tnl\t_draw.c:263)
	_tnl_draw_prims [0x1015F1E0+704] (e:\b\build\slave\webkit_win__dbg__2_\build\src\third_party\mesa\mesalib\src\mesa\tnl\t_draw.c:475)
	_tnl_vbo_draw_prims [0x1015EF08+72] (e:\b\build\slave\webkit_win__dbg__2_\build\src\third_party\mesa\mesalib\src\mesa\tnl\t_draw.c:384)
	vbo_validated_drawrangeelements [0x1001DBF7+391] (e:\b\build\slave\webkit_win__dbg__2_\build\src\third_party\mesa\mesalib\src\mesa\vbo\vbo_exec_array.c:732)
	vbo_exec_DrawElements [0x1001DD74+148] (e:\b\build\slave\webkit_win__dbg__2_\build\src\third_party\mesa\mesalib\src\mesa\vbo\vbo_exec_array.c:886)
	neutral_DrawElements [0x101737EF+239] (e:\b\build\slave\webkit_win__dbg__2_\build\src\third_party\mesa\mesalib\src\mesa\main\vtxfmt_tmp.h:334)
	glDrawElements [0x101BA0A4+36] (e:\b\build\slave\webkit_win__dbg__2_\build\src\third_party\mesa\mesalib\src\mapi\glapi\glapitemp.h:1658)
	gpu::gles2::GLES2DecoderImpl::HandleDrawElements [0x024E192E+558] (e:\b\build\slave\webkit_win__dbg__2_\build\src\gpu\command_buffer\service\gles2_cmd_decoder.cc:4626)
	gpu::gles2::GLES2DecoderImpl::DoCommand [0x024D8D8A+1626] (e:\b\build\slave\webkit_win__dbg__2_\build\src\gpu\command_buffer\service\gles2_cmd_decoder.cc:2663)
	gpu::CommandParser::ProcessCommand [0x024FB99E+446] (e:\b\build\slave\webkit_win__dbg__2_\build\src\gpu\command_buffer\service\cmd_parser.cc:57)
	gpu::GpuScheduler::PutChanged [0x024D1413+451] (e:\b\build\slave\webkit_win__dbg__2_\build\src\gpu\command_buffer\service\gpu_scheduler.cc:61)
	webkit::gpu::GLInProcessContext::PumpCommands [0x00BF5E5E+126] (e:\b\build\slave\webkit_win__dbg__2_\build\src\webkit\gpu\webgraphicscontext3d_in_process_command_buffer_impl.cc:299)
	DispatchToMethod<webkit::gpu::GLInProcessContext,void (__thiscall webkit::gpu::GLInProcessContext::*)(void)> [0x00C17FEC+12] (e:\b\build\slave\webkit_win__dbg__2_\build\src\base\tuple.h:537)
	CallbackImpl<webkit::gpu::GLInProcessContext,void (__thiscall webkit::gpu::GLInProcessContext::*)(void),Tuple0>::RunWithParams [0x00C17615+37] (e:\b\build\slave\webkit_win__dbg__2_\build\src\base\callback_old.h:119)
	CallbackRunner<Tuple0>::Run [0x00C0F510+48] (e:\b\build\slave\webkit_win__dbg__2_\build\src\base\callback_old.h:79)
	gpu::CommandBufferService::Flush [0x024F8305+85] (e:\b\build\slave\webkit_win__dbg__2_\build\src\gpu\command_buffer\service\command_buffer_service.cc:138)
	gpu::CommandBufferHelper::Flush [0x01C42042+66] (e:\b\build\slave\webkit_win__dbg__2_\build\src\gpu\command_buffer\client\cmd_buffer_helper.cc:64)
	gpu::CommandBufferHelper::WaitForAvailableEntries [0x01C429F9+841] (e:\b\build\slave\webkit_win__dbg__2_\build\src\gpu\command_buffer\client\cmd_buffer_helper.cc:170)
	gpu::CommandBufferHelper::GetSpace [0x01C42B95+53] (e:\b\build\slave\webkit_win__dbg__2_\build\src\gpu\command_buffer\client\cmd_buffer_helper.cc:175)
	gpu::CommandBufferHelper::GetCmdSpace<gpu::cmd::SetToken> [0x01C42D87+55] (e:\b\build\slave\webkit_win__dbg__2_\build\src\gpu\command_buffer\client\cmd_buffer_helper.h:96)
	gpu::CommandBufferHelper::InsertToken [0x01C42278+56] (e:\b\build\slave\webkit_win__dbg__2_\build\src\gpu\command_buffer\client\cmd_buffer_helper.cc:89)
	gpu::gles2::GLES2Implementation::BufferData [0x01FE382B+571] (e:\b\build\slave\webkit_win__dbg__2_\build\src\gpu\command_buffer\client\gles2_implementation.cc:1348)
	GLES2BufferData [0x027B384F+31] (e:\b\build\slave\webkit_win__dbg__2_\build\src\gpu\command_buffer\client\gles2_c_lib_autogen.h:54)
	GrGLVertexBuffer::updateData [0x022EA4E4+276] (e:\b\build\slave\webkit_win__dbg__2_\build\src\third_party\skia\gpu\src\grglvertexbuffer.cpp:108)
	GrBufferAllocPool::flushCpuData [0x022B80F1+369] (e:\b\build\slave\webkit_win__dbg__2_\build\src\third_party\skia\gpu\src\grbufferallocpool.cpp:333)
	GrBufferAllocPool::unlock [0x022B73C4+196] (e:\b\build\slave\webkit_win__dbg__2_\build\src\third_party\skia\gpu\src\grbufferallocpool.cpp:110)
	GrGpu::finalizeReservedVertices [0x022AEA44+68] (e:\b\build\slave\webkit_win__dbg__2_\build\src\third_party\skia\gpu\src\grgpu.cpp:766)
	GrGpuGL::setBuffers [0x022DB4B0+144] (e:\b\build\slave\webkit_win__dbg__2_\build\src\third_party\skia\gpu\src\grgpugl.cpp:2184)
	GrGpuGLShaders::setupGeometry [0x022CA0B5+149] (e:\b\build\slave\webkit_win__dbg__2_\build\src\third_party\skia\gpu\src\grgpuglshaders.cpp:743)
	GrGpu::onDrawNonIndexed [0x022AE993+131] (e:\b\build\slave\webkit_win__dbg__2_\build\src\third_party\skia\gpu\src\grgpu.cpp:758)
	GrDrawTarget::drawNonIndexed [0x022AA1BA+250] (e:\b\build\slave\webkit_win__dbg__2_\build\src\third_party\skia\gpu\src\grdrawtarget.cpp:820)
	GrDrawTarget::drawRect [0x022AAAD3+163] (e:\b\build\slave\webkit_win__dbg__2_\build\src\third_party\skia\gpu\src\grdrawtarget.cpp:1050)
	GrDrawTarget::drawSimpleRect [0x0228443E+46] (e:\b\build\slave\webkit_win__dbg__2_\build\src\third_party\skia\gpu\src\grdrawtarget.h:950)
	GrDefaultPathRenderer::onDrawPath [0x022CFEA9+1513] (e:\b\build\slave\webkit_win__dbg__2_\build\src\third_party\skia\gpu\src\grdefaultpathrenderer.cpp:532)
	GrDefaultPathRenderer::drawPath [0x022D00FC+28] (e:\b\build\slave\webkit_win__dbg__2_\build\src\third_party\skia\gpu\src\grdefaultpathrenderer.cpp:554)
	GrContext::drawPath [0x02286208+808] (e:\b\build\slave\webkit_win__dbg__2_\build\src\third_party\skia\gpu\src\grcontext.cpp:1549)
	SkGpuDevice::drawPath [0x0228F0DF+911] (e:\b\build\slave\webkit_win__dbg__2_\build\src\third_party\skia\src\gpu\skgpudevice.cpp:1098)
	SkCanvas::drawPath [0x00806EC9+329] (e:\b\build\slave\webkit_win__dbg__2_\build\src\third_party\skia\src\core\skcanvas.cpp:1280)
	WebCore::GraphicsContext::fillPath [0x019097FF+255] (e:\b\build\slave\webkit_win__dbg__2_\build\src\third_party\webkit\source\webcore\platform\graphics\skia\graphicscontextskia.cpp:762)
	WebCore::CanvasRenderingContext2D::fill [0x01AAC363+179] (e:\b\build\slave\webkit_win__dbg__2_\build\src\third_party\webkit\source\webcore\html\canvas\canvasrenderingcontext2d.cpp:946)
	WebCore::CanvasRenderingContext2DInternal::fillCallback [0x01101CD3+67] (e:\b\build\slave\webkit_win__dbg__2_\build\src\build\debug\obj\global_intermediate\webcore\bindings\v8canvasrenderingcontext2d.cpp:526)
	v8::internal::HandleApiCallHelper<0> [0x0068A77C+892] (e:\b\build\slave\webkit_win__dbg__2_\build\src\v8\src\builtins.cc:1163)
	v8::internal::Builtin_Impl_HandleApiCall [0x00685404+20] (e:\b\build\slave\webkit_win__dbg__2_\build\src\v8\src\builtins.cc:1180)
	v8::internal::Builtin_HandleApiCall [0x00685382+66] (e:\b\build\slave\webkit_win__dbg__2_\build\src\v8\src\builtins.cc:1179)
	(No symbol) [0x06B08C96]
	(No symbol) [0x06B526A6]
	(No symbol) [0x06B5168D]
	(No symbol) [0x06B1E0F9]
	(No symbol) [0x06B0D205]
	v8::internal::Invoke [0x004E9C53+387] (e:\b\build\slave\webkit_win__dbg__2_\build\src\v8\src\execution.cc:118)
	v8::internal::Execution::Call [0x004E99F4+420] (e:\b\build\slave\webkit_win__dbg__2_\build\src\v8\src\execution.cc:173)
	v8::Script::Run [0x00470BAE+526] (e:\b\build\slave\webkit_win__dbg__2_\build\src\v8\src\api.cc:1563)
	WebCore::V8Proxy::runScript [0x013806CE+398] (e:\b\build\slave\webkit_win__dbg__2_\build\src\third_party\webkit\source\webcore\bindings\v8\v8proxy.cpp:433)
	WebCore::V8Proxy::evaluate [0x013802BC+460] (e:\b\build\slave\webkit_win__dbg__2_\build\src\third_party\webkit\source\webcore\bindings\v8\v8proxy.cpp:387)
	WebCore::ScriptController::evaluate [0x013E49B6+214] (e:\b\build\slave\webkit_win__dbg__2_\build\src\third_party\webkit\source\webcore\bindings\v8\scriptcontroller.cpp:203)
	WebCore::ScriptElement::executeScript [0x01BD68C3+323] (e:\b\build\slave\webkit_win__dbg__2_\build\src\third_party\webkit\source\webcore\dom\scriptelement.cpp:297)
	WebCore::HTMLScriptRunner::executePendingScriptAndDispatchEvent [0x01B1EDFE+286] (e:\b\build\slave\webkit_win__dbg__2_\build\src\third_party\webkit\source\webcore\html\parser\htmlscriptrunner.cpp:140)
	WebCore::HTMLScriptRunner::executeParsingBlockingScript [0x01B1EA60+256] (e:\b\build\slave\webkit_win__dbg__2_\build\src\third_party\webkit\source\webcore\html\parser\htmlscriptrunner.cpp:119)
	WebCore::HTMLScriptRunner::executeParsingBlockingScripts [0x01B1F2CF+63] (e:\b\build\slave\webkit_win__dbg__2_\build\src\third_party\webkit\source\webcore\html\parser\htmlscriptrunner.cpp:196)
	WebCore::HTMLScriptRunner::executeScriptsWaitingForLoad [0x01B1F3CB+219] (e:\b\build\slave\webkit_win__dbg__2_\build\src\third_party\webkit\source\webcore\html\parser\htmlscriptrunner.cpp:207)
	WebCore::HTMLDocumentParser::notifyFinished [0x01AE904C+316] (e:\b\build\slave\webkit_win__dbg__2_\build\src\third_party\webkit\source\webcore\html\parser\htmldocumentparser.cpp:517)

[Kent Tamura]

At least the following tests sometimes crash since around 2011-10-12.

fast/canvas/canvas-fillPath-gradient-shadow.html = PASS CRASH
fast/canvas/canvas-strokePath-alpha-shadow.html = PASS CRASH
fast/canvas/canvas-arc-360-winding.html = PASS CRASH
fast/canvas/canvas-largedraws.html = PASS CRASH


Who is responsible for GPU Canvas?


http://test-results.appspot.com/dashboards/flakiness_dashboard.html#tests=fast%2Fcanvas%2Fcanvas-arc-360-winding.html&group=%40ToT%20GPU%20Mesa%20-%20chromium.org

[Justin Novosad]

(In reply to comment #0)
> Probable cause:
> Unknown

This could have been caused by a skia DEPS roll in chromium

[Brian Salomon]

(In reply to comment #3)
> (In reply to comment #0)
> > Probable cause:
> > Unknown
> 
> This could have been caused by a skia DEPS roll in chromium

I think that is likely. Assigning myself.

[Dimitri Glazkov (Google)]

Three more:

BUGWK70013 WIN GPU : fast/canvas/canvas-arc-connecting-line.html = CRASH
BUGWK70013 WIN GPU : fast/canvas/canvas-fillPath-pattern-shadow.html = CRASH
BUGWK70013 WIN GPU : fast/canvas/canvas-transforms-fillRect-shadow.html = CRASH

[Brian Salomon]

I've been able to reproduce it locally with a modified DRT that allows me to attach a debugger after the crash. It reproduces intermittently, seemingly only in Release builds, and not when the tests are run in isolation.

osmesa is crashing because a draw is reading beyond the end of of a vertex array. It is definitely related recent changes in skia that switched from using glBufferData to glBufferSubdata to populate vertex buffers. Previously the buffer stayed the same size and subdata updated a dynamically sized subregion. Now the buffer is dynamically resized with glBufferData. Rolling back this change stops the crash from reproducing.

I added a bunch of asserts to the skia gl release build around the draw calls to ensure we weren't reading beyond the buffer's current size. None of them have fired. My suspicion is that there is a bug lower in the stack, perhaps osmesa losing track of which draws read from which renamings of the the buffer. I think it is likely these crashes don't reproduce in the browser.  Continuing to investigate...

[Brian Salomon]

The root cause has been found and a fix checked into skia. After non-skia code uses the GL context and calls siia's resetContext function skia may leave vertex attrib arrays enabled that are not used by the current program. For most GL implementations this isn't a problem. They only read vertex data for attributes used by the current program. However, when osmesa moves vertex data around it looks at all enabled attrib arrays regardless of whether the program requires them. Depending on the current vertex attrib "pointer" and stride, this may cause an access violation. The skia change disables all vertex attribs up to GL_MAX_VERTEX_ATTRIB except attrib 0 which is always used by skia. This will hit chromium on the next skia DEPS roll, and WK on the subsequent chromium DEPS roll.

[Kent Tamura]

(In reply to comment #7)
> The root cause has been found and a fix checked into skia. After non-skia code uses the GL context and calls siia's resetContext function skia may leave vertex attrib arrays enabled that are not used by the current program. For most GL implementations this isn't a problem. They only read vertex data for attributes used by the current program. However, when osmesa moves vertex data around it looks at all enabled attrib arrays regardless of whether the program requires them. Depending on the current vertex attrib "pointer" and stride, this may cause an access violation. The skia change disables all vertex attribs up to GL_MAX_VERTEX_ATTRIB except attrib 0 which is always used by skia. This will hit chromium on the next skia DEPS roll, and WK on the subsequent chromium DEPS roll.

Thanks!

What revision of Skia fixed this issue? Probably M16 branch has this issue and we need to merge the Skia change.

[Brian Salomon]

(In reply to comment #8)
> (In reply to comment #7)
> > The root cause has been found and a fix checked into skia. After non-skia code uses the GL context and calls siia's resetContext function skia may leave vertex attrib arrays enabled that are not used by the current program. For most GL implementations this isn't a problem. They only read vertex data for attributes used by the current program. However, when osmesa moves vertex data around it looks at all enabled attrib arrays regardless of whether the program requires them. Depending on the current vertex attrib "pointer" and stride, this may cause an access violation. The skia change disables all vertex attribs up to GL_MAX_VERTEX_ATTRIB except attrib 0 which is always used by skia. This will hit chromium on the next skia DEPS roll, and WK on the subsequent chromium DEPS roll.
> 
> Thanks!
> 
> What revision of Skia fixed this issue? Probably M16 branch has this issue and we need to merge the Skia change.

It was fixed in skia r2494. It didn't make the branch (which is pulling skia r2480). I can make a skia branch for M16 at r2480 and cherry pick r2494 to it.