Bug 187257 - Flaky Test: fetch/fetch-worker-crash.html
Summary: Flaky Test: fetch/fetch-worker-crash.html
Status: NEW
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Robin Morisset
URL:
Keywords: InRadar
Depends on:
Blocks: 50856
  Show dependency treegraph
 
Reported: 2018-07-02 11:04 PDT by WebKit Commit Bot
Modified: 2019-10-23 14:21 PDT (History)
11 users (show)

See Also:


Attachments
Archive of layout-test-results from webkit-cq-01 (2.35 MB, application/zip)
2018-07-02 11:04 PDT, WebKit Commit Bot
no flags Details
Archive of layout-test-results from webkit-cq-01 (2.37 MB, application/zip)
2018-09-24 15:31 PDT, WebKit Commit Bot
no flags Details
Archive of layout-test-results from webkit-cq-02 (2.47 MB, application/zip)
2018-10-30 10:31 PDT, WebKit Commit Bot
no flags Details
Archive of layout-test-results from webkit-cq-01 (2.48 MB, application/zip)
2019-01-04 15:33 PST, WebKit Commit Bot
no flags Details
Archive of layout-test-results from webkit-cq-03 (2.53 MB, application/zip)
2019-01-15 11:00 PST, WebKit Commit Bot
no flags Details
Archive of layout-test-results from webkit-cq-03 (2.46 MB, application/zip)
2019-02-04 17:15 PST, WebKit Commit Bot
no flags Details
Archive of layout-test-results from webkit-cq-01 (2.49 MB, application/zip)
2019-02-13 04:20 PST, WebKit Commit Bot
no flags Details
Archive of layout-test-results from webkit-cq-01 (2.47 MB, application/zip)
2019-03-01 15:29 PST, WebKit Commit Bot
no flags Details
Archive of layout-test-results from webkit-cq-02 (2.55 MB, application/zip)
2019-03-15 16:59 PDT, WebKit Commit Bot
no flags Details
Archive of layout-test-results from webkit-cq-03 (2.68 MB, application/zip)
2019-04-03 14:14 PDT, WebKit Commit Bot
no flags Details
Archive of layout-test-results from webkit-cq-03 (2.47 MB, application/zip)
2019-04-03 16:29 PDT, WebKit Commit Bot
no flags Details
Archive of layout-test-results from webkit-cq-02 (3.14 MB, application/zip)
2019-04-16 17:17 PDT, WebKit Commit Bot
no flags Details
Archive of layout-test-results from webkit-cq-01 (3.16 MB, application/zip)
2019-04-17 15:50 PDT, WebKit Commit Bot
no flags Details
Archive of layout-test-results from webkit-cq-03 (3.12 MB, application/zip)
2019-04-26 11:09 PDT, WebKit Commit Bot
no flags Details
Archive of layout-test-results from webkit-cq-01 (3.11 MB, application/zip)
2019-04-30 15:24 PDT, WebKit Commit Bot
no flags Details
Archive of layout-test-results from webkit-cq-02 (3.12 MB, application/zip)
2019-05-03 16:12 PDT, WebKit Commit Bot
no flags Details
Archive of layout-test-results from webkit-cq-01 (3.15 MB, application/zip)
2019-05-08 14:13 PDT, WebKit Commit Bot
no flags Details
Archive of layout-test-results from webkit-cq-01 (3.14 MB, application/zip)
2019-05-22 13:57 PDT, WebKit Commit Bot
no flags Details
Archive of layout-test-results from webkit-cq-02 (3.12 MB, application/zip)
2019-05-22 14:02 PDT, WebKit Commit Bot
no flags Details
Archive of layout-test-results from webkit-cq-02 (3.11 MB, application/zip)
2019-05-30 11:38 PDT, WebKit Commit Bot
no flags Details
Archive of layout-test-results from webkit-cq-03 (3.13 MB, application/zip)
2019-06-02 16:34 PDT, WebKit Commit Bot
no flags Details
Archive of layout-test-results from webkit-cq-03 (3.11 MB, application/zip)
2019-06-04 16:30 PDT, WebKit Commit Bot
no flags Details
Archive of layout-test-results from webkit-cq-01 (3.21 MB, application/zip)
2019-06-14 19:22 PDT, WebKit Commit Bot
no flags Details
Archive of layout-test-results from webkit-cq-03 (3.24 MB, application/zip)
2019-06-20 18:20 PDT, WebKit Commit Bot
no flags Details
Archive of layout-test-results from webkit-cq-03 (3.22 MB, application/zip)
2019-06-23 08:44 PDT, WebKit Commit Bot
no flags Details
Archive of layout-test-results from webkit-cq-02 (3.26 MB, application/zip)
2019-08-07 10:21 PDT, WebKit Commit Bot
no flags Details
Archive of layout-test-results from webkit-cq-01 (3.24 MB, application/zip)
2019-08-29 13:32 PDT, WebKit Commit Bot
no flags Details
Archive of layout-test-results from webkit-cq-01 (3.30 MB, application/zip)
2019-09-20 07:51 PDT, WebKit Commit Bot
no flags Details
Archive of layout-test-results from webkit-cq-03 (3.43 MB, application/zip)
2019-09-23 13:41 PDT, WebKit Commit Bot
no flags Details
Archive of layout-test-results from webkit-cq-01 (3.95 MB, application/zip)
2019-10-10 13:28 PDT, WebKit Commit Bot
no flags Details
Archive of layout-test-results from webkit-cq-03 (3.55 MB, application/zip)
2019-10-11 19:56 PDT, WebKit Commit Bot
no flags Details
Archive of layout-test-results from webkit-cq-03 (3.51 MB, application/zip)
2019-10-17 13:30 PDT, WebKit Commit Bot
no flags Details
Archive of layout-test-results from webkit-cq-03 (3.50 MB, application/zip)
2019-10-18 09:19 PDT, WebKit Commit Bot
no flags Details
Archive of layout-test-results from webkit-cq-01 (3.68 MB, application/zip)
2019-10-21 17:21 PDT, WebKit Commit Bot
no flags Details
Archive of layout-test-results from webkit-cq-03 (3.63 MB, application/zip)
2019-10-23 14:21 PDT, WebKit Commit Bot
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description WebKit Commit Bot 2018-07-02 11:04:12 PDT
This is an automatically generated bug from the commit-queue.
fetch/fetch-worker-crash.html has been flaky on the commit-queue.

fetch/fetch-worker-crash.html was authored by youennf@gmail.com.
https://trac.webkit.org/browser/trunk/LayoutTests/fetch/fetch-worker-crash.html

The commit-queue just saw fetch/fetch-worker-crash.html flake (DumpRenderTree crashed) while processing attachment 344113 [details] on bug 187250.
Bot: webkit-cq-01  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.12.6

The bots will update this with information from each new failure.

If you believe this bug to be fixed or invalid, feel free to close.  The bots will re-open if the flake re-occurs.

If you would like to track this test fix with another bug, please close this bug as a duplicate.  The bots will follow the duplicate chain when making future comments.
Comment 1 WebKit Commit Bot 2018-07-02 11:04:13 PDT
Created attachment 344117 [details]
Archive of layout-test-results from webkit-cq-01
Comment 2 youenn fablet 2018-07-02 11:06:55 PDT
Crash log
Thread 0:: Dispatch queue: com.apple.main-thread
0   libsystem_kernel.dylib        	0x00007fffa453b43a swtch_pri + 10
1   libsystem_pthread.dylib       	0x00007fffa462e479 sched_yield + 11
2   com.apple.JavaScriptCore      	0x000000010d1e5fe1 WTF::WordLock::lockSlow() + 497 (WordLock.cpp:87)
3   com.apple.JavaScriptCore      	0x000000010d1cda3e WTF::installSignalHandler(WTF::Signal, WTF::Function<WTF::SignalAction (WTF::Signal, WTF::SigInfo&, __darwin_x86_thread_state64&)>&&) + 830 (WordLock.h:62)
4   com.apple.JavaScriptCore      	0x000000010dc6a84e _ZNSt3__117__call_once_proxyINS_5tupleIJOZN3JSC7VMTraps12SignalSenderC1ERKN3WTF14AbstractLockerERNS2_2VMEEUlvE_EEEEEvPv + 46 (memory:2733)
5   libc++.1.dylib                	0x00007fffa2fe6874 std::__1::__call_once(unsigned long volatile&, void*, void (*)(void*)) + 139
6   com.apple.JavaScriptCore      	0x000000010dc6a62c JSC::VMTraps::SignalSender::SignalSender(WTF::AbstractLocker const&, JSC::VM&) + 220 (VMTraps.cpp:221)
7   com.apple.JavaScriptCore      	0x000000010dc68607 JSC::VMTraps::fireTrap(JSC::VMTraps::EventType) + 167 (type_traits:3617)
8   com.apple.WebCore             	0x00000001113240b4 WebCore::WorkerThread::stop(WTF::Function<void ()>&&) + 116 (WorkerThread.cpp:287)
Comment 3 youenn fablet 2018-07-02 11:09:05 PDT
Keith, Mark, any idea what could be the issue there?
Comment 4 Saam Barati 2018-07-02 11:46:51 PDT
From the call stack, it seems like we might be installing a signal handler when we’re destroying a worker
Comment 5 WebKit Commit Bot 2018-09-24 15:31:06 PDT
The commit-queue just saw fetch/fetch-worker-crash.html flake (DumpRenderTree crashed) while processing attachment 350695 [details] on bug 189919.
Bot: webkit-cq-01  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.12.6
Comment 6 WebKit Commit Bot 2018-09-24 15:31:07 PDT
Created attachment 350700 [details]
Archive of layout-test-results from webkit-cq-01
Comment 7 WebKit Commit Bot 2018-10-30 10:31:05 PDT
The commit-queue just saw fetch/fetch-worker-crash.html flake (DumpRenderTree crashed) while processing attachment 353377 [details] on bug 190936.
Bot: webkit-cq-02  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.12.6
Comment 8 WebKit Commit Bot 2018-10-30 10:31:06 PDT
Created attachment 353383 [details]
Archive of layout-test-results from webkit-cq-02
Comment 9 WebKit Commit Bot 2019-01-04 15:33:08 PST
The commit-queue just saw fetch/fetch-worker-crash.html flake (DumpRenderTree crashed) while processing attachment 358374 [details] on bug 174980.
Bot: webkit-cq-01  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.12.6
Comment 10 WebKit Commit Bot 2019-01-04 15:33:09 PST
Created attachment 358383 [details]
Archive of layout-test-results from webkit-cq-01
Comment 11 WebKit Commit Bot 2019-01-15 11:00:58 PST
The commit-queue just saw fetch/fetch-worker-crash.html flake (DumpRenderTree crashed) while processing attachment 359176 [details] on bug 193361.
Bot: webkit-cq-03  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.13.6
Comment 12 WebKit Commit Bot 2019-01-15 11:00:59 PST
Created attachment 359180 [details]
Archive of layout-test-results from webkit-cq-03
Comment 13 WebKit Commit Bot 2019-02-04 17:15:50 PST
The commit-queue just saw fetch/fetch-worker-crash.html flake (DumpRenderTree crashed) while processing attachment 361130 [details] on bug 194158.
Bot: webkit-cq-03  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.13.6
Comment 14 WebKit Commit Bot 2019-02-04 17:15:52 PST
Created attachment 361135 [details]
Archive of layout-test-results from webkit-cq-03
Comment 15 WebKit Commit Bot 2019-02-13 04:20:24 PST
The commit-queue just saw fetch/fetch-worker-crash.html flake (DumpRenderTree crashed) while processing attachment 361909 [details] on bug 194536.
Bot: webkit-cq-01  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.13.6
Comment 16 WebKit Commit Bot 2019-02-13 04:20:25 PST
Created attachment 361910 [details]
Archive of layout-test-results from webkit-cq-01
Comment 17 WebKit Commit Bot 2019-03-01 15:29:31 PST
The commit-queue just saw fetch/fetch-worker-crash.html flake (DumpRenderTree crashed) while processing attachment 363377 [details] on bug 194612.
Bot: webkit-cq-01  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.13.6
Comment 18 WebKit Commit Bot 2019-03-01 15:29:32 PST
Created attachment 363382 [details]
Archive of layout-test-results from webkit-cq-01
Comment 19 Mark Lam 2019-03-01 15:48:56 PST
The crashing stack trace is actually:

Thread 79 Crashed:: WebCore: Worker
0   com.apple.JavaScriptCore      	0x000000010fcfa1ee WTF::registerThreadForMachExceptionHandling(WTF::Thread&) + 302 (Signals.cpp:208)
1   com.apple.JavaScriptCore      	0x00000001108187f7 JSC::JSLock::didAcquireLock() + 263 (JSLock.cpp:161)
2   com.apple.JavaScriptCore      	0x0000000110929d19 JSC::VM::VM(JSC::VM::VMType, JSC::HeapType) + 4793 (Threading.h:333)
3   com.apple.JavaScriptCore      	0x000000011092f98a JSC::VM::create(JSC::HeapType) + 42 (Ref.h:147)
4   com.apple.WebCore             	0x00000001149c907b WebCore::WorkerScriptController::WorkerScriptController(WebCore::WorkerGlobalScope*) + 27 (utility:896)
5   com.apple.WebCore             	0x0000000115662bca WebCore::WorkerGlobalScope::WorkerGlobalScope(WTF::URL const&, WTF::Ref<WebCore::SecurityOrigin, WTF::DumbPtrTraits<WebCore::SecurityOrigin> >&&, WTF::String const&, WTF::String const&, bool, WebCore::WorkerThread&, bool, WTF::Ref<WebCore::SecurityOrigin, WTF::DumbPtrTraits<WebCore::SecurityOrigin> >&&, WTF::MonotonicTime, WebCore::IDBClient::IDBConnectionProxy*, WebCore::SocketProvider*, PAL::SessionID) + 330 (memory:2097)
6   com.apple.WebCore             	0x0000000115663f01 WebCore::DedicatedWorkerThread::createWorkerGlobalScope(WTF::URL const&, WTF::Ref<WebCore::SecurityOrigin, WTF::DumbPtrTraits<WebCore::SecurityOrigin> >&&, WTF::String const&, WTF::String const&, WTF::String const&, bool, WebCore::ContentSecurityPolicyResponseHeaders const&, bool, WTF::Ref<WebCore::SecurityOrigin, WTF::DumbPtrTraits<WebCore::SecurityOrigin> >&&, WTF::MonotonicTime, PAL::SessionID) + 129 (DedicatedWorkerGlobalScope.cpp:53)
7   com.apple.WebCore             	0x000000011566cc01 WebCore::WorkerThread::workerThread() + 129 (WorkerThread.cpp:173)
8   com.apple.JavaScriptCore      	0x000000010fd0f482 WTF::Thread::entryPoint(WTF::Thread::NewThreadContext*) + 194 (memory:2597)
9   com.apple.JavaScriptCore      	0x000000010fd11719 WTF::wtfThreadEntryPoint(void*) + 9 (ThreadingPOSIX.cpp:201)
10  libsystem_pthread.dylib       	0x00007fff7d14e661 _pthread_body + 340
11  libsystem_pthread.dylib       	0x00007fff7d14e50d _pthread_start + 377
12  libsystem_pthread.dylib       	0x00007fff7d14dbf9 thread_start + 13

Signals.cpp:208 is:

inline void setExceptionPorts(const AbstractLocker& threadGroupLocker, Thread& thread)
{
    UNUSED_PARAM(threadGroupLocker);
    kern_return_t result = thread_set_exception_ports(thread.machThread(), activeExceptions, exceptionPort, EXCEPTION_STATE | MACH_EXCEPTION_CODES, MACHINE_THREAD_STATE);
    if (result != KERN_SUCCESS) {
        dataLogLn("thread set port failed due to ", mach_error_string(result));
        CRASH(); //<================= Signals.cpp:208
    }
}

Hence, the reason this crashed is because thread_set_exception_ports() failed.  The next step for debugging this would be to add some logging to see why thread_set_exception_ports() fails.  Keith, any ideas?
Comment 20 Radar WebKit Bug Importer 2019-03-01 16:23:26 PST
<rdar://problem/48527526>
Comment 21 Keith Miller 2019-03-04 09:29:15 PST
> 
> Hence, the reason this crashed is because thread_set_exception_ports()
> failed.  The next step for debugging this would be to add some logging to
> see why thread_set_exception_ports() fails.  Keith, any ideas?

In theory, the OS provides a failure reason. We could just make this a WTF crash with info and stuff values into registers.
Comment 22 WebKit Commit Bot 2019-03-15 16:59:23 PDT
The commit-queue just saw fetch/fetch-worker-crash.html flake (DumpRenderTree crashed) while processing attachment 364868 [details] on bug 195755.
Bot: webkit-cq-02  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.13.6
Comment 23 WebKit Commit Bot 2019-03-15 16:59:24 PDT
Created attachment 364881 [details]
Archive of layout-test-results from webkit-cq-02
Comment 24 WebKit Commit Bot 2019-04-03 14:14:38 PDT
The commit-queue just saw fetch/fetch-worker-crash.html flake (DumpRenderTree crashed) while processing attachment 366635 [details] on bug 196557.
Bot: webkit-cq-03  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.13.6
Comment 25 WebKit Commit Bot 2019-04-03 14:14:39 PDT
Created attachment 366640 [details]
Archive of layout-test-results from webkit-cq-03
Comment 26 WebKit Commit Bot 2019-04-03 16:29:30 PDT
The commit-queue just saw fetch/fetch-worker-crash.html flake (DumpRenderTree crashed) while processing attachment 366658 [details] on bug 196526.
Bot: webkit-cq-03  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.13.6
Comment 27 WebKit Commit Bot 2019-04-03 16:29:32 PDT
Created attachment 366666 [details]
Archive of layout-test-results from webkit-cq-03
Comment 28 WebKit Commit Bot 2019-04-16 17:17:28 PDT
The commit-queue just saw fetch/fetch-worker-crash.html flake (DumpRenderTree crashed) while processing attachment 367592 [details] on bug 196995.
Bot: webkit-cq-02  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.13.6
Comment 29 WebKit Commit Bot 2019-04-16 17:17:29 PDT
Created attachment 367595 [details]
Archive of layout-test-results from webkit-cq-02
Comment 30 WebKit Commit Bot 2019-04-17 15:50:38 PDT
The commit-queue just saw fetch/fetch-worker-crash.html flake (DumpRenderTree crashed) while processing attachment 367682 [details] on bug 196984.
Bot: webkit-cq-01  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.13.6
Comment 31 WebKit Commit Bot 2019-04-17 15:50:40 PDT
Created attachment 367691 [details]
Archive of layout-test-results from webkit-cq-01
Comment 32 Saam Barati 2019-04-17 15:50:48 PDT
Interesting, we're seeing this in stderr:
"thread set port failed due to (ipc/send) invalid port right"

and crashing in:
WTF::registerThreadForMachExceptionHandling
Comment 33 Robin Morisset 2019-04-24 16:26:37 PDT
Here is what I've found so far.
In every crash we have the same pattern, where the main thread is running
0   libsystem_kernel.dylib        	0x00007fff6b096306 swtch_pri + 10
1   libsystem_pthread.dylib       	0x00007fff6b26822e sched_yield + 11
2   com.apple.JavaScriptCore      	0x0000000104fc5d1a WTF::WordLock::lockSlow() + 506 (atomic:929)
3   com.apple.JavaScriptCore      	0x0000000104f7b735 WTF::installSignalHandler(WTF::Signal, WTF::Function<WTF::SignalAction (WTF::Signal, WTF::SigInfo&, __darwin_x86_thread_state64&)>&&) + 533 (Atomics.h:247)
4   com.apple.JavaScriptCore      	0x0000000105bfd7fe _ZNSt3__117__call_once_proxyINS_5tupleIJOZN3JSC7VMTraps12SignalSenderC1ERKN3WTF14AbstractLockerERNS2_2VMEEUlvE_EEEEEvPv + 46 (memory:2595)
5   libc++.1.dylib                	0x00007fff68eda36e std::__1::__call_once(unsigned long volatile&, void*, void (*)(void*)) + 139
6   com.apple.JavaScriptCore      	0x0000000105bfd5da JSC::VMTraps::SignalSender::SignalSender(WTF::AbstractLocker const&, JSC::VM&) + 202 (VMTraps.cpp:237)
7   com.apple.JavaScriptCore      	0x0000000105bfa7a8 JSC::VMTraps::fireTrap(JSC::VMTraps::EventType) + 200 (type_traits:4592)
8   com.apple.WebCore             	0x0000000109d4564b WebCore::WorkerThread::stop(WTF::Function<void ()>&&) + 203 (WorkerThread.cpp:290)

while the crashing thread is running
0   com.apple.JavaScriptCore      	0x0000000104f7b51e WTF::registerThreadForMachExceptionHandling(WTF::Thread&) + 302 (Signals.cpp:208)
1   com.apple.JavaScriptCore      	0x0000000105a379f6 JSC::JSLock::didAcquireLock() + 294 (JSLock.cpp:166)
2   com.apple.JavaScriptCore      	0x0000000105bf155e JSC::VM::VM(JSC::VM::VMType, JSC::HeapType) + 4718 (VM.cpp:350)
3   com.apple.JavaScriptCore      	0x0000000105bf708a JSC::VM::create(JSC::HeapType) + 42 (Ref.h:147)

The crash is not exactly in WTF::registerThreadForMachExceptionHandling, but in setExceptionPorts which is inlined into it. From the error message reported first by Saam, we know that the problem is an "invalid port right".
http://web.mit.edu/darwin/src/modules/xnu/osfmk/man/thread_set_exception_ports.html tells us that the "port right" is the third parameter to thread_set_exception_ports, which is exceptionPort in our case.

exceptionPort is a global variable defined at the top of Signals.cpp:
static mach_port_t exceptionPort;
It is set in exactly one place:
kern_return_t kr = mach_port_allocate(mach_task_self(), MACH_PORT_RIGHT_RECEIVE, &exceptionPort);
in a std::call_once, in startMachExceptionHandlerThread().
Note that startMachExceptionHandlerThread() is called from the main thread just before the point where we are stuck.. and there is no synchronization to make sure it completed and its effect is visible to the worker thread before it uses exceptionPort.

So I think the crash is due to this race between allocating exceptionPort and using it, resulting in an invalid exceptionPort being sometimes passed to the kernel.

I am still trying to understand why we don't hit this problem all the time, and how best to fix it. We must ensure that startMachExceptionHandlerThread() is called at least once before we ever take a JSLock() (such as in VM::VM()), since JSLock::didAcquireLock calls registerThreadForMachExceptionHandling. startMachExceptionHandlerThread is only called from installSignalHandler, which in turn is called from three places:
- VMTraps::SignalSender::SignalSender(..)
- installCrashHandler in SigIllCrashAnalyzer.cpp
- enableFastMemory() in WasmFaultSignalHandler.cpp, 
And new SignalHandler in turn only appears in fireTrap(), which is called from three places:
- void notifyNeedDebuggerBreak() { m_traps.fireTrap(VMTraps::NeedDebuggerBreak); }
- void notifyNeedTermination() { m_traps.fireTrap(VMTraps::NeedTermination); }
- void notifyNeedWatchdogCheck() { m_traps.fireTrap(VMTraps::NeedWatchdogCheck); }
I just don't see how any of these three functions can ever reliably be called before the constructor for VM runs!
As for SigillCrashAnalyzer, it is not enabled by default, and I don't think that fast memories for Wasm are enabled on all platforms. So I must be missing some element since we don't hit that crash literally all the time.
Comment 34 WebKit Commit Bot 2019-04-26 11:09:20 PDT
The commit-queue just saw fetch/fetch-worker-crash.html flake (DumpRenderTree crashed) while processing attachment 368329 [details] on bug 196807.
Bot: webkit-cq-03  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.13.6
Comment 35 WebKit Commit Bot 2019-04-26 11:09:22 PDT
Created attachment 368338 [details]
Archive of layout-test-results from webkit-cq-03
Comment 36 WebKit Commit Bot 2019-04-30 15:24:24 PDT
The commit-queue just saw fetch/fetch-worker-crash.html flake (DumpRenderTree crashed) while processing attachment 368615 [details] on bug 197228.
Bot: webkit-cq-01  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.13.6
Comment 37 WebKit Commit Bot 2019-04-30 15:24:25 PDT
Created attachment 368619 [details]
Archive of layout-test-results from webkit-cq-01
Comment 38 WebKit Commit Bot 2019-05-03 16:12:48 PDT
The commit-queue just saw fetch/fetch-worker-crash.html flake (DumpRenderTree crashed) while processing attachment 369003 [details] on bug 197566.
Bot: webkit-cq-02  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.13.6
Comment 39 WebKit Commit Bot 2019-05-03 16:12:49 PDT
Created attachment 369011 [details]
Archive of layout-test-results from webkit-cq-02
Comment 40 WebKit Commit Bot 2019-05-08 14:13:43 PDT
The commit-queue just saw fetch/fetch-worker-crash.html flake (DumpRenderTree crashed) while processing attachment 369414 [details] on bug 196315.
Bot: webkit-cq-01  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.13.6
Comment 41 WebKit Commit Bot 2019-05-08 14:13:44 PDT
Created attachment 369420 [details]
Archive of layout-test-results from webkit-cq-01
Comment 42 WebKit Commit Bot 2019-05-22 13:57:51 PDT
The commit-queue just saw fetch/fetch-worker-crash.html flake (DumpRenderTree crashed) while processing attachment 370437 [details] on bug 198137.
Bot: webkit-cq-01  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.13.6
Comment 43 WebKit Commit Bot 2019-05-22 13:57:53 PDT
Created attachment 370441 [details]
Archive of layout-test-results from webkit-cq-01
Comment 44 WebKit Commit Bot 2019-05-22 14:02:35 PDT
The commit-queue just saw fetch/fetch-worker-crash.html flake (DumpRenderTree crashed) while processing attachment 370436 [details] on bug 191253.
Bot: webkit-cq-02  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.13.6
Comment 45 WebKit Commit Bot 2019-05-22 14:02:36 PDT
Created attachment 370442 [details]
Archive of layout-test-results from webkit-cq-02
Comment 46 WebKit Commit Bot 2019-05-30 11:38:34 PDT
The commit-queue just saw fetch/fetch-worker-crash.html flake (DumpRenderTree crashed) while processing attachment 370954 [details] on bug 198359.
Bot: webkit-cq-02  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.13.6
Comment 47 WebKit Commit Bot 2019-05-30 11:38:36 PDT
Created attachment 370961 [details]
Archive of layout-test-results from webkit-cq-02
Comment 48 WebKit Commit Bot 2019-06-02 16:34:55 PDT
The commit-queue just saw fetch/fetch-worker-crash.html flake (DumpRenderTree crashed) while processing attachment 371166 [details] on bug 198228.
Bot: webkit-cq-03  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.13.6
Comment 49 WebKit Commit Bot 2019-06-02 16:34:56 PDT
Created attachment 371169 [details]
Archive of layout-test-results from webkit-cq-03
Comment 50 WebKit Commit Bot 2019-06-04 16:30:52 PDT
The commit-queue just saw fetch/fetch-worker-crash.html flake (DumpRenderTree crashed) while processing attachment 371346 [details] on bug 198534.
Bot: webkit-cq-03  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.13.6
Comment 51 WebKit Commit Bot 2019-06-04 16:30:53 PDT
Created attachment 371350 [details]
Archive of layout-test-results from webkit-cq-03
Comment 52 WebKit Commit Bot 2019-06-14 19:22:36 PDT
The commit-queue just saw fetch/fetch-worker-crash.html flake (DumpRenderTree crashed) while processing attachment 372161 [details] on bug 198185.
Bot: webkit-cq-01  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.13.6
Comment 53 WebKit Commit Bot 2019-06-14 19:22:37 PDT
Created attachment 372167 [details]
Archive of layout-test-results from webkit-cq-01
Comment 54 WebKit Commit Bot 2019-06-20 18:20:26 PDT
The commit-queue just saw fetch/fetch-worker-crash.html flake (DumpRenderTree crashed) while processing attachment 372603 [details] on bug 199082.
Bot: webkit-cq-03  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.13.6
Comment 55 WebKit Commit Bot 2019-06-20 18:20:27 PDT
Created attachment 372605 [details]
Archive of layout-test-results from webkit-cq-03
Comment 56 WebKit Commit Bot 2019-06-23 08:44:13 PDT
The commit-queue just saw fetch/fetch-worker-crash.html flake (DumpRenderTree crashed) while processing attachment 372697 [details] on bug 197009.
Bot: webkit-cq-03  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.13.6
Comment 57 WebKit Commit Bot 2019-06-23 08:44:15 PDT
Created attachment 372701 [details]
Archive of layout-test-results from webkit-cq-03
Comment 58 WebKit Commit Bot 2019-08-07 10:21:55 PDT
The commit-queue just saw fetch/fetch-worker-crash.html flake (DumpRenderTree crashed) while processing attachment 375703 [details] on bug 200505.
Bot: webkit-cq-02  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.13.6
Comment 59 WebKit Commit Bot 2019-08-07 10:21:58 PDT
Created attachment 375709 [details]
Archive of layout-test-results from webkit-cq-02
Comment 60 WebKit Commit Bot 2019-08-29 13:32:15 PDT
The commit-queue just saw fetch/fetch-worker-crash.html flake (DumpRenderTree crashed) while processing attachment 377478 [details] on bug 201232.
Bot: webkit-cq-01  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.13.6
Comment 61 WebKit Commit Bot 2019-08-29 13:32:16 PDT
Created attachment 377618 [details]
Archive of layout-test-results from webkit-cq-01
Comment 62 WebKit Commit Bot 2019-09-20 07:51:22 PDT
The commit-queue just saw fetch/fetch-worker-crash.html flake (DumpRenderTree crashed) while processing attachment 379236 [details] on bug 200391.
Bot: webkit-cq-01  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.13.6
Comment 63 WebKit Commit Bot 2019-09-20 07:51:23 PDT
Created attachment 379240 [details]
Archive of layout-test-results from webkit-cq-01
Comment 64 WebKit Commit Bot 2019-09-23 13:41:32 PDT
The commit-queue just saw fetch/fetch-worker-crash.html flake (DumpRenderTree crashed) while processing attachment 379380 [details] on bug 202113.
Bot: webkit-cq-03  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.13.6
Comment 65 WebKit Commit Bot 2019-09-23 13:41:33 PDT
Created attachment 379391 [details]
Archive of layout-test-results from webkit-cq-03
Comment 66 WebKit Commit Bot 2019-10-10 13:28:37 PDT
The commit-queue just saw fetch/fetch-worker-crash.html flake (DumpRenderTree crashed) while processing attachment 380672 [details] on bug 202817.
Bot: webkit-cq-01  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.13.6
Comment 67 WebKit Commit Bot 2019-10-10 13:28:38 PDT
Created attachment 380675 [details]
Archive of layout-test-results from webkit-cq-01
Comment 68 WebKit Commit Bot 2019-10-11 19:56:23 PDT
The commit-queue just saw fetch/fetch-worker-crash.html flake (DumpRenderTree crashed) while processing attachment 380806 [details] on bug 202872.
Bot: webkit-cq-03  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.13.6
Comment 69 WebKit Commit Bot 2019-10-11 19:56:24 PDT
Created attachment 380812 [details]
Archive of layout-test-results from webkit-cq-03
Comment 70 WebKit Commit Bot 2019-10-17 13:30:49 PDT
The commit-queue just saw fetch/fetch-worker-crash.html flake (DumpRenderTree crashed) while processing attachment 381201 [details] on bug 203108.
Bot: webkit-cq-03  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.13.6
Comment 71 WebKit Commit Bot 2019-10-17 13:30:50 PDT
Created attachment 381219 [details]
Archive of layout-test-results from webkit-cq-03
Comment 72 WebKit Commit Bot 2019-10-18 09:19:28 PDT
The commit-queue just saw fetch/fetch-worker-crash.html flake (DumpRenderTree crashed) while processing attachment 381289 [details] on bug 202919.
Bot: webkit-cq-03  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.13.6
Comment 73 WebKit Commit Bot 2019-10-18 09:19:30 PDT
Created attachment 381306 [details]
Archive of layout-test-results from webkit-cq-03
Comment 74 WebKit Commit Bot 2019-10-21 17:21:21 PDT
The commit-queue just saw fetch/fetch-worker-crash.html flake (DumpRenderTree crashed) while processing attachment 381467 [details] on bug 203202.
Bot: webkit-cq-01  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.13.6
Comment 75 WebKit Commit Bot 2019-10-21 17:21:22 PDT
Created attachment 381486 [details]
Archive of layout-test-results from webkit-cq-01
Comment 76 WebKit Commit Bot 2019-10-23 14:21:37 PDT
The commit-queue just saw fetch/fetch-worker-crash.html flake (DumpRenderTree crashed) while processing attachment 381663 [details] on bug 203280.
Bot: webkit-cq-03  Port: <class 'webkitpy.common.config.ports.MacPort'>  Platform: Mac OS X 10.13.6
Comment 77 WebKit Commit Bot 2019-10-23 14:21:39 PDT
Created attachment 381727 [details]
Archive of layout-test-results from webkit-cq-03