118602 2013-07-12 06:38:50 -0700 [NRWT] run-webkit-tests sometimes reports a wrong test crashing 2014-12-19 14:36:50 -0800 1 1 1 Unclassified WebKit Tools / Tests 528+ (Nightly build) Unspecified Unspecified NEW https://bugs.webkit.org/show_bug.cgi?id=98345 NRWT P2 Normal --- 1 spena webkit-unassigned ap brian.holt dpranke lforschler rniwa oldest_to_newest 907939 0 spena 2013-07-12 06:38:50 -0700 This can be easily reproduced by doing: run-webkit-tests --gtk -2 --debug --iterations=2 editing/selection/selection-in-iframe-removed-crash.html editing/selection/selection-invalid-offset.html The first test, selection-in-iframe-removed-crash.html, is crashing, but run-webkit-tests reports that it is the second one, selection-invalid-offset.html. The second test in this example can be any: as long as it is run after the first one, it will be reported as crashing. (See related issues in bug #111451 and bug #111521) I took a look at this, and in driver.py run_test, self.has_crashed() reports False for the first test, later reporting True for the second. When something delays the call to has_crashed() (for example, adding a import pdb and pdb.set_trace() to debug manually, or adding a time.sleep(2)), the call is able to correctly report the crash. I am not aware of any other crashing test that has this effect on the one coming after it. 907940 1 spena 2013-07-12 06:40:35 -0700 In https://bugs.webkit.org/show_bug.cgi?id=111521#c3 and https://bugs.webkit.org/show_bug.cgi?id=111451#c3 Ryosuke has already experienced this. 909902 2 dpranke 2013-07-18 16:19:57 -0700 Are you saying that the crash is occurring between the time the first test completes (we get the #EOFs) and the second test starts? That's the only way I can imagine offhand how putting in a sleep or a breakpoint would change the result you're seeing, but I haven't looked at the code lately. It is true that if DRT/WTR crashes after the first test has "completed" we'll misattribute things. I'm not sure that this is a solvable problem. It's particularly true that if the first test corrupts state or otherwise triggers something that doesn't show up until a subsequent test (e.g., a delayed task runs, or a GC occurs), you'll have problems. 910056 3 spena 2013-07-19 02:01:39 -0700 (In reply to comment #2) > Are you saying that the crash is occurring between the time the first test completes (we get the #EOFs) and the second test starts? That's the only way I can imagine offhand how putting in a sleep or a breakpoint would change the result you're seeing, but I haven't looked at the code lately. If I run the test directly with WTR (this is the GTK port) $ Tools/jhbuild/jhbuild-wrapper --gtk run WebKitBuild/Debug/Programs/WebKitTestRunner LayoutTests/editing/selection/selection-in-iframe-removed-crash.html Gtk-Message: Failed to load module "overlay-scrollbar" Gtk-Message: Failed to load module "canberra-gtk-module" Gtk-Message: Failed to load module "overlay-scrollbar" Gtk-Message: Failed to load module "canberra-gtk-module" Content-Type: text/plain Test passes if it does not crash. #EOF #EOF #EOF ASSERTION FAILED: commonScope ../../Source/WebCore/editing/htmlediting.cpp(79) : int WebCore::comparePositions(const WebCore::Position&, const WebCore::Position&) 1 0x7ffcd581dd94 /home/SERILOCAL/simon.pena/Projects/WebKit/WebKitBuild/Debug/.libs/libjavascriptcoregtk-3.0.so.0(WTFCrash+0x1e) [0x7ffcd581dd94] 2 0x7ffcd6f9a29a /home/SERILOCAL/simon.pena/Projects/WebKit/WebKitBuild/Debug/.libs/libwebkit2gtk-3.0.so.25(+0xc7529a) [0x7ffcd6f9a29a] 3 0x7ffcd6fdfce6 /home/SERILOCAL/simon.pena/Projects/WebKit/WebKitBuild/Debug/.libs/libwebkit2gtk-3.0.so.25(+0xcbace6) [0x7ffcd6fdfce6] 4 0x7ffcd6abb02d /home/SERILOCAL/simon.pena/Projects/WebKit/WebKitBuild/Debug/.libs/libwebkit2gtk-3.0.so.25(+0x79602d) [0x7ffcd6abb02d] 5 0x7ffcd6abad17 /home/SERILOCAL/simon.pena/Projects/WebKit/WebKitBuild/Debug/.libs/libwebkit2gtk-3.0.so.25(+0x795d17) [0x7ffcd6abad17] 6 0x7ffcd6ac8e4e /home/SERILOCAL/simon.pena/Projects/WebKit/WebKitBuild/Debug/.libs/libwebkit2gtk-3.0.so.25(+0x7a3e4e) [0x7ffcd6ac8e4e] 7 0x7ffcd6f7e368 /home/SERILOCAL/simon.pena/Projects/WebKit/WebKitBuild/Debug/.libs/libwebkit2gtk-3.0.so.25(+0xc59368) [0x7ffcd6f7e368] 8 0x7ffcd6f8bf39 /home/SERILOCAL/simon.pena/Projects/WebKit/WebKitBuild/Debug/.libs/libwebkit2gtk-3.0.so.25(+0xc66f39) [0x7ffcd6f8bf39] 9 0x7ffcd6f90e99 /home/SERILOCAL/simon.pena/Projects/WebKit/WebKitBuild/Debug/.libs/libwebkit2gtk-3.0.so.25(+0xc6be99) [0x7ffcd6f90e99] 10 0x7ffcd6f9764f /home/SERILOCAL/simon.pena/Projects/WebKit/WebKitBuild/Debug/.libs/libwebkit2gtk-3.0.so.25(+0xc7264f) [0x7ffcd6f9764f] 11 0x7ffcd6f90e5a /home/SERILOCAL/simon.pena/Projects/WebKit/WebKitBuild/Debug/.libs/libwebkit2gtk-3.0.so.25(+0xc6be5a) [0x7ffcd6f90e5a] 12 0x7ffcd6f90c8d /home/SERILOCAL/simon.pena/Projects/WebKit/WebKitBuild/Debug/.libs/libwebkit2gtk-3.0.so.25(+0xc6bc8d) [0x7ffcd6f90c8d] 13 0x7ffcd739a919 /home/SERILOCAL/simon.pena/Projects/WebKit/WebKitBuild/Debug/.libs/libwebkit2gtk-3.0.so.25(+0x1075919) [0x7ffcd739a919] 14 0x7ffcd782dba8 /home/SERILOCAL/simon.pena/Projects/WebKit/WebKitBuild/Debug/.libs/libwebkit2gtk-3.0.so.25(+0x1508ba8) [0x7ffcd782dba8] 15 0x7ffc87f000e5 [0x7ffc87f000e5] #CRASHED - WebProcess LEAK: 1 WebPageProxy LEAK: 1 WebContext You are right: it seems that the crash happens after we get the #EOFs. I checked some other tests crashing, and the crashes seem to happen always before we get the EOFs. > It is true that if DRT/WTR crashes after the first test has "completed" we'll misattribute things. I'm not sure that this is a solvable problem. It's particularly true that if the first test corrupts state or otherwise triggers something that doesn't show up until a subsequent test (e.g., a delayed task runs, or a GC occurs), you'll have problems. I see. As I mentioned earlier, I only experienced this behaviour with this test, so maybe there are no other tests crashing after being reported "completed".