Bug 156133

Summary: DumpRenderTree, WebKitTestRunner should be more strict about waitUntilDone()/notifyDone()
Product: WebKit Reporter: David Kilzer (:ddkilzer) <ddkilzer>
Component: Tools / TestsAssignee: Nobody <webkit-unassigned>
Status: NEW ---    
Severity: Normal CC: ap, buildbot, joepeck, lforschler, rniwa, ryanhaddad, simon.fraser, webkit-bug-importer
Priority: P2 Keywords: InRadar
Version: WebKit Nightly Build   
Hardware: Unspecified   
OS: Unspecified   
Bug Depends on: 149319, 167598    
Bug Blocks:    
Attachments:
Description Flags
Patch v1 (proof-of-concept; not for landing)
none
Patch v2 (proof-of-concept; not for landing)
buildbot: commit-queue-
Archive of layout-test-results from ews104 for mac-yosemite-wk2
none
Archive of layout-test-results from ews125 for ios-simulator-wk2
none
Archive of layout-test-results from ews115 for mac-yosemite
none
Archive of layout-test-results from ews103 for mac-yosemite none

Description David Kilzer (:ddkilzer) 2016-04-02 07:53:54 PDT
The fix for Bug 149319 (r198963) included a broken test that only called notifyDone(), but never called waitUntilDone():

Bug 149319: Backdrop Filter should repaint when changed via script
<https://bugs.webkit.org/show_bug.cgi?id=149319>
<http://trac.webkit.org/changeset/198963>

It was fixed in r198971:  <http://trac.webkit.org/changeset/198971>

However, this got me to thinking about whether we could detect such a bad test before it lands by being stricter (meaning crash DumpRenderTree or WebKitTestRunner with an RELEASE_ASSERT) about two conditions:

1. Calling notifyDone() before calling waitUntilDone().
2. Calling waitUntilDone() after calling notifyDone().
Comment 1 David Kilzer (:ddkilzer) 2016-04-02 08:22:30 PDT
Created attachment 275467 [details]
Patch v1 (proof-of-concept; not for landing)
Comment 2 David Kilzer (:ddkilzer) 2016-04-02 08:24:13 PDT
(In reply to comment #1)
> Created attachment 275467 [details]
> Patch v1 (proof-of-concept; not for landing)

If there are any DumpRenderTree or WebKitTestRunner crashes with this patch, that means a layout test exists that calls waitUntilDone()/notifyDone() in the wrong order.
Comment 3 David Kilzer (:ddkilzer) 2016-04-02 13:10:15 PDT
Created attachment 275478 [details]
Patch v2 (proof-of-concept; not for landing)
Comment 4 Build Bot 2016-04-02 13:36:24 PDT
Comment on attachment 275478 [details]
Patch v2 (proof-of-concept; not for landing)

Attachment 275478 [details] did not pass mac-wk2-ews (mac-wk2):
Output: http://webkit-queues.webkit.org/results/1088391

Number of test failures exceeded the failure limit.
Comment 5 Build Bot 2016-04-02 13:36:27 PDT
Created attachment 275481 [details]
Archive of layout-test-results from ews104 for mac-yosemite-wk2

The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: ews104  Port: mac-yosemite-wk2  Platform: Mac OS X 10.10.5
Comment 6 David Kilzer (:ddkilzer) 2016-04-02 13:49:26 PDT
Okay, this proof-of-concept indicates there might be some work to do here, but I don't have time to spend on it immediately.
Comment 7 Radar WebKit Bug Importer 2016-04-02 13:49:42 PDT
<rdar://problem/25511581>
Comment 8 Build Bot 2016-04-02 13:51:59 PDT
Comment on attachment 275478 [details]
Patch v2 (proof-of-concept; not for landing)

Attachment 275478 [details] did not pass ios-sim-ews (ios-simulator-wk2):
Output: http://webkit-queues.webkit.org/results/1088437

Number of test failures exceeded the failure limit.
Comment 9 Build Bot 2016-04-02 13:52:03 PDT
Created attachment 275482 [details]
Archive of layout-test-results from ews125 for ios-simulator-wk2

The attached test failures were seen while running run-webkit-tests on the ios-sim-ews.
Bot: ews125  Port: ios-simulator-wk2  Platform: Mac OS X 10.10.5
Comment 10 Build Bot 2016-04-02 13:52:07 PDT
Comment on attachment 275478 [details]
Patch v2 (proof-of-concept; not for landing)

Attachment 275478 [details] did not pass mac-debug-ews (mac):
Output: http://webkit-queues.webkit.org/results/1088435

Number of test failures exceeded the failure limit.
Comment 11 Build Bot 2016-04-02 13:52:10 PDT
Created attachment 275483 [details]
Archive of layout-test-results from ews115 for mac-yosemite

The attached test failures were seen while running run-webkit-tests on the mac-debug-ews.
Bot: ews115  Port: mac-yosemite  Platform: Mac OS X 10.10.5
Comment 12 Build Bot 2016-04-02 15:22:11 PDT
Comment on attachment 275478 [details]
Patch v2 (proof-of-concept; not for landing)

Attachment 275478 [details] did not pass mac-ews (mac):
Output: http://webkit-queues.webkit.org/results/1088818

Number of test failures exceeded the failure limit.
Comment 13 Build Bot 2016-04-02 15:22:15 PDT
Created attachment 275486 [details]
Archive of layout-test-results from ews103 for mac-yosemite

The attached test failures were seen while running run-webkit-tests on the mac-ews.
Bot: ews103  Port: mac-yosemite  Platform: Mac OS X 10.10.5