DumpRenderTree, WebKitTestRunner should be more strict about waitUntilDone()/notifyDone()

• Component: Tools / Tests
• Status: NEW ---

[David Kilzer (:ddkilzer)]

The fix for Bug 149319 (r198963) included a broken test that only called notifyDone(), but never called waitUntilDone():

Bug 149319: Backdrop Filter should repaint when changed via script
<https://bugs.webkit.org/show_bug.cgi?id=149319>
<http://trac.webkit.org/changeset/198963>

It was fixed in r198971:  <http://trac.webkit.org/changeset/198971>

However, this got me to thinking about whether we could detect such a bad test before it lands by being stricter (meaning crash DumpRenderTree or WebKitTestRunner with an RELEASE_ASSERT) about two conditions:

1. Calling notifyDone() before calling waitUntilDone().
2. Calling waitUntilDone() after calling notifyDone().

[David Kilzer (:ddkilzer)]

Created attachment 275467 [details]
Patch v1 (proof-of-concept; not for landing)

[David Kilzer (:ddkilzer)]

(In reply to comment #1)
> Created attachment 275467 [details]
> Patch v1 (proof-of-concept; not for landing)

If there are any DumpRenderTree or WebKitTestRunner crashes with this patch, that means a layout test exists that calls waitUntilDone()/notifyDone() in the wrong order.

[David Kilzer (:ddkilzer)]

Created attachment 275478 [details]
Patch v2 (proof-of-concept; not for landing)

[Build Bot]

Comment on attachment 275478 [details]
Patch v2 (proof-of-concept; not for landing)

Attachment 275478 [details] did not pass mac-wk2-ews (mac-wk2):
Output: http://webkit-queues.webkit.org/results/1088391

Number of test failures exceeded the failure limit.

[Build Bot]

Created attachment 275481 [details]
Archive of layout-test-results from ews104 for mac-yosemite-wk2

The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: ews104  Port: mac-yosemite-wk2  Platform: Mac OS X 10.10.5

[David Kilzer (:ddkilzer)]

Okay, this proof-of-concept indicates there might be some work to do here, but I don't have time to spend on it immediately.

[Radar WebKit Bug Importer]

<rdar://problem/25511581>

[Build Bot]

Comment on attachment 275478 [details]
Patch v2 (proof-of-concept; not for landing)

Attachment 275478 [details] did not pass ios-sim-ews (ios-simulator-wk2):
Output: http://webkit-queues.webkit.org/results/1088437

Number of test failures exceeded the failure limit.

[Build Bot]

Created attachment 275482 [details]
Archive of layout-test-results from ews125 for ios-simulator-wk2

The attached test failures were seen while running run-webkit-tests on the ios-sim-ews.
Bot: ews125  Port: ios-simulator-wk2  Platform: Mac OS X 10.10.5

[Build Bot]

Comment on attachment 275478 [details]
Patch v2 (proof-of-concept; not for landing)

Attachment 275478 [details] did not pass mac-debug-ews (mac):
Output: http://webkit-queues.webkit.org/results/1088435

Number of test failures exceeded the failure limit.

[Build Bot]

Created attachment 275483 [details]
Archive of layout-test-results from ews115 for mac-yosemite

The attached test failures were seen while running run-webkit-tests on the mac-debug-ews.
Bot: ews115  Port: mac-yosemite  Platform: Mac OS X 10.10.5

[Build Bot]

Comment on attachment 275478 [details]
Patch v2 (proof-of-concept; not for landing)

Attachment 275478 [details] did not pass mac-ews (mac):
Output: http://webkit-queues.webkit.org/results/1088818

Number of test failures exceeded the failure limit.

[Build Bot]

Created attachment 275486 [details]
Archive of layout-test-results from ews103 for mac-yosemite

The attached test failures were seen while running run-webkit-tests on the mac-ews.
Bot: ews103  Port: mac-yosemite  Platform: Mac OS X 10.10.5