Bug 95144 - Skipping tests on platform mac due to WebCore::JSEventListener::jsFunction(WebCore::ScriptExecutionContext*) crashes
Summary: Skipping tests on platform mac due to WebCore::JSEventListener::jsFunction(We...
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: 528+ (Nightly build)
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Nobody
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2012-08-27 15:41 PDT by Mark Lam
Modified: 2012-08-27 17:01 PDT (History)
4 users (show)

See Also:


Attachments
gardening. (2.07 KB, patch)
2012-08-27 15:47 PDT, Mark Lam
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Mark Lam 2012-08-27 15:41:56 PDT
Skipping tests on platform mac due to WebCore::JSEventListener::jsFunction(WebCore::ScriptExecutionContext*) crashes as detailed in https://bugs.webkit.org/show_bug.cgi?id=93878.  The tests that will be skipped are:
    - fast/events/keyevent-iframe-removed-crash.html
    - fullscreen/full-screen-iframe-zIndex.html
    - fullscreen/full-screen-restrictions.html
    - http/tests/security/MessagePort/event-listener-context.html
    - platform/mac/accessibility/search-with-frames.html
    - svg/custom/use-instanceRoot-as-event-target.xhtml
Comment 1 Mark Lam 2012-08-27 15:47:46 PDT
Created attachment 160833 [details]
gardening.
Comment 2 Mark Lam 2012-08-27 16:58:17 PDT
How did I choose which tests to skip?
=======================
I manually sampled hundreds of the bot test failure results including going back to very early tests whose results are still on the server but are not conveniently presented in the waterfall results.  I did this by going to the following URL for Apple Lion WK1 debug:

http://build.webkit.org/builders/Apple%20Lion%20Debug%20WK1%20%28Tests%29/builds/<build number>

This is how I found build 1713 which was the earliest build I saw this crash manifest on.

For each occurrence of the crash stack trace I encountered, I recorded the build number and the failing test that the crash stack trace was observed on.  Based on these records, I was able to observe that there is a consistency in which tests fail due to this crash.

Subsequently, I run each of those tests individually and repeatedly on my local machine like so:

oss/webkit4$ perl ./Tools/Scripts/run-webkit-tests --no-launch-safari --no-new-test-results --no-sample-on-timeout --results-directory ./layout-test-results --use-remote-links-to-tests --debug --no-build --exit-after-n-crashes-or-timeouts 20 --exit-after-n-failures 500 -v --repeat-each 1000 <test name>

If the test crashes as expected with the same stack trace, then I consider it a worthy candidate to be skipped.  The fact that it crashes suggests that this test has the trigger / originating condition for manifesting the crash.

There is no guarantee that once we exclude these tests that the crashes will go away altogether.  Since the crashes are intermittent, there may be a few more straggles who just haven't manifested with this symptom yet.  We would expect the set of the stragglers to be less in count than the set I am now skipping.

Alternatively, there's a chance that the symptom will just start manifesting on other tests.  But if that case happens, then we will see a whole new set of failures with this crash of about the same size as the original set.  If that occurs, we will now have evidence that the crash can occur just anywhere, and we can and should revert this current change set for skipping these tests.

Note: there were some tests that were seen to manifest the crash failure, but when run alone, do not manifest the crash.  This suggests that there are tests that do not have the trigger / originating condition of the crash.  As a result, if we successfully skip all the tests that do trigger the crash, there is a possibility that the crash will cease to manifest.

Also, the test that do not manifest the crash when run alone, though previously seen in reported bot failures, are excluded from the skip list.
Comment 3 WebKit Review Bot 2012-08-27 17:01:47 PDT
Comment on attachment 160833 [details]
gardening.

Clearing flags on attachment: 160833

Committed r126821: <http://trac.webkit.org/changeset/126821>
Comment 4 WebKit Review Bot 2012-08-27 17:01:50 PDT
All reviewed patches have been landed.  Closing bug.