Bug 56047

Summary: new-run-webkit-tests causes several tests to fail on mac port
Product: WebKit Reporter: Tony Gentilcore <tonyg>
Component: Tools / TestsAssignee: Dirk Pranke <dpranke>
Status: RESOLVED FIXED    
Severity: Normal CC: dpranke, mihaip
Priority: P2    
Version: 528+ (Nightly build)   
Hardware: PC   
OS: OS X 10.5   
Bug Depends on: 56760, 57662    
Bug Blocks: 34984    

Description Tony Gentilcore 2011-03-09 13:28:32 PST
new-run-webkit-tests is waaay faster than run-webkit-tests. I'd love to be able to switch to it. However, there are several tests that fail or timeout on new-run-webkit-tests, but pass under run-webkit-tests. This is probably closely related to bug 35358, but that only refers to timeouts. Bug 35266 might also help the ones that timeout.

Here's a list from r79638 on the mac port:

fast/canvas/webgl/uninitialized-test.html
  Test timed out
  Text diff mismatch expected actual diff wdiff pretty diff

fast/css/font-face-in-shadow-DOM.html
  Text diff mismatch expected actual diff wdiff pretty diff

fast/forms/restore-selection-after-layout.html
  Text diff mismatch expected actual diff wdiff pretty diff

fast/frames/lots-of-iframes.html
  Test timed out
  Text diff mismatch expected actual diff wdiff pretty diff

fast/frames/lots-of-objects.html
  Test timed out
  Text diff mismatch expected actual diff wdiff pretty diff

fast/overflow/lots-of-sibling-inline-boxes.html
  Test timed out
  Text diff mismatch expected actual diff wdiff pretty diff

fast/parser/block-nesting-cap.html
  Test timed out
  Text diff mismatch expected actual diff wdiff pretty diff

fast/wcss/wap-input-format.xhtml
  Text diff mismatch expected actual diff wdiff pretty diff

fast/wcss/wap-input-required.xhtml
  Text diff mismatch expected actual diff wdiff pretty diff

fast/workers/storage/use-same-database-in-page-and-workers.html
  DumpRenderTree crashed stack

http/tests/navigation/image-load-in-unload-handler.html
  Text diff mismatch expected actual diff wdiff pretty diff

http/tests/navigation/ping-cookie.html
  Text diff mismatch expected actual diff wdiff pretty diff

http/tests/navigation/ping-same-origin.html
  Text diff mismatch expected actual diff wdiff pretty diff

http/tests/security/isolatedWorld/all-window-properties.html
  Text diff mismatch expected actual diff wdiff pretty diff

http/tests/security/isolatedWorld/all-window-prototypes.html
  Text diff mismatch expected actual diff wdiff pretty diff

http/tests/xmlhttprequest/basic-auth-nopassword.html
  Text diff mismatch expected actual diff wdiff pretty diff

java/lc3/JavaObject/JavaObjectToBoolean-001-n.html
  Text diff mismatch expected actual diff wdiff pretty diff

java/lc3/JavaObject/JavaObjectToBoolean-002-n.html
  Text diff mismatch expected actual diff wdiff pretty diff

java/lc3/JavaObject/JavaObjectToDouble-001.html
  Text diff mismatch expected actual diff wdiff pretty diff

media/invalid-media-url-crash.html
  DumpRenderTree crashed stack

platform/chromium-linux/fast/text/international/complex-joining-using-gpos.html
  Text diff mismatch expected actual diff wdiff pretty diff

sputnik/Conformance/07_Lexical_Conventions/7.8_Literals/7.8.5_Regular_Expression_Literals/S7.8.5_A1.1_T2.html
  Test timed out
  Text diff mismatch expected actual diff wdiff pretty diff

sputnik/Conformance/07_Lexical_Conventions/7.8_Literals/7.8.5_Regular_Expression_Literals/S7.8.5_A1.4_T2.html
  Test timed out
  Text diff mismatch expected actual diff wdiff pretty diff

sputnik/Conformance/07_Lexical_Conventions/7.8_Literals/7.8.5_Regular_Expression_Literals/S7.8.5_A2.1_T2.html
  Test timed out
  Text diff mismatch expected actual diff wdiff pretty diff

sputnik/Conformance/07_Lexical_Conventions/7.8_Literals/7.8.5_Regular_Expression_Literals/S7.8.5_A2.4_T2.html
  Test timed out
  Text diff mismatch expected actual diff wdiff pretty diff

sputnik/Unicode/Unicode_218/S7.6_A5.3_T2.html
  Test timed out
  Text diff mismatch expected actual diff wdiff pretty diff

transitions/cancel-transition.html
  Text diff mismatch expected actual diff wdiff pretty diff
Comment 1 Dirk Pranke 2011-03-09 13:33:56 PST
Yes, I've noticed this as well.
Comment 2 Dirk Pranke 2011-03-21 11:57:59 PDT
A number of tests appear to be sensitive to the order they are run in, and for some reason NRWT is running the tests in descending alphabetical order inside a directory. I've filed bug 56760 to fix that.

There are still some tests that are failing for other reasons, so I'm leaving this open for now.
Comment 3 Dirk Pranke 2011-04-01 19:45:15 PDT
I believe all of the issues that were causing NRWT to not run cleanly (in a single thread, at least) on Mac SL have been resolved. I expect we will still need to shake out some flakiness as a result of contention between multiple processes, but I'll mark this as closed because I've fixed the main issues.