Bug 55909

Summary: new-run-webkit-tests --run-singly option is busted
Product: WebKit Reporter: James Robinson <jamesr>
Component: Tools / TestsAssignee: Dirk Pranke <dpranke>
Severity: Normal CC: abarth, dpranke, eric, ojan, senorblanco, tony
Priority: P2    
Version: 528+ (Nightly build)   
Hardware: PC   
OS: OS X 10.5   
Description Flags
simplify by reading an attribute on SingleTestThread() instead
mask out unused args tony: review+

Description James Robinson 2011-03-07 15:33:56 PST
When running with --run-singly option tests that fail and are expected to fail are producing 'unexpected pass' output.  Looking at the actual output from the test they seem to be really failing.

For example, canvas/philip/tests/2d.composite.operation.clear.html is marked as an expected TEXT diff and actually does fail with a TEXT diff.  When run with --run-singly --verbose on chromium linux I get this output:

2011-03-07 15:32:45,836 13188 executive.py:365 DEBUG "/usr/bin/wdiff --start-delete=##WDIFF_DEL## --end-delete=##WDIFF_END## --start-insert=##WDIFF_ADD## --end-insert=##WDIFF_END## /usr/local/google/home/jamesr/chrome/src/webkit/Release/layout-test-results/canvas/philip/tests/2d.composite.operation.clear-expected.txt /usr/local/google/home/jamesr/chrome/src/webkit/Release/layout-test-results/canvas/philip/tests/2d.composite.operation.clear-actual.txt" took 0.01s
2011-03-07 15:32:45,883 13188 executive.py:365 DEBUG "ruby -I /usr/local/google/home/jamesr/chrome/src/third_party/WebKit/Websites/bugs.webkit.org/PrettyPatch /usr/local/google/home/jamesr/chrome/src/third_party/WebKit/Websites/bugs.webkit.org/PrettyPatch/prettify.rb /usr/local/google/home/jamesr/chrome/src/webkit/Release/layout-test-results/canvas/philip/tests/2d.composite.operation.clear-diff.txt" took 0.05s
2011-03-07 15:32:45,989 13188 worker_mixin.py:141 DEBUG worker/0 canvas/philip/tests/2d.composite.operation.clear.html passed
2011-03-07 15:32:45,990 13184 printing.py:541 INFO   canvas/philip/tests/2d.composite.operation.clear.html -> unexpected pass
2011-03-07 15:32:45,990 13188 worker.py:83 DEBUG worker/0 done

but if I diff the output on disk locally...

$ diff /usr/local/google/home/jamesr/chrome/src/webkit/Release/layout-test-results/canvas/philip/tests/2d.composite.operation.clear-expected.txt /usr/local/google/home/jamesr/chrome/src/webkit/Release/layout-test-results/canvas/philip/tests/2d.composite.operation.clear-actual.txt
< Passed
> Failed assertion ctx.globalCompositeOperation == 'xor' (got clear[string], expected xor[string])
Comment 1 Dirk Pranke 2011-04-06 19:45:11 PDT
I don't think it should block bug 34984 and I plan to clear the blocking flag on Mon 4/11. If you disagree, now would be a good time to speak up :).
Comment 2 Dirk Pranke 2011-04-12 14:54:57 PDT
Created attachment 89278 [details]
Comment 3 Dirk Pranke 2011-04-12 18:04:38 PDT
clearing blocking flag.
Comment 4 Tony Chang 2011-04-13 10:28:07 PDT
Comment on attachment 89278 [details]

View in context: https://bugs.webkit.org/attachment.cgi?id=89278&action=review

> Tools/Scripts/webkitpy/layout_tests/layout_package/worker_mixin.py:170
> +            def run(other):
> +                self._result = worker._run_single_test(driver, test_input)

I think it's more common to do something like:

def run(self, worker_mixin=self):
    worker_mixin._result = ...

> Tools/Scripts/webkitpy/layout_tests/run_webkit_tests_unittest.py:328
> +        res, out, err, user = logging_run(['--run-singly', 'failures/unexpected'])

Nit: maybe replace the unused outputs with _
Comment 5 Dirk Pranke 2011-04-13 14:17:29 PDT
Created attachment 89453 [details]
simplify by reading an attribute on SingleTestThread() instead
Comment 6 Dirk Pranke 2011-04-13 14:18:30 PDT
Created attachment 89456 [details]
mask out unused args
Comment 7 Dirk Pranke 2011-04-13 18:37:48 PDT
Committed r83801