Bug 154764

Summary: [iOS Simulator] Reftests don't work
Product: WebKit Reporter: Alexey Proskuryakov <ap>
Component: Tools / TestsAssignee: Alexey Proskuryakov <ap>
Status: RESOLVED FIXED    
Severity: Normal CC: commit-queue, dbates, glenn, lforschler, mmaxfield, ryanhaddad, simon.fraser, thorton, zalan
Priority: P2    
Version: Other   
Hardware: Unspecified   
OS: Unspecified   
See Also: https://bugs.webkit.org/show_bug.cgi?id=154761
Attachments:
Description Flags
proposed fix none

Description Alexey Proskuryakov 2016-02-26 17:23:43 PST 
ImageDiff doesn't work in iOS simulator - all failing tests say "ref test hashes didn't match but diff passed".

Looks like the problem is with launching it:

$ xcrun -sdk iphonesimulator sim --environment=preserve --adopt-pid /Volumes/Data/slave/ios-simulator-9-release-tests-wk1/build/WebKitBuild/Release-iphonesimulator/ImageDiff --tolerance 0
xcrun: error: unable to find utility "sim", not a developer tool or in PATH
Comment 1 Alexey Proskuryakov 2016-02-26 18:04:38 PST 
Created attachment 272389 [details]
proposed fix
Comment 2 Simon Fraser (smfr) 2016-02-26 18:14:19 PST 
(In reply to comment #0)
> ImageDiff doesn't work in iOS simulator - all failing tests say "ref test
> hashes didn't match but diff passed".

All of them? Don't we have an "expected mismatch" test that fails when ref tests aren't working?
Comment 3 WebKit Commit Bot 2016-02-26 18:53:38 PST 
Comment on attachment 272389 [details]
proposed fix

Clearing flags on attachment: 272389

Committed r197227: <http://trac.webkit.org/changeset/197227>
Comment 4 WebKit Commit Bot 2016-02-26 18:53:43 PST 
All reviewed patches have been landed.  Closing bug.
Comment 5 Alexey Proskuryakov 2016-02-26 18:56:53 PST 
Presumably all expected mismatch tests are already marked as failing or skipped, because of this bug.
Comment 6 Alexey Proskuryakov 2016-02-27 12:23:46 PST 
Now that reftests work, there is a number of failing tests.

https://build.webkit.org/results/Apple%20iOS%209%20Simulator%20Release%20WK2%20(Tests)/r197255%20(3477)/results.html

Mass marked them as failing in <http://trac.webkit.org/r197258>. They need triage, it's mostly text and layout issues.
Comment 7 Simon Fraser (smfr) 2016-02-28 20:18:59 PST 
We should probably have a directory of unskippable tests (like fast/harness) that make sure the harness is working.
Comment 8 Alexey Proskuryakov 2016-02-29 09:26:08 PST 
We didn't really have much trouble detecting that something was wrong, and guessing that reftests were completely broken. The problem was with finding the time to investigate the issue.