Bug 154764

Summary: [iOS Simulator] Reftests don't work
Product: WebKit Reporter: Alexey Proskuryakov <ap>
Component: Tools / TestsAssignee: Alexey Proskuryakov <ap>
Status: RESOLVED FIXED    
Severity: Normal CC: commit-queue, dbates, glenn, lforschler, mmaxfield, ryanhaddad, simon.fraser, thorton, zalan
Priority: P2    
Version: Other   
Hardware: Unspecified   
OS: Unspecified   
See Also: https://bugs.webkit.org/show_bug.cgi?id=154761
Attachments:
Description Flags
proposed fix none

Alexey Proskuryakov
Reported 2016-02-26 17:23:43 PST
ImageDiff doesn't work in iOS simulator - all failing tests say "ref test hashes didn't match but diff passed". Looks like the problem is with launching it: $ xcrun -sdk iphonesimulator sim --environment=preserve --adopt-pid /Volumes/Data/slave/ios-simulator-9-release-tests-wk1/build/WebKitBuild/Release-iphonesimulator/ImageDiff --tolerance 0 xcrun: error: unable to find utility "sim", not a developer tool or in PATH
Attachments
proposed fix (1.80 KB, patch)
2016-02-26 18:04 PST, Alexey Proskuryakov 		no flags
DetailsFormatted DiffDiff
View All Add attachment proposed patch, testcase, etc.
Alexey Proskuryakov
Comment 1 2016-02-26 18:04:38 PST
Created attachment 272389 [details] proposed fix
Simon Fraser (smfr)
Comment 2 2016-02-26 18:14:19 PST
(In reply to comment #0) > ImageDiff doesn't work in iOS simulator - all failing tests say "ref test > hashes didn't match but diff passed". All of them? Don't we have an "expected mismatch" test that fails when ref tests aren't working?
WebKit Commit Bot
Comment 3 2016-02-26 18:53:38 PST
Comment on attachment 272389 [details] proposed fix Clearing flags on attachment: 272389 Committed r197227: <http://trac.webkit.org/changeset/197227>
WebKit Commit Bot
Comment 4 2016-02-26 18:53:43 PST
All reviewed patches have been landed. Closing bug.
Alexey Proskuryakov
Comment 5 2016-02-26 18:56:53 PST
Presumably all expected mismatch tests are already marked as failing or skipped, because of this bug.
Alexey Proskuryakov
Comment 6 2016-02-27 12:23:46 PST
Now that reftests work, there is a number of failing tests. https://build.webkit.org/results/Apple%20iOS%209%20Simulator%20Release%20WK2%20(Tests)/r197255%20(3477)/results.html Mass marked them as failing in <http://trac.webkit.org/r197258>. They need triage, it's mostly text and layout issues.
Simon Fraser (smfr)
Comment 7 2016-02-28 20:18:59 PST
We should probably have a directory of unskippable tests (like fast/harness) that make sure the harness is working.
Alexey Proskuryakov
Comment 8 2016-02-29 09:26:08 PST
We didn't really have much trouble detecting that something was wrong, and guessing that reftests were completely broken. The problem was with finding the time to investigate the issue.
Note You need to log in before you can comment on or make changes to this bug.