ImageDiff doesn't work in iOS simulator - all failing tests say "ref test hashes didn't match but diff passed".
Looks like the problem is with launching it:
$ xcrun -sdk iphonesimulator sim --environment=preserve --adopt-pid /Volumes/Data/slave/ios-simulator-9-release-tests-wk1/build/WebKitBuild/Release-iphonesimulator/ImageDiff --tolerance 0
xcrun: error: unable to find utility "sim", not a developer tool or in PATH
Created attachment 272389 [details]
(In reply to comment #0)
> ImageDiff doesn't work in iOS simulator - all failing tests say "ref test
> hashes didn't match but diff passed".
All of them? Don't we have an "expected mismatch" test that fails when ref tests aren't working?
Comment on attachment 272389 [details]
Clearing flags on attachment: 272389
Committed r197227: <http://trac.webkit.org/changeset/197227>
All reviewed patches have been landed. Closing bug.
Presumably all expected mismatch tests are already marked as failing or skipped, because of this bug.
Now that reftests work, there is a number of failing tests.
Mass marked them as failing in <http://trac.webkit.org/r197258>. They need triage, it's mostly text and layout issues.
We should probably have a directory of unskippable tests (like fast/harness) that make sure the harness is working.
We didn't really have much trouble detecting that something was wrong, and guessing that reftests were completely broken. The problem was with finding the time to investigate the issue.