Bug 154764 – [iOS Simulator] Reftests don't work

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Alexey Proskuryakov]

ImageDiff doesn't work in iOS simulator - all failing tests say "ref test hashes didn't match but diff passed".

Looks like the problem is with launching it:

$ xcrun -sdk iphonesimulator sim --environment=preserve --adopt-pid /Volumes/Data/slave/ios-simulator-9-release-tests-wk1/build/WebKitBuild/Release-iphonesimulator/ImageDiff --tolerance 0
xcrun: error: unable to find utility "sim", not a developer tool or in PATH

[Alexey Proskuryakov]

Created attachment 272389 [details]
proposed fix

[Simon Fraser (smfr)]

(In reply to comment #0)
> ImageDiff doesn't work in iOS simulator - all failing tests say "ref test
> hashes didn't match but diff passed".

All of them? Don't we have an "expected mismatch" test that fails when ref tests aren't working?

[WebKit Commit Bot]

Comment on attachment 272389 [details]
proposed fix

Clearing flags on attachment: 272389

Committed r197227: <http://trac.webkit.org/changeset/197227>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Alexey Proskuryakov]

Presumably all expected mismatch tests are already marked as failing or skipped, because of this bug.

[Alexey Proskuryakov]

Now that reftests work, there is a number of failing tests.

https://build.webkit.org/results/Apple%20iOS%209%20Simulator%20Release%20WK2%20(Tests)/r197255%20(3477)/results.html

Mass marked them as failing in <http://trac.webkit.org/r197258>. They need triage, it's mostly text and layout issues.

[Simon Fraser (smfr)]

We should probably have a directory of unskippable tests (like fast/harness) that make sure the harness is working.

[Alexey Proskuryakov]

We didn't really have much trouble detecting that something was wrong, and guessing that reftests were completely broken. The problem was with finding the time to investigate the issue.