Bug 154764 - [iOS Simulator] Reftests don't work
Summary: [iOS Simulator] Reftests don't work
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: Other
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Alexey Proskuryakov
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-02-26 17:23 PST by Alexey Proskuryakov
Modified: 2016-02-29 09:26 PST (History)
9 users (show)

See Also:


Attachments
proposed fix (1.80 KB, patch)
2016-02-26 18:04 PST, Alexey Proskuryakov
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Alexey Proskuryakov 2016-02-26 17:23:43 PST
ImageDiff doesn't work in iOS simulator - all failing tests say "ref test hashes didn't match but diff passed".

Looks like the problem is with launching it:

$ xcrun -sdk iphonesimulator sim --environment=preserve --adopt-pid /Volumes/Data/slave/ios-simulator-9-release-tests-wk1/build/WebKitBuild/Release-iphonesimulator/ImageDiff --tolerance 0
xcrun: error: unable to find utility "sim", not a developer tool or in PATH
Comment 1 Alexey Proskuryakov 2016-02-26 18:04:38 PST
Created attachment 272389 [details]
proposed fix
Comment 2 Simon Fraser (smfr) 2016-02-26 18:14:19 PST
(In reply to comment #0)
> ImageDiff doesn't work in iOS simulator - all failing tests say "ref test
> hashes didn't match but diff passed".

All of them? Don't we have an "expected mismatch" test that fails when ref tests aren't working?
Comment 3 WebKit Commit Bot 2016-02-26 18:53:38 PST
Comment on attachment 272389 [details]
proposed fix

Clearing flags on attachment: 272389

Committed r197227: <http://trac.webkit.org/changeset/197227>
Comment 4 WebKit Commit Bot 2016-02-26 18:53:43 PST
All reviewed patches have been landed.  Closing bug.
Comment 5 Alexey Proskuryakov 2016-02-26 18:56:53 PST
Presumably all expected mismatch tests are already marked as failing or skipped, because of this bug.
Comment 6 Alexey Proskuryakov 2016-02-27 12:23:46 PST
Now that reftests work, there is a number of failing tests.

https://build.webkit.org/results/Apple%20iOS%209%20Simulator%20Release%20WK2%20(Tests)/r197255%20(3477)/results.html

Mass marked them as failing in <http://trac.webkit.org/r197258>. They need triage, it's mostly text and layout issues.
Comment 7 Simon Fraser (smfr) 2016-02-28 20:18:59 PST
We should probably have a directory of unskippable tests (like fast/harness) that make sure the harness is working.
Comment 8 Alexey Proskuryakov 2016-02-29 09:26:08 PST
We didn't really have much trouble detecting that something was wrong, and guessing that reftests were completely broken. The problem was with finding the time to investigate the issue.