154764 2016-02-26 17:23:43 -0800 [iOS Simulator] Reftests don't work 2016-02-29 09:26:08 -0800 1 1 1 Unclassified WebKit Tools / Tests Other Unspecified Unspecified RESOLVED FIXED https://bugs.webkit.org/show_bug.cgi?id=154761 P2 Normal --- 1 ap ap commit-queue dbates glenn lforschler mmaxfield ryanhaddad simon.fraser thorton zalan oldest_to_newest 1168762 0 ap 2016-02-26 17:23:43 -0800 ImageDiff doesn't work in iOS simulator - all failing tests say "ref test hashes didn't match but diff passed". Looks like the problem is with launching it: $ xcrun -sdk iphonesimulator sim --environment=preserve --adopt-pid /Volumes/Data/slave/ios-simulator-9-release-tests-wk1/build/WebKitBuild/Release-iphonesimulator/ImageDiff --tolerance 0 xcrun: error: unable to find utility "sim", not a developer tool or in PATH 1168774 1 272389 ap 2016-02-26 18:04:38 -0800 Created attachment 272389 proposed fix 1168778 2 simon.fraser 2016-02-26 18:14:19 -0800 (In reply to comment #0) > ImageDiff doesn't work in iOS simulator - all failing tests say "ref test > hashes didn't match but diff passed". All of them? Don't we have an "expected mismatch" test that fails when ref tests aren't working? 1168779 3 272389 commit-queue 2016-02-26 18:53:38 -0800 Comment on attachment 272389 proposed fix Clearing flags on attachment: 272389 Committed r197227: <http://trac.webkit.org/changeset/197227> 1168780 4 commit-queue 2016-02-26 18:53:43 -0800 All reviewed patches have been landed. Closing bug. 1168781 5 ap 2016-02-26 18:56:53 -0800 Presumably all expected mismatch tests are already marked as failing or skipped, because of this bug. 1168891 6 ap 2016-02-27 12:23:46 -0800 Now that reftests work, there is a number of failing tests. https://build.webkit.org/results/Apple%20iOS%209%20Simulator%20Release%20WK2%20(Tests)/r197255%20(3477)/results.html Mass marked them as failing in <http://trac.webkit.org/r197258>. They need triage, it's mostly text and layout issues. 1169065 7 simon.fraser 2016-02-28 20:18:59 -0800 We should probably have a directory of unskippable tests (like fast/harness) that make sure the harness is working. 1169158 8 ap 2016-02-29 09:26:08 -0800 We didn't really have much trouble detecting that something was wrong, and guessing that reftests were completely broken. The problem was with finding the time to investigate the issue. 272389 2016-02-26 18:04:38 -0800 2016-02-26 18:53:38 -0800 proposed fix iOSImageDiff.txt text/plain 1846 ap SW5kZXg6IFRvb2xzL0NoYW5nZUxvZwo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBUb29scy9DaGFuZ2VMb2cJKHJl dmlzaW9uIDE5NzIyNCkKKysrIFRvb2xzL0NoYW5nZUxvZwkod29ya2luZyBjb3B5KQpAQCAtMSwz ICsxLDEzIEBACisyMDE2LTAyLTI2ICBBbGV4ZXkgUHJvc2t1cnlha292ICA8YXBAYXBwbGUuY29t PgorCisgICAgICAgIFtpT1MgU2ltdWxhdG9yXSBSZWZ0ZXN0cyBkb24ndCB3b3JrCisgICAgICAg IGh0dHBzOi8vYnVncy53ZWJraXQub3JnL3Nob3dfYnVnLmNnaT9pZD0xNTQ3NjQKKworICAgICAg ICBSZXZpZXdlZCBieSBOT0JPRFkgKE9PUFMhKS4KKworICAgICAgICAqIFNjcmlwdHMvd2Via2l0 cHkvcG9ydC9pbWFnZV9kaWZmLnB5OiAoSU9TU2ltdWxhdG9ySW1hZ2VEaWZmZXIuX3N0YXJ0KToK KyAgICAgICAgVXNlIHNpbWN0bCBpbnN0ZWFkIG9mIHNpbS4KKwogMjAxNi0wMi0yNiAgQ29tbWl0 IFF1ZXVlICA8Y29tbWl0LXF1ZXVlQHdlYmtpdC5vcmc+CiAKICAgICAgICAgVW5yZXZpZXdlZCwg cm9sbGluZyBvdXQgcjE5NzE1OC4KSW5kZXg6IFRvb2xzL1NjcmlwdHMvd2Via2l0cHkvcG9ydC9p bWFnZV9kaWZmLnB5Cj09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT0KLS0tIFRvb2xzL1NjcmlwdHMvd2Via2l0cHkvcG9ydC9p bWFnZV9kaWZmLnB5CShyZXZpc2lvbiAxOTcxODEpCisrKyBUb29scy9TY3JpcHRzL3dlYmtpdHB5 L3BvcnQvaW1hZ2VfZGlmZi5weQkod29ya2luZyBjb3B5KQpAQCAtMTE3LDcgKzExNywxMCBAQCBj bGFzcyBJbWFnZURpZmZlcihvYmplY3QpOgogCiBjbGFzcyBJT1NTaW11bGF0b3JJbWFnZURpZmZl cihJbWFnZURpZmZlcik6CiAgICAgZGVmIF9zdGFydChzZWxmLCB0b2xlcmFuY2UpOgotICAgICAg ICBjb21tYW5kID0gWyd4Y3J1bicsICctc2RrJywgJ2lwaG9uZXNpbXVsYXRvcicsICdzaW0nLCAn LS1lbnZpcm9ubWVudD1wcmVzZXJ2ZScsICctLWFkb3B0LXBpZCcsIHNlbGYuX3BvcnQuX3BhdGhf dG9faW1hZ2VfZGlmZigpLCAnLS10b2xlcmFuY2UnLCBzdHIodG9sZXJhbmNlKV0KKyAgICAgICAg IyBGSVhNRTogVXNpbmcgImJvb3RlZCIgaXMgbm90IDEwMCUgY29ycmVjdCwgd2Ugc2hvdWxkIHVz ZSBhIGRldmljZSBJRCB0aGF0IHdlIGNyZWF0ZWQuCisgICAgICAgICMgRklYTUU6IElzIGl0IGEg cHJvYmxlbSB0aGF0IEltYWdlRGlmZiBhbmQgc2ltY3RsIHJlbWFpbiBzZXBhcmF0ZSBwcm9jZXNz ZXM/IFRoaXMgY29kZSB1c2VkIHRvIHBhc3MgLS1hZG9wdC1waWQgdG8gc2ltIHRvb2wuCisgICAg ICAgICMgV2Ugc2hvdWxkIHByb2JhYmx5IGp1c3QgYnVpbGQgYW5kIHVzZSBhIGhvc3QgdmVyc2lv biBvZiBJbWFnZURpZmYgaW5zdGVhZCBvZiBhIHNpbXVsYXRvciBvbmUgdG8gc29sdmUgYm90aC4K KyAgICAgICAgY29tbWFuZCA9IFsneGNydW4nLCAnLXNkaycsICdpcGhvbmVzaW11bGF0b3InLCAn c2ltY3RsJywgJ3NwYXduJywgJ2Jvb3RlZCcsIHNlbGYuX3BvcnQuX3BhdGhfdG9faW1hZ2VfZGlm ZigpLCAnLS10b2xlcmFuY2UnLCBzdHIodG9sZXJhbmNlKV0KICAgICAgICAgZW52aXJvbm1lbnQg PSBzZWxmLl9wb3J0LnNldHVwX2Vudmlyb25fZm9yX3NlcnZlcignSW1hZ2VEaWZmJykKICAgICAg ICAgc2VsZi5fcHJvY2VzcyA9IHNlbGYuX3BvcnQuX3NlcnZlcl9wcm9jZXNzX2NvbnN0cnVjdG9y KHNlbGYuX3BvcnQsICdJbWFnZURpZmYnLCBjb21tYW5kLCBlbnZpcm9ubWVudCkKICAgICAgICAg c2VsZi5fcHJvY2Vzcy5zdGFydCgpCg==