Change set http://trac.webkit.org/changeset/118566 added new Acid3 tests and baselines. However, the bots are reporting the files as MISSING despite the fallback rules: http://test-results.appspot.com/dashboards/flakiness_dashboard.html#tests=http%2Ftests%2Fmisc%2Facid3.html dpranke mentions (in chat) that "acid3-expected.png doesn't have an embedded checksum and it looks like we don't handle that very well"
Created attachment 144383 [details] Patch
Created attachment 144479 [details] Patch
Committed r118741: <http://trac.webkit.org/changeset/118741>
Sorry, forgot to land with --no-close. This bug still exists.
MISSING seems correct for a png missing a checksum. We do printout "No expected image hash found" for that test. I suppose we should make that text more clear like "No embedded checksum in the png file". Arguably that test should still pass and just log a warning, but I think I prefer it failing hard like this. There simply shouldn't ever be png files without checksums. We should see how this one got committed and resolve that issue.
Looks like the break happened in https://bugs.webkit.org/show_bug.cgi?id=87187. Andras, did you use run-webkit-tests to generate the png files for that patch? We should figure out how you ended up with png files that lack embedded checksums. On a tooling note, we should add a presubmit check for this. It would be super easy. And we should add a big warning in the code-review tool. Also super-easy.
(In reply to comment #6) > Looks like the break happened in https://bugs.webkit.org/show_bug.cgi?id=87187. Andras, did you use run-webkit-tests to generate the png files for that patch? We should figure out how you ended up with png files that lack embedded checksums. D'oh, sorry for that. Those png files need to be regenerated. Can the actual png's be recovered from the build bots or the ews? > > On a tooling note, we should add a presubmit check for this. It would be super easy. And we should add a big warning in the code-review tool. Also super-easy. Agree on that.
(In reply to comment #7) > (In reply to comment #6) > > Looks like the break happened in https://bugs.webkit.org/show_bug.cgi?id=87187. Andras, did you use run-webkit-tests to generate the png files for that patch? We should figure out how you ended up with png files that lack embedded checksums. > > D'oh, sorry for that. Those png files need to be regenerated. > Can the actual png's be recovered from the build bots or the ews? > You should be able to fetch the chromium png's from the bots. I tend to agree with Ojan that this should stay as a hard failure, although I am open to suggestions to make it clearer what's happened. I think the missing hash log message gets lost in the --verbose logging.
(In reply to comment #7) > (In reply to comment #6) > > Looks like the break happened in https://bugs.webkit.org/show_bug.cgi?id=87187. Andras, did you use run-webkit-tests to generate the png files for that patch? We should figure out how you ended up with png files that lack embedded checksums. > > D'oh, sorry for that. Those png files need to be regenerated. Do you know how this happened? If it's a bug in the tooling, I'd like to address it so it doesn't happen again. What process did you use to generate the png files?
https://bugs.webkit.org/show_bug.cgi?id=87791 for prettydiff.
https://bugs.webkit.org/show_bug.cgi?id=87793 for the linter error. So, this bug can focus on the following: 1) How these pngs were generated in the first place. 2) Giving better output from run-webkit-tests.
Created attachment 144776 [details] proposed patch This adds platform independent png with checksum and reverts the chromium expectations since layoutTestController.keepWebHistory() does not seem to work for the chromium test infrastructure. I opened a bug for the issue: https://bugs.webkit.org/show_bug.cgi?id=87839
(In reply to comment #9) > (In reply to comment #7) > Do you know how this happened? If it's a bug in the tooling, I'd like to address it so it doesn't happen again. What process did you use to generate the png files? It was a human error, not a bug in the tools. Sorry for the inconvenience.
(In reply to comment #13) > (In reply to comment #9) > > (In reply to comment #7) > > Do you know how this happened? If it's a bug in the tooling, I'd like to address it so it doesn't happen again. What process did you use to generate the png files? Loos like the mac-wk2 result prior r118566 does not have a checksum, that is what I copied to be a platform independent result.
(In reply to comment #14) > Loos like the mac-wk2 result prior r118566 does not have a checksum, that is what I copied to be a platform independent result. In fact it did have a checksum, which means I ended up copying my mock-up png. Sorry for the noise.
Comment on attachment 144776 [details] proposed patch This patch looks good, but please upload it to a new bug. In general, webkit has a one patch per bug policy and this bug is about the tooling not giving good feedback when this error is hit. R=me if you upload this to a new bug with the appropriate title.
Anything else actionable on this bug? If so, please re-open.