59188 2011-04-22 04:33:37 -0700 Reftests can not work on Mac build of WebKit. 2011-06-13 23:31:36 -0700 1 1 1 Unclassified WebKit Tools / Tests 528+ (Nightly build) PC OS X 10.5 RESOLVED FIXED P2 Normal --- 36065 58858 1 hayato hayato dpranke hayato ojan webkit.review.bot oldest_to_newest 390707 0 hayato 2011-04-22 04:33:37 -0700 Please see the following bug for the detail: https://bugs.webkit.org/show_bug.cgi?id=58858 The issue comes from the different behavior between ChrominDriver and WebKitDriver which new-run-webkit-tests uses. I'll update this entry with more details later. 401737 1 hayato 2011-05-10 20:51:29 -0700 Let me update the status. It seems that WebKitDriver.run_test() returns 'None' as an image_hash value when an actual image_hash matches the given driver input's image_hash. Whereas, ChromiumDriver.run_test() returns an actual image_hash value as is when the actual image_hash matches the given driver input's image_hash. I think that this different behavior doesn't cause any problems to existing pixel tests tests because existing pixel tests skips 'comparing images' if an actual image_hash is 'None'. But reftests always try to compare image_hash values of both HTML files. That is the root cause. 401740 2 hayato 2011-05-10 21:05:09 -0700 I guess there are two ways to fix the issue: A). Update WebKitDriver, maybe including a DumpRenderTree itself used by WebKitDriver, so that it returns an actual image_hash value even when image_hash_values matches. B). Update reftests runner so that it can handle 'None' value from WebKitDriver. 401742 3 hayato 2011-05-10 21:14:04 -0700 Hi Dirk, Ojan, (In reply to comment #1) > Let me update the status. > > It seems that WebKitDriver.run_test() returns 'None' as an image_hash value when an actual image_hash matches the given driver input's image_hash. > Whereas, ChromiumDriver.run_test() returns an actual image_hash value as is when the actual image_hash matches the given driver input's image_hash. Could you tell me that this difference between WebKitDriver and ChromiumDriver is an expected one? (In reply to comment #2) > I guess there are two ways to fix the issue: > > A). Update WebKitDriver, maybe including a DumpRenderTree itself used by WebKitDriver, so that it returns an actual image_hash value even when image_hash_values matches. > > B). Update reftests runner so that it can handle 'None' value from WebKitDriver. If this different behavior is unexpected one, I'd like to choose A) to fix the issue rather than B) 401796 4 dpranke 2011-05-10 23:32:46 -0700 (In reply to comment #3) > Hi Dirk, Ojan, > > (In reply to comment #1) > > Let me update the status. > > > > It seems that WebKitDriver.run_test() returns 'None' as an image_hash value when an actual image_hash matches the given driver input's image_hash. > > Whereas, ChromiumDriver.run_test() returns an actual image_hash value as is when the actual image_hash matches the given driver input's image_hash. > > Could you tell me that this difference between WebKitDriver and ChromiumDriver is an expected one? > No, it's a bug. The Driver should always return an image hash if it got one from DRT. Feel free to upload a patch to fix this :) Seems like pixel tests couldn't possibly work on the Mac port if this was true, but maybe there's something else at play as well. Then again, I haven't tried pixel tests on the Mac port in a long time. - Dirk 401805 5 hayato 2011-05-10 23:44:29 -0700 Thank you for the replay, Dirk. Okay. I'll take a look further. I suspect DumpRenderTree itself doesn't return image_hash on Mac Port. (In reply to comment #4) > (In reply to comment #3) > > Hi Dirk, Ojan, > > > > (In reply to comment #1) > > > Let me update the status. > > > > > > It seems that WebKitDriver.run_test() returns 'None' as an image_hash value when an actual image_hash matches the given driver input's image_hash. > > > Whereas, ChromiumDriver.run_test() returns an actual image_hash value as is when the actual image_hash matches the given driver input's image_hash. > > > > Could you tell me that this difference between WebKitDriver and ChromiumDriver is an expected one? > > > > No, it's a bug. The Driver should always return an image hash if it got one from DRT. Feel free to upload a patch to fix this :) > > Seems like pixel tests couldn't possibly work on the Mac port if this was true, but maybe there's something else at play as well. Then again, I haven't tried pixel tests on the Mac port in a long time. > > - Dirk 402049 6 dpranke 2011-05-11 09:39:16 -0700 (In reply to comment #5) > Thank you for the replay, Dirk. > > Okay. I'll take a look further. I suspect DumpRenderTree itself doesn't return image_hash on Mac Port. > If it doesn't, you can always return the value that is passed in. Note that you don't (and shouldn't) need to return a value if you are not doing pixel tests (and I don't think one is even passed in in that case). 417936 7 96568 hayato 2011-06-09 03:31:22 -0700 Created attachment 96568 Patch 418302 8 96568 ojan 2011-06-09 14:28:12 -0700 Comment on attachment 96568 Patch View in context: https://bugs.webkit.org/attachment.cgi?id=96568&action=review > Tools/ChangeLog:12 > + No tests since I'll add sample reftests to make sure mismatch reftests work later, > + which should be in a separate patch. I'd actually prefer putting the sample reftests in this patch, but a separate patch is fine too. 420239 9 96568 webkit.review.bot 2011-06-13 23:31:31 -0700 Comment on attachment 96568 Patch Clearing flags on attachment: 96568 Committed r88780: <http://trac.webkit.org/changeset/88780> 420240 10 webkit.review.bot 2011-06-13 23:31:36 -0700 All reviewed patches have been landed. Closing bug. 96568 2011-06-09 03:31:22 -0700 2011-06-13 23:31:31 -0700 Patch bug-59188-20110609193120.patch text/plain 1724 hayato U3VidmVyc2lvbiBSZXZpc2lvbjogODc5NzkKZGlmZiAtLWdpdCBhL1Rvb2xzL0NoYW5nZUxvZyBi L1Rvb2xzL0NoYW5nZUxvZwppbmRleCA2YzNjZjViZmY1ZTQ2MjllY2QzY2I3MDI1NWZhMTdkZThl MDFlZDYwLi4zOWYxNzIwYjllYmMxYzdkNmE0MjhkYmJkZWFlM2RhNjNmMzZkZGRiIDEwMDY0NAot LS0gYS9Ub29scy9DaGFuZ2VMb2cKKysrIGIvVG9vbHMvQ2hhbmdlTG9nCkBAIC0xLDMgKzEsMTgg QEAKKzIwMTEtMDYtMDkgIEhheWF0byBJdG8gIDxoYXlhdG9AY2hyb21pdW0ub3JnPgorCisgICAg ICAgIFJldmlld2VkIGJ5IE5PQk9EWSAoT09QUyEpLgorCisgICAgICAgIFtOUldUXSBGaXggYSBX ZWJLaXREcml2ZXIgc28gdGhhdCBpdCByZXR1cm5zIGFuIGFjdHVhbCBoYXNoIHZhbHVlIGlmIERS VCByZXR1cm5zIHRoYXQuCisgICAgICAgIGh0dHBzOi8vYnVncy53ZWJraXQub3JnL3Nob3dfYnVn LmNnaT9pZD01OTE4OAorCisgICAgICAgIEEgY3VycmVudCBXZWJLaXREcml2ZXIgZG9lc24ndCB0 cnkgdG8gcmV0dXJuIGFuIGFjdHVhbCBoYXNoIHZhbHVlIGlmIERSVCBkb2VzIG5vdCByZXR1cm4g YW4gaW1hZ2UuCisgICAgICAgIFdlYktpdERyaXZlciBzaG91bGQgYWx3YXlzIHJldHVybiBhbiBh Y3R1YWwgaGFzaCB2YWx1ZSBpZiBEUlQgcmV0dXJucyB0aGF0LgorCisgICAgICAgIE5vIHRlc3Rz IHNpbmNlIEknbGwgYWRkIHNhbXBsZSByZWZ0ZXN0cyB0byBtYWtlIHN1cmUgbWlzbWF0Y2ggcmVm dGVzdHMgd29yayBsYXRlciwKKyAgICAgICAgd2hpY2ggc2hvdWxkIGJlIGluIGEgc2VwYXJhdGUg cGF0Y2guCisKKyAgICAgICAgKiBTY3JpcHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9wb3J0L3dl YmtpdC5weToKKwogMjAxMS0wNi0wMiAgRGlyayBQcmFua2UgIDxkcHJhbmtlQGNocm9taXVtLm9y Zz4KIAogICAgICAgICBSZXZpZXdlZCBieSBUb255IENoYW5nLgpkaWZmIC0tZ2l0IGEvVG9vbHMv U2NyaXB0cy93ZWJraXRweS9sYXlvdXRfdGVzdHMvcG9ydC93ZWJraXQucHkgYi9Ub29scy9TY3Jp cHRzL3dlYmtpdHB5L2xheW91dF90ZXN0cy9wb3J0L3dlYmtpdC5weQppbmRleCBiYmM1NjNkYmE0 ZDc4NDM4MGQ3MzUzNDU3MDM4YTg5MTJlNjliNGM2Li44YjIxM2E4OGYyN2MzN2YyNDJjYmViY2Q1 ZWIyZmZiMDhmMWY2MDBiIDEwMDY0NAotLS0gYS9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91 dF90ZXN0cy9wb3J0L3dlYmtpdC5weQorKysgYi9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L2xheW91 dF90ZXN0cy9wb3J0L3dlYmtpdC5weQpAQCAtNDQxLDYgKzQ0MSw4IEBAIGNsYXNzIFdlYktpdERy aXZlcihiYXNlLkRyaXZlcik6CiAgICAgICAgICAgICBpZiBibG9jay5jb250ZW50IGFuZCBibG9j ay5jb250ZW50X3R5cGUgPT0gJ2ltYWdlL3BuZyc6CiAgICAgICAgICAgICAgICAgaW1hZ2UgPSBi bG9jay5kZWNvZGVkX2NvbnRlbnQKICAgICAgICAgICAgICAgICBhY3R1YWxfaW1hZ2VfaGFzaCA9 IGJsb2NrLmNvbnRlbnRfaGFzaAorICAgICAgICAgICAgZWxpZiBibG9jay5jb250ZW50X2hhc2g6 CisgICAgICAgICAgICAgICAgYWN0dWFsX2ltYWdlX2hhc2ggPSBibG9jay5jb250ZW50X2hhc2gK IAogICAgICAgICBlcnJvcl9saW5lcyA9IHNlbGYuX3NlcnZlcl9wcm9jZXNzLmVycm9yLnNwbGl0 bGluZXMoKQogICAgICAgICAjIEZJWE1FOiBUaGlzIGlzIGEgaGFjay4gIEl0IGlzIHVuY2xlYXIg d2h5IHNvbWV0aW1lcwo=