Bug 176686 – Remove duplicated test results from LayoutTest platform directories

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Carlos Garcia Campos]

After several rebaselines we might end up with the same result in the platform directory than the common result without noticing it. I've written a script to detect those cases and remove all the duplicated results from the repository. The current status is:

$ Tools/Scripts/check-for-duplicated-platform-test-results -n | grep found
5 found in platform mac-yosemite
15 found in platform win-7sp0
2 found in platform win-future
110 found in platform wpe
2 found in platform mac-highsierra
8 found in platform mac-elcapitan
142 found in platform win
2 found in platform mac-wk1
21 found in platform ios-wk2
147 found in platform mac
3 found in platform mac-wk2
2 found in platform ios-11
20 found in platform ios-wk1
109 found in platform ios
237 found in platform gtk

[Carlos Garcia Campos]

Created attachment 320421 [details]
Patch

This patch removes the files and includes the script too.

$ Tools/Scripts/check-for-duplicated-platform-test-results
5 found in platform mac-yosemite
15 found in platform win-7sp0
2 found in platform win-future
110 found in platform wpe
2 found in platform mac-highsierra
8 found in platform mac-elcapitan
142 found in platform win
2 found in platform mac-wk1
21 found in platform ios-wk2
147 found in platform mac
3 found in platform mac-wk2
2 found in platform ios-11
20 found in platform ios-wk1
109 found in platform ios
237 found in platform gtk
825 files have been removed from the repository, check the status and commit if everything is correct

[Carlos Garcia Campos]

Created attachment 320422 [details]
Patch

I forgot to use --binary

[Build Bot]

Comment on attachment 320422 [details]
Patch

Attachment 320422 [details] did not pass mac-ews (mac):
Output: http://webkit-queues.webkit.org/results/4511410

New failing tests:
compositing/webgl/webgl-no-alpha.html
editing/mac/spelling/autocorrection-at-beginning-of-word-2.html
fast/forms/search/search-padding-cancel-results-buttons.html
fast/forms/auto-fill-button/input-contacts-auto-fill-button.html
platform/mac/fast/dom/HTMLImageElement/dashboard-src-quirk.html
fast/forms/auto-fill-button/input-auto-fill-button.html
editing/mac/spelling/autocorrection-at-beginning-of-word-1.html

[Build Bot]

Created attachment 320424 [details]
Archive of layout-test-results from ews102 for mac-elcapitan

The attached test failures were seen while running run-webkit-tests on the mac-ews.
Bot: ews102  Port: mac-elcapitan  Platform: Mac OS X 10.11.6

[Build Bot]

Comment on attachment 320422 [details]
Patch

Attachment 320422 [details] did not pass mac-wk2-ews (mac-wk2):
Output: http://webkit-queues.webkit.org/results/4511421

New failing tests:
compositing/webgl/webgl-no-alpha.html
http/tests/security/cross-origin-local-storage-wk1.html
editing/mac/spelling/autocorrection-at-beginning-of-word-2.html
fast/forms/search/search-padding-cancel-results-buttons.html
fast/forms/auto-fill-button/input-contacts-auto-fill-button.html
platform/mac/fast/dom/HTMLImageElement/dashboard-src-quirk.html
fast/forms/auto-fill-button/input-auto-fill-button.html
editing/mac/spelling/autocorrection-at-beginning-of-word-1.html

[Build Bot]

Created attachment 320425 [details]
Archive of layout-test-results from ews107 for mac-elcapitan-wk2

The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: ews107  Port: mac-elcapitan-wk2  Platform: Mac OS X 10.11.6

[Carlos Garcia Campos]

It seems the script is not smart enough...

[Carlos Garcia Campos]

I guess it's because when removing the expected results for mac-elcapitan, it's comparing to what there's in mac platform instead of common results. The script needs to know the relationships between platforms and check for theses particular cases. Or maybe in these cases we should add the different result to the most specific platform dir that produces a different result.

[Build Bot]

Comment on attachment 320422 [details]
Patch

Attachment 320422 [details] did not pass mac-debug-ews (mac):
Output: http://webkit-queues.webkit.org/results/4511444

New failing tests:
compositing/webgl/webgl-no-alpha.html
editing/mac/spelling/autocorrection-at-beginning-of-word-2.html
fast/forms/search/search-padding-cancel-results-buttons.html
fast/forms/auto-fill-button/input-contacts-auto-fill-button.html
platform/mac/fast/dom/HTMLImageElement/dashboard-src-quirk.html
fast/forms/auto-fill-button/input-auto-fill-button.html
editing/mac/spelling/autocorrection-at-beginning-of-word-1.html

[Build Bot]

Created attachment 320426 [details]
Archive of layout-test-results from ews117 for mac-elcapitan

The attached test failures were seen while running run-webkit-tests on the mac-debug-ews.
Bot: ews117  Port: mac-elcapitan  Platform: Mac OS X 10.11.6

[Build Bot]

Comment on attachment 320422 [details]
Patch

Attachment 320422 [details] did not pass ios-sim-ews (ios-simulator-wk2):
Output: http://webkit-queues.webkit.org/results/4511457

New failing tests:
imported/w3c/web-platform-tests/html/semantics/forms/the-label-element/labelable-elements.html
imported/w3c/web-platform-tests/html/semantics/forms/form-control-infrastructure/form.html
imported/w3c/web-platform-tests/html/semantics/interfaces.html
compositing/webgl/webgl-no-alpha.html

[Build Bot]

Created attachment 320428 [details]
Archive of layout-test-results from ews126 for ios-simulator-wk2

The attached test failures were seen while running run-webkit-tests on the ios-sim-ews.
Bot: ews126  Port: ios-simulator-wk2  Platform: Mac OS X 10.12.5

[Carlos Garcia Campos]

Does this work the other way around too? I mean, for example for platform mac-elcapitan the search path is:

mac-elcapitan -> mac-sierra-wk2 -> mac-sierra -> mac-wk2 -> mac -> wk2

so, any rebaseline in mac-elcapitan subdir that is exactly the same as the one in mac could be removed too as long as there isn't a different result in mac-sierra? even if it's not the same as the main result? If my script is not wrong there are currently 267 of those.

[Carlos Garcia Campos]

(In reply to Carlos Garcia Campos from comment #13)
> Does this work the other way around too? I mean, for example for platform
> mac-elcapitan the search path is:
> 
> mac-elcapitan -> mac-sierra-wk2 -> mac-sierra -> mac-wk2 -> mac -> wk2
> 
> so, any rebaseline in mac-elcapitan subdir that is exactly the same as the
> one in mac could be removed too as long as there isn't a different result in
> mac-sierra? even if it's not the same as the main result? If my script is
> not wrong there are currently 267 of those.

Nevermind, my script was indeed wrong :-P

[Carlos Garcia Campos]

With a new version of the patch that checks the baseline search paths this is what I'm getting now:

$ ./check-for-duplicated-platform-test-results -n | grep "found in"
240 found in gtk -> wk2 -> generic
13 found in mac-yosemite -> mac-elcapitan -> mac-sierra -> mac-wk1 -> mac -> generic
5 found in win-7sp0 -> win -> mac -> generic
193 found in win -> mac -> generic
110 found in wpe -> wk2 -> generic
3 found in mac-wk1 -> mac -> generic
7 found in mac-elcapitan -> mac-sierra -> mac-wk1 -> mac -> generic
3 found in mac-wk2 -> mac -> wk2 -> generic

I'm not sure about the win tests, and they can't be tested with EWS, so I'll submit a new patch with the new version of the script and the duplicated removes for all platforms except win.

[Carlos Garcia Campos]

Created attachment 320551 [details]
Updated patch

[Michael Catanzaro]

Comment on attachment 320551 [details]
Updated patch

View in context: https://bugs.webkit.org/attachment.cgi?id=320551&action=review

r=me if it passes all the EWS bots

> Tools/Scripts/check-for-duplicated-platform-test-results:29
> +layout_tests_directory = os.path.join(checkout_root, "LayoutTests")
> +platform_directory = os.path.join(layout_tests_directory, "platform")

Nit: please don't switch between using single quotes and double quotes for strings in the same script. I would stick with single quotes, since that's what most of our WebKit scripts use, and using single quotes also matches the GNOME style to use double quotes only for translatable strings.

> Tools/Scripts/check-for-duplicated-platform-test-results:110
> +    print "%d found in %s -> generic" % (duplicates_len, " -> ".join([os.path.basename(p) for p in baseline_search_path[:-1]]))

And here you've switched back to double quotes, though you used single quotes for most of the script.

[Carlos Garcia Campos]

Committed r221928: <http://trac.webkit.org/changeset/221928>

[Radar WebKit Bug Importer]

<rdar://problem/34693476>