Bug 104850 – Fix detection of ref test files in old and new run webkit tests

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Roger Fong]

Previously skipped ref tests here:
http://trac.webkit.org/changeset/125604
But only where results used the -expected tag.

Some ref tests now use the -ref tag instead of -expected.
Changing old-run-webkit-tests to reflect this.

[Roger Fong]

Created attachment 179147 [details]
patch

[Roger Fong]

Created attachment 179169 [details]
patch

[Tim Horton]

Comment on attachment 179169 [details]
patch

View in context: https://bugs.webkit.org/attachment.cgi?id=179169&action=review

> Tools/Scripts/old-run-webkit-tests:2638
> +    my $prefixExtensionExpression = "^(($refTag|$notrefTag)-)";

Don't need the outer parens, I think.

[Roger Fong]

http://trac.webkit.org/changeset/137553

[Dirk Pranke]

Comment on attachment 179169 [details]
patch

Looks like I was mistaken, and the W3C does not use ref- or noref- as prefixes for the references, just -ref and -notref instead of our -expected and -expected-mismatch.

What they *do* use ref- for is for actual common reference files, which would live in a references/ directory (e.g., css2.1/reference/ref-filled-green-100px-square.htm ). However, in this case, we'd probably want to actually treat such tests as pixel tests, if we ran them at all. Note that we haven't actually imported any such references yet.

See http://wiki.csswg.org/test/reftest and https://bugs.webkit.org/show_bug.cgi?id=83048 .

[Elliott Sprehn]

This broke run-webkit-tests for me:

Using port 'chromium-linux-x86_64'
Test configuration: <lucid, x86_64, debug>
Placing test results in /media/Checkouts/chrome1/src/webkit/Debug/layout-test-results
Baseline search path: chromium-linux -> chromium-win -> chromium -> generic
Using Debug build
Pixel tests enabled
Regular timeout: 12000, slow test timeout: 60000
Command line: /media/Checkouts/chrome1/src/out/Debug/DumpRenderTree -

Collecting tests ...
AttributeError raised: 'str' object has no attribute 'startsWith'
Traceback (most recent call last):
  File "/media/Checkouts/chrome1/src/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 407, in main
    run_details = run(port, options, args, stderr)
  File "/media/Checkouts/chrome1/src/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 105, in run
    run_details = manager.run(args)
  File "/media/Checkouts/chrome1/src/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 346, in run
    paths, test_names = self._collect_tests(args)
  File "/media/Checkouts/chrome1/src/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 265, in _collect_tests
    return self._finder.find_tests(self._options, args)
  File "/media/Checkouts/chrome1/src/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_finder.py", line 50, in find_tests
    test_files = self._port.tests(paths)
  File "/media/Checkouts/chrome1/src/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py", line 580, in tests
    tests = self._real_tests(expanded_paths)
  File "/media/Checkouts/chrome1/src/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py", line 600, in _real_tests
    files = find_files.find(self._filesystem, self.layout_tests_dir(), paths, skipped_directories, Port._is_test_file, self.test_key)
  File "/media/Checkouts/chrome1/src/third_party/WebKit/Tools/Scripts/webkitpy/common/find_files.py", line 61, in find
    return _normalized_find(filesystem, _normalize(filesystem, base_dir, paths), skipped_directories, file_filter, directory_sort_key)
  File "/media/Checkouts/chrome1/src/third_party/WebKit/Tools/Scripts/webkitpy/common/find_files.py", line 83, in _normalized_find
    all_files = itertools.chain(*(sort_by_directory_key(filesystem.files_under(path, skipped_directories, file_filter)) for path in paths_to_walk))
  File "/media/Checkouts/chrome1/src/third_party/WebKit/Tools/Scripts/webkitpy/common/find_files.py", line 83, in <genexpr>
    all_files = itertools.chain(*(sort_by_directory_key(filesystem.files_under(path, skipped_directories, file_filter)) for path in paths_to_walk))
  File "/media/Checkouts/chrome1/src/third_party/WebKit/Tools/Scripts/webkitpy/common/system/filesystem.py", line 109, in files_under
    if file_filter(self, dirpath, filename):
  File "/media/Checkouts/chrome1/src/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py", line 626, in _is_test_file
    return Port._has_supported_extension(filesystem, filename) and not Port.is_reference_html_file(filesystem, dirname, filename)
  File "/media/Checkouts/chrome1/src/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py", line 610, in is_reference_html_file
    if filename.startswith('ref-') or filename.startsWith('notref-'):
AttributeError: 'str' object has no attribute 'startsWith'
Failed to execute Tools/Scripts/new-run-webkit-tests at ./Tools/Scripts/run-webkit-tests line 126.

[Elliott Sprehn]

(In reply to comment #6)
> This broke run-webkit-tests for me:
> 


Note this is when I do

./Tools/Scripts/run-webkit-tests --debug --chromium LayoutTests/

[Elliott Sprehn]

Comment on attachment 179169 [details]
patch

View in context: https://bugs.webkit.org/attachment.cgi?id=179169&action=review

> Tools/Scripts/webkitpy/layout_tests/port/base.py:610
> +        if filename.startswith('ref-') or filename.startsWith('notref-'):

This is a typo, the method in python is all lower case.

[Roger Fong]

Sorry fixed it r137558