Bug 137441 – Add a test expectations file specific to W3C web platform tests

• Status: RESOLVED
• Resolution: WONTFIX
• Component: Tools / Tests

[youenn fablet]

Ideally, W3C test expectations would be automatically computed each time the version of the web platform test suite is bumped or each time a new module of the web platform test suite is activated. Between bumps, test expectations would be manually edited by patches as currently done.
To ease the automation of these test expectations, it may be desirable to create a separate file for them.

[youenn fablet]

Bug 134767 implements automatic test expectation update computed from conformance results, the update being based on a specific organization of test expectations: test expectations are grouped by submodules (more or less top-level directories of WPT folder).

[youenn fablet]

Created attachment 239320 [details]
Patch

[Ryosuke Niwa]

Comment on attachment 239320 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=239320&action=review

> Tools/Scripts/webkitpy/port/base.py:1032
> -        return [self.path_to_generic_test_expectations_file()] + self._port_specific_expectations_files()
> +        return [self.path_to_generic_test_expectations_file()] + self._port_specific_expectations_files() + [self.path_to_w3c_expectations_file()]

I don't think W3C test expectations should override port specific test expectations.
That would be super confusing.

> LayoutTests/imported/w3c/TestExpectationsW3C:1
> +#imported/w3c/wpt [ Skip ]

This expectation doesn't work because it starts with #.
Does this really need to be at /w3c/?  Can't we move it to imported/w3c/resources/wpt?
If not, we can add it to the list of names we skip instead.

[youenn fablet]

(In reply to comment #3)
> (From update of attachment 239320 [details])
> View in context: https://bugs.webkit.org/attachment.cgi?id=239320&action=review
> 
> > Tools/Scripts/webkitpy/port/base.py:1032
> > -        return [self.path_to_generic_test_expectations_file()] + self._port_specific_expectations_files()
> > +        return [self.path_to_generic_test_expectations_file()] + self._port_specific_expectations_files() + [self.path_to_w3c_expectations_file()]
> 
> I don't think W3C test expectations should override port specific test expectations.
> That would be super confusing.

Right.
For port diverging results, handling may be done manually at first, one port (mac, mac-wk2?) being set as the baseline port.
Introducing a specific W3C test expectation file per port may be useful to automate diverging results handling.

> > LayoutTests/imported/w3c/TestExpectationsW3C:1
> > +#imported/w3c/wpt [ Skip ]
> 
> This expectation doesn't work because it starts with #.

The idea is to uncomment this expectation when doing the first WPT import.
Removing # currently causes a warning because imported/w3c/wpt does not exist yet.

> Does this really need to be at /w3c/?  Can't we move it to imported/w3c/resources/wpt?
> If not, we can add it to the list of names we skip instead.

There is no particular constraint, except that putting it into w3c folder or any of its subfolder allows using the dedicated ChangeLog.
I am fine with renaming it to imported/w3c/resources/wpt/TestExpectations

[youenn fablet]

Created attachment 240011 [details]
Moved W3C test expectations next to generic test expectations

[Ryosuke Niwa]

(In reply to comment #4)
>
> > > LayoutTests/imported/w3c/TestExpectationsW3C:1
> > > +#imported/w3c/wpt [ Skip ]
> > 
> > This expectation doesn't work because it starts with #.
> 
> The idea is to uncomment this expectation when doing the first WPT import.
> Removing # currently causes a warning because imported/w3c/wpt does not
> exist yet.

For what purpose?

[youenn fablet]

(In reply to comment #6)
> (In reply to comment #4)
> >
> > > > LayoutTests/imported/w3c/TestExpectationsW3C:1
> > > > +#imported/w3c/wpt [ Skip ]
> > > 
> > > This expectation doesn't work because it starts with #.
> > 
> > The idea is to uncomment this expectation when doing the first WPT import.
> > Removing # currently causes a warning because imported/w3c/wpt does not
> > exist yet.
> 
> For what purpose?

Document the scope/reason of this file between its initial check-in and the first import of WPT tests.

[Ryosuke Niwa]

I don't understand why we need test expectations for newly imported tests.  They should all pass because all expected results are generated from scratch.

[youenn fablet]

(In reply to comment #8)
> I don't understand why we need test expectations for newly imported tests. 
> They should all pass because all expected results are generated from scratch.
That is a good point, I should have stated the reason upfront.

The current WPT baseline generator imports all tests from a given WPT folder. In this folder, one will find:
- tests that crash, timeout or fail: these tests are marked as skip. Running these tests (especially the ones that timeout) does not provide much and the total run time becomes very long.
- tests that pass partially (some assertions passing, some not). These tests are put in the test expectations and marked as pass. This allows keeping all tests that need some fixes in the test expectations.

Importing only tests that pass or partially pass is not feasible as some ports have different features and different bugs. http://youennf.github.io/w3c-reports/12092014/w3c_conformance_results.html gives details about that. Issues would also arise when bumping version of thevtest suite.

One will also find additional test expectation lines:
imported/w3c/wpt is marked as skip so that, by default, all tests are skipped. This applies for instance to files in the 'tools' folder that should not be run as tests.

Each enabled WPT folder has a line marked explicitly as pass so that tests inside it are run.

At the end, the whole test suite could be regularly imported, each port deciding to enable/skip specific folders.

This process is described with some additional information at https://github.com/canonresearchfrance/webkit/wiki/Running-W3C-Tests

[youenn fablet]

Anyone to review the patch?

[Bem Jones-Bey]

(In reply to comment #10)
> Anyone to review the patch?

I'm going to have to defer to rniwa on this one to make sure you've satisfied all of his concerns. If he doesn't get back to you here soon, I'd suggest pinging him on IRC.

[Ryosuke Niwa]

I don't think we should add TestExpectations for subdirectories of LayoutTests/imported/  We already have too many TestExpectations files, and it's hard to reason about.  We shouldn't make that even worse.

[youenn fablet]

Latest patches that generate baselines directly work on the generic and platform-specific TestExpectations. Closing this bug as wontfix.