Bug 204469 – Add support for generating wpt's MANIFEST.json when running tests

• Component: Tools / Tests
• Status: ASSIGNED

[Simon Fraser (smfr)]

Add support for generating wpt's MANIFEST.json when importing tests

[Simon Fraser (smfr)]

Created attachment 384085 [details]
Patch

[youenn fablet]

Patch is big so difficult to provide feedback inline.
Some comments:

the new command line option does not seem to allow disabling generation.
Should be store_false and something like --no-manifest-generation?
Or, we could provide a way to only trigger manifest generation?

I would use the 'wpt' script from LayoutTests/imported/w3c instead of the checkout.
That will make it stable and we can make sure this version is passing our unit tests so that at some point, this would be done by bots.

We should add some unit tests like we probably should have for tests-options.json

Is there a way to only generate the manifest to the updated folders?
We would then do the merge like we are doing for tests-options.json.

Tests labelled as manual should be skipped or not imported.
Currently, we are not importing them based on a heuristic.
We can tighten that later on (remove manuals from MANIFEST.json or import manual tests but not run them). It seems that some of the labelled tests as manual might be ref tests ("css/css-text/hanging-punctuation/hanging-punctuation-first-001.xht").

[Simon Fraser (smfr)]

(In reply to youenn fablet from comment #2)
> Patch is big so difficult to provide feedback inline.

I can post a version without the MANIFEST.json

> Some comments:
> 
> the new command line option does not seem to allow disabling generation.

Good point.

> Should be store_false and something like --no-manifest-generation?
> Or, we could provide a way to only trigger manifest generation?

I tested that and it works.

> 
> I would use the 'wpt' script from LayoutTests/imported/w3c instead of the
> checkout.

Oh, I didn't know we'd imported LayoutTests/imported/w3c/web-platform-tests/tools 

> That will make it stable and we can make sure this version is passing our
> unit tests so that at some point, this would be done by bots.
> 
> We should add some unit tests like we probably should have for
> tests-options.json

Sure.

> Is there a way to only generate the manifest to the updated folders?
> We would then do the merge like we are doing for tests-options.json.

I can try merging if the wpt scripts makes this possible. I don't want to write custom marging code.
> Tests labelled as manual should be skipped or not imported.

I think it's OK for them to be in the MANIFEST though. I didn't touch the logic about which tests were imported.

> Currently, we are not importing them based on a heuristic.
> We can tighten that later on (remove manuals from MANIFEST.json or import
> manual tests but not run them). It seems that some of the labelled tests as
> manual might be ref tests
> ("css/css-text/hanging-punctuation/hanging-punctuation-first-001.xht").

Yes, we've definitely made that mistake. After we have a working MANIFEST.json we need to go back and re-import a bunch of stuff.

[Simon Fraser (smfr)]

(In reply to Simon Fraser (smfr) from comment #3)
> (In reply to youenn fablet from comment #2)
> > Patch is big so difficult to provide feedback inline.
> 
> I can post a version without the MANIFEST.json
> 
> > Some comments:
> > 
> > the new command line option does not seem to allow disabling generation.
> 
> Good point.
> 
> > Should be store_false and something like --no-manifest-generation?
> > Or, we could provide a way to only trigger manifest generation?
> 
> I tested that and it works.
> 
> > 
> > I would use the 'wpt' script from LayoutTests/imported/w3c instead of the
> > checkout.
> 
> Oh, I didn't know we'd imported
> LayoutTests/imported/w3c/web-platform-tests/tools 

I tried that version and it was much slower, and generated paths with an unwanted prefix. I'll have to re-import tools/ first I guess.

[Simon Fraser (smfr)]

This patch just needs testing again now that bug 204728 is fixed.

We also need some warnings from the style checker when someone adds a new WPT with a reference, saying that the manifest needs to be regenerated.

[Carlos Alberto Lopez Perez]

(In reply to Simon Fraser (smfr) from comment #5)
> This patch just needs testing again now that bug 204728 is fixed.
> 

It seems a few days ago there has been a major change in the manifest format used by WPT.

Generating the manifest its now much faster (around 150% faster) and the generated json its smaller as well (around 40% smaller).

See:
https://github.com/web-platform-tests/rfcs/blob/master/rfcs/manifest-path-trie.md
https://github.com/web-platform-tests/wpt/commit/31c0f5efba38b7d1d7f45ac449bcbc892e8771ce

So I think it makes sense to update the tools directory (again) and use this new format

[Sam Sneddon [:gsnedders]]

(In reply to Carlos Alberto Lopez Perez from comment #6)
> Generating the manifest its now much faster (around 150% faster) and the
> generated json its smaller as well (around 40% smaller).
> 
> See:
> https://github.com/web-platform-tests/rfcs/blob/master/rfcs/manifest-path-
> trie.md
> https://github.com/web-platform-tests/wpt/commit/
> 31c0f5efba38b7d1d7f45ac449bcbc892e8771ce
> 
> So I think it makes sense to update the tools directory (again) and use this
> new format

The manifest generation now builds with a fair degree of parallelism (essentially any parsing/etc now happens in parallel).

(In reply to Simon Fraser (smfr) from comment #4)
> I tried that version and it was much slower, and generated paths with an
> unwanted prefix. I'll have to re-import tools/ first I guess.

As has been the case for a while it will use git's cache to check whether the file has changed, which clearly this doesn't help when you're in a SVN checkout which you might be with the WebKit checkout.

For the paths you probably want --tests-root or --url-base or something.

(In reply to Simon Fraser (smfr) from comment #5)
> We also need some warnings from the style checker when someone adds a new
> WPT with a reference, saying that the manifest needs to be regenerated.

I guess the other question is whether the manifest is quick enough to run at test execution time nowadays… Making it substantially quicker is hard without (at least optionally) using some compiled code.

[Radar WebKit Bug Importer]

<rdar://problem/69332068>

[Sam Sneddon [:gsnedders]]

Current plan:

Run the manifest generation at test run time (this will become more important once we have the automated export working, but even now modifying tests and then using the manual exporter we should be getting this right).

It is _probably_ desirable to run it at import time as well to make the first test run quicker, but that may or may not be worthwhile depending how much we care about performance in that case. (As far as I'm aware, the bots don't clean between runs, so they'll only have to do an incremental update.)

For reference, a clean manifest build on my MacBook Pro (16-inch, 2019) 2.4 GHz 8-Core Intel Core i9 takes about 20s, and a no-op incremental build takes about 2s.