Bug 186132 – export-w3c-test-changes can not export reftests

• Status: NEW
• Component: Tools / Tests

[Frédéric Wang (:fredw)]

In WebKit, reftests are

NAME.html
NAME-expected.html

Moreover, the <link rel="match"> points to the non-existing NAME-ref.html file.

When export-w3c-test-changes runs the lint tool, it complains about

1) NAME-expected.html being a support file in the wrong directory.
2) NAME.html pointing to non-existent ref file.

The support script should rename NAME-expected.html to NAME-ref.html before exporting in order to fix these errors.

[Frédéric Wang (:fredw)]

Created attachment 341663 [details]
Patch

[youenn fablet]

The issue with ref tests is that one ref file in WPT might be used for several test files, so I am not sure we can always update correctly all ref files.
In the simple one test/one ref case, this should work so maybe we could add a warning or add a check for this multi case

[Frédéric Wang (:fredw)]

So you mean that when we have a single ref for multiple reftests, the import script will generate multiple -expected.html?

I guess it's a problem with the import / way WebKit handles reftest then. I saw a FIXME in the import script about that.

[Frédéric Wang (:fredw)]

I suppose for new reftests, it's safe to just rename -expected => -ref.

For existing reftests using a single ref for multiple files or when upstream names are not of the form *-ref.html or similar (e.g. ./css/css-fill-stroke/reference/paint-order-001-ref.tentative.html) I guess the patch will just fail to apply.

[Frédéric Wang (:fredw)]

Cancelling review since it seems we don't have a clear agreement on how we will handle the mismatch between WebKit / WPT formats...

[Philip Jägenstedt]

Is it Tools/Scripts/import-w3c-tests that renames references to -expected.html?

[Frédéric Wang (:fredw)]

(In reply to Philip Jägenstedt from comment #6)
> Is it Tools/Scripts/import-w3c-tests that renames references to
> -expected.html?

Yes it is.

[Philip Jägenstedt]

That'll definitely make it challenging to work effectively with reftests, and it looks like plenty of WebKit-specific failures in WPT are reftests.

Is there any possible fix here short of getting rid of the renaming, with all that would entail?

[Radar WebKit Bug Importer]

<rdar://problem/105269974>

[Antoine Quint]

Not sure the name of this Radar is completely correct. You can export reftests so long as you have both -expected.html and -ref.html files in your WebKit repository. It's far from ideal though.

[Tim Nguyen (:ntim)]

(In reply to Antoine Quint from comment #10)
> Not sure the name of this Radar is completely correct. You can export
> reftests so long as you have both -expected.html and -ref.html files in your
> WebKit repository. It's far from ideal though.

It wasn't possible before because we renamed the -ref.html file and deleted the original -ref.html.

Now we have duplicates, which allows exporting, but is far from ideal.