Bug 94815 – Support ignore-metrics baselines

• Component: Tools / Tests
• Status: RESOLVED WONTFIX

[Allan Sandfeld Jensen]

It would be valuable to be able to generate baselines that does not depend on specific layout metrics, for tests where layout is not important, such as CSS selectors or HTML parsing.

It is possible to create new metric-less baseline with by generating the baseline using NRWT with --ignore-metrics. Unfortunately these baseline will only PASS if NRWT is always run with --ignore-metrics, and will always FAIL if it is not.

I propose to detect that a baseline has been generate with --ignore-metrics, and if it was automatically strip metrics from the output result before comparing output and expected result. 

Note this will only work NRWT, since as far as I know only NRWT support ignore-metrics. On the other hand stripping is only a small set of regular expressions, and shouldn't be too hard to import to ORWT if necessary.

[Allan Sandfeld Jensen]

The reason this is valuable is of course that more baselines can be shared between ports.

[Allan Sandfeld Jensen]

(In reply to comment #0)
> Note this will only work NRWT, since as far as I know only NRWT support ignore-metrics. On the other hand stripping is only a small set of regular expressions, and shouldn't be too hard to import to ORWT if necessary.

Skip that. It is already there. Will post patch soon.

[Allan Sandfeld Jensen]

Created attachment 160376 [details]
Patch

[Allan Sandfeld Jensen]

Created attachment 160379 [details]
LayoutTests Patch

Cleanup of fast/selectors test using the new features. Split off from the first patch to make the feature patch smaller and easier to review.

[Allan Sandfeld Jensen]

Comment on attachment 160379 [details]
LayoutTests Patch

No review yet, the bots will choke without the first patch applied.

[Allan Sandfeld Jensen]

Created attachment 160383 [details]
LayoutTests Patch

Now with new baselines.

[Balazs Kelemen]

I think this is an excellent idea, others?

[Csaba Osztrogonác]

The idea is good to make tests less metric dependent where layout 
isn't important at all and share expected files between ports.

But I think it would be better to refactor this kind of tests to dumpAsText() tests.

Using --ignore-metrics is risky a little bit, because we still depend
on the layout, we only hide some minor pixel differences. Tests will
still fail with --ignore-metrics if different platforms use different
font size/type and the line breaks become different.

And it is big change in test infrastructure which needs wider consensus
in the community. Could you send a mail about this proposel to webkit-dev
with referencing to this bug?

[Andreas Kling]

Comment on attachment 160376 [details]
Patch

Clearing review flag on patches from before 2014. If this patch is still relevant, please reset the r? flag.

[Allan Sandfeld Jensen]

Was decided against on webkit-dev.