Bug 86628 – [EFL][DRT] Structure test_expectations.txt into meaningful sections

• Status: RESOLVED
• Resolution: FIXED
• Component: WebKit EFL

[Dominik Röttsches (drott)]

We come to realize that we're facing a couple of features and testcases that don't necessarily apply or make sense on EFL.
Let's reflect that in our test_expectations.txt - and while we're at it, introduce three more categories:

    a) Expected Failures
    b) Crashes
    c) Flaky tests
    d) Interim failures (failures that show up on the buildbot which are expected to be fixed within a short timeframe).

[Dominik Röttsches (drott)]

Created attachment 142266 [details]
Structuring test_expectations.txt

[Raphael Kubo da Costa (:rakuco)]

Awesome, I fully support this.

[Thiago Marcos P. Santos]

I like the idea. Some suggestions:

- If the feature is not relevant, i.e something that is Mac specific, we should move to a WONTFIX-kind of section and skip the test. Otherwise we are just making the test run slower.

- Make it clear that we should not add comments on top of tests that are associated to a bug (people should make the observations on the bug instead and not pollute the test_expectation.txt file)

[Dominik Röttsches (drott)]

(In reply to comment #3)
> I like the idea. Some suggestions:
> 
> - If the feature is not relevant, i.e something that is Mac specific, we should move to a WONTFIX-kind of section and skip the test. Otherwise we are just making the test run slower.

Wouldn't such a WONTFIX test fit in the first category?

> - Make it clear that we should not add comments on top of tests that are associated to a bug (people should make the observations on the bug instead and not pollute the test_expectation.txt file)

We can agree on that and delete the comments for issue where we have bugs. Opinions welcome. Personally, I don't mind to have a one-line comment to get an idea what the bug is about.

[Thiago Marcos P. Santos]

(In reply to comment #4)
> (In reply to comment #3)
> > I like the idea. Some suggestions:
> > 
> > - If the feature is not relevant, i.e something that is Mac specific, we should move to a WONTFIX-kind of section and skip the test. Otherwise we are just making the test run slower.
> 
> Wouldn't such a WONTFIX test fit in the first category?
> 

Hmmm... yes. But we should skip tests related to features that we don't want to activate. Most of them will timeout and consume precious time of developers testing before submitting patches.

> > - Make it clear that we should not add comments on top of tests that are associated to a bug (people should make the observations on the bug instead and not pollute the test_expectation.txt file)
> 
> We can agree on that and delete the comments for issue where we have bugs. Opinions welcome. Personally, I don't mind to have a one-line comment to get an idea what the bug is about.

I'm afraid of information scattering. We have some entries with comments but no bugs. If you have something useful to add, open a bug and set the entry pointing to the bug.

We should make this a rule.

[Chris Dumez]

Comment on attachment 142266 [details]
Structuring test_expectations.txt

LGTM. Exactly what we need.

[Sudarsana Nagineni (babu)]

I also like this idea. How about having a separate section or information in the section d) for the tests which requires new results?

[Dominik Röttsches (drott)]

Created attachment 143270 [details]
Structuring test_expectations.txt, updated

[Dominik Röttsches (drott)]

Created attachment 143271 [details]
Structuring test_expectations.txt, updated

[Gyuyoung Kim]

Comment on attachment 143271 [details]
Structuring test_expectations.txt, updated

Looks fine to me.

[WebKit Review Bot]

Comment on attachment 143271 [details]
Structuring test_expectations.txt, updated

Clearing flags on attachment: 143271

Committed r117955: <http://trac.webkit.org/changeset/117955>

[WebKit Review Bot]

All reviewed patches have been landed.  Closing bug.