We come to realize that we're facing a couple of features and testcases that don't necessarily apply or make sense on EFL. Let's reflect that in our test_expectations.txt - and while we're at it, introduce three more categories: a) Expected Failures b) Crashes c) Flaky tests d) Interim failures (failures that show up on the buildbot which are expected to be fixed within a short timeframe).
Created attachment 142266 [details] Structuring test_expectations.txt
Awesome, I fully support this.
I like the idea. Some suggestions: - If the feature is not relevant, i.e something that is Mac specific, we should move to a WONTFIX-kind of section and skip the test. Otherwise we are just making the test run slower. - Make it clear that we should not add comments on top of tests that are associated to a bug (people should make the observations on the bug instead and not pollute the test_expectation.txt file)
(In reply to comment #3) > I like the idea. Some suggestions: > > - If the feature is not relevant, i.e something that is Mac specific, we should move to a WONTFIX-kind of section and skip the test. Otherwise we are just making the test run slower. Wouldn't such a WONTFIX test fit in the first category? > - Make it clear that we should not add comments on top of tests that are associated to a bug (people should make the observations on the bug instead and not pollute the test_expectation.txt file) We can agree on that and delete the comments for issue where we have bugs. Opinions welcome. Personally, I don't mind to have a one-line comment to get an idea what the bug is about.
(In reply to comment #4) > (In reply to comment #3) > > I like the idea. Some suggestions: > > > > - If the feature is not relevant, i.e something that is Mac specific, we should move to a WONTFIX-kind of section and skip the test. Otherwise we are just making the test run slower. > > Wouldn't such a WONTFIX test fit in the first category? > Hmmm... yes. But we should skip tests related to features that we don't want to activate. Most of them will timeout and consume precious time of developers testing before submitting patches. > > - Make it clear that we should not add comments on top of tests that are associated to a bug (people should make the observations on the bug instead and not pollute the test_expectation.txt file) > > We can agree on that and delete the comments for issue where we have bugs. Opinions welcome. Personally, I don't mind to have a one-line comment to get an idea what the bug is about. I'm afraid of information scattering. We have some entries with comments but no bugs. If you have something useful to add, open a bug and set the entry pointing to the bug. We should make this a rule.
Comment on attachment 142266 [details] Structuring test_expectations.txt LGTM. Exactly what we need.
I also like this idea. How about having a separate section or information in the section d) for the tests which requires new results?
Created attachment 143270 [details] Structuring test_expectations.txt, updated
Created attachment 143271 [details] Structuring test_expectations.txt, updated
Comment on attachment 143271 [details] Structuring test_expectations.txt, updated Looks fine to me.
Comment on attachment 143271 [details] Structuring test_expectations.txt, updated Clearing flags on attachment: 143271 Committed r117955: <http://trac.webkit.org/changeset/117955>
All reviewed patches have been landed. Closing bug.