media/controls-without-preload.html failing on Windows

• Component: Media
• Status: RESOLVED CONFIGURATION CHANGED

[Jessie Berlin]

It has been failing on both Windows 7 and XP:
http://build.webkit.org/results/Windows%207%20Release%20(Tests)/r73278%20(7105)/media/controls-without-preload-actual.txt

The failures have been happening since the test results were committed:
http://trac.webkit.org/changeset/73278

Which were separate from the tests being added:
http://trac.webkit.org/changeset/73257

[Jessie Berlin]

Committed the windows-specific results in r73288
http://trac.webkit.org/changeset/73288

[Philippe Normand]

Can r73278 please be reverted? The test expectations are platform-specific.

[Jessie Berlin]

Still failing on XP. I think the XP failures started with r78690:

http://build.webkit.org/old-results/Windows%20XP%20Debug%20(Tests)/r78690%20(25314)/results.html
http://build.webkit.org/old-results/Windows%20XP%20Debug%20(Tests)/r78689%20(25313)/results.html

I will commit the expected failing results for XP soon.

[Eric Carlson]

(In reply to comment #3)
> Still failing on XP. I think the XP failures started with r78690:
> 
> http://build.webkit.org/old-results/Windows%20XP%20Debug%20(Tests)/r78690%20(25314)/results.html
> http://build.webkit.org/old-results/Windows%20XP%20Debug%20(Tests)/r78689%20(25313)/results.html
> 
> I will commit the expected failing results for XP soon.

I think it would be better to add the test to the skipped list and file a bug instead of committing failing result.s

[Jessie Berlin]

Hrm, well I comitted the failing results in http://trac.webkit.org/changeset/79716 before your comment.

They don't seem to be flakey, just consistently failing with those results. Would it really be more useful to remove these expected results and just add it to the skip lists?

[Eric Carlson]

(In reply to comment #5)
> Hrm, well I comitted the failing results in http://trac.webkit.org/changeset/79716 before your comment.
> 
> They don't seem to be flakey, just consistently failing with those results. Would it really be more useful to remove these expected results and just add it to the skip lists?

Yes, I do think it is better to have a failing file in the skipped list rather than using incorrect results.

[Adam Roben (:aroben)]

(In reply to comment #4)
> (In reply to comment #3)
> > Still failing on XP. I think the XP failures started with r78690:
> > 
> > http://build.webkit.org/old-results/Windows%20XP%20Debug%20(Tests)/r78690%20(25314)/results.html
> > http://build.webkit.org/old-results/Windows%20XP%20Debug%20(Tests)/r78689%20(25313)/results.html
> > 
> > I will commit the expected failing results for XP soon.
> 
> I think it would be better to add the test to the skipped list and file a bug instead of committing failing result.s

A bug has been filed: this one.

Committing expected failure results has been our policy for a while now. We do this when a test is failing in a consistent way and is not crashing or timing out. The advantage of committing expected failure results is that we will notice if someone causes a change in the test's behavior (including fixing it!).

A number of people say that it's better to have the test in the Skipped file because then there's a record that it's failing. However, this bug records that the test is failing, and people don't seem to look through the Skipped file to see what tests are failing much in practice.

[Jessie Berlin]

<rdar://problem/9055504>

[Ahmad Saleem]

This expectation does not exist for win-cairo and only have for GTK and tracked with separate bug.

It might be for 'AppleWin' port, which is gone.

Marking this as 'RESOLVED CONFIGURATION CHANGED' since nothing to do here.