Bug 160707 - [macOS Sierra] Fix flaky test: media/controls/picture-in-picture.html
Summary: [macOS Sierra] Fix flaky test: media/controls/picture-in-picture.html
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Ada Chan
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2016-08-09 13:41 PDT by Ada Chan
Modified: 2016-08-26 13:20 PDT (History)
1 user (show)

See Also:


Attachments
Patch (3.21 KB, patch)
2016-08-09 13:52 PDT, Ada Chan
eric.carlson: review+
Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Ada Chan 2016-08-09 13:41:59 PDT
media/controls/picture-in-picture.html is flaky on macOS Sierra.

@@ -21,13 +21,13 @@
 Test for the pip placeholder visibility in pip mode
 
 PASS: Should be in pip mode
-PASS: Inline placeholder should be visible at this point
+FAIL: Inline placeholder should be visible at this point Expected to not contain "hidden". Actual: "hidden picture-in-picture"
 
 Test for the pip placeholder visibility in pip mode without a 'controls' attribute
 
 PASS: Should still be in pip mode
 PASS: No controls attribute
-PASS: Inline placeholder should still be visible
+FAIL: Inline placeholder should still be visible Expected to not contain "hidden". Actual: "hidden picture-in-picture"
 
 Testing finished.

Comment 1 Ada Chan 2016-08-09 13:42:21 PDT
<rdar://problem/27574303>
Comment 2 Ada Chan 2016-08-09 13:52:47 PDT
Created attachment 285672 [details]
Patch
Comment 3 Eric Carlson 2016-08-09 14:34:22 PDT
Comment on attachment 285672 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=285672&action=review

> LayoutTests/media/controls/picture-in-picture.html:75
> +                if (stateForPlaceholder.className.indexOf("hidden") >= 0) {
> +                    // Use 33 to match PlaceholderPollingDelay in mediaControlsApple.js.
> +                    setTimeout(pollPIPPlaceholderVisibilityChange, 33);
> +                    return;
> +                }

Nit: in case something goes wrong I think it would be better to fail and log an error message after some maximum number of retrys, rather than trying until the test times out.
Comment 4 Ada Chan 2016-08-09 14:54:29 PDT
(In reply to comment #3)
> Comment on attachment 285672 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=285672&action=review
> 
> > LayoutTests/media/controls/picture-in-picture.html:75
> > +                if (stateForPlaceholder.className.indexOf("hidden") >= 0) {
> > +                    // Use 33 to match PlaceholderPollingDelay in mediaControlsApple.js.
> > +                    setTimeout(pollPIPPlaceholderVisibilityChange, 33);
> > +                    return;
> > +                }
> 
> Nit: in case something goes wrong I think it would be better to fail and log
> an error message after some maximum number of retrys, rather than trying
> until the test times out.

OK, I'll set a maximum number retrys to 10.

Thanks for the review!
Comment 5 Ada Chan 2016-08-09 14:57:11 PDT
Committed:
https://trac.webkit.org/changeset/204304
Comment 6 Alexey Proskuryakov 2016-08-09 20:44:05 PDT
Comment on attachment 285672 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=285672&action=review

>>> LayoutTests/media/controls/picture-in-picture.html:75
>>> +                }
>> 
>> Nit: in case something goes wrong I think it would be better to fail and log an error message after some maximum number of retrys, rather than trying until the test times out.
> 
> OK, I'll set a maximum number retrys to 10.
> 
> Thanks for the review!

10 retries is just 330 ms. I suggest waiting ~10 seconds before failing.
Comment 7 Ada Chan 2016-08-26 13:20:10 PDT
(In reply to comment #6)
> Comment on attachment 285672 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=285672&action=review
> 
> >>> LayoutTests/media/controls/picture-in-picture.html:75
> >>> +                }
> >> 
> >> Nit: in case something goes wrong I think it would be better to fail and log an error message after some maximum number of retrys, rather than trying until the test times out.
> > 
> > OK, I'll set a maximum number retrys to 10.
> > 
> > Thanks for the review!
> 
> 10 retries is just 330 ms. I suggest waiting ~10 seconds before failing.

The placeholder is not guaranteed to be visible right when the presentation mode changes but it should be visible basically very soon after.  If it takes in the order of seconds to become visible, then that's a bug.