Bug 217414 – REGRESSION (r267402): [ iOS Mojave+ Release ] imported/w3c/web-platform-tests/user-timing/measure-l3.any.html is a flaky failure

• Status: RESOLVED
• Resolution: FIXED
• Component: New Bugs

[Karl Rackler]

Description:
imported/w3c/web-platform-tests/user-timing/measure-l3.any.html

The first failure that I see this on the dashboard was 07/29/20 at r265034 for iOS 13, then 08/25/20 began to be consistent passing for iOS and macOS, with a few flaky failures on iOS.  On 09/22/20 at r267403 macOS and then iOS began to flaky fail together, but this commit does not appear to be relevant to what would have caused this.

I can reproduce this on r267403, but unable to reproduce on r267401 or earlier.
This test was modified in the layout test import in r267402, so that probably introduced the flakiness.

REPRODUCTION STEPS
Command: 
run-webkit-tests --exit-after-n-failures 1 --exit-after-n-crashes-or-timeouts 1 --iterations 50 imported/w3c/web-platform-tests/user-timing/measure-l3.any.html

Result: 
Regressions: Unexpected text-only failures (1)
  imported/w3c/web-platform-tests/user-timing/measure-l3.any.html [ Failure ]

History:
https://results.webkit.org/?suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Fuser-timing%2Fmeasure-l3.any.html&platform=ios&platform=mac&limit=50000

Diff Log:
Catalina wk2
--- /Volumes/Data/slave/catalina-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/user-timing/measure-l3.any-expected.txt
+++ /Volumes/Data/slave/catalina-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/user-timing/measure-l3.any-actual.txt
@@ -1,5 +1,5 @@
 
 PASS When the end mark is given and the start is unprovided, the end time of the measure entry should be the end mark's time, the start time should be 0.
-PASS When the start mark is given and the end is unprovided, the start time of the measure entry should be the start mark's time, the end should be now.
+FAIL When the start mark is given and the end is unprovided, the start time of the measure entry should be the start mark's time, the end should be now. assert_greater_than_equal: expected a number greater than or equal to 9.000000000000002 but got 9
 PASS When start and end mark are both given, the start time and end time of the measure entry should be the the marks' time, repectively

Mojave wk2
--- /Volumes/Data/slave/mojave-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/user-timing/measure-l3.any-expected.txt
+++ /Volumes/Data/slave/mojave-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/user-timing/measure-l3.any-actual.txt
@@ -1,5 +1,5 @@
 
 PASS When the end mark is given and the start is unprovided, the end time of the measure entry should be the end mark's time, the start time should be 0.
-PASS When the start mark is given and the end is unprovided, the start time of the measure entry should be the start mark's time, the end should be now.
+FAIL When the start mark is given and the end is unprovided, the start time of the measure entry should be the start mark's time, the end should be now. assert_greater_than_equal: expected a number greater than or equal to 9.000000000000002 but got 9
 PASS When start and end mark are both given, the start time and end time of the measure entry should be the the marks' time, repectively
 
iOS wk2
--- /Volumes/Data/slave/ios-simulator-14-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/user-timing/measure-l3.any-expected.txt
+++ /Volumes/Data/slave/ios-simulator-14-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/user-timing/measure-l3.any-actual.txt
@@ -1,5 +1,5 @@
 
 PASS When the end mark is given and the start is unprovided, the end time of the measure entry should be the end mark's time, the start time should be 0.
-PASS When the start mark is given and the end is unprovided, the start time of the measure entry should be the start mark's time, the end should be now.
+FAIL When the start mark is given and the end is unprovided, the start time of the measure entry should be the start mark's time, the end should be now. assert_greater_than_equal: expected a number greater than or equal to 51.00000000000001 but got 51
 PASS When start and end mark are both given, the start time and end time of the measure entry should be the the marks' time, repectively

[Radar WebKit Bug Importer]

<rdar://problem/70024016>

[Karl Rackler]

I have marked this test as failing while this issue is investigated.
https://trac.webkit.org/changeset/268093/webkit

[Diego Pino]

Created attachment 412095 [details]
Patch

[EWS Watchlist]

This patch modifies the imported WPT tests. Please ensure that any changes on the tests (not coming from a WPT import) are exported to WPT. Please see https://trac.webkit.org/wiki/WPTExportProcess

[Diego Pino]

Link to same change in WPT repo: https://github.com/web-platform-tests/wpt/pull/26192

[Diego Pino]

Created attachment 412129 [details]
Patch

[Alex Christensen]

Comment on attachment 412129 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=412129&action=review

> LayoutTests/imported/w3c/ChangeLog:8
> +        Sometimes lhs of comparison is higher by an infinitisimal margin.

infinitisimal isn't quite correct but whatevs.

[EWS]

Committed r268928: <https://trac.webkit.org/changeset/268928>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 412129 [details].

[Alex Christensen]

(In reply to Diego Pino from comment #5)
> Link to same change in WPT repo:
> https://github.com/web-platform-tests/wpt/pull/26192

Diego, would you mind including https://bugs.webkit.org/show_bug.cgi?id=217118 in your PR?

[Diego Pino]

(In reply to Alex Christensen from comment #9)
> (In reply to Diego Pino from comment #5)
> > Link to same change in WPT repo:
> > https://github.com/web-platform-tests/wpt/pull/26192
> 
> Diego, would you mind including
> https://bugs.webkit.org/show_bug.cgi?id=217118 in your PR?

Thank you Alex, I added your changes to the PR https://github.com/web-platform-tests/wpt/pull/26192