Bug 139123 – REGRESSION: mozilla-tests.yaml/ecma/Date/15.9.5.8.js fails intermittently

• Component: JavaScriptCore
• Status: NEW

[Csaba Osztrogonác]

It fails everywhere in this time interval:
- Apple Yosemite WK1: https://build.webkit.org/builders/Apple%20Yosemite%20Release%20WK1%20%28Tests%29/builds/985
- Apple Yosemite WK2: https://build.webkit.org/builders/Apple%20Yosemite%20Release%20WK2%20%28Tests%29/builds/982
- Apple Mavericks WK1: https://build.webkit.org/builders/Apple%20Mavericks%20Release%20WK1%20%28Tests%29/builds/10295
- Apple Mavericks WK2: https://build.webkit.org/builders/Apple%20Mavericks%20Release%20WK2%20%28Tests%29/builds/9718
- EFL: https://build.webkit.org/builders/EFL%20Linux%2064-bit%20Release%20WK2/builds/17969
- GTK: https://build.webkit.org/builders/GTK%20Linux%2064-bit%20Release%20%28Tests%29/builds/4517

[Csaba Osztrogonác]

mozilla-tests.yaml/ecma/Date/15.9.5.8.js.mozilla: (new Date(1446364102494)).getMonth() = 10 FAILED! expected: 9
mozilla-tests.yaml/ecma/Date/15.9.5.8.js.mozilla: (new Date(1446364102495)).getMonth() = 10 FAILED! expected: 9
mozilla-tests.yaml/ecma/Date/15.9.5.8.js.mozilla: (new Date(1446364102493)).getMonth() = 10 FAILED! expected: 9

[Csaba Osztrogonác]

maybe it is related to bug138303 somehow, but it is unlikely.

[Alexey Proskuryakov]

The tests are still failing today

[Csaba Osztrogonác]

strange, it must be a flakiness:

Nov 30 23:46 - fail - https://build.webkit.org/builders/Apple%20Mavericks%20Release%20WK1%20%28Tests%29/builds/10295
Dec 01 05:26 - pass - https://build.webkit.org/builders/Apple%20Mavericks%20Release%20WK1%20%28Tests%29/builds/10296
Dec 01 06:59 - fail - https://build.webkit.org/builders/Apple%20Mavericks%20Release%20WK1%20%28Tests%29/builds/10297
Dec 01 10:40 - pass - https://build.webkit.org/builders/Apple%20Mavericks%20Release%20WK1%20%28Tests%29/builds/10298

[Csaba Osztrogonác]

and again:
- https://build.webkit.org/builders/Apple%20Mavericks%20Release%20WK2%20%28Tests%29/builds/11978
- https://build.webkit.org/builders/Apple%20Mavericks%20Release%20WK1%20%28Tests%29/builds/12570
- https://build.webkit.org/builders/EFL%20Linux%2064-bit%20Release%20WK2/builds/20120
- https://build.webkit.org/builders/Apple%20Yosemite%20Release%20WK1%20%28Tests%29/builds/3241
- https://build.webkit.org/builders/Apple%20Yosemite%20Release%20WK2%20%28Tests%29/builds/3228

[Csaba Osztrogonác]

It seems this test fails only on the first day of months between 22:00-24:00.

[Alexey Proskuryakov]

The tests just failed again on all bots around 8am on March 2nd.

[Myles C. Maxfield]

Perhaps related to daylight savings?

[Filip Pizlo]

(In reply to comment #8)
> Perhaps related to daylight savings?

Yes :-)

[Myles C. Maxfield]

Looking into this now.

[Myles C. Maxfield]

This test uses the current time, then performs date math on it in JS code, and compares to the same date math being performed by the platform. However, this date math is sensitive to daylight savings code. I propose that we perform the tests on hardcoded dates and not on "now"

[Filip Pizlo]

(In reply to comment #11)
> This test uses the current time, then performs date math on it in JS code,
> and compares to the same date math being performed by the platform. However,
> this date math is sensitive to daylight savings code. I propose that we
> perform the tests on hardcoded dates and not on "now"

We could do that, but before we do, you should check what other coverage we have on the behavior of Date.now().

Also, have you checked if these tests overlap the more modern tests in sputnik and ietestcenter, or even Test262?  Fixing these ancient Mozilla tests if there is a modern test with similar coverage is probably not worthwhile.

[Alexey Proskuryakov]

*** Bug 142180 has been marked as a duplicate of this bug. ***

[Csaba Osztrogonác]

similar bug: bug138303

[Brent Fulgham]

The daylight-savings sensitivity is actually a useful thing to test. After all, real world code needs to deal with this stuff! However, I do think it would be better to have specific Daylight Savings Time tests so it would be very clear when these start failing that it is related to the time of year.

What makes this more difficult is that the DST logic can change from year-to-year (based on law).

But I agree that all of our stock date tests should use specific dates and times. To exercise 'now' we should probably place this in its own test with DST logic or special code to account for DST.

[Myles C. Maxfield]

The consensus that Mark Lam and I came up with is that we should test all cases with specific test cases, but that we should not test "now." That way, we maintain our coverage without having possible future flakes if governments change DST behavior.

[Mark Lam]

(In reply to comment #16)
> The consensus that Mark Lam and I came up with is that we should test all
> cases with specific test cases, but that we should not test "now." That way,
> we maintain our coverage without having possible future flakes if
> governments change DST behavior.

To clarify, I think Brent’s suggestion is a good one i.e. to put the DST test cases in a DST specific test.  That makes it easier to understand at a glance what could have gone wrong.  Brent’s suggestion is not in conflict with the approach Myles and I discussed.

[Filip Pizlo]

I'm opposed to removing these tests, as they are some of the more complex pieces of code that use Date and we should at least run them for crash test coverage.  It shouldn't be hard to arrange for them to be run just for seeing that they don't crash.

I haven't seen any comments regarding overlap with sputnik.  Sputnik is a comprehensive and modern JS test suite that we run in LayoutTests.  What we do about these tests depends on whether or not sputnik has Date.now tests.  Have you looked at this, and what did you find?

[Aakash Jain]

This test flaked in https://ews-build.webkit.org/#/builders/26/builds/652 It failed in jscore-test, but passed on retry (in immediately next jscore-test-rerun step).

mozilla-tests.yaml/ecma/Date/15.9.5.8.js.mozilla-dfg-eager-no-cjit-validate-phases: (new Date(1604216191497)).getMonth() = 10 FAILED! expected: 9

mozilla-tests.yaml/ecma/Date/15.9.5.8.js.mozilla-no-ftl: (new Date(1604216191382)).getMonth() = 10 FAILED! expected: 9