Bug 186638 – [GTK] Tests that are failing on the EWS test queue

• Status: RESOLVED
• Resolution: FIXED
• Component: WebKitGTK

[Carlos Alberto Lopez Perez]

On r232844 <https://trac.webkit.org/r232844> we enabled tests on the GTK EWS test queue.
Since that this are some of the tests failing on the queue https://webkit-queues.webkit.org/queue-status/gtk-wk2-ews

printing/print-close-crash.html [ Timeout ]
fast/media/print-restores-previous-mediatype.html [ Timeout ]
fast/hidpi/broken-image-icon-very-hidpi.html [ Crash ]
editing/execCommand/print.html [ Timeout ]
animations/play-state-paused.html [ Failure ]
imported/w3c/web-platform-tests/streams/piping/error-propagation-forward.html [ Failure Pass ]
http/tests/misc/cached-scripts.html [ Crash Pass ]
imported/w3c/web-platform-tests/FileAPI/FileReader/workers.html [ Crash Pass ]
imported/mozilla/svg/dynamic-stroke-opacity-01.svg [ Crash Pass ]
imported/mozilla/css-transitions/test_animation-finished.html [ Timeout Pass ]
imported/blink/fast/text/international/complex-text-space-in-second-font.html [ Crash Pass ]
fast/mediastream/change-tracks-media-stream-being-played.html [ Crash Pass ]
compositing/repaint/repaint-on-layer-grouping-change.html [ Failure  Pass ]

[Carlos Alberto Lopez Perez]

Expectations updated on https://trac.webkit.org/changeset/232861

[Carlos Alberto Lopez Perez]

Some of this failures may have been caused by some revision on the range [232841-232860] that is the interval between almost zero-failures (when testing on the EWS was enabled) and the above gardening commit

[Carlos Alberto Lopez Perez]

More expectations updated on https://trac.webkit.org/changeset/232865

[Michael Catanzaro]

(In reply to Carlos Alberto Lopez Perez from comment #1)
> Expectations updated on https://trac.webkit.org/changeset/232861

You added them to the part of the file where we keep the "expected" expected failures (desirable failures or tests for unimplemented features)... they should be moved.

[Michael Catanzaro]

I noticed a huge number of issues today that were caused by the new expectations added here.

I've audited all the crash expectations added here. There are no recent recorded crashes on the GTK release bot for most of the tests with added crash expectations. For two of the tests, there are recorded crashes, but I investigated and determined they were both random events caused by the memory pressure handler. In my judgment, none of these should have crash expectations. If they are producing a real crash backtrace on the EWS, we should file separate bug reports for those with a backtrace.

The only new crash expectation I am not removing are these two:

fast/mediastream/change-tracks-media-stream-being-played.html
media/video-currentTime-delay.html

These two I will split out into new bugs.

I also audited all three of the new non-flaky timeouts. In each case, the tests are not timing out on the release bot, and in fact have never timed out recently. In these cases, since we do know they've recently timed out on EWS and can't blame the memory pressure monitor, I'll add Pass expectations rather than removing the new expectations.

In the case of the non-flaky failure added for animations/play-state-paused.html, the test does have one recent failure, but it almost always passes. I'll add a  Pass expectation.

(In reply to Michael Catanzaro from comment #4)
> You added them to the part of the file where we keep the "expected" expected
> failures (desirable failures or tests for unimplemented features)... they
> should be moved.

I've move them.

[Carlos Alberto Lopez Perez]

And another attempt at making the EWS happy landed in r232889: <https://trac.webkit.org/changeset/232889>

[Michael Catanzaro]

I feel like we're taking two steps forward, one step back. :)

I've landed another follow-up in https://trac.webkit.org/changeset/232895, partially reverting r232889. I want to make sure these tests are really crashing (i.e. have real backtraces and associated bug reports, and aren't just victims of the memory pressure limit you removed earlier today).

I'll watch the EWS later tonight to see if it reports any more crashes from these tests, but note the EWS seems to be broken now (not running any tests).

[Carlos Alberto Lopez Perez]

(In reply to Michael Catanzaro from comment #7)
> I feel like we're taking two steps forward, one step back. :)
> 

And I feel like unless you are willing to compromise on marking all the flaky tests we are seeing on the EWS (even without a crash log), we will have to revert this back ASAP.

> I've landed another follow-up in https://trac.webkit.org/changeset/r,
> partially reverting r232889. I want to make sure these tests are really
> crashing (i.e. have real backtraces and associated bug reports, and aren't
> just victims of the memory pressure limit you removed earlier today).
> 

I'm pretty sure they are not. I ensured to update the EWS manually to r232882 and restarted it manually before looking at the changes I commited into r232888.

> I'll watch the EWS later tonight to see if it reports any more crashes from
> these tests, but note the EWS seems to be broken now (not running any tests).

I see it broken all the time because its unable to pass tests, but I can see the output of each text run at https://webkit-queues.webkit.org/queue-status/gtk-wk2-ews

[Carlos Alberto Lopez Perez]

(In reply to Carlos Alberto Lopez Perez from comment #8)

> I see it broken all the time because its unable to pass tests, but I can see
> the output of each text run at
> https://webkit-queues.webkit.org/queue-status/gtk-wk2-ews

I stand corrected.. there was something wrong with the test step there.. trying to fix it now.

[Carlos Alberto Lopez Perez]

(In reply to Carlos Alberto Lopez Perez from comment #9)
> (In reply to Carlos Alberto Lopez Perez from comment #8)
> 
> > I see it broken all the time because its unable to pass tests, but I can see
> > the output of each text run at
> > https://webkit-queues.webkit.org/queue-status/gtk-wk2-ews
> 
> I stand corrected.. there was something wrong with the test step there..
> trying to fix it now.

Fixed.. but issue remains the same.. random crashes everywhere:

https://webkit-queues.webkit.org/results/8230086


Regressions: Unexpected crashes (3)
  fast/hidpi/broken-image-icon-very-hidpi.html [ Crash ]
  http/tests/misc/cached-scripts.html [ Crash ]
  legacy-animation-engine/fast/css/getFloatValueForUnit.html [ Crash ]

[Carlos Alberto Lopez Perez]

(In reply to Carlos Alberto Lopez Perez from comment #10)

> Regressions: Unexpected crashes (3)
>   fast/hidpi/broken-image-icon-very-hidpi.html [ Crash ]
>   http/tests/misc/cached-scripts.html [ Crash ]
>   legacy-animation-engine/fast/css/getFloatValueForUnit.html [ Crash ]

Crash log for fast/hidpi/broken-image-icon-very-hidpi.html obtained on the EWS bot: http://termbin.com/a4of

[Carlos Alberto Lopez Perez]

(In reply to Carlos Alberto Lopez Perez from comment #11)
> (In reply to Carlos Alberto Lopez Perez from comment #10)
> 
> > Regressions: Unexpected crashes (3)
> >   fast/hidpi/broken-image-icon-very-hidpi.html [ Crash ]
> >   http/tests/misc/cached-scripts.html [ Crash ]
> >   legacy-animation-engine/fast/css/getFloatValueForUnit.html [ Crash ]
> 
> Crash log for fast/hidpi/broken-image-icon-very-hidpi.html obtained on the
> EWS bot: http://termbin.com/a4of

stderr: g_bytes_get_size: assertion 'bytes != NULL' failed

[Carlos Alberto Lopez Perez]

(In reply to Carlos Alberto Lopez Perez from comment #12)
> (In reply to Carlos Alberto Lopez Perez from comment #11)
> > (In reply to Carlos Alberto Lopez Perez from comment #10)
> > 
> > > Regressions: Unexpected crashes (3)
> > >   fast/hidpi/broken-image-icon-very-hidpi.html [ Crash ]
> > >   http/tests/misc/cached-scripts.html [ Crash ]
> > >   legacy-animation-engine/fast/css/getFloatValueForUnit.html [ Crash ]
> > 
> > Crash log for fast/hidpi/broken-image-icon-very-hidpi.html obtained on the
> > EWS bot: http://termbin.com/a4of
> 
> stderr: g_bytes_get_size: assertion 'bytes != NULL' failed

Ok.. I have determined that this one is caused because the package gnome-icon-theme is not installed in the EWS machine (but it is on the bot).
We should add this to the install-dependencies script

[Michael Catanzaro]

(In reply to Carlos Alberto Lopez Perez from comment #11)
> (In reply to Carlos Alberto Lopez Perez from comment #10)
> 
> > Regressions: Unexpected crashes (3)
> >   fast/hidpi/broken-image-icon-very-hidpi.html [ Crash ]
> >   http/tests/misc/cached-scripts.html [ Crash ]
> >   legacy-animation-engine/fast/css/getFloatValueForUnit.html [ Crash ]

Yes, as expected. I was planning to watch the EWS this weekend to try to find backtraces for the crashing tests, but it was broken all weekend. Anyway, I intend to help with this. I don't think it's too much effort to take the backtrace and report a bug when EWS finds a crashing test.

> Crash log for fast/hidpi/broken-image-icon-very-hidpi.html obtained on the
> EWS bot: http://termbin.com/a4of

Weird... although you should probably install gnome-icon-theme, I think this is a WebKit bug worth reporting. Our current code assumes that we have missingImage icons in the GResource, which is apparently not true. See here:

static Ref<Image> loadMissingImageIconFromTheme(const char* name)
{
    // ...
    if (iconInfo) {
        // Is the icon installed?
        return WTFMove(icon);
    }

    return loadImageFromGResource(name);
}

That's not right, we shouldn't continue on to loadImageFromGResource if the image is not guaranteed to be in the resource.

[Michael Catanzaro]

Looking over the current state of the EWS queue, I think we are both wrong here:

(In reply to Carlos Alberto Lopez Perez from comment #8) 
> And I feel like unless you are willing to compromise on marking all the
> flaky tests we are seeing on the EWS (even without a crash log), we will
> have to revert this back ASAP.

We won't have to do this, because I don't see any recent bugs where our EWS went red, even though it is fixed and running tests again. At least, there are no such bugs currently displayed on the queue page.

But getting backtraces is much harder than I expected, because the EWS does not attach test results unless it goes red. It is impossible to get a backtrace from just the results log. We can still get a backtrace each time the EWS actually goes red, but it would be nice to have access to test results for every run. :/

[Adrian Perez]

(In reply to Michael Catanzaro from comment #14)
> (In reply to Carlos Alberto Lopez Perez from comment #11)
> > (In reply to Carlos Alberto Lopez Perez from comment #10)
> > 
> > > Regressions: Unexpected crashes (3)
> > >   fast/hidpi/broken-image-icon-very-hidpi.html [ Crash ]
> > >   http/tests/misc/cached-scripts.html [ Crash ]
> > >   legacy-animation-engine/fast/css/getFloatValueForUnit.html [ Crash ]
> 
> Yes, as expected. I was planning to watch the EWS this weekend to try to
> find backtraces for the crashing tests, but it was broken all weekend.
> Anyway, I intend to help with this. I don't think it's too much effort to
> take the backtrace and report a bug when EWS finds a crashing test.
> 
> > Crash log for fast/hidpi/broken-image-icon-very-hidpi.html obtained on the
> > EWS bot: http://termbin.com/a4of
> 
> Weird... although you should probably install gnome-icon-theme, I think this
> is a WebKit bug worth reporting. Our current code assumes that we have
> missingImage icons in the GResource, which is apparently not true. See here:
> 
> static Ref<Image> loadMissingImageIconFromTheme(const char* name)
> {
>     // ...
>     if (iconInfo) {
>         // Is the icon installed?
>         return WTFMove(icon);
>     }
> 
>     return loadImageFromGResource(name);
> }
> 
> That's not right, we shouldn't continue on to loadImageFromGResource if the
> image is not guaranteed to be in the resource.

I think that WebKitGTK+ should include within itself a copy of the
resources it uses which currently end up being loaded from the GNOME
theme, to avoid the runtime dependency on the “gnome-icon-theme”
package. Using icons which are part of GTK+ itself would still be
fine, of course. Should we open a new bug for this?

[Carlos Alberto Lopez Perez]

(In reply to Adrian Perez from comment #16)
> (In reply to Michael Catanzaro from comment #14)
> > (In reply to Carlos Alberto Lopez Perez from comment #11)
> > > (In reply to Carlos Alberto Lopez Perez from comment #10)
> > > 
> > > > Regressions: Unexpected crashes (3)
> > > >   fast/hidpi/broken-image-icon-very-hidpi.html [ Crash ]
> > > >   http/tests/misc/cached-scripts.html [ Crash ]
> > > >   legacy-animation-engine/fast/css/getFloatValueForUnit.html [ Crash ]
> > 
> > Yes, as expected. I was planning to watch the EWS this weekend to try to
> > find backtraces for the crashing tests, but it was broken all weekend.
> > Anyway, I intend to help with this. I don't think it's too much effort to
> > take the backtrace and report a bug when EWS finds a crashing test.
> > 
> > > Crash log for fast/hidpi/broken-image-icon-very-hidpi.html obtained on the
> > > EWS bot: http://termbin.com/a4of
> > 
> > Weird... although you should probably install gnome-icon-theme, I think this
> > is a WebKit bug worth reporting. Our current code assumes that we have
> > missingImage icons in the GResource, which is apparently not true. See here:
> > 
> > static Ref<Image> loadMissingImageIconFromTheme(const char* name)
> > {
> >     // ...
> >     if (iconInfo) {
> >         // Is the icon installed?
> >         return WTFMove(icon);
> >     }
> > 
> >     return loadImageFromGResource(name);
> > }
> > 
> > That's not right, we shouldn't continue on to loadImageFromGResource if the
> > image is not guaranteed to be in the resource.
> 
> I think that WebKitGTK+ should include within itself a copy of the
> resources it uses which currently end up being loaded from the GNOME
> theme, to avoid the runtime dependency on the “gnome-icon-theme”
> package. Using icons which are part of GTK+ itself would still be
> fine, of course. Should we open a new bug for this?

Yes, please.

[Adrian Perez]

(In reply to Carlos Alberto Lopez Perez from comment #17)
> (In reply to Adrian Perez from comment #16)
>
> > [...]
> > 
> > I think that WebKitGTK+ should include within itself a copy of the
> > resources it uses which currently end up being loaded from the GNOME
> > theme, to avoid the runtime dependency on the “gnome-icon-theme”
> > package. Using icons which are part of GTK+ itself would still be
> > fine, of course. Should we open a new bug for this?
> 
> Yes, please.

Done, that's bug #186767

[Michael Catanzaro]

(In reply to Carlos Alberto Lopez Perez from comment #10) 
> Regressions: Unexpected crashes (3)
>   fast/hidpi/broken-image-icon-very-hidpi.html [ Crash ]
>   http/tests/misc/cached-scripts.html [ Crash ]
>   legacy-animation-engine/fast/css/getFloatValueForUnit.html [ Crash ]

I've pushed expectations for the first two (bug #176767 and bug #186778). Let's keep watching for legacy-animation-engine/fast/css/getFloatValueForUnit.html and any others.

[Michael Catanzaro]

I've noted the following EWS failures today:

https://bugs.webkit.org/show_bug.cgi?id=186765#c11
https://bugs.webkit.org/show_bug.cgi?id=184911#c8
https://bugs.webkit.org/show_bug.cgi?id=186291#c51
https://bugs.webkit.org/show_bug.cgi?id=186715#c6
https://bugs.webkit.org/show_bug.cgi?id=186608#c2
https://bugs.webkit.org/show_bug.cgi?id=186608#c2

It's only three distinct tests, because four of them are the same flaky tests. We should address these.

[Diego Pino]

I've cleaned up many of the failure entries filed under this bug.

Many tests, although marked as flaky, having consistently passing at least in the last 4000 revisions. Other tests are consistently crashing or failing. As for the remaining flaky tests, I updated their actual state based on the test bot results of the last 4000 revisions.

I think at some point the remaining failures still filed under this bug should be filed under a new bug(s), since this bug refers to a situation which is no longer true.

[Diego Pino]

Reusing this bug to keep track of the tests failing on EWS but not in the post-commit bot:

Unexpected flakiness: text-only failures (4)
  editing/pasteboard/paste-line-endings-008.html [ Failure ImageOnlyFailure Pass ]
  fast/preloader/document-write.html [ Failure Pass ]
  http/wpt/service-workers/cors-preflight-star.any-serviceworker.html [ Failure Pass ]
  inspector/debugger/breakpoints/resolved-dump-all-pause-locations.html [ Failure Pass ]

Unexpected flakiness: timeouts (1)
  scrollbars/scrollbar-selectors.html [ Pass Timeout Missing ]

Regressions: Unexpected text-only failures (19)
  fast/canvas/webgl/webgl-compressed-texture-astc.html [ Failure ]
  fast/canvas/webgl/webgl-depth-texture.html [ Failure ]
  fast/loader/local-CSS-from-local.html [ Failure ]
  fast/loader/local-JavaScript-from-local.html [ Failure ]
  fast/loader/local-image-from-local.html [ Failure ]
  fast/xmlhttprequest/xmlhttprequest-nonexistent-file.html [ Failure ]
  http/tests/appcache/history-test.html [ Failure ]
  http/tests/security/local-user-CSS-from-remote.html [ Failure ]
  imported/w3c/web-platform-tests/IndexedDB/interleaved-cursors-small.html [ Failure ]
  imported/w3c/web-platform-tests/IndexedDB/large-requests-abort.html [ Failure ]
  imported/w3c/web-platform-tests/IndexedDB/request-event-ordering.html [ Failure ]
  imported/w3c/web-platform-tests/svg/import/animate-elem-77-t-manual.svg [ Failure ]
  imported/w3c/web-platform-tests/svg/import/text-tspan-01-b-manual.svg [ Failure ]
  storage/domstorage/sessionstorage/blocked-file-access.html [ Failure ]
  svg/W3C-SVG-1.1-SE/text-tspan-02-b.svg [ Failure ]
  svg/W3C-SVG-1.1-SE/types-dom-05-b.svg [ Failure ]
  svg/custom/glyph-transformation-with-hkern.svg [ Failure ]
  webgl/1.0.3/conformance/glsl/bugs/sampler-array-using-loop-index.html [ Failure ]
  webgl/1.0.3/conformance/textures/texture-size-limit.html [ Failure ]

Regressions: Unexpected image-only failures (1)
  compositing/video/video-border-radius-clipping.html [ ImageOnlyFailure ]

Regressions: Unexpected audio failures (1)
  webaudio/codec-tests/aac/vbr-128kbps-44khz.html [ Failure ]

[Diego Pino]

I recreated the EWS GTK-WK2 containers out of the GTK Release Tests container. The number of tests failing on the EWS GTK-WK2 cointainer is much smaller:

webrtc/release-after-getting-track.html [ Crash ]
imported/w3c/web-platform-tests/css/css-ui/text-overflow-022.html [ ImageOnlyFailure ]
fast/forms/select/popup-closes-on-blur.html [ Failure Pass ]

These tests should be marked as flaky.

As for the current failures filed under this bug in GTK TestExpectations, those are legit failures in the post-commit bot and should be filed under a different bug.

[Diego Pino]

> As for the current failures filed under this bug in GTK TestExpectations,
> those are legit failures in the post-commit bot and should be filed under a
> different bug.

All the tests marked as flaky under this bug were actually valid flaky failures in the post-commit bot.

For some of these tests there were already similar bugs created for other platforms, or the test was a flaky timeout in other platforms and marked as skipped. Other tests were fixed failures and actually they required a rebaseline.

Changes committed in:
  * https://trac.webkit.org/changeset/281130/webkit
  * https://trac.webkit.org/changeset/281131/webkit

I think this bug should be closed now since there are no more test failures filed under this bug. If we think it's still useful to have a meta-bug where to track failures happening in EWS GTK-WK2 bot but not in the post-commit bot, I'd suggest to open a new bug and mark this one as RESOLVED MOVED.

[Diego Pino]

There are no references to this bug in any TestExpectations. It's probable this bug was solved at some point but it wasn't marked as closed. I'm closing this bug now. If you think this bug report is still valid, please reopen it and add an entry to TestExpectations.