Bug 220922 – REGRESSION(r270074): Broke all GLib unit tests using a SoupServer

• Status: RESOLVED
• Resolution: FIXED
• Component: New Bugs

[WebKit Commit Bot]

https://trac.webkit.org/changeset/270074 broke the build:
Broke all GLib unit tests using a SoupServer

This is an automatic bug report generated by webkitbot. If this bug
report was created because of a flaky test, please file a bug for the flaky
test (if we don't already have one on file) and dup this bug against that bug
so that we can track how often these flaky tests fail.

[WebKit Commit Bot]

Created attachment 418288 [details]
REVERT of r270074

Any committer can land this patch automatically by marking it commit-queue+.  The commit-queue will build and test the patch before landing to ensure that the revert will be successful.  This process takes approximately 15 minutes.

If you would like to land the revert faster, you can use the following command:

  webkit-patch land-attachment ATTACHMENT_ID

where ATTACHMENT_ID is the ID of this attachment.

[EWS]

Committed r271794: <https://trac.webkit.org/changeset/271794>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 418288 [details].

[Radar WebKit Bug Importer]

<rdar://problem/73570236>

[Lauro Moura]

A bunch of GTK/WPE API failures after this revert. I'm gardening them under this bug for now.

Unexpected failures (10)
    /WebKit2Gtk/TestWebKitSettings
        /webkit/WebKitSettings/webkit-settings
        /webkit/WebKitSettings/javascript-markup
    /WebKit2Gtk/TestAuthentication
        /webkit/Authentication/authentication-empty-realm
        /webkit/Authentication/authentication-no-credential
        /webkit/Authentication/authentication-failure
        /webkit/Authentication/authentication-success
    /WebKit2Gtk/TestSSL
        /webkit/WebKitWebView/load-failed-with-tls-errors
    /WebKit2Gtk/TestBackForwardList
        /webkit/BackForwardList/navigation
    /WebKit2Gtk/TestWebKitWebView
        /webkit/WebKitWebView/title-change
    /WebKit2Gtk/TestLoaderClient
        /webkit/WebKitWebView/title

Unexpected timeouts (1)
    /WebKit2Gtk/TestWebKitFaviconDatabase
        /webkit/WebKitFaviconDatabase/get-favicon

[Michael Catanzaro]

Something is seriously wrong here. Carlos and I are both convinced that all these tests should have been guaranteed to fail *without* the revert, because there was no way the WebKitTestServer could have started. And yet the tests were passing. Now that the change is reverted to get back to the original known-good state, and the tests have all started to fail.

There must be something more going on that we don't understand.

[Lauro Moura]

(In reply to Michael Catanzaro from comment #5)
> Something is seriously wrong here. Carlos and I are both convinced that all
> these tests should have been guaranteed to fail *without* the revert,
> because there was no way the WebKitTestServer could have started. And yet
> the tests were passing. Now that the change is reverted to get back to the
> original known-good state, and the tests have all started to fail.
> 
> There must be something more going on that we don't understand.

Yeah. The tests weren't being run at all, it seems.

Test log from the revision *before* the revert: https://build.webkit.org/builders/GTK-Linux-64-bit-Release-Tests/builds/17973/steps/API-tests/logs/stdio

Test log from the revert: https://build.webkit.org/builders/GTK-Linux-64-bit-Release-Tests/builds/17974/steps/API-tests/logs/stdio

There are many tests (the failing ones among them) not appearing in the run before the revert.

And indeed, trying to run the title-change one, for example, before the revert, shows only the "1 of 1 passed" message without actually running it.

It looks like an issue with the glib test runner failing to report some kinds of failures/crashes. And due to this, there's a chance these tests weren't actually being tested all this time.

[Lauro Moura]

Gardened in r271803