Bug 196905 - API test WKAttachmentTests.AddAttachmentToConnectedImageElement is a flaky failure on Mac Release builds
Summary: API test WKAttachmentTests.AddAttachmentToConnectedImageElement is a flaky fa...
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: New Bugs (show other bugs)
Version: WebKit Local Build
Hardware: Mac All
: P2 Normal
Assignee: Wenson Hsieh
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2019-04-14 08:37 PDT by Andy Estes
Modified: 2019-04-14 21:47 PDT (History)
8 users (show)

See Also:


Attachments
Patch (4.25 KB, patch)
2019-04-14 18:24 PDT, Wenson Hsieh
no flags Details | Formatted Diff | Diff
Add radar link to ChangeLog (4.27 KB, patch)
2019-04-14 18:27 PDT, Wenson Hsieh
no flags Details | Formatted Diff | Diff
Also reenable the test (4.46 KB, patch)
2019-04-14 19:00 PDT, Wenson Hsieh
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Andy Estes 2019-04-14 08:37:51 PDT
On macOS Mojave, in a Release build of r244241, the API test WKAttachmentTests.AddAttachmentToConnectedImageElement failed on 6 out of 10 runs of the WKAttachmentTests API test suite. It also sometimes fails on api-mac EWS bots in a way that makes it appear like a seemingly-unrelated patch regressed the test (e.g., bug #177484).

Steps to reproduce:
1. Create a Mac Release build of r244241.
2. run-api-tests WKAttachmentTests.

Expected results:
All 31 tests pass.

Actual results:
On 6 out of 10 runs, WKAttachmentTests.AddAttachmentToConnectedImageElement failed.

    TestWebKitAPI.WKAttachmentTests.AddAttachmentToConnectedImageElement
        _RegisterApplication(), FAILED TO establish the default connection to the WindowServer, _CGSDefaultConnection() is NULL.
        2019-04-14 08:23:20.508 TestWebKitAPI[59497:767481] Expected inserted attachments: (
        ) to match (
            "<_WKAttachment 0x7fbcc7d01340 id='2414efeb-bbf1-4650-83ce-a167b0a11484'>"
        ).
        
        /Volumes/Data/Projects/OpenSource/Tools/TestWebKitAPI/Tests/WebKitCocoa/WKAttachmentTests.mm:150
        Value of: insertedAttachmentsMatch
          Actual: false
        Expected: true
Comment 1 Radar WebKit Bug Importer 2019-04-14 09:34:26 PDT
<rdar://problem/49886096>
Comment 2 Aakash Jain 2019-04-14 15:47:37 PDT
This flaky test is slowing down EWS (since EWS has to re-run API tests few times to confirm that the failure is not because of the patch being tested).

e.g.:
https://ews-build.webkit.org/#/builders/3/builds/535
https://ews-build.webkit.org/#/builders/3/builds/530
https://ews-build.webkit.org/#/builders/3/builds/528
https://ews-build.webkit.org/#/builders/3/builds/503
https://ews-build.webkit.org/#/builders/3/builds/500
https://ews-build.webkit.org/#/builders/3/builds/467
https://ews-build.webkit.org/#/builders/3/builds/465
https://ews-build.webkit.org/#/builders/3/builds/460

https://ews-build.webkit.org/#/builders/3/builds/541
https://ews-build.webkit.org/#/builders/3/builds/527
https://ews-build.webkit.org/#/builders/3/builds/526
https://ews-build.webkit.org/#/builders/3/builds/518
https://ews-build.webkit.org/#/builders/3/builds/515
https://ews-build.webkit.org/#/builders/3/builds/509
https://ews-build.webkit.org/#/builders/3/builds/499
https://ews-build.webkit.org/#/builders/3/builds/497
https://ews-build.webkit.org/#/builders/3/builds/494
https://ews-build.webkit.org/#/builders/3/builds/493
https://ews-build.webkit.org/#/builders/3/builds/490
https://ews-build.webkit.org/#/builders/3/builds/480
https://ews-build.webkit.org/#/builders/3/builds/479
https://ews-build.webkit.org/#/builders/3/builds/477
https://ews-build.webkit.org/#/builders/3/builds/476
https://ews-build.webkit.org/#/builders/3/builds/474
https://ews-build.webkit.org/#/builders/3/builds/473
https://ews-build.webkit.org/#/builders/3/builds/470



In following builds, the test was so flaky that EWS incorrectly thought that the failure was introduced by the patch being tested (this test failed twice with the patch, and didn't fail on clean tree):
https://ews-build.webkit.org/#/builders/3/builds/542
https://ews-build.webkit.org/#/builders/3/builds/546
https://ews-build.webkit.org/#/builders/3/builds/512
https://ews-build.webkit.org/#/builders/3/builds/495
https://ews-build.webkit.org/#/builders/3/builds/492
https://ews-build.webkit.org/#/builders/3/builds/488
https://ews-build.webkit.org/#/builders/3/builds/484
https://ews-build.webkit.org/#/builders/3/builds/471
https://ews-build.webkit.org/#/builders/3/builds/468
https://ews-build.webkit.org/#/builders/3/builds/464
https://ews-build.webkit.org/#/builders/3/builds/454
https://ews-build.webkit.org/#/builders/3/builds/440
https://ews-build.webkit.org/#/builders/3/builds/439
https://ews-build.webkit.org/#/builders/3/builds/429
https://ews-build.webkit.org/#/builders/3/builds/426
https://ews-build.webkit.org/#/builders/3/builds/414


Observation: This test seems to be failing/flaky since https://ews-build.webkit.org/#/builders/3/builds/414
Comment 3 Wenson Hsieh 2019-04-14 16:44:26 PDT
I can't seem to hit this when running the test as-is, but this is reasonably easy to reproduce by running it in a loop like so:

--- a/Tools/TestWebKitAPI/Tests/WebKitCocoa/WKAttachmentTests.mm
+++ b/Tools/TestWebKitAPI/Tests/WebKitCocoa/WKAttachmentTests.mm
@@ -1348,6 +1348,8 @@ TEST(WKAttachmentTests, ChangeFileWrapperForPastedImage)

 TEST(WKAttachmentTests, AddAttachmentToConnectedImageElement)
 {
+    for (int i = 0; i < 50; i++) {
+    NSLog(@"<WK>: Running iteration: %tu", i);
     auto webView = webViewForTestingAttachments();
     [webView _synchronouslyExecuteEditCommand:@"InsertHTML" argument:@"<img></img>"];

@@ -1370,6 +1372,7 @@ TEST(WKAttachmentTests, AddAttachmentToConnectedImageElement)

     [attachment synchronouslySetFileWrapper:firstImage.get() newContentType:nil error:nil];
     [webView waitForImageElementSizeToBecome:CGSizeMake(215, 174)];
+    }
 }
Comment 4 Wenson Hsieh 2019-04-14 18:24:51 PDT Comment hidden (obsolete)
Comment 5 Wenson Hsieh 2019-04-14 18:27:38 PDT
Created attachment 367409 [details]
Add radar link to ChangeLog
Comment 6 Wenson Hsieh 2019-04-14 19:00:59 PDT
Created attachment 367410 [details]
Also reenable the test
Comment 7 WebKit Commit Bot 2019-04-14 21:47:44 PDT
Comment on attachment 367410 [details]
Also reenable the test

Clearing flags on attachment: 367410

Committed r244251: <https://trac.webkit.org/changeset/244251>
Comment 8 WebKit Commit Bot 2019-04-14 21:47:46 PDT
All reviewed patches have been landed.  Closing bug.