Bug 196905 – API test WKAttachmentTests.AddAttachmentToConnectedImageElement is a flaky failure on Mac Release builds

• Status: RESOLVED
• Resolution: FIXED
• Component: New Bugs

[Andy Estes]

On macOS Mojave, in a Release build of r244241, the API test WKAttachmentTests.AddAttachmentToConnectedImageElement failed on 6 out of 10 runs of the WKAttachmentTests API test suite. It also sometimes fails on api-mac EWS bots in a way that makes it appear like a seemingly-unrelated patch regressed the test (e.g., bug #177484).

Steps to reproduce:
1. Create a Mac Release build of r244241.
2. run-api-tests WKAttachmentTests.

Expected results:
All 31 tests pass.

Actual results:
On 6 out of 10 runs, WKAttachmentTests.AddAttachmentToConnectedImageElement failed.

    TestWebKitAPI.WKAttachmentTests.AddAttachmentToConnectedImageElement
        _RegisterApplication(), FAILED TO establish the default connection to the WindowServer, _CGSDefaultConnection() is NULL.
        2019-04-14 08:23:20.508 TestWebKitAPI[59497:767481] Expected inserted attachments: (
        ) to match (
            "<_WKAttachment 0x7fbcc7d01340 id='2414efeb-bbf1-4650-83ce-a167b0a11484'>"
        ).
        
        /Volumes/Data/Projects/OpenSource/Tools/TestWebKitAPI/Tests/WebKitCocoa/WKAttachmentTests.mm:150
        Value of: insertedAttachmentsMatch
          Actual: false
        Expected: true

[Radar WebKit Bug Importer]

<rdar://problem/49886096>

[Aakash Jain]

This flaky test is slowing down EWS (since EWS has to re-run API tests few times to confirm that the failure is not because of the patch being tested).

e.g.:
https://ews-build.webkit.org/#/builders/3/builds/535
https://ews-build.webkit.org/#/builders/3/builds/530
https://ews-build.webkit.org/#/builders/3/builds/528
https://ews-build.webkit.org/#/builders/3/builds/503
https://ews-build.webkit.org/#/builders/3/builds/500
https://ews-build.webkit.org/#/builders/3/builds/467
https://ews-build.webkit.org/#/builders/3/builds/465
https://ews-build.webkit.org/#/builders/3/builds/460

https://ews-build.webkit.org/#/builders/3/builds/541
https://ews-build.webkit.org/#/builders/3/builds/527
https://ews-build.webkit.org/#/builders/3/builds/526
https://ews-build.webkit.org/#/builders/3/builds/518
https://ews-build.webkit.org/#/builders/3/builds/515
https://ews-build.webkit.org/#/builders/3/builds/509
https://ews-build.webkit.org/#/builders/3/builds/499
https://ews-build.webkit.org/#/builders/3/builds/497
https://ews-build.webkit.org/#/builders/3/builds/494
https://ews-build.webkit.org/#/builders/3/builds/493
https://ews-build.webkit.org/#/builders/3/builds/490
https://ews-build.webkit.org/#/builders/3/builds/480
https://ews-build.webkit.org/#/builders/3/builds/479
https://ews-build.webkit.org/#/builders/3/builds/477
https://ews-build.webkit.org/#/builders/3/builds/476
https://ews-build.webkit.org/#/builders/3/builds/474
https://ews-build.webkit.org/#/builders/3/builds/473
https://ews-build.webkit.org/#/builders/3/builds/470



In following builds, the test was so flaky that EWS incorrectly thought that the failure was introduced by the patch being tested (this test failed twice with the patch, and didn't fail on clean tree):
https://ews-build.webkit.org/#/builders/3/builds/542
https://ews-build.webkit.org/#/builders/3/builds/546
https://ews-build.webkit.org/#/builders/3/builds/512
https://ews-build.webkit.org/#/builders/3/builds/495
https://ews-build.webkit.org/#/builders/3/builds/492
https://ews-build.webkit.org/#/builders/3/builds/488
https://ews-build.webkit.org/#/builders/3/builds/484
https://ews-build.webkit.org/#/builders/3/builds/471
https://ews-build.webkit.org/#/builders/3/builds/468
https://ews-build.webkit.org/#/builders/3/builds/464
https://ews-build.webkit.org/#/builders/3/builds/454
https://ews-build.webkit.org/#/builders/3/builds/440
https://ews-build.webkit.org/#/builders/3/builds/439
https://ews-build.webkit.org/#/builders/3/builds/429
https://ews-build.webkit.org/#/builders/3/builds/426
https://ews-build.webkit.org/#/builders/3/builds/414


Observation: This test seems to be failing/flaky since https://ews-build.webkit.org/#/builders/3/builds/414

[Wenson Hsieh]

I can't seem to hit this when running the test as-is, but this is reasonably easy to reproduce by running it in a loop like so:

--- a/Tools/TestWebKitAPI/Tests/WebKitCocoa/WKAttachmentTests.mm
+++ b/Tools/TestWebKitAPI/Tests/WebKitCocoa/WKAttachmentTests.mm
@@ -1348,6 +1348,8 @@ TEST(WKAttachmentTests, ChangeFileWrapperForPastedImage)

 TEST(WKAttachmentTests, AddAttachmentToConnectedImageElement)
 {
+    for (int i = 0; i < 50; i++) {
+    NSLog(@"<WK>: Running iteration: %tu", i);
     auto webView = webViewForTestingAttachments();
     [webView _synchronouslyExecuteEditCommand:@"InsertHTML" argument:@"<img></img>"];

@@ -1370,6 +1372,7 @@ TEST(WKAttachmentTests, AddAttachmentToConnectedImageElement)

     [attachment synchronouslySetFileWrapper:firstImage.get() newContentType:nil error:nil];
     [webView waitForImageElementSizeToBecome:CGSizeMake(215, 174)];
+    }
 }

[Wenson Hsieh]

Created attachment 367408 [details]
Patch

[Wenson Hsieh]

Created attachment 367409 [details]
Add radar link to ChangeLog

[Wenson Hsieh]

Created attachment 367410 [details]
Also reenable the test

[WebKit Commit Bot]

Comment on attachment 367410 [details]
Also reenable the test

Clearing flags on attachment: 367410

Committed r244251: <https://trac.webkit.org/changeset/244251>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.