Bug 196905

Summary: API test WKAttachmentTests.AddAttachmentToConnectedImageElement is a flaky failure on Mac Release builds
Product: WebKit Reporter: Andy Estes <aestes>
Component: New BugsAssignee: Wenson Hsieh <wenson_hsieh>
Status: RESOLVED FIXED    
Severity: Normal CC: aakash_jain, aestes, bdakin, commit-queue, rniwa, thorton, webkit-bug-importer, wenson_hsieh
Priority: P2 Keywords: InRadar
Version: WebKit Local Build   
Hardware: Macintosh   
OS: All   
See Also: https://bugs.webkit.org/show_bug.cgi?id=177484
https://bugs.webkit.org/show_bug.cgi?id=196893
https://bugs.webkit.org/show_bug.cgi?id=196909
Attachments:
Description Flags
Patch
none
Add radar link to ChangeLog
none
Also reenable the test none

Description Andy Estes 2019-04-14 08:37:51 PDT
On macOS Mojave, in a Release build of r244241, the API test WKAttachmentTests.AddAttachmentToConnectedImageElement failed on 6 out of 10 runs of the WKAttachmentTests API test suite. It also sometimes fails on api-mac EWS bots in a way that makes it appear like a seemingly-unrelated patch regressed the test (e.g., bug #177484).

Steps to reproduce:
1. Create a Mac Release build of r244241.
2. run-api-tests WKAttachmentTests.

Expected results:
All 31 tests pass.

Actual results:
On 6 out of 10 runs, WKAttachmentTests.AddAttachmentToConnectedImageElement failed.

    TestWebKitAPI.WKAttachmentTests.AddAttachmentToConnectedImageElement
        _RegisterApplication(), FAILED TO establish the default connection to the WindowServer, _CGSDefaultConnection() is NULL.
        2019-04-14 08:23:20.508 TestWebKitAPI[59497:767481] Expected inserted attachments: (
        ) to match (
            "<_WKAttachment 0x7fbcc7d01340 id='2414efeb-bbf1-4650-83ce-a167b0a11484'>"
        ).
        
        /Volumes/Data/Projects/OpenSource/Tools/TestWebKitAPI/Tests/WebKitCocoa/WKAttachmentTests.mm:150
        Value of: insertedAttachmentsMatch
          Actual: false
        Expected: true
Comment 1 Radar WebKit Bug Importer 2019-04-14 09:34:26 PDT
<rdar://problem/49886096>
Comment 2 Aakash Jain 2019-04-14 15:47:37 PDT
This flaky test is slowing down EWS (since EWS has to re-run API tests few times to confirm that the failure is not because of the patch being tested).

e.g.:
https://ews-build.webkit.org/#/builders/3/builds/535
https://ews-build.webkit.org/#/builders/3/builds/530
https://ews-build.webkit.org/#/builders/3/builds/528
https://ews-build.webkit.org/#/builders/3/builds/503
https://ews-build.webkit.org/#/builders/3/builds/500
https://ews-build.webkit.org/#/builders/3/builds/467
https://ews-build.webkit.org/#/builders/3/builds/465
https://ews-build.webkit.org/#/builders/3/builds/460

https://ews-build.webkit.org/#/builders/3/builds/541
https://ews-build.webkit.org/#/builders/3/builds/527
https://ews-build.webkit.org/#/builders/3/builds/526
https://ews-build.webkit.org/#/builders/3/builds/518
https://ews-build.webkit.org/#/builders/3/builds/515
https://ews-build.webkit.org/#/builders/3/builds/509
https://ews-build.webkit.org/#/builders/3/builds/499
https://ews-build.webkit.org/#/builders/3/builds/497
https://ews-build.webkit.org/#/builders/3/builds/494
https://ews-build.webkit.org/#/builders/3/builds/493
https://ews-build.webkit.org/#/builders/3/builds/490
https://ews-build.webkit.org/#/builders/3/builds/480
https://ews-build.webkit.org/#/builders/3/builds/479
https://ews-build.webkit.org/#/builders/3/builds/477
https://ews-build.webkit.org/#/builders/3/builds/476
https://ews-build.webkit.org/#/builders/3/builds/474
https://ews-build.webkit.org/#/builders/3/builds/473
https://ews-build.webkit.org/#/builders/3/builds/470



In following builds, the test was so flaky that EWS incorrectly thought that the failure was introduced by the patch being tested (this test failed twice with the patch, and didn't fail on clean tree):
https://ews-build.webkit.org/#/builders/3/builds/542
https://ews-build.webkit.org/#/builders/3/builds/546
https://ews-build.webkit.org/#/builders/3/builds/512
https://ews-build.webkit.org/#/builders/3/builds/495
https://ews-build.webkit.org/#/builders/3/builds/492
https://ews-build.webkit.org/#/builders/3/builds/488
https://ews-build.webkit.org/#/builders/3/builds/484
https://ews-build.webkit.org/#/builders/3/builds/471
https://ews-build.webkit.org/#/builders/3/builds/468
https://ews-build.webkit.org/#/builders/3/builds/464
https://ews-build.webkit.org/#/builders/3/builds/454
https://ews-build.webkit.org/#/builders/3/builds/440
https://ews-build.webkit.org/#/builders/3/builds/439
https://ews-build.webkit.org/#/builders/3/builds/429
https://ews-build.webkit.org/#/builders/3/builds/426
https://ews-build.webkit.org/#/builders/3/builds/414


Observation: This test seems to be failing/flaky since https://ews-build.webkit.org/#/builders/3/builds/414
Comment 3 Wenson Hsieh 2019-04-14 16:44:26 PDT
I can't seem to hit this when running the test as-is, but this is reasonably easy to reproduce by running it in a loop like so:

--- a/Tools/TestWebKitAPI/Tests/WebKitCocoa/WKAttachmentTests.mm
+++ b/Tools/TestWebKitAPI/Tests/WebKitCocoa/WKAttachmentTests.mm
@@ -1348,6 +1348,8 @@ TEST(WKAttachmentTests, ChangeFileWrapperForPastedImage)

 TEST(WKAttachmentTests, AddAttachmentToConnectedImageElement)
 {
+    for (int i = 0; i < 50; i++) {
+    NSLog(@"<WK>: Running iteration: %tu", i);
     auto webView = webViewForTestingAttachments();
     [webView _synchronouslyExecuteEditCommand:@"InsertHTML" argument:@"<img></img>"];

@@ -1370,6 +1372,7 @@ TEST(WKAttachmentTests, AddAttachmentToConnectedImageElement)

     [attachment synchronouslySetFileWrapper:firstImage.get() newContentType:nil error:nil];
     [webView waitForImageElementSizeToBecome:CGSizeMake(215, 174)];
+    }
 }
Comment 4 Wenson Hsieh 2019-04-14 18:24:51 PDT Comment hidden (obsolete)
Comment 5 Wenson Hsieh 2019-04-14 18:27:38 PDT
Created attachment 367409 [details]
Add radar link to ChangeLog
Comment 6 Wenson Hsieh 2019-04-14 19:00:59 PDT
Created attachment 367410 [details]
Also reenable the test
Comment 7 WebKit Commit Bot 2019-04-14 21:47:44 PDT
Comment on attachment 367410 [details]
Also reenable the test

Clearing flags on attachment: 367410

Committed r244251: <https://trac.webkit.org/changeset/244251>
Comment 8 WebKit Commit Bot 2019-04-14 21:47:46 PDT
All reviewed patches have been landed.  Closing bug.