Bug 196006

Summary: Flaky API Test TestWebKitAPI.KeyboardInputTests.KeyboardTypeForInput
Product: WebKit Reporter: Aakash Jain <aakash_jain>
Component: Tools / TestsAssignee: Wenson Hsieh <wenson_hsieh>
Status: RESOLVED FIXED    
Severity: Normal CC: aakash_jain, ap, lforschler, pxlcoder, ryanhaddad, thorton, tsavell, webkit-bot-watchers-bugzilla, webkit-bug-importer, wenson_hsieh
Priority: P2 Keywords: InRadar
Version: Other   
Hardware: Unspecified   
OS: Unspecified   
See Also: https://bugs.webkit.org/show_bug.cgi?id=196007
https://bugs.webkit.org/show_bug.cgi?id=196630

Description Aakash Jain 2019-03-20 07:43:08 PDT
TestWebKitAPI.KeyboardInputTests.KeyboardTypeForInput seems flaky. In https://ews-build.webkit-uat.org/#/builders/20/builds/2040, the test Failed in run-api-tests step. However, in the immediately next retry step (re-run-api-tests), it passed.
Comment 1 Aakash Jain 2019-03-20 09:17:47 PDT
This test was added in https://trac.webkit.org/changeset/235201
Comment 2 Aakash Jain 2019-03-21 14:00:32 PDT
Happened again in https://ews-build.webkit-uat.org/#/builders/20/builds/2198



TestWebKitAPI.KeyboardInputTests.KeyboardTypeForInput
    2019-03-21 12:08:07.471 TestWebKitAPI[32628:65791725] Displayed 7 for <input type='search' inputmode='tel' pattern=''>. Expected 5.
    2019-03-21 12:08:09.257 TestWebKitAPI[32628:65791725] Displayed 5 for <input type='tel' inputmode='numeric' pattern=''>. Expected 2.
        
    /Volumes/Data/worker/iOS-11-Simulator-Build-EWS/build/Tools/TestWebKitAPI/Tests/ios/KeyboardInputTestsIOS.mm:413
    Value of: runTest(inputType, inputMode, pattern, (UIKeyboardType)keyboardType.intValue)
      Actual: false
      Expected: true


    /Volumes/Data/worker/iOS-11-Simulator-Build-EWS/build/Tools/TestWebKitAPI/Tests/ios/KeyboardInputTestsIOS.mm:413
    Value of: runTest(inputType, inputMode, pattern, (UIKeyboardType)keyboardType.intValue)
      Actual: false
      Expected: true
Comment 3 Aakash Jain 2019-03-22 06:17:09 PDT
Happening very frequently, e.g.:

https://ews-build.webkit-uat.org/#/builders/20/builds/2232
https://ews-build.webkit-uat.org/#/builders/20/builds/2224
https://ews-build.webkit-uat.org/#/builders/20/builds/2205


TestWebKitAPI.KeyboardInputTests.KeyboardTypeForInput
    2019-03-21 17:39:00.638 TestWebKitAPI[20388:66464848] Displayed 8 for <input type='search' inputmode='url' pattern=''>. Expected 3.
        
    /Volumes/Data/worker/iOS-12-Simulator-Build-EWS/build/Tools/TestWebKitAPI/Tests/ios/KeyboardInputTestsIOS.mm:413
    Value of: runTest(inputType, inputMode, pattern, (UIKeyboardType)keyboardType.intValue)
      Actual: false
      Expected: true


TestWebKitAPI.KeyboardInputTests.KeyboardTypeForInput
    2019-03-21 13:58:33.025 TestWebKitAPI[5458:274706] Displayed 3 for <input type='text' inputmode='' pattern=''>. Expected 0.
        
    /Volumes/Data/worker/iOS-12-Simulator-Build-EWS/build/Tools/TestWebKitAPI/Tests/ios/KeyboardInputTestsIOS.mm:413
    Value of: runTest(inputType, inputMode, pattern, (UIKeyboardType)keyboardType.intValue)
      Actual: false
      Expected: true
Comment 4 Aakash Jain 2019-03-24 04:52:24 PDT
Can we prioritize this?
I am working on EWS for API tests, and this flaky failure is creating problem (e.g.: https://ews-build.webkit-uat.org/#/builders/20/builds/2256).
Comment 5 Wenson Hsieh 2019-03-25 14:41:42 PDT
I can repro this somewhat reliably by running the test for a dozen or so iterations in a loop.
Comment 6 Aakash Jain 2019-04-04 04:41:36 PDT
Happening very frequently, Recent occurrences:
https://ews-build.webkit-uat.org/#/builders/20/builds/2630
https://ews-build.webkit-uat.org/#/builders/20/builds/2624
https://ews-build.webkit-uat.org/#/builders/20/builds/2623
https://ews-build.webkit-uat.org/#/builders/20/builds/2616
https://ews-build.webkit-uat.org/#/builders/20/builds/2614
https://ews-build.webkit-uat.org/#/builders/20/builds/2612
https://ews-build.webkit-uat.org/#/builders/20/builds/2604
https://ews-build.webkit-uat.org/#/builders/20/builds/2599
https://ews-build.webkit-uat.org/#/builders/20/builds/2594
https://ews-build.webkit-uat.org/#/builders/20/builds/2593
https://ews-build.webkit-uat.org/#/builders/20/builds/2591
https://ews-build.webkit-uat.org/#/builders/20/builds/2588
https://ews-build.webkit-uat.org/#/builders/20/builds/2584
https://ews-build.webkit-uat.org/#/builders/20/builds/2564
https://ews-build.webkit-uat.org/#/builders/20/builds/2563
https://ews-build.webkit-uat.org/#/builders/20/builds/2554
https://ews-build.webkit-uat.org/#/builders/20/builds/2552
https://ews-build.webkit-uat.org/#/builders/20/builds/2551
https://ews-build.webkit-uat.org/#/builders/20/builds/2544
https://ews-build.webkit-uat.org/#/builders/20/builds/2535
https://ews-build.webkit-uat.org/#/builders/20/builds/2530
https://ews-build.webkit-uat.org/#/builders/20/builds/2521
https://ews-build.webkit-uat.org/#/builders/20/builds/2506
https://ews-build.webkit-uat.org/#/builders/20/builds/2497
https://ews-build.webkit-uat.org/#/builders/20/builds/2496
https://ews-build.webkit-uat.org/#/builders/20/builds/2492
https://ews-build.webkit-uat.org/#/builders/20/builds/2490
https://ews-build.webkit-uat.org/#/builders/20/builds/2480
https://ews-build.webkit-uat.org/#/builders/20/builds/2478
https://ews-build.webkit-uat.org/#/builders/20/builds/2473
https://ews-build.webkit-uat.org/#/builders/20/builds/2450
https://ews-build.webkit-uat.org/#/builders/20/builds/2446
https://ews-build.webkit-uat.org/#/builders/20/builds/2429
https://ews-build.webkit-uat.org/#/builders/20/builds/2426
https://ews-build.webkit-uat.org/#/builders/20/builds/2415
https://ews-build.webkit-uat.org/#/builders/20/builds/2413
https://ews-build.webkit-uat.org/#/builders/20/builds/2407
https://ews-build.webkit-uat.org/#/builders/20/builds/2403
https://ews-build.webkit-uat.org/#/builders/20/builds/2389
https://ews-build.webkit-uat.org/#/builders/20/builds/2385
https://ews-build.webkit-uat.org/#/builders/20/builds/2384
https://ews-build.webkit-uat.org/#/builders/20/builds/2376
https://ews-build.webkit-uat.org/#/builders/20/builds/2374
https://ews-build.webkit-uat.org/#/builders/20/builds/2366
https://ews-build.webkit-uat.org/#/builders/20/builds/2359
https://ews-build.webkit-uat.org/#/builders/20/builds/2335
https://ews-build.webkit-uat.org/#/builders/20/builds/2334


In following builds, this test was flaky in a manner that EWS thought that the patch being tested introduced this test failure (failed twice with patch, passed without patch):
https://ews-build.webkit-uat.org/#/builders/20/builds/2498
https://ews-build.webkit-uat.org/#/builders/20/builds/2378
https://ews-build.webkit-uat.org/#/builders/20/builds/2337
https://ews-build.webkit-uat.org/#/builders/20/builds/2329
https://ews-build.webkit-uat.org/#/builders/20/builds/2311
https://ews-build.webkit-uat.org/#/builders/20/builds/2308
https://ews-build.webkit-uat.org/#/builders/20/builds/2256
Comment 7 Radar WebKit Bug Importer 2019-05-06 13:11:21 PDT
<rdar://problem/50509880>
Comment 8 Aakash Jain 2019-05-06 13:33:26 PDT
This test was disabled in https://bugs.webkit.org/show_bug.cgi?id=196630. It would be nice to fix and re-enable it.
Comment 9 Wenson Hsieh 2019-05-17 20:46:57 PDT
I reenabled this in https://trac.webkit.org/changeset/245338/webkit.