Bug 196006

Summary:	Flaky API Test TestWebKitAPI.KeyboardInputTests.KeyboardTypeForInput
Product:	WebKit	Reporter:	Aakash Jain <aakash_jain>
Component:	Tools / Tests	Assignee:	Wenson Hsieh <wenson_hsieh>
Status:	RESOLVED FIXED
Severity:	Normal	CC:	aakash_jain, ap, lforschler, pxlcoder, ryanhaddad, thorton, tsavell, webkit-bot-watchers-bugzilla, webkit-bug-importer, wenson_hsieh
Priority:	P2	Keywords:	InRadar
Version:	Other
Hardware:	Unspecified
OS:	Unspecified
See Also:	https://bugs.webkit.org/show_bug.cgi?id=196007 https://bugs.webkit.org/show_bug.cgi?id=196630

Description Aakash Jain 2019-03-20 07:43:08 PDT

TestWebKitAPI.KeyboardInputTests.KeyboardTypeForInput seems flaky. In https://ews-build.webkit-uat.org/#/builders/20/builds/2040, the test Failed in run-api-tests step. However, in the immediately next retry step (re-run-api-tests), it passed.

Comment 1 Aakash Jain 2019-03-20 09:17:47 PDT

This test was added in https://trac.webkit.org/changeset/235201

Comment 2 Aakash Jain 2019-03-21 14:00:32 PDT

Happened again in https://ews-build.webkit-uat.org/#/builders/20/builds/2198



TestWebKitAPI.KeyboardInputTests.KeyboardTypeForInput
    2019-03-21 12:08:07.471 TestWebKitAPI[32628:65791725] Displayed 7 for <input type='search' inputmode='tel' pattern=''>. Expected 5.
    2019-03-21 12:08:09.257 TestWebKitAPI[32628:65791725] Displayed 5 for <input type='tel' inputmode='numeric' pattern=''>. Expected 2.
        
    /Volumes/Data/worker/iOS-11-Simulator-Build-EWS/build/Tools/TestWebKitAPI/Tests/ios/KeyboardInputTestsIOS.mm:413
    Value of: runTest(inputType, inputMode, pattern, (UIKeyboardType)keyboardType.intValue)
      Actual: false
      Expected: true


    /Volumes/Data/worker/iOS-11-Simulator-Build-EWS/build/Tools/TestWebKitAPI/Tests/ios/KeyboardInputTestsIOS.mm:413
    Value of: runTest(inputType, inputMode, pattern, (UIKeyboardType)keyboardType.intValue)
      Actual: false
      Expected: true

Comment 3 Aakash Jain 2019-03-22 06:17:09 PDT

Happening very frequently, e.g.:

https://ews-build.webkit-uat.org/#/builders/20/builds/2232
https://ews-build.webkit-uat.org/#/builders/20/builds/2224
https://ews-build.webkit-uat.org/#/builders/20/builds/2205


TestWebKitAPI.KeyboardInputTests.KeyboardTypeForInput
    2019-03-21 17:39:00.638 TestWebKitAPI[20388:66464848] Displayed 8 for <input type='search' inputmode='url' pattern=''>. Expected 3.
        
    /Volumes/Data/worker/iOS-12-Simulator-Build-EWS/build/Tools/TestWebKitAPI/Tests/ios/KeyboardInputTestsIOS.mm:413
    Value of: runTest(inputType, inputMode, pattern, (UIKeyboardType)keyboardType.intValue)
      Actual: false
      Expected: true


TestWebKitAPI.KeyboardInputTests.KeyboardTypeForInput
    2019-03-21 13:58:33.025 TestWebKitAPI[5458:274706] Displayed 3 for <input type='text' inputmode='' pattern=''>. Expected 0.
        
    /Volumes/Data/worker/iOS-12-Simulator-Build-EWS/build/Tools/TestWebKitAPI/Tests/ios/KeyboardInputTestsIOS.mm:413
    Value of: runTest(inputType, inputMode, pattern, (UIKeyboardType)keyboardType.intValue)
      Actual: false
      Expected: true

Comment 4 Aakash Jain 2019-03-24 04:52:24 PDT

Can we prioritize this?
I am working on EWS for API tests, and this flaky failure is creating problem (e.g.: https://ews-build.webkit-uat.org/#/builders/20/builds/2256).

Comment 5 Wenson Hsieh 2019-03-25 14:41:42 PDT

I can repro this somewhat reliably by running the test for a dozen or so iterations in a loop.

Comment 6 Aakash Jain 2019-04-04 04:41:36 PDT

Happening very frequently, Recent occurrences:
https://ews-build.webkit-uat.org/#/builders/20/builds/2630
https://ews-build.webkit-uat.org/#/builders/20/builds/2624
https://ews-build.webkit-uat.org/#/builders/20/builds/2623
https://ews-build.webkit-uat.org/#/builders/20/builds/2616
https://ews-build.webkit-uat.org/#/builders/20/builds/2614
https://ews-build.webkit-uat.org/#/builders/20/builds/2612
https://ews-build.webkit-uat.org/#/builders/20/builds/2604
https://ews-build.webkit-uat.org/#/builders/20/builds/2599
https://ews-build.webkit-uat.org/#/builders/20/builds/2594
https://ews-build.webkit-uat.org/#/builders/20/builds/2593
https://ews-build.webkit-uat.org/#/builders/20/builds/2591
https://ews-build.webkit-uat.org/#/builders/20/builds/2588
https://ews-build.webkit-uat.org/#/builders/20/builds/2584
https://ews-build.webkit-uat.org/#/builders/20/builds/2564
https://ews-build.webkit-uat.org/#/builders/20/builds/2563
https://ews-build.webkit-uat.org/#/builders/20/builds/2554
https://ews-build.webkit-uat.org/#/builders/20/builds/2552
https://ews-build.webkit-uat.org/#/builders/20/builds/2551
https://ews-build.webkit-uat.org/#/builders/20/builds/2544
https://ews-build.webkit-uat.org/#/builders/20/builds/2535
https://ews-build.webkit-uat.org/#/builders/20/builds/2530
https://ews-build.webkit-uat.org/#/builders/20/builds/2521
https://ews-build.webkit-uat.org/#/builders/20/builds/2506
https://ews-build.webkit-uat.org/#/builders/20/builds/2497
https://ews-build.webkit-uat.org/#/builders/20/builds/2496
https://ews-build.webkit-uat.org/#/builders/20/builds/2492
https://ews-build.webkit-uat.org/#/builders/20/builds/2490
https://ews-build.webkit-uat.org/#/builders/20/builds/2480
https://ews-build.webkit-uat.org/#/builders/20/builds/2478
https://ews-build.webkit-uat.org/#/builders/20/builds/2473
https://ews-build.webkit-uat.org/#/builders/20/builds/2450
https://ews-build.webkit-uat.org/#/builders/20/builds/2446
https://ews-build.webkit-uat.org/#/builders/20/builds/2429
https://ews-build.webkit-uat.org/#/builders/20/builds/2426
https://ews-build.webkit-uat.org/#/builders/20/builds/2415
https://ews-build.webkit-uat.org/#/builders/20/builds/2413
https://ews-build.webkit-uat.org/#/builders/20/builds/2407
https://ews-build.webkit-uat.org/#/builders/20/builds/2403
https://ews-build.webkit-uat.org/#/builders/20/builds/2389
https://ews-build.webkit-uat.org/#/builders/20/builds/2385
https://ews-build.webkit-uat.org/#/builders/20/builds/2384
https://ews-build.webkit-uat.org/#/builders/20/builds/2376
https://ews-build.webkit-uat.org/#/builders/20/builds/2374
https://ews-build.webkit-uat.org/#/builders/20/builds/2366
https://ews-build.webkit-uat.org/#/builders/20/builds/2359
https://ews-build.webkit-uat.org/#/builders/20/builds/2335
https://ews-build.webkit-uat.org/#/builders/20/builds/2334


In following builds, this test was flaky in a manner that EWS thought that the patch being tested introduced this test failure (failed twice with patch, passed without patch):
https://ews-build.webkit-uat.org/#/builders/20/builds/2498
https://ews-build.webkit-uat.org/#/builders/20/builds/2378
https://ews-build.webkit-uat.org/#/builders/20/builds/2337
https://ews-build.webkit-uat.org/#/builders/20/builds/2329
https://ews-build.webkit-uat.org/#/builders/20/builds/2311
https://ews-build.webkit-uat.org/#/builders/20/builds/2308
https://ews-build.webkit-uat.org/#/builders/20/builds/2256

Comment 7 Radar WebKit Bug Importer 2019-05-06 13:11:21 PDT

<rdar://problem/50509880>

Comment 8 Aakash Jain 2019-05-06 13:33:26 PDT

This test was disabled in https://bugs.webkit.org/show_bug.cgi?id=196630. It would be nice to fix and re-enable it.

Comment 9 Wenson Hsieh 2019-05-17 20:46:57 PDT

I reenabled this in https://trac.webkit.org/changeset/245338/webkit.