Bug 242128 – REGRESSION (251741@main-251739@main?): [ iOS ] TestWebKitAPI.AutocorrectionTests.AutocorrectionContextBeforeAndAfterEditing is a flaky failure

• Status: NEW
• Component: New Bugs

[Karl Rackler]

Description:
TestWebKitAPI.AutocorrectionTests.AutocorrectionContextBeforeAndAfterEditing

The first failure that I saw on the dashboard was on 6/22/2022 at 251749@main.

REPRODUCTION STEPS
I can reproduce this on 251741@main, but unable to reproduce it on 251738@main or earlier.  This affects the iOS model iPhone 8 only. 

Command: 
run-api-tests --release --verbose --iterations 10 --model iPhone8 TestWebKitAPI.AutocorrectionTests.AutocorrectionContextBeforeAndAfterEditing

Result: 
Same as bot below

History:
https://results.webkit.org/?suite=api-tests&test=TestWebKitAPI.AutocorrectionTests.AutocorrectionContextBeforeAndAfterEditing&limit=50000&platform=ios&model=iPhone%208

run-api-tests stdio:
Failed
    TestWebKitAPI.AutocorrectionTests.AutocorrectionContextBeforeAndAfterEditing
        
        /Volumes/Data/worker/Apple-iOS-15-Simulator-Release-Build/build/Tools/TestWebKitAPI/Tests/ios/AutocorrectionTestsIOS.mm:182
        Expected equality of these values:
          "hello"
          [contextAfterInsertingText contextBeforeSelection]
            Which is: ""
        
        
        /Volumes/Data/worker/Apple-iOS-15-Simulator-Release-Build/build/Tools/TestWebKitAPI/Tests/ios/AutocorrectionTestsIOS.mm:187
        Expected equality of these values:
          "hello"
          [contextAfterSelecting selectedText]
            Which is: ""

[Radar WebKit Bug Importer]

<rdar://problem/96161911>

[Karl Rackler]

I have disabled this test while this issue is investigated.

[EWS]

Test gardening commit 251961@main (9aa946bf2540): <https://commits.webkit.org/251961@main>

Reviewed commits have been landed. Closing PR #1912 and removing active labels.

[Karl Rackler]

The test failure is referencing contextAfterInsertingText and contextAfterSelecting from 246659@main.  

The flakiness dashboard shows a clear change from consistently passing to flaky failure first appearing on the dashboard at 251749@main.  

Reproduction steps further reduced the range(251741@main-251739@main), but it is unclear what in this range may have caused this.

[Wenson Hsieh]

(In reply to Karl Rackler from comment #4)
> The test failure is referencing contextAfterInsertingText and
> contextAfterSelecting from 246659@main.  
> 
> The flakiness dashboard shows a clear change from consistently passing to
> flaky failure first appearing on the dashboard at 251749@main.  
> 
> Reproduction steps further reduced the range(251741@main-251739@main), but
> it is unclear what in this range may have caused this.

Yeah, neither of the commits in that range remotely seem like they could've caused these test failures.

Were there any interesting configuration changes might've affected the bots at around the same time? (i.e. upgrading to a different version, or using new simulator device models, etc.)

[Ryan Haddad]

(In reply to Wenson Hsieh from comment #5)
> Were there any interesting configuration changes might've affected the bots
> at around the same time? (i.e. upgrading to a different version, or using
> new simulator device models, etc.)

We have not made any changes in this area recently, we're also seeing this failure on internal bots in roughly the same regression range http://commits.webkit.org/compare/251744@main...251736@main

[Ryan Haddad]

Not sure what happened with my URL in comment 6, it should be showing a range of 251744@main - 251736@main
http://commits.webkit.org/compare/251744@main...251736@main