Some time around August 20, a few tests became much slower - usually taking around 30 seconds, and thus often timing out. E.g.: https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=http%2Ftests%2Fxmlhttprequest%2Fworkers%2Fmethods.html https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=http/tests/xmlhttprequest/methods-async.html You can see that before August 20th or so, the tests were taking under a second, or a few seconds at most, then gradually grew, and now always take very substantial time.
There are a couple reasons for such behavior that I can think of: 1. Disk caching going crazy, and storing way too many files to be performant (we've seen that with media loads before). 2. Us gradually adding more and more unrelated tests that globally slow down the system in some way, perhaps leaking VM entries or confusing JSC garbage collection code.
The regression is only observed with WebKit2 at this time.
I guess #1 is unlikely, as this is happening on both iOS and macOS, and we are supposed to always start with a clean simulator on iOS.
https://bugs.webkit.org/show_bug.cgi?id=161699 should enable spindump on timeout for only iOS 9 WK2 release simulators. Just needs a review.
<rdar://problem/28219123>
Looks like rebooting clears this up.
Tracked this down to an issue below WebKit, will continue tracking in Radar. We'll need to reboot the bots periodically for now.