249444 2022-12-15 18:29:54 -0800 WK1 tests are taking a long time to run 'extract-test-results' 2022-12-22 18:30:17 -0800 1 1 1 Unclassified WebKit Tools / Tests WebKit Nightly Build Unspecified Unspecified NEW InRadar P2 Normal --- 249614 249658 249724 1 mattwoodrow webkit-unassigned achristensen ap jbedard ryanhaddad webkit-bug-importer oldest_to_newest 1919789 0 mattwoodrow 2022-12-15 18:29:54 -0800 It looks like extracting test results is frequently taking 10+ minutes, adding up to a lot when we often run tests twice. My build: https://ews-build.webkit.org/#/builders/73/builds/22639 Others picked at random: https://ews-build.webkit.org/#/builders/73/builds/22631 (17 minutes!) https://ews-build.webkit.org/#/builders/73/builds/22619 1920048 1 ap 2022-12-16 13:20:47 -0800 Looks like we have 35K files in the archive, most of them stderr output of this kind: 2022-12-15 18:20:33.500 DumpRenderTree[27762:191348898] nil host used in call to allowsSpecificHTTPSCertificateForHost 2022-12-15 18:20:33.500 DumpRenderTree[27762:191348898] nil host used in call to allowsAnyHTTPSCertificateForHost: Not sure if something changed recently, but we should definitely fix this. 1920054 2 ap 2022-12-16 13:46:23 -0800 What's also weird is that these are not in results.html. Are we just failing to clean the results directory between runs? 1920375 3 ap 2022-12-18 14:25:36 -0800 No - checking the dates of -stderr.txt files on a bot, that's not what is happening, they are all recent. 1920476 4 jbedard 2022-12-19 08:11:47 -0800 I also was able to reproduce this problem very easily on a local build. I'm going to see if I can figure out the cause of the problem with code inspection, but we might consider attempting to bisect. 1920547 5 jbedard 2022-12-19 12:29:56 -0800 From testing locally, looks like there are 2 problems: 1) It's somehow possible for us to get empty allowed hosts or local host aliases. I still haven't figured out _how_ that's possible, though, because according to the code, it shouldn't be possible. 2) Clearly out our database caches is racy because these caches are shared between all workers. This race seems so obvious, though, that I'm unclear what I'm missing. 1920614 6 jbedard 2022-12-19 15:57:56 -0800 Not convinced https://bugs.webkit.org/show_bug.cgi?id=249614 is problem, but it does seem to be a problem. 1920811 7 ap 2022-12-20 09:33:12 -0800 Turns out that these were a known bug fixed in macOS Ventura, so we should just filter out output on older OSes: > 2022-12-15 18:20:33.500 DumpRenderTree[27762:191348898] nil host used in call to allowsSpecificHTTPSCertificateForHost > 2022-12-15 18:20:33.500 DumpRenderTree[27762:191348898] nil host used in call to allowsAnyHTTPSCertificateForHost: I didn't check if this resolves enough of the spurious logs. 1921580 8 webkit-bug-importer 2022-12-22 18:30:17 -0800 <rdar://problem/103657108>