Bug 48374

Summary: run-webkit-tests takes 20 minutes longer on a Debug build than on a Release build on Windows and Leopard
Product: WebKit Reporter: Adam Roben (:aroben) <aroben>
Component: Tools / TestsAssignee: Nobody <webkit-unassigned>
Status: RESOLVED CONFIGURATION CHANGED    
Severity: Normal CC: ap, barraclough, bfulgham, bweinstein, lforschler, loislo, pfeldman, sfalken, simon.fraser, yurys
Priority: P2 Keywords: InRadar, PlatformOnly, ToolsHitList
Version: 528+ (Nightly build)   
Hardware: PC   
OS: Windows XP   
Attachments:
Description Flags
Time it took to run each test in a Debug build
none
Time it took to run each test in a Release build none

Adam Roben (:aroben)
Reported 2010-10-26 14:22:23 PDT
run-webkit-tests takes 20 minutes longer on a Debug build than on a Release build on Windows. Here's an example of two test runs for the same revision on identical hardware: Debug: http://build.webkit.org/builders/Windows%20Debug%20%28Tests%29/builds/21592 Release: http://build.webkit.org/builders/Windows%20Release%20%28Tests%29/builds/5601 You can see that run-webkit-tests took nearly 20 minutes longer in the Debug build. This is making our Debug test bots fall way behind.
Attachments
Time it took to run each test in a Debug build (1.26 MB, text/plain)
2010-10-26 14:22 PDT, Adam Roben (:aroben) 		no flags
Details
Time it took to run each test in a Release build (1.26 MB, text/plain)
2010-10-26 14:26 PDT, Adam Roben (:aroben) 		no flags
Details
View All Add attachment proposed patch, testcase, etc.
Adam Roben (:aroben)
Comment 1 2010-10-26 14:22:57 PDT
Created attachment 71937 [details] Time it took to run each test in a Debug build
Adam Roben (:aroben)
Comment 2 2010-10-26 14:26:21 PDT
Created attachment 71938 [details] Time it took to run each test in a Release build I instrumented run-webkit-tests to record the amount of time (in seconds) that it took to run each tests. Here are the results for Debug and Release.
Adam Roben (:aroben)
Comment 3 2010-10-26 14:27:16 PDT
I looked at the delta between Debug and Release for each test. Here are the top 40 tests with their deltas, ordered by how much longer they took in Debug: 19.36 http/tests/misc/acid3.html 15.12 editing/selection/extend-selection.html 13.65 fast/js/sputnik/Conformance/07_Lexical_Conventions/7.8_Literals/7.8.5_Regular_Expression_Literals/S7.8.5_A2.1_T2.html 13.48 fast/js/sputnik/Conformance/07_Lexical_Conventions/7.8_Literals/7.8.5_Regular_Expression_Literals/S7.8.5_A2.4_T2.html 13.36 fast/js/sputnik/Conformance/07_Lexical_Conventions/7.8_Literals/7.8.5_Regular_Expression_Literals/S7.8.5_A1.4_T2.html 13.05 fast/js/sputnik/Conformance/07_Lexical_Conventions/7.8_Literals/7.8.5_Regular_Expression_Literals/S7.8.5_A1.1_T2.html 12.38 fast/parser/block-nesting-cap.html 10.44 fast/js/sputnik/Unicode/Unicode_218/S7.6_A5.3_T2.html 9.55 ietestcenter/Javascript/15.4.4.16-7-c-ii-2.html 9.54 ietestcenter/Javascript/15.4.4.18-7-c-ii-1.html 9.52 ietestcenter/Javascript/15.4.4.17-7-c-ii-2.html 9.52 ietestcenter/Javascript/15.4.4.19-8-c-ii-1.html 9.51 ietestcenter/Javascript/15.4.4.20-9-c-ii-1.html 8.59 fast/js/sputnik/Conformance/07_Lexical_Conventions/7.4_Comments/S7.4_A5.html 8.41 fast/js/sputnik/Conformance/07_Lexical_Conventions/7.4_Comments/S7.4_A6.html 8.08 fast/frames/lots-of-objects.html 8.07 editing/selection/doubleclick-beside-cr-span.html 7.94 ietestcenter/Javascript/15.4.4.15-9-1.html 7.85 fast/js/toString-and-valueOf-override.html 7.83 fast/js/sputnik/Unicode/Unicode_218/S7.6_A5.3_T1.html 7.25 fast/workers/worker-cloneport.html 6.84 fast/parser/element-nesting-cap.html 6.61 ietestcenter/Javascript/15.4.4.14-10-1.html 6.37 fast/js/sputnik/Implementation_Diagnostics/S8.6_D1.4.html 6.16 fast/frames/lots-of-iframes.html 5.77 fast/js/function-apply-many-args.html 5.51 html5lib/webkit-resumer.html 4.98 fast/css/visited-link-hang.html 4.86 fast/js/sputnik/Unicode/Unicode_320/S7.6_A5.3_T2.html 4.64 fast/js/sputnik/Unicode/Unicode_320/S7.6_A3.1.html 4.45 fast/js/sputnik/Unicode/Unicode_410/S7.6_A5.3_T2.html 4.37 fast/js/sputnik/Unicode/Unicode_500/S7.6_A5.3_T2.html 4.37 fast/workers/storage/interrupt-database.html 4.30 fast/js/exception-sequencing-binops.html 4.19 fast/js/sputnik/Unicode/Unicode_320/S7.6_A5.3_T1.html 4.16 inspector/debugger-breakpoints-not-activated-on-reload.html 4.13 fast/css/css-selector-deeply-nested.html 4.13 fast/js/sputnik/Unicode/Unicode_410/S7.6_A3.1.html 4.11 fast/js/excessive-comma-usage.html 4.10 fast/js/sputnik/Conformance/15_Native_Objects/15.1_The_Global_Object/15.1.3/15.1.3.2_decodeURIComponent/S15.1.3.2_A1.12_T2.html
Adam Roben (:aroben)
Comment 4 2010-10-26 14:28:14 PDT
I also tried looking at which directories take the most extra time in Debug. Here are the top 40 directories and their deltas: 205.24 inspector 100.92 ietestcenter/Javascript 78.54 fast/js 53.84 fast/js/sputnik/Conformance/07_Lexical_Conventions/7.8_Literals/7.8.5_Regular_Expression_Literals 37.28 fast/js/sputnik/Conformance/15_Native_Objects/15.1_The_Global_Object/15.1.3/15.1.3.2_decodeURIComponent 37.19 fast/js/sputnik/Conformance/15_Native_Objects/15.1_The_Global_Object/15.1.3/15.1.3.1_decodeURI 34.01 editing/selection 26.47 fast/parser 25.39 dom/xhtml/level2/html 24.15 dom/html/level2/html 22.06 dom/xhtml/level3/core 20.89 http/tests/misc 20.57 fast/js/sputnik/Unicode/Unicode_218 20.14 fast/js/sputnik/Unicode/Unicode_320 19.46 fast/js/sputnik/Unicode/Unicode_500 19.23 fast/js/sputnik/Unicode/Unicode_410 18.71 fast/frames 17.84 fast/css 17.02 fast/js/sputnik/Conformance/07_Lexical_Conventions/7.4_Comments 16.73 fast/js/sputnik/Unicode/Unicode_510 16.19 fast/forms 14.84 fast/workers 14.76 http/tests/inspector 13.34 storage 12.60 fast/workers/storage 12.07 svg/dynamic-updates 11.81 http/tests/xmlhttprequest 9.56 fast/js/sputnik/Implementation_Diagnostics 8.56 editing/execCommand 8.23 fast/dom 8.08 dom/html/level1/core 7.98 dom/xhtml/level1/core 7.74 html5lib 7.48 http/tests/security/xssAuditor 7.00 fast/events 6.89 editing/pasteboard 6.63 fast/canvas 6.36 fast/js/sputnik/Conformance/15_Native_Objects/15.1_The_Global_Object/15.1.3/15.1.3.3_encodeURI 6.35 fast/js/sputnik/Conformance/15_Native_Objects/15.1_The_Global_Object/15.1.3/15.1.3.4_encodeURIComponent 5.91 media
Adam Roben (:aroben)
Comment 5 2010-10-26 14:31:08 PDT
<rdar://problem/8596732>
Adam Roben (:aroben)
Comment 6 2011-02-03 16:24:12 PST
I realized that just looking at proportional time increases might be helpful. So I divided the amount of time the test took in a Debug build by how long it took in a Release build, and found the 40 tests with the highest ratio: 137.80 fast/parser/element-nesting-cap.html 63.50 fast/xpath/attr-namespace.html 41.67 fast/parser/block-nesting-cap-table.html 29.79 fast/parser/block-nesting-cap.html 27.50 ietestcenter/Javascript/15.4.4.18-7-c-ii-1.html 27.42 ietestcenter/Javascript/15.4.4.20-9-c-ii-1.html 26.81 ietestcenter/Javascript/15.4.4.16-7-c-ii-2.html 26.05 ietestcenter/Javascript/15.4.4.19-8-c-ii-1.html 26.05 ietestcenter/Javascript/15.4.4.17-7-c-ii-2.html 25.12 fast/js/sputnik/Conformance/15_Native_Objects/15.1_The_Global_Object/15.1.3/15.1.3.2_decodeURIComponent/S15.1.3.2_A1.12_T2.html 25.06 fast/js/sputnik/Conformance/15_Native_Objects/15.1_The_Global_Object/15.1.3/15.1.3.1_decodeURI/S15.1.3.1_A1.12_T2.html 25.00 fast/js/sputnik/Conformance/15_Native_Objects/15.1_The_Global_Object/15.1.3/15.1.3.2_decodeURIComponent/S15.1.3.2_A1.10_T1.html 24.18 fast/js/sputnik/Conformance/15_Native_Objects/15.1_The_Global_Object/15.1.3/15.1.3.1_decodeURI/S15.1.3.1_A1.2_T2.html 24.18 fast/js/sputnik/Conformance/15_Native_Objects/15.1_The_Global_Object/15.1.3/15.1.3.2_decodeURIComponent/S15.1.3.2_A1.2_T2.html 24.00 fast/js/sputnik/Conformance/15_Native_Objects/15.1_The_Global_Object/15.1.3/15.1.3.2_decodeURIComponent/S15.1.3.2_A1.11_T1.html 23.94 fast/js/sputnik/Conformance/15_Native_Objects/15.1_The_Global_Object/15.1.3/15.1.3.1_decodeURI/S15.1.3.1_A1.12_T3.html 23.94 fast/js/sputnik/Conformance/15_Native_Objects/15.1_The_Global_Object/15.1.3/15.1.3.2_decodeURIComponent/S15.1.3.2_A1.11_T2.html 23.94 fast/js/sputnik/Conformance/15_Native_Objects/15.1_The_Global_Object/15.1.3/15.1.3.1_decodeURI/S15.1.3.1_A1.12_T1.html 23.94 fast/js/sputnik/Conformance/15_Native_Objects/15.1_The_Global_Object/15.1.3/15.1.3.1_decodeURI/S15.1.3.1_A1.11_T1.html 23.88 fast/js/sputnik/Conformance/15_Native_Objects/15.1_The_Global_Object/15.1.3/15.1.3.2_decodeURIComponent/S15.1.3.2_A1.12_T1.html 23.88 fast/js/sputnik/Conformance/15_Native_Objects/15.1_The_Global_Object/15.1.3/15.1.3.2_decodeURIComponent/S15.1.3.2_A1.12_T3.html 23.82 fast/js/sputnik/Conformance/15_Native_Objects/15.1_The_Global_Object/15.1.3/15.1.3.2_decodeURIComponent/S15.1.3.2_A1.2_T1.html 23.82 fast/js/sputnik/Conformance/15_Native_Objects/15.1_The_Global_Object/15.1.3/15.1.3.1_decodeURI/S15.1.3.1_A1.2_T1.html 23.65 fast/js/sputnik/Conformance/15_Native_Objects/15.1_The_Global_Object/15.1.3/15.1.3.1_decodeURI/S15.1.3.1_A1.10_T1.html 23.20 http/tests/xmlhttprequest/origin-whitelisting-all.html 23.03 ietestcenter/Javascript/15.4.4.14-10-1.html 22.56 fast/js/sputnik/Conformance/15_Native_Objects/15.1_The_Global_Object/15.1.3/15.1.3.1_decodeURI/S15.1.3.1_A1.11_T2.html 22.46 ietestcenter/Javascript/15.4.4.15-9-1.html 22.37 fast/js/function-apply-many-args.html 20.13 fast/js/function-apply.html 19.50 fast/xmlhttprequest/xmlhttprequest-bad-mimetype.html 19.45 fast/js/sputnik/Unicode/Unicode_500/S7.6_A3.2.html 18.67 fast/js/sputnik/Conformance/15_Native_Objects/15.1_The_Global_Object/15.1.3/15.1.3.4_encodeURIComponent/S15.1.3.4_A1.3_T1.html 18.67 fast/js/sputnik/Conformance/15_Native_Objects/15.1_The_Global_Object/15.1.3/15.1.3.3_encodeURI/S15.1.3.3_A1.3_T1.html 18.53 fast/js/sputnik/Conformance/07_Lexical_Conventions/7.4_Comments/S7.4_A5.html 18.52 fast/js/sputnik/Unicode/Unicode_410/S7.6_A3.2.html 18.21 fast/js/sputnik/Unicode/Unicode_410/S7.6_A3.1.html 18.07 fast/js/toString-and-valueOf-override.html 17.88 fast/js/sputnik/Unicode/Unicode_510/S7.6_A3.2.html 17.78 fast/js/large-expressions.html For example, if fast/js/large-expressions.html takes 1s in a Release build, it takes 17.78s in a Debug build.
Simon Fraser (smfr)
Comment 7 2011-02-03 16:29:42 PST
sputnik :(
Adam Roben (:aroben)
Comment 8 2011-03-17 01:32:56 PDT
It looks like Leopard suffers from this problem, too. Gavin hypothesizes that there is something about the JSVALUE32_64 implementation (which is used in 32-bit builds) that makes it much slower in Debug than Release. Perhaps it has some assertions that are slowing things down a lot.
Adam Roben (:aroben)
Comment 9 2011-03-21 15:37:43 PDT
I did some sunspider comparisons: 64-bit Debug -> 64-bit Debug with assertions disabled: 1.47x as fast 64-bit Debug -> 64-bit Release with assertions enabled: 5.76x as fast The vast majority of the slowdown in Debug mode seems to come from having compiler optimizations turned off. We could use a Release build with assertions enabled to speed up test times on the bots. Maybe that would be just as good as using a Debug build? Some possible downsides: 1) We wouldn't be testing with the system malloc 2) Backtraces from the bots might be harder to interpret
Brent Fulgham
Comment 10 2021-09-21 14:43:57 PDT
Whatever was going on here, it's not really relevant more than a decade later.
Note You need to log in before you can comment on or make changes to this bug.