75499 2012-01-03 15:17:15 -0800 Test results server is repeating runs 2012-10-12 11:32:29 -0700 1 1 1 Unclassified WebKit Tools / Tests 528+ (Nightly build) Unspecified Unspecified RESOLVED DUPLICATE 97643 P2 Normal --- 1 ojan webkit-unassigned jparent tony oldest_to_newest 529092 0 ojan 2012-01-03 15:17:15 -0800 http://test-results.appspot.com/dashboards/flakiness_dashboard.html#tests=svg%2Fhixie%2Ftext%2F003.html See for example the Webkit Mac10.6 (CG)(dbg) row. Each set of data for a run is repeated five times. The problem is that the data is successfully merging, but appengine is returning a 500 because we're exceeding the memory limit, so we retry and the same problem happens again. (see the 500s at the end http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac10.6%20%28CG%29%28dbg%29/builds/2310/steps/webkit_tests/logs/stdio) As best I can tell here is the corresponding 500: 2012-01-03 13:48:34.411 /testfile/upload 500 39738ms 0kb Python-urllib/2.6 See details 74.125.248.75 - - [03/Jan/2012:13:48:34 -0800] "POST /testfile/upload HTTP/1.1" 500 0 - "Python-urllib/2.6" "test-results.appspot.com" ms=39739 cpu_ms=31712 api_cpu_ms=398 cpm_usd=0.993279 exit_code=105 instance=00c61b117c4a2afa8703f545ea5fa594fcd8e5d9 C 2012-01-03 13:48:34.387 Exceeded soft private memory limit with 156.695 MB after servicing 7 requests total W 2012-01-03 13:48:34.387 While handling this request, the process that handled this request was found to be using too much memory and was terminated. This is likely to cause a new process to be used for the next request to your application. If you see this message frequently, you may have a memory leak in your application. The long-term solution to this is to change the JSON data format to use less memory (e.g. use a single string as a run-length-encoding format, store data chunked by 100 runs at a time, etc). In the short-term we can add a check to make sure that the run currently being uploaded wasn't just uploaded. 529736 1 ojan 2012-01-04 14:04:38 -0800 Got a better log of the error. We're hitting AppEngine's request deadline. Move this to a TaskQueue or using an AppEngine Backend would fix this. 2012-01-04 13:18:13.510 /testfile/upload 500 78616ms 0kb Python-urllib/2.6 D 2012-01-04 13:17:20.669 Processing upload request, master: ChromiumWebkit, builder: Webkit Mac10.6 (CG)(dbg), test_type: layout-tests. D 2012-01-04 13:17:20.669 test: [FieldStorage(u'file', u'incremental_results.json'), FieldStorage(u'file', u'full_results.json'), FieldStorage(u'file', u'times_ms.json')], type I 2012-01-04 13:17:20.682 Loading data for key: agx0ZXN0LXJlc3VsdHNyEgsSCURhdGFFbnRyeRidosMBDA. I 2012-01-04 13:17:20.712 Loading incremental json... I 2012-01-04 13:17:21.836 Checking incremental json... I 2012-01-04 13:17:21.836 Loading existing aggregated json... I 2012-01-04 13:17:24.210 Checking existing aggregated json... I 2012-01-04 13:17:24.211 Merging json results... D 2012-01-04 13:17:24.211 Merging build 2329, incremental json index: 0. I 2012-01-04 13:17:27.142 Data saved: agx0ZXN0LXJlc3VsdHNyEgsSCURhdGFFbnRyeRiO6cIBDA. I 2012-01-04 13:17:27.167 Loading data for key: agx0ZXN0LXJlc3VsdHNyEgsSCURhdGFFbnRyeRiQ6cIBDA. I 2012-01-04 13:17:27.326 Loading data for key: agx0ZXN0LXJlc3VsdHNyEgsSCURhdGFFbnRyeRiCtOcBDA. I 2012-01-04 13:17:27.419 Loading data for key: agx0ZXN0LXJlc3VsdHNyEgsSCURhdGFFbnRyeRiVuoACDA. I 2012-01-04 13:17:27.486 Loading data for key: agx0ZXN0LXJlc3VsdHNyEgsSCURhdGFFbnRyeRil-ucCDA. I 2012-01-04 13:17:27.561 Loading data for key: agx0ZXN0LXJlc3VsdHNyEgsSCURhdGFFbnRyeRi1uJYDDA. I 2012-01-04 13:17:27.577 Loading incremental json... I 2012-01-04 13:17:28.716 Checking incremental json... I 2012-01-04 13:17:28.716 Loading existing aggregated json... I 2012-01-04 13:17:51.448 Checking existing aggregated json... I 2012-01-04 13:17:51.474 Merging json results... D 2012-01-04 13:17:51.474 Merging build 2329, incremental json index: 0. E 2012-01-04 13:18:13.494 <class 'google.appengine.runtime.DeadlineExceededError'>: Traceback (most recent call last): File "/base/data/home/apps/test-results/r103983.355847 C 2012-01-04 13:18:13.495 Exceeded soft private memory limit with 142.629 MB after servicing 1 requests total I 2012-01-04 13:18:13.495 This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This requ W 2012-01-04 13:18:13.495 A serious problem was encountered with the process that handled this request, causing it to exit. This is likely to cause a new process to be used for 741057 2 ojan 2012-10-12 11:32:29 -0700 *** This bug has been marked as a duplicate of bug 97643 ***