75024 2011-12-21 11:54:38 -0800 EWS bubble-status error on attachment 120136 2013-01-30 21:19:42 -0800 1 1 1 Unclassified WebKit Tools / Tests 528+ (Nightly build) Unspecified Unspecified RESOLVED DUPLICATE 107184 P2 Normal --- 1 eric alancutter abarth alancutter ojan simon.fraser oldest_to_newest 525417 0 eric 2011-12-21 11:54:38 -0800 EWS bubble-status error on attachment 120136 Not sure what's going on yet. Traceback (most recent call last): File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/_webapp25.py", line 701, in __call__ handler.get(*groups) File "/base/data/home/apps/webkit-commit-queue/1.353964625221040786/handlers/statusbubble.py", line 78, in get bubbles, show_submit_to_ews = self._build_bubbles_for_attachment(attachment) File "/base/data/home/apps/webkit-commit-queue/1.353964625221040786/handlers/statusbubble.py", line 62, in _build_bubbles_for_attachment if not self._have_status_for(attachment, queue): File "/base/data/home/apps/webkit-commit-queue/1.353964625221040786/handlers/statusbubble.py", line 56, in _have_status_for return bool(queue.is_ews() and attachment.status_for_queue(queue)) File "/base/data/home/apps/webkit-commit-queue/1.353964625221040786/model/attachment.py", line 95, in status_for_queue queue_summary = self.summary().get(queue.name_with_underscores()) File "/base/data/home/apps/webkit-commit-queue/1.353964625221040786/model/attachment.py", line 73, in summary memcache.set(str(self.id), self._summary, namespace="attachment-summary") File "/base/python_runtime/python_lib/versions/1/google/appengine/api/memcache/__init__.py", line 767, in set namespace=namespace) File "/base/python_runtime/python_lib/versions/1/google/appengine/api/memcache/__init__.py", line 872, in _set_with_policy time, '', namespace) File "/base/python_runtime/python_lib/versions/1/google/appengine/api/memcache/__init__.py", line 951, in _set_multi_async_with_policy stored_value, flags = _validate_encode_value(value, self._do_pickle) File "/base/python_runtime/python_lib/versions/1/google/appengine/api/memcache/__init__.py", line 236, in _validate_encode_value 'received %d bytes' % (MAX_VALUE_SIZE, len(stored_value))) ValueError: Values may not be more than 1000000 bytes in length; received 1052506 bytes 525432 1 abarth 2011-12-21 12:03:50 -0800 its one of the zombie queues. 525436 2 eric 2011-12-21 12:07:01 -0800 Maybe it finally has so many attachemnts queud that it's killing the system? 525504 3 abarth 2011-12-21 13:11:55 -0800 Yes. I cleaned it out manually before. We just need to nuke the cr-mac-ews. 590302 4 eric 2012-03-28 12:50:00 -0700 I understand the error now. It occurs more frequently as of late now that we've added more queues to the system. The problem is that the Attachment class wrongly caches the full status message text for each queue for their most recent status message in its "summary" object. The summary* methods have big FIXMEs next to them about how they're bad design, but now that design is coming back to haunt us. The error is temporary and goes away once queues start to report back with their final "PASS" or "FAIL" message. We've always had this bug, it just has started happening way more often now that we have more queues, and thus the "Processing ... attachment" or "Validating Changelog... " or whatever long-ish in-progress messages add up to exceed the memcache limit very quickly. 590331 5 eric 2012-03-28 13:15:11 -0700 http://trac.webkit.org/browser/trunk/Tools/QueueStatusServer/model/attachment.py#L114 http://trac.webkit.org/browser/trunk/Tools/QueueStatusServer/model/attachment.py#L93 810147 6 alancutter 2013-01-17 19:58:09 -0800 Another case of this cropped up today. https://webkit-commit-queue.appspot.com/status-bubble/183293 > Traceback (most recent call last): > File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/_webapp25.py", line 714, in __call__ > handler.get(*groups) > File "/base/data/home/apps/webkit-commit-queue/1.364503684603662849/handlers/statusbubble.py", line 78, in get > bubbles, show_submit_to_ews = self._build_bubbles_for_attachment(attachment) > File "/base/data/home/apps/webkit-commit-queue/1.364503684603662849/handlers/statusbubble.py", line 62, in _build_bubbles_for_attachment > if not self._have_status_for(attachment, queue): > File "/base/data/home/apps/webkit-commit-queue/1.364503684603662849/handlers/statusbubble.py", line 56, in _have_status_for > return bool(queue.is_ews() and attachment.status_for_queue(queue)) > File "/base/data/home/apps/webkit-commit-queue/1.364503684603662849/model/attachment.py", line 95, in status_for_queue > queue_summary = self.summary().get(queue.name_with_underscores()) > File "/base/data/home/apps/webkit-commit-queue/1.364503684603662849/model/attachment.py", line 73, in summary > memcache.set(str(self.id), self._summary, namespace="attachment-summary") > File "/base/python_runtime/python_lib/versions/1/google/appengine/api/memcache/__init__.py", line 793, in set > namespace=namespace) > File "/base/python_runtime/python_lib/versions/1/google/appengine/api/memcache/__init__.py", line 898, in _set_with_policy > time, '', namespace) > File "/base/python_runtime/python_lib/versions/1/google/appengine/api/memcache/__init__.py", line 977, in _set_multi_async_with_policy > stored_value, flags = _validate_encode_value(value, self._do_pickle) > File "/base/python_runtime/python_lib/versions/1/google/appengine/api/memcache/__init__.py", line 236, in _validate_encode_value > 'received %d bytes' % (MAX_VALUE_SIZE, len(stored_value))) > ValueError: Values may not be more than 1000000 bytes in length; received 1053158 bytes 821054 7 alancutter 2013-01-30 21:19:42 -0800 Bug 107184 is a duplicate of this and has been resolved. *** This bug has been marked as a duplicate of bug 107184 ***