Bug 75024 – EWS bubble-status error on attachment 120136 [details]

• Status: RESOLVED
• Resolution: DUPLICATE
• Component: Tools / Tests

[Eric Seidel (no email)]

EWS bubble-status error on attachment 120136 [details]

Not sure what's going on yet.

Traceback (most recent call last):
  File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/_webapp25.py", line 701, in __call__
    handler.get(*groups)
  File "/base/data/home/apps/webkit-commit-queue/1.353964625221040786/handlers/statusbubble.py", line 78, in get
    bubbles, show_submit_to_ews = self._build_bubbles_for_attachment(attachment)
  File "/base/data/home/apps/webkit-commit-queue/1.353964625221040786/handlers/statusbubble.py", line 62, in _build_bubbles_for_attachment
    if not self._have_status_for(attachment, queue):
  File "/base/data/home/apps/webkit-commit-queue/1.353964625221040786/handlers/statusbubble.py", line 56, in _have_status_for
    return bool(queue.is_ews() and attachment.status_for_queue(queue))
  File "/base/data/home/apps/webkit-commit-queue/1.353964625221040786/model/attachment.py", line 95, in status_for_queue
    queue_summary = self.summary().get(queue.name_with_underscores())
  File "/base/data/home/apps/webkit-commit-queue/1.353964625221040786/model/attachment.py", line 73, in summary
    memcache.set(str(self.id), self._summary, namespace="attachment-summary")
  File "/base/python_runtime/python_lib/versions/1/google/appengine/api/memcache/__init__.py", line 767, in set
    namespace=namespace)
  File "/base/python_runtime/python_lib/versions/1/google/appengine/api/memcache/__init__.py", line 872, in _set_with_policy
    time, '', namespace)
  File "/base/python_runtime/python_lib/versions/1/google/appengine/api/memcache/__init__.py", line 951, in _set_multi_async_with_policy
    stored_value, flags = _validate_encode_value(value, self._do_pickle)
  File "/base/python_runtime/python_lib/versions/1/google/appengine/api/memcache/__init__.py", line 236, in _validate_encode_value
    'received %d bytes' % (MAX_VALUE_SIZE, len(stored_value)))
ValueError: Values may not be more than 1000000 bytes in length; received 1052506 bytes

[Adam Barth]

its one of the zombie queues.

[Eric Seidel (no email)]

Maybe it finally has so many attachemnts queud that it's killing the system?

[Adam Barth]

Yes.  I cleaned it out manually before.  We just need to nuke the cr-mac-ews.

[Eric Seidel (no email)]

I understand the error now.  It occurs more frequently as of late now that we've added more queues to the system.

The problem is that the Attachment class wrongly caches the full status message text for each queue for their most recent status message in its "summary" object.  The summary* methods have big FIXMEs next to them about how they're bad design, but now that design is coming back to haunt us.

The error is temporary and goes away once queues start to report back with their final "PASS" or "FAIL" message.  We've always had this bug, it just has started happening way more often now that we have more queues, and thus the "Processing ... attachment" or "Validating Changelog... " or whatever long-ish in-progress messages add up to exceed the memcache limit very quickly.

[Eric Seidel (no email)]

http://trac.webkit.org/browser/trunk/Tools/QueueStatusServer/model/attachment.py#L114
http://trac.webkit.org/browser/trunk/Tools/QueueStatusServer/model/attachment.py#L93

[Alan Cutter]

Another case of this cropped up today.
https://webkit-commit-queue.appspot.com/status-bubble/183293

> Traceback (most recent call last):
>   File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/_webapp25.py", line 714, in __call__
>     handler.get(*groups)
>   File "/base/data/home/apps/webkit-commit-queue/1.364503684603662849/handlers/statusbubble.py", line 78, in get
>     bubbles, show_submit_to_ews = self._build_bubbles_for_attachment(attachment)
>   File "/base/data/home/apps/webkit-commit-queue/1.364503684603662849/handlers/statusbubble.py", line 62, in _build_bubbles_for_attachment
>     if not self._have_status_for(attachment, queue):
>   File "/base/data/home/apps/webkit-commit-queue/1.364503684603662849/handlers/statusbubble.py", line 56, in _have_status_for
>     return bool(queue.is_ews() and attachment.status_for_queue(queue))
>   File "/base/data/home/apps/webkit-commit-queue/1.364503684603662849/model/attachment.py", line 95, in status_for_queue
>     queue_summary = self.summary().get(queue.name_with_underscores())
>   File "/base/data/home/apps/webkit-commit-queue/1.364503684603662849/model/attachment.py", line 73, in summary
>     memcache.set(str(self.id), self._summary, namespace="attachment-summary")
>   File "/base/python_runtime/python_lib/versions/1/google/appengine/api/memcache/__init__.py", line 793, in set
>     namespace=namespace)
>   File "/base/python_runtime/python_lib/versions/1/google/appengine/api/memcache/__init__.py", line 898, in _set_with_policy
>     time, '', namespace)
>   File "/base/python_runtime/python_lib/versions/1/google/appengine/api/memcache/__init__.py", line 977, in _set_multi_async_with_policy
>     stored_value, flags = _validate_encode_value(value, self._do_pickle)
>   File "/base/python_runtime/python_lib/versions/1/google/appengine/api/memcache/__init__.py", line 236, in _validate_encode_value
>     'received %d bytes' % (MAX_VALUE_SIZE, len(stored_value)))
> ValueError: Values may not be more than 1000000 bytes in length; received 1053158 bytes

[Alan Cutter]

Bug 107184 is a duplicate of this and has been resolved.

*** This bug has been marked as a duplicate of bug 107184 ***