Tools/ChangeLog

112018-06-19 Daniel Bates <dabates@apple.com>
22
 3 Implement EWS real-time patch status updates and tail logging
 4 https://bugs.webkit.org/show_bug.cgi?id=186823
 5
 6 Reviewed by NOBODY (OOPS!).
 7
 8 Makes the patch status page (e.g. https://webkit-queues.webkit.org/patch/342789/mac-ews) display
 9 real-time status updates, including a tail(1) of the standard output and standard error streams
 10 from the bot as it runs commands to build and test a patch. You can either access the patch status
 11 page directly using a URL similar to the example one above or, even easier, just click on the EWS
 12 bubble of the queue you want to view on the Bugzilla bug page the patch was uploaded to.
 13
 14 Real-time updates are written to a TailLog, which is implemented using the AppEngine Channel API.
 15 Each TailLog stores the Channel API token (used by the patch status page to connect to the back-end)
 16 in memcache keyed off the queue name and attachment id. We use memcache to store this association
 17 instead of storing it in the database to complement the transient nature of the Channel API. Memcache
 18 may be flushed or items may be evicted from it at any time.
 19
 20 A side effect of this change this that the patch status patch now lists events in chronological
 21 order from oldest to newest. Currently it lists status updates in reverse chronological order.
 22 The change in ordering was made so that the standard output and standard error streams are
 23 written as a person would expect in a terminal.
 24
 25 * QueueStatusServer/handlers/patch.py:
 26 (Patch.get): Always order the status messages in chronological order from oldest to newest.
 27 If we have an existing tail log for the specified queue and attachment then include its token
 28 as a template variable so that the rendered web page will be able to connect to the stream.
 29 * QueueStatusServer/handlers/releaselock.py:
 30 (ReleaseLock.post): Expire the tail log when we release the lock for the patch.
 31 * QueueStatusServer/handlers/releasepatch.py:
 32 (ReleasePatch.post): Expire the tail log when the patch is released from the queue.
 33 * QueueStatusServer/handlers/updatestatus.py:
 34 (UpdateStatus.post): Create a tail log for the specified queue and attachment if one does not
 35 already exist and write the status details to it. We only persist the status details to the
 36 database (disk) if the update is not is transient. Transient updates are only written to the
 37 tail log and are best used for data that is unnecessary to keep around. We use transient
 38 update for streamed standard output and standard error text because the value of such text
 39 is time-sensitive (usually only valuable while iterating on a patch before it lands) and
 40 the output can be excessive (tens of MBs). EWS will persist a trailing portion of these
 41 streams to the database if a patch fails to build of causes tests failures.
 42 * QueueStatusServer/index.yaml: Remove the sort order from the database index.
 43 * QueueStatusServer/model/taillog.py: Added.
 44 (TailLog):
 45 (TailLog.__init__):
 46 (TailLog.lookup_or_create): Query memcache for an existing channel token. Otherwise, create
 47 a new channel and store the token in memcache.
 48 (TailLog.lookup): Query memcache for the channel token.
 49 (TailLog.expire): Remove the key from memcache.
 50 (TailLog.write_message): Writes a message to the channel.
 51 (TailLog._generate_key): Compute a key from the name of the queue and the attachment id.
 52 * QueueStatusServer/stylesheets/common.css:
 53 (.code): Stylize transient messages using a monospace font to resemble console output.
 54 * QueueStatusServer/templates/patch.html: Connect to to the underlying Channel API to stream
 55 tail log updates.
 56 * QueueStatusServer/templates/updatestatus.html: Add a checkbox as to whether the update is
 57 considered transient and should only appear in the tail log.
 58 * Scripts/webkitpy/common/net/statusserver.py:
 59 (StatusServer._post_status_to_server): Modified to take a boolean, is_transient, as to whether
 60 the status is considered transient.
 61 (StatusServer.update_status): Ditto.
 62
 63 (WritableStatusServerStatusFileObject):
 64 (WritableStatusServerStatusFileObject.__init__):
 65 (WritableStatusServerStatusFileObject.__eq__):
 66 (WritableStatusServerStatusFileObject.write):
 67 File-like object that forwards writes to the status
 68 server as transient status updates.
 69
 70 * Scripts/webkitpy/common/net/statusserver_mock.py:
 71 (MockStatusServer.update_status):
 72 * Scripts/webkitpy/common/net/web_mock.py:
 73 (MockBrowser.find_control): Added.
 74 * Scripts/webkitpy/common/system/executive.py:
 75 (Executive.run_and_throw_if_fail): Extract out logic into run_with_teed_output_and_throw_if_fail()
 76 and write this function in terms of it.
 77 (Executive.run_with_teed_output_and_throw_if_fail): Extracted from Executive.run_and_throw_if_fail().
 78 * Scripts/webkitpy/common/system/executive_mock.py:
 79 (MockExecutive.run_with_teed_output_and_throw_if_fail): Added.
 80
 81 * Scripts/webkitpy/tool/bot/commitqueuetask_unittest.py:
 82 (MockCommitQueue.run_command):
 83 (FailingTestCommitQueue.run_command):
 84 (MockSimpleTestPlanCommitQueue.run_command):
 85 * Scripts/webkitpy/tool/bot/patchanalysistask.py:
 86 (PatchAnalysisTaskDelegate.run_command):
 87 (PatchAnalysisTask._run_command):
 88 * Scripts/webkitpy/tool/commands/earlywarningsystem.py:
 89 (AbstractEarlyWarningSystem.run_command):
 90 Pass the Attachment object through so that we can log the output of the command with respect to a particular patch.
 91
 92 * Scripts/webkitpy/tool/commands/earlywarningsystem_unittest.py:
 93 (EarlyWarningSystemTest._default_expected_logs): Update expected result now that we send updates for transient events.
 94 * Scripts/webkitpy/tool/commands/perfalizer.py:
 95 (PerfalizerTask.run_command): Pass the Attachment object through so that we can log the output of the command with
 96 respect to a particular patch.
 97 * Scripts/webkitpy/tool/commands/queues.py:
 98 (AbstractQueue.run_webkit_patch):
 99 (CommitQueue.run_command): Pass the Attachment object through so that we can log the output of the command with respect
 100 to a particular patch.
 101 (StyleQueue.run_command): Ditto.
 102 * Scripts/webkitpy/tool/commands/queues_unittest.py:
 103 (AbstractQueueTest._assert_run_webkit_patch): Update test now that we send updates for transient events.
 104
 1052018-06-19 Daniel Bates <dabates@apple.com>
 106
3107 EWS for security bugs
4108 https://bugs.webkit.org/show_bug.cgi?id=186291
5109 <rdar://problem/40829658>

Tools/QueueStatusServer/handlers/patch.py

2929from google.appengine.ext import webapp
3030from google.appengine.ext.webapp import template
3131
 32from model.taillog import TailLog
3233from model.queuestatus import QueueStatus
3334
3435
3536class Patch(webapp.RequestHandler):
3637 def get(self, attachment_id_string, queue_name=None):
3738 attachment_id = int(attachment_id_string)
38  statuses = QueueStatus.all().filter("active_patch_id =", attachment_id).order("-date")
 39 statuses = QueueStatus.all().filter("active_patch_id =", attachment_id).order("date")
3940
4041 bug_id = None
4142 queue_status = {}

@@class Patch(webapp.RequestHandler):
5152 "bug_id" : bug_id,
5253 "queue_status" : queue_status,
5354 }
 55 tail_log = TailLog.lookup(queue_name, attachment_id) if queue_name else None
 56 if tail_log:
 57 template_values["tail_log_token"] = tail_log.token
5458 self.response.out.write(template.render("templates/patch.html", template_values))

Tools/QueueStatusServer/handlers/releaselock.py

@@from google.appengine.ext.webapp import template
2727from config.queues import work_item_lock_timeout
2828from handlers.updatebase import UpdateBase
2929from model.queues import Queue
 30from model.taillog import TailLog
3031
3132
3233class ReleaseLock(UpdateBase):

@@class ReleaseLock(UpdateBase):
4344
4445 attachment_id = self._int_from_request("attachment_id")
4546 queue.active_work_items().expire_item(attachment_id)
 47 TailLog.expire(queue_name, attachment_id)
4648
4749 # ReleaseLock is used when a queue neither succeeded nor failed, so it silently releases the patch.
4850 # Let's try other patches before retrying this one, in the interest of fairness, and also because

Tools/QueueStatusServer/handlers/releasepatch.py

@@from loggers.recordpatchevent import RecordPatchEvent
3434from model.attachment import Attachment
3535from model.attachmentdata import AttachmentData
3636from model.queues import Queue
 37from model.taillog import TailLog
3738
3839
3940class ReleasePatch(UpdateBase):

@@class ReleasePatch(UpdateBase):
6061 # WorkItems and ActiveWorkItems.
6162
6263 queue.work_items().remove_work_item(attachment_id)
 64 TailLog.expire(queue_name, attachment_id)
6365 RecordPatchEvent.stopped(attachment_id, queue_name, last_status.message)
6466
6567 queue.active_work_items().expire_item(attachment_id)

Tools/QueueStatusServer/handlers/updatestatus.py

@@from loggers.recordbotevent import RecordBotEvent
3535from loggers.recordpatchevent import RecordPatchEvent
3636from model.attachment import Attachment
3737from model.queuestatus import QueueStatus
 38from model.taillog import TailLog
3839
3940
4041class UpdateStatus(UpdateBase):

@@class UpdateStatus(UpdateBase):
6364
6465 def post(self):
6566 queue_status = self._queue_status_from_request()
 67 tail_log = None
 68 if queue_status.queue_name and queue_status.active_patch_id:
 69 tail_log = TailLog.lookup_or_create(queue_status.queue_name, queue_status.active_patch_id)
 70 if tail_log and bool(self.request.get("is_transient")):
 71 tail_log.write_message(queue_status.message, queue_status.bot_id, is_transient=True)
 72 self.response.out.write("-1")
 73 return
6674 queue_status.put()
 75 if tail_log:
 76 tail_log.write_message(queue_status.message, queue_status.bot_id, time=queue_status.date, results_file_id=(queue_status.key().id() if queue_status.results_file else None))
6777 RecordBotEvent.record_activity(queue_status.queue_name, queue_status.bot_id)
6878 if queue_status.active_patch_id:
6979 RecordPatchEvent.updated(queue_status.active_patch_id, queue_status.queue_name, queue_status.message, queue_status.bot_id)

Tools/QueueStatusServer/index.yaml

@@indexes:
1717 properties:
1818 - name: active_patch_id
1919 - name: date
20  direction: desc
2120
2221- kind: QueueStatus
2322 properties:

Tools/QueueStatusServer/model/taillog.py

 1# Copyright (C) 2018 Apple Inc. All rights reserved.
 2#
 3# Redistribution and use in source and binary forms, with or without
 4# modification, are permitted provided that the following conditions
 5# are met:
 6# 1. Redistributions of source code must retain the above copyright
 7# notice, this list of conditions and the following disclaimer.
 8# 2. Redistributions in binary form must reproduce the above copyright
 9# notice, this list of conditions and the following disclaimer in the
 10# documentation and/or other materials provided with the distribution.
 11#
 12# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND ANY
 13# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
 14# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
 15# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR ANY
 16# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
 17# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
 18# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
 19# ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
 20# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
 21# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 22
 23import json
 24
 25from datetime import datetime
 26from google.appengine.api import channel
 27from google.appengine.api import memcache
 28
 29
 30class TailLog(object):
 31 _MEMCACHE_PREFIX = "tail-log"
 32
 33 def __init__(self, key, token):
 34 self._key = key
 35 self.token = token
 36
 37 @classmethod
 38 def lookup_or_create(cls, queue_name, attachment_id):
 39 key = cls._generate_key(queue_name, attachment_id)
 40 token = memcache.get(key)
 41 if token:
 42 return cls(key, token)
 43 # Three hours should be a reasonable amount of time to build and test a WebKit change.
 44 token = channel.create_channel(key, duration_minutes=3 * 60)
 45 memcache.set(key, token)
 46 return cls(key, token)
 47
 48 @classmethod
 49 def lookup(cls, queue_name, attachment_id):
 50 key = cls._generate_key(queue_name, attachment_id)
 51 token = memcache.get(key)
 52 if token:
 53 return cls(key, token)
 54 return None
 55
 56 @classmethod
 57 def expire(cls, queue_name, attachment_id):
 58 key = cls._generate_key(queue_name, attachment_id)
 59 if not memcache.get(key):
 60 return
 61 memcache.delete(key)
 62
 63 def write_message(self, message, bot_id, time=datetime.now(), results_file_id=None, is_transient=False):
 64 if not memcache.get(self._key):
 65 return False
 66 message_dict = {
 67 "message": message,
 68 "timestamp": time,
 69 "bot-id": bot_id,
 70 "is-transient": bool(is_transient),
 71 }
 72 if results_file_id:
 73 message_dict["results-file-id"] = results_file_id
 74 dthandler = lambda obj: obj.isoformat() + "Z" if isinstance(obj, datetime) or isinstance(obj, datetime.date) else None
 75 channel.send_message(self._key, json.dumps(message_dict, default=dthandler))
 76 return True
 77
 78 @classmethod
 79 def _generate_key(cls, queue_name, attachment_id):
 80 return "-".join([cls._MEMCACHE_PREFIX, queue_name, str(attachment_id)])

Tools/QueueStatusServer/stylesheets/common.css

@@tr:hover, li:hover {
4545 background-color: #EEE;
4646}
4747
 48.code {
 49 font-family: monospace;
 50}
 51
4852.status-group {
4953 font-size: 90%;
5054}

Tools/QueueStatusServer/templates/patch.html

22<html>
33<head>
44<title>Patch Status</title>
5 <link type="text/css" rel="stylesheet" href="/stylesheets/common.css" />
 5<link rel="stylesheet" href="/stylesheets/common.css">
 6<script src="/_ah/channel/jsapi"></script>
 7<script>
 8const tailLogToken = "{{ tail_log_token }}";
 9var statusLogElement;
 10var channel;
 11
 12function buildLogLine(messageData)
 13{
 14 let statusItem = document.createElement("li");
 15
 16 let statusBotId = document.createElement("span");
 17 statusItem.appendChild(statusBotId);
 18 statusBotId.className = "status-bot"
 19 statusBotId.textContent = messageData["bot-id"];
 20 statusBotId.innerHTML += "&nbsp;";
 21
 22 let statusMessage = document.createElement("span");
 23 statusItem.appendChild(statusMessage);
 24 statusMessage.className = "status-message";
 25 if (messageData["is-transient"])
 26 statusMessage.classList.add("code");
 27 statusMessage.textContent = messageData["message"];
 28
 29 if (messageData["results-file-id"]) {
 30 let statusResult = document.createElement("span");
 31 statusItem.appendChild(statusResult);
 32 statusResult.className = "status-results";
 33
 34 statusResult.appendChild(document.createTextNode("["));
 35 let statusResultLink = document.createElement("a");
 36 statusResult.appendChild(statusResultLink);
 37 statusResultLink.href = "/results/" + messageData["results-file-id"];
 38 statusResultLink.textContent = "results";
 39 statusResult.appendChild(document.createTextNode("]"));
 40 }
 41
 42 let statusDate = document.createElement("span");
 43 statusItem.appendChild(statusDate);
 44 statusDate.className = "status-date";
 45 statusDate.textContent = new Date(messageData["timestamp"]).toLocaleString();
 46
 47 return statusItem;
 48}
 49
 50function buildErrorLogLine(message)
 51{
 52 let statusItem = document.createElement("li");
 53 statusItem.textContent = message;
 54 statusItem.innerHTML += "&nbsp;";
 55
 56 statusItem.appendChild(document.createTextNode("["))
 57 let reloadLink = document.createElement("a");
 58 statusItem.appendChild(reloadLink);
 59 reloadLink.href="javascript:reload()";
 60 reloadLink.textContent = "reload";
 61 statusItem.appendChild(document.createTextNode("]"))
 62
 63 return statusItem;
 64}
 65
 66function reload()
 67{
 68 if (confirm("Are you sure you want to reload?\n\nAll monospace-typed messages will be lost.")) {
 69 window.location.reload();
 70 }
 71}
 72
 73function tailLog()
 74{
 75 if (!tailLogToken)
 76 return;
 77 channel = new goog.appengine.Channel(tailLogToken);
 78 let handlers = {
 79 "onmessage": (messageEvent) => { statusLogElement.appendChild(buildLogLine(JSON.parse(messageEvent.data))); },
 80 "onclose": () => { statusLogElement.appendChild(buildErrorLogLine("Disconnected.")); },
 81 "onerror": (errorEvent) => { statusLogElement.appendChild(buildErrorLogLine(`Error: ${errorEvent.description} (${errorEvent.code}).`)); },
 82 }
 83 channel.open(handlers);
 84}
 85
 86window.onload = function ()
 87{
 88 statusLogElement = document.getElementById("status-log");
 89 tailLog()
 90}
 91</script>
692</head>
793<body>
894<h1>

14100{% for queue_name, statuses in queue_status %}
15101<div class="status-details">
16102 <h2>{{ queue_name }}</h2>
17  <ul>{% for status in statuses %}
 103 <ul id="status-log">{% for status in statuses %}
18104 <li>
19  <span class="status-bot">{{ status.bot_id }}&nbsp</span>
 105 <span class="status-bot">{{ status.bot_id }}&nbsp;</span>
20106 <span class="status-message">{{ status.message|force_escape|urlize|webkit_linkify|safe }}</span>{% if status.results_file %}
21107 <span class="status-results">[{{ status.key.id|results_link|safe }}]</span>{% endif %}
22108 <span class="status-date">{{ status.date|timesince }} ago</span>

Tools/QueueStatusServer/templates/updatestatus.html

@@Update status for a queue: <input name="queue_name">
1212 Active Patch Id:
1313 <input name="patch_id">
1414 </div>
 15 <div>
 16 Is transient (only show in tail log):
 17 <input name="is_transient" type="checkbox" value="1">
 18 </div>
1519 <div>
16  Status Text:<br>
17  <textarea name="status" rows="3" cols="60"></textarea>
 20 Status Text:<br>
 21 <textarea name="status" rows="3" cols="60"></textarea>
1822 </div>
1923 <div>Results file: <input type="file" name="results_file"></div>
2024 <div><input type="submit" value="Add Status"></div>

Tools/Scripts/webkitpy/common/net/statusserver.py

@@class StatusServer:
9797 _log.warn("Attempted to set %s to value exceeding %s characters, truncating." % (field_name, limit))
9898 self._browser[field_name] = value[:limit]
9999
100  def _post_status_to_server(self, queue_name, status, patch, results_file):
 100 def _post_status_to_server(self, queue_name, status, patch, results_file, is_transient):
101101 if results_file:
102102 # We might need to re-wind the file if we've already tried to post it.
103103 results_file.seek(0)

@@class StatusServer:
106106 self._browser.open(update_status_url)
107107 self._browser.select_form(name="update_status")
108108 self._browser["queue_name"] = queue_name
 109 self._browser.find_control('is_transient').items[0].selected = bool(is_transient)
109110 if self.bot_id:
110111 self._browser["bot_id"] = self.bot_id
111112 self._add_patch(patch)

@@class StatusServer:
192193 _log.info("Recording work items: %s for %s" % (high_priority_work_items + work_items, queue_name))
193194 return NetworkTransaction().run(lambda: self._post_work_items_to_server(queue_name, high_priority_work_items, work_items))
194195
195  def update_status(self, queue_name, status, patch=None, results_file=None):
 196 def update_status(self, queue_name, status, patch=None, results_file=None, is_transient=False):
196197 _log.info(status)
197  return NetworkTransaction().run(lambda: self._post_status_to_server(queue_name, status, patch, results_file))
 198 return NetworkTransaction().run(lambda: self._post_status_to_server(queue_name, status, patch, results_file, is_transient))
198199
199200 def update_svn_revision(self, svn_revision_number, broken_bot):
200201 _log.info("SVN revision: %s broke %s" % (svn_revision_number, broken_bot))

@@class StatusServer:
235236 def svn_revision(self, svn_revision_number):
236237 svn_revision_url = '{}/svn-revision/{}'.format(self._server_url(), svn_revision_number)
237238 return self._fetch_url(svn_revision_url)
 239
 240
 241class WritableStatusServerStatusFileObject(object):
 242 def __init__(self, status_server, queue_name, patch):
 243 self._status_server = status_server
 244 self._queue_name = queue_name
 245 self._patch = patch
 246
 247 # For unit testing
 248 def __eq__(self, other):
 249 return self._status_server == other._status_server and self._queue_name == other._queue_name and self._patch == other._patch
 250
 251 # Callers should pass an already encoded string for writing.
 252 def write(self, bytes):
 253 self._status_server.update_status(self._queue_name, bytes, patch=self._patch, is_transient=True)

Tools/Scripts/webkitpy/common/net/statusserver_mock.py

@@class MockStatusServer(object):
6767 def submit_to_ews(self, patch_id):
6868 _log.info("MOCK: submit_to_ews: %s" % (patch_id))
6969
70  def update_status(self, queue_name, status, patch=None, results_file=None):
71  _log.info("MOCK: update_status: %s %s" % (queue_name, status))
 70 def update_status(self, queue_name, status, patch=None, results_file=None, is_transient=False):
 71 _log.info('MOCK: update_status: {}{} {}'.format(('(transient) ' if is_transient else ''), queue_name, status))
7272 return 187
7373
7474 def update_svn_revision(self, svn_revision, broken_bot):

Tools/Scripts/webkitpy/common/net/web_mock.py

2929import StringIO
3030import urllib2
3131
 32from webkitpy.thirdparty.mock import Mock
 33
 34
3235class MockWeb(object):
3336 def __init__(self, urls=None, responses=[]):
3437 self.urls = urls or {}

@@class MockBrowser(object):
8386
8487 def set_handle_robots(self, value):
8588 pass
 89
 90 def find_control(self, name):
 91 control = Mock()
 92 control.items = [Mock()]
 93 return control

Tools/Scripts/webkitpy/common/system/executive.py

@@class Executive(AbstractExecutive):
125125 # like "build-webkit" where we want to display to the user that we're building
126126 # but still have the output to stuff into a log file.
127127 def run_and_throw_if_fail(self, args, quiet=False, decode_output=True, **kwargs):
128  # Cache the child's output locally so it can be used for error reports.
129  child_out_file = StringIO.StringIO()
130128 tee_stdout = sys.stdout
131129 if quiet:
132  dev_null = open(os.devnull, "w") # FIXME: Does this need an encoding?
 130 dev_null = open(os.devnull, 'w') # FIXME: Does this need an encoding?
133131 tee_stdout = dev_null
 132 try:
 133 command_output = self.run_with_teed_output_and_throw_if_fail(args, tee_stdout, decode_output=decode_output, **kwargs)
 134 finally:
 135 if quiet:
 136 dev_null.close()
 137 return command_output
 138
 139 def run_with_teed_output_and_throw_if_fail(self, args, tee_stdout, decode_output, **kwargs):
 140 # Cache the child's output locally so it can be used for error reports.
 141 child_out_file = StringIO.StringIO()
134142 child_stdout = Tee(child_out_file, tee_stdout)
135143 exit_code = self._run_command_with_teed_output(args, child_stdout, **kwargs)
136  if quiet:
137  dev_null.close()
138144
139145 child_output = child_out_file.getvalue()
140146 child_out_file.close()

Tools/Scripts/webkitpy/common/system/executive_mock.py

@@class MockExecutive(object):
9999 raise ScriptError("Exception for %s" % args, output="MOCK command output")
100100 return "MOCK output of child process"
101101
 102 def run_with_teed_output_and_throw_if_fail(self, args, tee_stdout, quiet=False, cwd=None, env=None):
 103 if self._should_log:
 104 env_string = ""
 105 if env:
 106 env_string = ", env=%s" % env
 107 _log.info("MOCK run_with_teed_output_and_throw_if_fail: %s, cwd=%s%s" % (args, cwd, env_string))
 108 if self._should_throw_when_run.intersection(args):
 109 raise ScriptError("Exception for %s" % args, output="MOCK command output")
 110 return "MOCK output of child process"
 111
102112 def command_for_printing(self, args):
103113 string_args = map(unicode, args)
104114 return " ".join(string_args)

Tools/Scripts/webkitpy/tool/bot/commitqueuetask_unittest.py

@@class MockCommitQueue(CommitQueueTaskDelegate):
4949 self._failure_status_id = 0
5050 self._flaky_tests = []
5151
52  def run_command(self, command):
 52 def run_command(self, command, patch):
5353 _log.info("run_webkit_patch: %s" % command)
5454 if self._error_plan:
5555 error = self._error_plan.pop(0)

@@class FailingTestCommitQueue(MockCommitQueue):
9999 self._test_run_counter = -1 # Special value to indicate tests have never been run.
100100 self._test_failure_plan = test_failure_plan
101101
102  def run_command(self, command):
 102 def run_command(self, command, patch):
103103 if command[0] == "build-and-test":
104104 self._test_run_counter += 1
105  MockCommitQueue.run_command(self, command)
 105 MockCommitQueue.run_command(self, command, patch)
106106
107107 def _mock_test_result(self, testname):
108108 return test_results.TestResult(testname, [test_failures.FailureTextMismatch()])

@@class MockSimpleTestPlanCommitQueue(MockCommitQueue):
130130 self._clean_test_results = [clean_test_failures]
131131 self._current_test_results = []
132132
133  def run_command(self, command):
134  MockCommitQueue.run_command(self, command)
 133 def run_command(self, command, patch):
 134 MockCommitQueue.run_command(self, command, patch)
135135 if command[0] == "build-and-test":
136136 if "--no-clean" in command:
137137 self._current_test_results = self._patch_test_results.pop(0)

Tools/Scripts/webkitpy/tool/bot/patchanalysistask.py

@@class PatchAnalysisTaskDelegate(object):
5555 def parent_command(self):
5656 raise NotImplementedError("subclasses must implement")
5757
58  def run_command(self, command):
 58 def run_command(self, command, patch):
5959 raise NotImplementedError("subclasses must implement")
6060
6161 def command_passed(self, message, patch):

@@class PatchAnalysisTask(object):
9797 if not self.validate():
9898 raise PatchIsNotValid(self._patch, self.error)
9999 try:
100  self._delegate.run_command(command)
 100 self._delegate.run_command(command, patch=self._patch)
101101 self._delegate.command_passed(success_message, patch=self._patch)
102102 return True
103103 except ScriptError as e:

Tools/Scripts/webkitpy/tool/commands/earlywarningsystem.py

@@class AbstractEarlyWarningSystem(AbstractReviewQueue, EarlyWarningSystemTaskDele
128128 def parent_command(self):
129129 return self.name
130130
131  def run_command(self, command):
132  self.run_webkit_patch(command + [self._deprecated_port.flag()] + (['--architecture=%s' % self._port.architecture()] if self._port.architecture() and self._port.did_override_architecture else []))
 131 def run_command(self, command, patch):
 132 self.run_webkit_patch(command + [self._deprecated_port.flag()] + (['--architecture=%s' % self._port.architecture()] if self._port.architecture() and self._port.did_override_architecture else []), patch=patch)
133133
134134 def command_passed(self, message, patch):
135135 self._update_status(message, patch=patch)

Tools/Scripts/webkitpy/tool/commands/earlywarningsystem_unittest.py

@@class EarlyWarningSystemTest(QueuesTest):
132132 }
133133
134134 if ews.should_build:
135  build_line = "Running: webkit-patch --status-host=example.com build --no-clean --no-update --build-style=%(build_style)s --group=%(group)s --port=%(port)s%(architecture)s\nMOCK: update_status: %(name)s Built patch\n" % string_replacements
 135 build_line = """Running: webkit-patch --status-host=example.com build --no-clean --no-update --build-style=%(build_style)s --group=%(group)s --port=%(port)s%(architecture)s
 136MOCK: update_status: (transient) %(name)s Running: webkit-patch --status-host=example.com build --no-clean --no-update --build-style=%(build_style)s --group=%(group)s --port=%(port)s%(architecture)s
 137MOCK: update_status: %(name)s Built patch
 138""" % string_replacements
136139 else:
137140 build_line = ""
138141 string_replacements['build_line'] = build_line
139142
140143 if ews.run_tests:
141  run_tests_line = "Running: webkit-patch --status-host=example.com build-and-test --no-clean --no-update --test --non-interactive --build-style=%(build_style)s --group=%(group)s --port=%(port)s%(architecture)s\nMOCK: update_status: %(name)s Passed tests\n" % string_replacements
 144 run_tests_line = """Running: webkit-patch --status-host=example.com build-and-test --no-clean --no-update --test --non-interactive --build-style=%(build_style)s --group=%(group)s --port=%(port)s%(architecture)s
 145MOCK: update_status: (transient) %(name)s Running: webkit-patch --status-host=example.com build-and-test --no-clean --no-update --test --non-interactive --build-style=%(build_style)s --group=%(group)s --port=%(port)s%(architecture)s
 146MOCK: update_status: %(name)s Passed tests
 147""" % string_replacements
142148 else:
143149 run_tests_line = ""
144150 string_replacements['run_tests_line'] = run_tests_line

@@class EarlyWarningSystemTest(QueuesTest):
153159 "begin_work_queue": self._default_begin_work_queue_logs(ews.name),
154160 "process_work_item": """MOCK: update_status: %(name)s Started processing patch
155161Running: webkit-patch --status-host=example.com clean --port=%(port)s%(architecture)s
 162MOCK: update_status: (transient) %(name)s Running: webkit-patch --status-host=example.com clean --port=%(port)s%(architecture)s
156163MOCK: update_status: %(name)s Cleaned working directory
157164Running: webkit-patch --status-host=example.com update --port=%(port)s%(architecture)s
 165MOCK: update_status: (transient) %(name)s Running: webkit-patch --status-host=example.com update --port=%(port)s%(architecture)s
158166MOCK: update_status: %(name)s Updated working directory
159167Running: webkit-patch --status-host=example.com apply-attachment --no-update --non-interactive 10000 --port=%(port)s%(architecture)s
 168MOCK: update_status: (transient) %(name)s Running: webkit-patch --status-host=example.com apply-attachment --no-update --non-interactive 10000 --port=%(port)s%(architecture)s
160169MOCK: update_status: %(name)s Applied patch
161170Running: webkit-patch --status-host=example.com check-patch-relevance --quiet --group=%(group)s --port=%(port)s%(architecture)s
 171MOCK: update_status: (transient) %(name)s Running: webkit-patch --status-host=example.com check-patch-relevance --quiet --group=%(group)s --port=%(port)s%(architecture)s
162172MOCK: update_status: %(name)s Checked relevance of patch
163173%(build_line)s%(run_tests_line)s%(result_lines)s""" % string_replacements,
164174 "handle_unexpected_error": "Mock error message\n",

Tools/Scripts/webkitpy/tool/commands/perfalizer.py

@@class PerfalizerTask(PatchAnalysisTask):
141141 '--output-json-path', self._json_path(), '--description', description]
142142 return self._tool.executive.run_and_throw_if_fail(perf_test_runner_args, cwd=self._tool.scm().checkout_root)
143143
144  def run_command(self, command):
 144 def run_command(self, command, patch):
145145 self.run_webkit_patch(command)
146146
147147 def command_passed(self, message, patch):

Tools/Scripts/webkitpy/tool/commands/queues.py

@@from StringIO import StringIO
4141from webkitpy.common.config.committervalidator import CommitterValidator
4242from webkitpy.common.config.ports import DeprecatedPort
4343from webkitpy.common.net.bugzilla import Bugzilla, Attachment
 44from webkitpy.common.net.statusserver import WritableStatusServerStatusFileObject
4445from webkitpy.common.system.executive import ScriptError
4546from webkitpy.tool.bot.botinfo import BotInfo
4647from webkitpy.tool.bot.commitqueuetask import CommitQueueTask, CommitQueueTaskDelegate

@@class AbstractQueue(Command, QueueEngineDelegate):
8384 traceback.print_exc()
8485 _log.error("Failed to CC watchers.")
8586
86  def run_webkit_patch(self, args):
 87 def run_webkit_patch(self, args, patch):
8788 webkit_patch_args = [self._tool.path()]
8889 # FIXME: This is a hack, we should have a more general way to pass global options.
8990 # FIXME: We must always pass global options and their value in one argument

@@class AbstractQueue(Command, QueueEngineDelegate):
99100 try:
100101 args_for_printing = list(webkit_patch_args)
101102 args_for_printing[0] = 'webkit-patch' # Printing our path for each log is redundant.
102  _log.info("Running: %s" % self._tool.executive.command_for_printing(args_for_printing))
103  command_output = self._tool.executive.run_command(webkit_patch_args, cwd=self._tool.scm().checkout_root)
 103 message = "Running: %s" % self._tool.executive.command_for_printing(args_for_printing)
 104 _log.info(message)
 105
 106 tee_stdout = WritableStatusServerStatusFileObject(self._tool.status_server, self.name, patch)
 107 tee_stdout.write(message)
 108 command_output = self._tool.executive.run_with_teed_output_and_throw_if_fail(webkit_patch_args, tee_stdout, cwd=self._tool.scm().checkout_root)
104109 except ScriptError as e:
105110 # Make sure the whole output gets printed if the command failed.
106111 _log.error(e.message_with_output(output_limit=None))

@@class CommitQueue(PatchProcessingQueue, StepSequenceErrorHandler, CommitQueueTas
380385
381386 # CommitQueueTaskDelegate methods
382387
383  def run_command(self, command):
384  self.run_webkit_patch(command + [self._deprecated_port.flag()])
 388 def run_command(self, command, patch):
 389 self.run_webkit_patch(command + [self._deprecated_port.flag()], patch=patch)
385390
386391 def command_passed(self, message, patch):
387392 self._update_status(message, patch=patch)

@@class StyleQueue(AbstractReviewQueue, StyleQueueTaskDelegate):
498503
499504 # StyleQueueTaskDelegate methods
500505
501  def run_command(self, command):
502  self.run_webkit_patch(command)
 506 def run_command(self, command, patch):
 507 self.run_webkit_patch(command, patch=patch)
503508
504509 def command_passed(self, message, patch):
505510 self._update_status(message, patch=patch)

Tools/Scripts/webkitpy/tool/commands/queues_unittest.py

@@from webkitpy.common.checkout.scm import CheckoutNeedsUpdate
3333from webkitpy.common.checkout.scm.scm_mock import MockSCM
3434from webkitpy.common.net.layouttestresults import LayoutTestResults
3535from webkitpy.common.net.bugzilla import Attachment
 36from webkitpy.common.net.statusserver import WritableStatusServerStatusFileObject
3637from webkitpy.common.system.outputcapture import OutputCapture
3738from webkitpy.layout_tests.models import test_results
3839from webkitpy.layout_tests.models import test_failures

@@class AbstractQueueTest(CommandsTest):
8485 queue._options = Mock()
8586 queue._options.port = port
8687
87  queue.run_webkit_patch(run_args)
 88 patch = Attachment({'id': 1}, None)
 89 teed_stdout = WritableStatusServerStatusFileObject(tool.status_server, queue.name, patch)
 90
 91 queue.run_webkit_patch(run_args, patch=patch)
8892 expected_run_args = ["echo", "--status-host=example.com", "--bot-id=gort"]
8993 if port:
9094 expected_run_args.append("--port=%s" % port)
9195 expected_run_args.extend(run_args)
92  tool.executive.run_command.assert_called_with(expected_run_args, cwd='/mock-checkout')
 96 tool.executive.run_with_teed_output_and_throw_if_fail.assert_called_with(expected_run_args, teed_stdout, cwd='/mock-checkout')
9397
9498 def test_run_webkit_patch(self):
9599 self._assert_run_webkit_patch([1])

@@class SecondThoughtsCommitQueue(TestCommitQueue):
210214 self._reject_patch = False
211215 TestCommitQueue.__init__(self, tool)
212216
213  def run_command(self, command):
 217 def run_command(self, command, patch):
214218 # We want to reject the patch after the first validation,
215219 # so wait to reject it until after some other command has run.
216220 self._reject_patch = True
217  return CommitQueue.run_command(self, command)
 221 return CommitQueue.run_command(self, command, patch)
218222
219223 def refetch_patch(self, patch):
220224 if not self._reject_patch:

@@class CommitQueueTest(QueuesTest):
246250 expected_logs = {
247251 "begin_work_queue": self._default_begin_work_queue_logs("commit-queue"),
248252 "process_work_item": """Running: webkit-patch --status-host=example.com clean --port=mac
 253MOCK: update_status: (transient) commit-queue Running: webkit-patch --status-host=example.com clean --port=mac
249254MOCK: update_status: commit-queue Cleaned working directory
250255Running: webkit-patch --status-host=example.com update --port=mac
 256MOCK: update_status: (transient) commit-queue Running: webkit-patch --status-host=example.com update --port=mac
251257MOCK: update_status: commit-queue Updated working directory
252258Running: webkit-patch --status-host=example.com apply-attachment --no-update --non-interactive 10000 --port=mac
 259MOCK: update_status: (transient) commit-queue Running: webkit-patch --status-host=example.com apply-attachment --no-update --non-interactive 10000 --port=mac
253260MOCK: update_status: commit-queue Applied patch
254261Running: webkit-patch --status-host=example.com validate-changelog --check-oops --non-interactive 10000 --port=mac
 262MOCK: update_status: (transient) commit-queue Running: webkit-patch --status-host=example.com validate-changelog --check-oops --non-interactive 10000 --port=mac
255263MOCK: update_status: commit-queue ChangeLog validated
256264Running: webkit-patch --status-host=example.com build --no-clean --no-update --build-style=release --port=mac
 265MOCK: update_status: (transient) commit-queue Running: webkit-patch --status-host=example.com build --no-clean --no-update --build-style=release --port=mac
257266MOCK: update_status: commit-queue Built patch
258267Running: webkit-patch --status-host=example.com build-and-test --no-clean --no-update --test --non-interactive --build-style=release --port=mac
 268MOCK: update_status: (transient) commit-queue Running: webkit-patch --status-host=example.com build-and-test --no-clean --no-update --test --non-interactive --build-style=release --port=mac
259269MOCK: update_status: commit-queue Passed tests
260270Running: webkit-patch --status-host=example.com land-attachment --force-clean --non-interactive --parent-command=commit-queue 10000 --port=mac
 271MOCK: update_status: (transient) commit-queue Running: webkit-patch --status-host=example.com land-attachment --force-clean --non-interactive --parent-command=commit-queue 10000 --port=mac
261272MOCK: update_status: commit-queue Landed patch
262273MOCK: update_status: commit-queue Pass
263274MOCK: release_work_item: commit-queue 10000

@@MOCK: release_work_item: commit-queue 10000
283294 }
284295 queue = CommitQueue()
285296
286  def mock_run_webkit_patch(command):
 297 def mock_run_webkit_patch(command, patch):
287298 if command[0] == 'clean' or command[0] == 'update':
288299 # We want cleaning to succeed so we can error out on a step
289300 # that causes the commit-queue to reject the patch.

@@MOCK: release_work_item: commit-queue 10000
315326
316327 queue = CommitQueue(MockCommitQueueTask)
317328
318  def mock_run_webkit_patch(command):
 329 def mock_run_webkit_patch(command, patch):
319330 if command[0] == 'clean' or command[0] == 'update':
320331 # We want cleaning to succeed so we can error out on a step
321332 # that causes the commit-queue to reject the patch.

@@MOCK: release_work_item: commit-queue 10000
333344 expected_logs = {
334345 "begin_work_queue": self._default_begin_work_queue_logs("commit-queue"),
335346 "process_work_item": """Running: webkit-patch --status-host=example.com clean --port=%(port)s
 347MOCK: update_status: (transient) commit-queue Running: webkit-patch --status-host=example.com clean --port=%(port)s
336348MOCK: update_status: commit-queue Cleaned working directory
337349Running: webkit-patch --status-host=example.com update --port=%(port)s
 350MOCK: update_status: (transient) commit-queue Running: webkit-patch --status-host=example.com update --port=%(port)s
338351MOCK: update_status: commit-queue Updated working directory
339352Running: webkit-patch --status-host=example.com apply-attachment --no-update --non-interactive 10000 --port=%(port)s
 353MOCK: update_status: (transient) commit-queue Running: webkit-patch --status-host=example.com apply-attachment --no-update --non-interactive 10000 --port=%(port)s
340354MOCK: update_status: commit-queue Applied patch
341355Running: webkit-patch --status-host=example.com validate-changelog --check-oops --non-interactive 10000 --port=%(port)s
 356MOCK: update_status: (transient) commit-queue Running: webkit-patch --status-host=example.com validate-changelog --check-oops --non-interactive 10000 --port=%(port)s
342357MOCK: update_status: commit-queue ChangeLog validated
343358Running: webkit-patch --status-host=example.com build --no-clean --no-update --build-style=release --port=%(port)s
 359MOCK: update_status: (transient) commit-queue Running: webkit-patch --status-host=example.com build --no-clean --no-update --build-style=release --port=%(port)s
344360MOCK: update_status: commit-queue Built patch
345361Running: webkit-patch --status-host=example.com build-and-test --no-clean --no-update --test --non-interactive --build-style=release --port=%(port)s
 362MOCK: update_status: (transient) commit-queue Running: webkit-patch --status-host=example.com build-and-test --no-clean --no-update --test --non-interactive --build-style=release --port=%(port)s
346363MOCK: update_status: commit-queue Passed tests
347364Running: webkit-patch --status-host=example.com land-attachment --force-clean --non-interactive --parent-command=commit-queue 10000 --port=%(port)s
 365MOCK: update_status: (transient) commit-queue Running: webkit-patch --status-host=example.com land-attachment --force-clean --non-interactive --parent-command=commit-queue 10000 --port=%(port)s
348366MOCK: update_status: commit-queue Landed patch
349367MOCK: update_status: commit-queue Pass
350368MOCK: release_work_item: commit-queue 10000

@@MOCK: release_work_item: commit-queue 10000
362380 expected_logs = {
363381 "begin_work_queue": self._default_begin_work_queue_logs("commit-queue"),
364382 "process_work_item": """Running: webkit-patch --status-host=example.com clean --port=%(port)s
 383MOCK: update_status: (transient) commit-queue Running: webkit-patch --status-host=example.com clean --port=%(port)s
365384MOCK: update_status: commit-queue Cleaned working directory
366385Running: webkit-patch --status-host=example.com update --port=%(port)s
 386MOCK: update_status: (transient) commit-queue Running: webkit-patch --status-host=example.com update --port=%(port)s
367387MOCK: update_status: commit-queue Updated working directory
368388Running: webkit-patch --status-host=example.com apply-attachment --no-update --non-interactive 10005 --port=%(port)s
 389MOCK: update_status: (transient) commit-queue Running: webkit-patch --status-host=example.com apply-attachment --no-update --non-interactive 10005 --port=%(port)s
369390MOCK: update_status: commit-queue Applied patch
370391Running: webkit-patch --status-host=example.com validate-changelog --check-oops --non-interactive 10005 --port=%(port)s
 392MOCK: update_status: (transient) commit-queue Running: webkit-patch --status-host=example.com validate-changelog --check-oops --non-interactive 10005 --port=%(port)s
371393MOCK: update_status: commit-queue ChangeLog validated
372394Running: webkit-patch --status-host=example.com land-attachment --force-clean --non-interactive --parent-command=commit-queue 10005 --port=%(port)s
 395MOCK: update_status: (transient) commit-queue Running: webkit-patch --status-host=example.com land-attachment --force-clean --non-interactive --parent-command=commit-queue 10005 --port=%(port)s
373396MOCK: update_status: commit-queue Landed patch
374397MOCK: update_status: commit-queue Pass
375398MOCK: release_work_item: commit-queue 10005

@@MOCK: update_status: commit-queue Tests passed, but commit failed (checkout out
415438 queue._options = Mock()
416439 queue._options.port = None
417440 expected_logs = """Running: webkit-patch --status-host=example.com clean --port=mac
 441MOCK: update_status: (transient) commit-queue Running: webkit-patch --status-host=example.com clean --port=mac
418442MOCK: update_status: commit-queue Cleaned working directory
419443MOCK: update_status: commit-queue Error: commit-queue did not process patch. Reason: Patch is obsolete.
420444MOCK: release_work_item: commit-queue 10000

@@class StyleQueueTest(QueuesTest):
478502 "begin_work_queue": self._default_begin_work_queue_logs("style-queue"),
479503 "process_work_item": """MOCK: update_status: style-queue Started processing patch
480504Running: webkit-patch --status-host=example.com clean
 505MOCK: update_status: (transient) style-queue Running: webkit-patch --status-host=example.com clean
481506MOCK: update_status: style-queue Cleaned working directory
482507Running: webkit-patch --status-host=example.com update
 508MOCK: update_status: (transient) style-queue Running: webkit-patch --status-host=example.com update
483509MOCK: update_status: style-queue Updated working directory
484510Running: webkit-patch --status-host=example.com apply-attachment --no-update --non-interactive 10000
 511MOCK: update_status: (transient) style-queue Running: webkit-patch --status-host=example.com apply-attachment --no-update --non-interactive 10000
485512MOCK: update_status: style-queue Applied patch
486513Running: webkit-patch --status-host=example.com apply-watchlist-local 50000
 514MOCK: update_status: (transient) style-queue Running: webkit-patch --status-host=example.com apply-watchlist-local 50000
487515MOCK: update_status: style-queue Watchlist applied
488516Running: webkit-patch --status-host=example.com check-style-local --non-interactive --quiet
 517MOCK: update_status: (transient) style-queue Running: webkit-patch --status-host=example.com check-style-local --non-interactive --quiet
489518MOCK: update_status: style-queue Style checked
490519MOCK: update_status: style-queue Pass
491520MOCK: release_work_item: style-queue 10000

@@MOCK: release_work_item: style-queue 10000
501530 "begin_work_queue": self._default_begin_work_queue_logs("style-queue"),
502531 "process_work_item": """MOCK: update_status: style-queue Started processing patch
503532Running: webkit-patch --status-host=example.com clean
 533MOCK: update_status: (transient) style-queue Running: webkit-patch --status-host=example.com clean
504534MOCK: update_status: style-queue Cleaned working directory
505535Running: webkit-patch --status-host=example.com update
 536MOCK: update_status: (transient) style-queue Running: webkit-patch --status-host=example.com update
506537MOCK: update_status: style-queue Updated working directory
507538Running: webkit-patch --status-host=example.com apply-attachment --no-update --non-interactive 10000
 539MOCK: update_status: (transient) style-queue Running: webkit-patch --status-host=example.com apply-attachment --no-update --non-interactive 10000
508540MOCK: update_status: style-queue Applied patch
509541Running: webkit-patch --status-host=example.com apply-watchlist-local 50000
 542MOCK: update_status: (transient) style-queue Running: webkit-patch --status-host=example.com apply-watchlist-local 50000
510543Exception for ['echo', '--status-host=example.com', 'apply-watchlist-local', 50000]
511544
512545MOCK command output
513546MOCK: update_status: style-queue Unabled to apply watchlist
514547Running: webkit-patch --status-host=example.com check-style-local --non-interactive --quiet
 548MOCK: update_status: (transient) style-queue Running: webkit-patch --status-host=example.com check-style-local --non-interactive --quiet
515549MOCK: update_status: style-queue Style checked
516550MOCK: update_status: style-queue Pass
517551MOCK: release_work_item: style-queue 10000