COMMIT_MESSAGE

EWS for security bugs 

https://bugs.webkit.org/show_bug.cgi?id=186291

<rdar://problem/40829658>

Reviewed by NOBODY (OOPS!).

Part 1 of 2.

Implements support for EWS processing of patches on security sensitive bugs. We add new

endpoints to the status server to support uploading and downloading of patches and associated

metadata. When webkit-patch submits a patch for EWS processing it will now upload the contents

and metadata for the patch to the status server if the patch is on a security sensitive bug.

We teach the EWS machinery in webkitpy to query the status server for a patch only if fetching

the patch from Bugzilla is not permitted due to an authorization error.

Fetching patches from the status server requires an API key. The API key is read from the

environment variable WEBKIT_STATUS_API_KEY or the value of the Git configuration key webkit.status_api_key

(in that order). Contact me or another Apple engineer for an API key.

Additionally, expose an optional command line option called --status-host-uses-https to

query the status server over HTTPS as opposed to HTTP.

* QueueStatusServer/config/authorization.py: Added.

(_path_to_authorized_api_keys_file): Returns the absolute filesystem path to the file authorized_api_keys.txt.

(_parse_authorized_api_keys):

(authorized_api_keys):

(_parse_authorization_header): Parses the API key from the Authorization header. We use a

custom authentication scheme: "APIKey". See remark below for more details.

(is_authorized): Checks if the request includes an API key and whether that API key is in the

list of authorized keys (performs a case-sensitive match). The API key may be specified either

in a HTTP header Authorization or in the query string argument "api-key". When using the HTTP

headers approach the Authorization header should have the form: "Authorization: APIKey X" where

X is the case-sensitive API key.

* QueueStatusServer/handlers/fetchattachment.py: Added.

(FetchAttachment):

(FetchAttachment.get):

* QueueStatusServer/handlers/releasepatch.py:

(ReleasePatch.check_complete): Returns whether the specified attachment was processed by all the queues.

(ReleasePatch.post): Delete the patch from AppEngine (if we have it) once the patch was processed

by all the queues.

* QueueStatusServer/handlers/submittoews.py:

(SubmitToEWS._should_add_to_ews_queue): Fix a typo in a comment while I am working in this code.

* QueueStatusServer/handlers/uploadattachment.py: Added.

(UploadAttachment):

(UploadAttachment.get):

(UploadAttachment.post):

* QueueStatusServer/main.py: Add new routes /upload-attachment and /attachment to upload an attachment

and view an attachment (or its metadata), respectively.

* QueueStatusServer/model/attachmentdata.py: Added.

(AttachmentData):

(AttachmentData.add_attachment_data):

(AttachmentData.lookup_if_exists):

(AttachmentData.remove_attachment_data):

* QueueStatusServer/templates/uploadattachment.html: Added.

* Scripts/webkitpy/common/net/bugzilla/attachment.py:

(Attachment.committer):

(Attachment):

(Attachment.to_json): Serialize to JSON so that we can upload it to AppEngine.

(Attachment.from_json): Deserialize from JSON. This is used as part of downloading a patch from AppEngine.

* Scripts/webkitpy/common/net/bugzilla/attachment_unittest.py: Added.

(AttachmentTest):

(AttachmentTest.test_convert_to_json_and_back):

* Scripts/webkitpy/common/net/bugzilla/bug.py:

(Bug.group): Returns the group that bug is in or the empty string.

(Bug.is_security_sensitive): Returns whether the bug is in group Security-Sensitive.

* Scripts/webkitpy/common/net/bugzilla/bugzilla.py:

(BugzillaQueries.fetch_attachment_ids_from_review_queue): Modified to take an optional boolean, only_security_bugs,

as to whether to only fetch attachment ids for unreviewed patches associated with security bugs. By default, we

keep the current behavior and query for the attachment ids of all unreviewed patches that the currently logged in

Bugzilla user can see, which may include patches associated with security bugs.

(Bugzilla._parse_date): Update for moved and renamed constant. See remark for class Bugzilla.

(Bugzilla._parse_bug_dictionary_from_xml): Modified to return an empty dictionary if we do not have access to view the bug.

Otherwise, extract the name of the group the bug is in.

(Bugzilla.fetch_bug): Modified to return None if we do not have access to view the bug.

(Bugzilla._parse_bug_title_from_attachment_page): Extracted out logic to parse the title of the Attachment page

from _parse_bug_id_from_attachment_page() so that it can be used from both _parse_bug_id_from_attachment_page()

and get_bug_id_for_attachment_id().

(Bugzilla): Moved class constant _bugzilla_date_format to Scripts/webkitpy/common/net/bugzilla/constants.py

and renamed it to BUGZILLA_DATE_FORMAT.

(Bugzilla.AccessError):

(Bugzilla.AccessError.__init__):

(Bugzilla._parse_bug_id_from_attachment_page): Modified to return a tuple of ("bug id", "error code") so that

the caller can know the reason the parse failed if it did. The parse will fail if we do not have access to view

the bug.

(Bugzilla.bug_id_for_attachment_id): Modified to take a boolean throw_on_access_error (default: False)

as to whether to raise a Bugzilla.AccessError exception and pass it through to get_bug_id_for_attachment_id().

(Bugzilla.get_bug_id_for_attachment_id): Modified to take a boolean throw_on_access_error (default: False)

as to whether to raise a Bugzilla.AccessError exception if we do not have access to the bug associated with

the specified attachment id.

(Bugzilla.fetch_attachment):

* Scripts/webkitpy/common/net/bugzilla/bugzilla_mock.py:

(MockBugzillaQueries.fetch_attachment_ids_from_review_queue):

(MockBugzilla):

(MockBugzilla.fetch_attachment):

(MockBugzilla.fetch_attachment_contents):

(MockBugzilla.add_patch_to_bug):

* Scripts/webkitpy/common/net/bugzilla/bugzilla_unittest.py:

* Scripts/webkitpy/common/net/bugzilla/constants.py: Added.

* Scripts/webkitpy/common/net/statusserver.py:

(StatusServer.set_host): Modified to take an boolean use_https as to whether to query the server using

HTTPS (default: False - use HTTP; our current behavior).

(StatusServer.set_api_key): Added.

(StatusServer._upload_attachment_to_server): Added.

(StatusServer.upload_attachment): Added.

(StatusServer._fetch_attachment_page): Added.

(StatusServer.fetch_attachment): Added.

* Scripts/webkitpy/common/net/statusserver_mock.py:

(MockStatusServer.upload_attachment): Added.

(MockStatusServer.fetch_attachment): Added.

* Scripts/webkitpy/tool/bot/feeders.py:

(EWSFeeder.feed): Modified to download patches on security bugs and upload them to the status server (AppEngine).

* Scripts/webkitpy/tool/commands/download.py:

(ProcessAttachmentsMixin._fetch_list_of_patches_to_process): Modified to handle the case when fetching the

bug details from Bugzilla fail, say because we are not allowed to the view the bug.

(ProcessBugsMixin._fetch_list_of_patches_to_process): Filter out None values for attachments that we failed

to fetch, say because we are not allowed to the view the bug the attachment is on.

* Scripts/webkitpy/tool/commands/earlywarningsystem.py:

(AbstractEarlyWarningSystem.refetch_patch): For now, refetch the patch from the status server. Ideally, we

need a way to ask the status server to fetch the patch again from Bugzilla (or at least its metadata) so

that the EWS can check the current state of the patch (i.e. is it still marked r?).

* Scripts/webkitpy/tool/commands/queries_unittest.py:

(QueryCommandsTest.test_patches_to_review): Update expected result.

* Scripts/webkitpy/tool/commands/queues.py:

(AbstractPatchQueue._next_patch): Fetch the patch  from the status server if we failed to fetch it from

Bugzilla because we do not have permission to view it.

* Scripts/webkitpy/tool/commands/queues_unittest.py:

* Scripts/webkitpy/tool/commands/upload_unittest.py:

(test_upload_of_security_sensitive_patch_with_no_review_and_ews): Added.

* Scripts/webkitpy/tool/main.py:

(WebKitPatch):

(WebKitPatch._status_server_api_key_from_git): Read the API key from the Git configuration key webkit.status_api_key.

(WebKitPatch._status_server_api_key): Read the API key from the environment variable WEBKIT_STATUS_API_KEY.

(WebKitPatch.handle_global_options): Read the API key and update the state of the StatusServer object, if applicable.

* Scripts/webkitpy/tool/steps/obsoletepatches.py:

(ObsoletePatches.run): Modified to handle the case when fetching the bug details from Bugzilla fail, say because we

are not allowed to the view the bug.

* Scripts/webkitpy/tool/steps/submittoews.py:

(SubmitToEWS.run): Upload the contents of the patch and the Bugzilla metadata about it to the status server

if the patch was posted to a security bug.

Tools/ChangeLog

2018-06-15  Daniel Bates  <dabates@apple.com>

        EWS for security bugs

        https://bugs.webkit.org/show_bug.cgi?id=186291

        <rdar://problem/40829658>

        Reviewed by NOBODY (OOPS!).

        Part 1 of 2.

        Implements support for EWS processing of patches on security sensitive bugs. We add new

        endpoints to the status server to support uploading and downloading of patches and associated

        metadata. When webkit-patch submits a patch for EWS processing it will now upload the contents

        and metadata for the patch to the status server if the patch is on a security sensitive bug.

        We teach the EWS machinery in webkitpy to query the status server for a patch only if fetching

        the patch from Bugzilla is not permitted due to an authorization error.

        Fetching patches from the status server requires an API key. The API key is read from the

        environment variable WEBKIT_STATUS_API_KEY or the value of the Git configuration key webkit.status_api_key

        (in that order). Contact me or another Apple engineer for an API key.

        Additionally, expose an optional command line option called --status-host-uses-https to

        query the status server over HTTPS as opposed to HTTP.

        * QueueStatusServer/config/authorization.py: Added.

        (_path_to_authorized_api_keys_file): Returns the absolute filesystem path to the file authorized_api_keys.txt.

        (_parse_authorized_api_keys):

        (authorized_api_keys):

        (_parse_authorization_header): Parses the API key from the Authorization header. We use a

        custom authentication scheme: "APIKey". See remark below for more details.

        (is_authorized): Checks if the request includes an API key and whether that API key is in the

        list of authorized keys (performs a case-sensitive match). The API key may be specified either

        in a HTTP header Authorization or in the query string argument "api-key". When using the HTTP

        headers approach the Authorization header should have the form: "Authorization: APIKey X" where

        X is the case-sensitive API key.

        * QueueStatusServer/handlers/fetchattachment.py: Added.

        (FetchAttachment):

        (FetchAttachment.get):

        * QueueStatusServer/handlers/releasepatch.py:

        (ReleasePatch.check_complete): Returns whether the specified attachment was processed by all the queues.

        (ReleasePatch.post): Delete the patch from AppEngine (if we have it) once the patch was processed

        by all the queues.

        * QueueStatusServer/handlers/submittoews.py:

        (SubmitToEWS._should_add_to_ews_queue): Fix a typo in a comment while I am working in this code.

        * QueueStatusServer/handlers/uploadattachment.py: Added.

        (UploadAttachment):

        (UploadAttachment.get):

        (UploadAttachment.post):

        * QueueStatusServer/main.py: Add new routes /upload-attachment and /attachment to upload an attachment

        and view an attachment (or its metadata), respectively.

        * QueueStatusServer/model/attachmentdata.py: Added.

        (AttachmentData):

        (AttachmentData.add_attachment_data):

        (AttachmentData.lookup_if_exists):

        (AttachmentData.remove_attachment_data):

        * QueueStatusServer/templates/uploadattachment.html: Added.

        * Scripts/webkitpy/common/net/bugzilla/attachment.py:

        (Attachment.committer):

        (Attachment):

        (Attachment.to_json): Serialize to JSON so that we can upload it to AppEngine.

        (Attachment.from_json): Deserialize from JSON. This is used as part of downloading a patch from AppEngine.

        * Scripts/webkitpy/common/net/bugzilla/attachment_unittest.py: Added.

        (AttachmentTest):

        (AttachmentTest.test_convert_to_json_and_back):

        * Scripts/webkitpy/common/net/bugzilla/bug.py:

        (Bug.group): Returns the group that bug is in or the empty string.

        (Bug.is_security_sensitive): Returns whether the bug is in group Security-Sensitive.

        * Scripts/webkitpy/common/net/bugzilla/bugzilla.py:

        (BugzillaQueries.fetch_attachment_ids_from_review_queue): Modified to take an optional boolean, only_security_bugs,

        as to whether to only fetch attachment ids for unreviewed patches associated with security bugs. By default, we

        keep the current behavior and query for the attachment ids of all unreviewed patches that the currently logged in

        Bugzilla user can see, which may include patches associated with security bugs.

        (Bugzilla._parse_date): Update for moved and renamed constant. See remark for class Bugzilla.

        (Bugzilla._parse_bug_dictionary_from_xml): Modified to return an empty dictionary if we do not have access to view the bug.

        Otherwise, extract the name of the group the bug is in.

        (Bugzilla.fetch_bug): Modified to return None if we do not have access to view the bug.

        (Bugzilla._parse_bug_title_from_attachment_page): Extracted out logic to parse the title of the Attachment page

        from _parse_bug_id_from_attachment_page() so that it can be used from both _parse_bug_id_from_attachment_page()

        and get_bug_id_for_attachment_id().

        (Bugzilla): Moved class constant _bugzilla_date_format to Scripts/webkitpy/common/net/bugzilla/constants.py

        and renamed it to BUGZILLA_DATE_FORMAT.

        (Bugzilla.AccessError):

        (Bugzilla.AccessError.__init__):

        (Bugzilla._parse_bug_id_from_attachment_page): Modified to return a tuple of ("bug id", "error code") so that

        the caller can know the reason the parse failed if it did. The parse will fail if we do not have access to view

        the bug.

        (Bugzilla.bug_id_for_attachment_id): Modified to take a boolean throw_on_access_error (default: False)

        as to whether to raise a Bugzilla.AccessError exception and pass it through to get_bug_id_for_attachment_id().

        (Bugzilla.get_bug_id_for_attachment_id): Modified to take a boolean throw_on_access_error (default: False)

        as to whether to raise a Bugzilla.AccessError exception if we do not have access to the bug associated with

        the specified attachment id.

        (Bugzilla.fetch_attachment):

        * Scripts/webkitpy/common/net/bugzilla/bugzilla_mock.py:

        (MockBugzillaQueries.fetch_attachment_ids_from_review_queue):

        (MockBugzilla):

        (MockBugzilla.fetch_attachment):

        (MockBugzilla.fetch_attachment_contents):

        (MockBugzilla.add_patch_to_bug):

        * Scripts/webkitpy/common/net/bugzilla/bugzilla_unittest.py:

        * Scripts/webkitpy/common/net/bugzilla/constants.py: Added.

        * Scripts/webkitpy/common/net/statusserver.py:

        (StatusServer.set_host): Modified to take an boolean use_https as to whether to query the server using

        HTTPS (default: False - use HTTP; our current behavior).

        (StatusServer.set_api_key): Added.

        (StatusServer._upload_attachment_to_server): Added.

        (StatusServer.upload_attachment): Added.

        (StatusServer._fetch_attachment_page): Added.

        (StatusServer.fetch_attachment): Added.

        * Scripts/webkitpy/common/net/statusserver_mock.py:

        (MockStatusServer.upload_attachment): Added.

        (MockStatusServer.fetch_attachment): Added.

        * Scripts/webkitpy/tool/bot/feeders.py:

        (EWSFeeder.feed): Modified to download patches on security bugs and upload them to the status server (AppEngine).

        * Scripts/webkitpy/tool/commands/download.py:

        (ProcessAttachmentsMixin._fetch_list_of_patches_to_process): Modified to handle the case when fetching the

        bug details from Bugzilla fail, say because we are not allowed to the view the bug.

        (ProcessBugsMixin._fetch_list_of_patches_to_process): Filter out None values for attachments that we failed

        to fetch, say because we are not allowed to the view the bug the attachment is on.

        * Scripts/webkitpy/tool/commands/earlywarningsystem.py:

        (AbstractEarlyWarningSystem.refetch_patch): For now, refetch the patch from the status server. Ideally, we

        need a way to ask the status server to fetch the patch again from Bugzilla (or at least its metadata) so

        that the EWS can check the current state of the patch (i.e. is it still marked r?).

        * Scripts/webkitpy/tool/commands/queries_unittest.py:

        (QueryCommandsTest.test_patches_to_review): Update expected result.

        * Scripts/webkitpy/tool/commands/queues.py:

        (AbstractPatchQueue._next_patch): Fetch the patch  from the status server if we failed to fetch it from

        Bugzilla because we do not have permission to view it.

        * Scripts/webkitpy/tool/commands/queues_unittest.py:

        * Scripts/webkitpy/tool/commands/upload_unittest.py:

        (test_upload_of_security_sensitive_patch_with_no_review_and_ews): Added.

        * Scripts/webkitpy/tool/main.py:

        (WebKitPatch):

        (WebKitPatch._status_server_api_key_from_git): Read the API key from the Git configuration key webkit.status_api_key.

        (WebKitPatch._status_server_api_key): Read the API key from the environment variable WEBKIT_STATUS_API_KEY.

        (WebKitPatch.handle_global_options): Read the API key and update the state of the StatusServer object, if applicable.

        * Scripts/webkitpy/tool/steps/obsoletepatches.py:

        (ObsoletePatches.run): Modified to handle the case when fetching the bug details from Bugzilla fail, say because we

        are not allowed to the view the bug.

        * Scripts/webkitpy/tool/steps/submittoews.py:

        (SubmitToEWS.run): Upload the contents of the patch and the Bugzilla metadata about it to the status server

        if the patch was posted to a security bug.

2018-06-14  Carlos Alberto Lopez Perez  <clopez@igalia.com>

        [GTK] Enable tests on the GTK EWS queue

Tools/QueueStatusServer/config/authorization.py

# Copyright (C) 2018 Apple Inc. All rights reserved.

#

# Redistribution and use in source and binary forms, with or without

# modification, are permitted provided that the following conditions

# are met:

# 1.  Redistributions of source code must retain the above copyright

#     notice, this list of conditions and the following disclaimer.

# 2.  Redistributions in binary form must reproduce the above copyright

#     notice, this list of conditions and the following disclaimer in the

#     documentation and/or other materials provided with the distribution.

#

# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND

# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED

# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE

# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR

# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL

# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR

# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER

# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,

# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE

# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

import os

_cached_authorized_api_keys = frozenset([])

def _path_to_authorized_api_keys_file():

    return os.path.abspath(os.path.join(os.path.dirname(__file__), "authorized_api_keys.txt"))

def _parse_authorized_api_keys(file):

    api_keys = set()

    for line in file:

        line = line.strip()

        if not line or line.startswith("#"):  # Skip empty lines and comments

            continue

        api_keys.add(line)

    return frozenset(api_keys)

def authorized_api_keys():

    global _cached_authorized_api_keys

    if _cached_authorized_api_keys:

        return _cached_authorized_api_keys

    authorized_api_keys_path = _path_to_authorized_api_keys_file()

    with open(authorized_api_keys_path, "r") as file:

        _cached_authorized_api_keys = _parse_authorized_api_keys(file)

    return _cached_authorized_api_keys

def _parse_authorization_header(credentials):

    # See <https://tools.ietf.org/html/rfc7235#section-4.2>.

    parts = credentials.split(" ", 1)

    if len(parts) < 2:

        return ""

    scheme = parts[0]

    token68_encoded_value = parts[1]

    if scheme.lower() == "apikey":

        return token68_encoded_value

    return ""

def is_authorized(request):

    api_key = ''

    credentials = request.headers.get("Authorization")

    if credentials:

        api_key = _parse_authorization_header(credentials)

    if not api_key:

        api_key = request.get("api-key")

    return api_key in authorized_api_keys()

Tools/QueueStatusServer/handlers/fetchattachment.py

# Copyright (C) 2018 Apple Inc. All rights reserved.

#

# Redistribution and use in source and binary forms, with or without

# modification, are permitted provided that the following conditions

# are met:

# 1.  Redistributions of source code must retain the above copyright

#     notice, this list of conditions and the following disclaimer.

# 2.  Redistributions in binary form must reproduce the above copyright

#     notice, this list of conditions and the following disclaimer in the

#     documentation and/or other materials provided with the distribution.

#

# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND

# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED

# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE

# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR

# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL

# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR

# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER

# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,

# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE

# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

from config import authorization

from google.appengine.ext import webapp

from model.attachmentdata import AttachmentData

class FetchAttachment(webapp.RequestHandler):

    def get(self, action, attachment_id):

        if action not in ["data", "metadata"]:

            self.error(400)

            return

        if not authorization.is_authorized(self.request):

            # Return an HTTP 404 response code instead of an HTTP 401 to avoid leaking whether

            # the attachment id is known.

            self.error(404)

            return

        attachment_data = AttachmentData.lookup_if_exists(attachment_id)

        if not attachment_data:

            self.error(404)

            return

        if action == "metadata":

            self.response.out.write(attachment_data.metadata)

        else:

            self.response.out.write(attachment_data.data)

Tools/QueueStatusServer/handlers/releasepatch.py

from google.appengine.ext import webapp, db

from google.appengine.ext.webapp import template

32from config.queues import all_queue_names

from handlers.updatebase import UpdateBase

from loggers.recordpatchevent import RecordPatchEvent

from model.attachment import Attachment

36from model.attachmentdata import AttachmentData

from model.queues import Queue

@@class ReleasePatch(UpdateBase):

    def get(self):

        self.response.out.write(template.render("templates/releasepatch.html", None))

    @staticmethod

    def check_complete(attachment_id):

        for queue_name in all_queue_names:

            queue = Queue.queue_with_name(queue_name)

            if queue.work_items().display_position_for_attachment(attachment_id) is not None:

                return False

        return True

    def post(self):

        queue_name = self.request.get("queue_name")

        # FIXME: This queue lookup should be shared between handlers.

@@class ReleasePatch(UpdateBase):

        RecordPatchEvent.stopped(attachment_id, queue_name, last_status.message)

        queue.active_work_items().expire_item(attachment_id)

        if self.check_complete(attachment_id):

            AttachmentData.remove_attachment_data(attachment_id)

Tools/QueueStatusServer/handlers/submittoews.py

@@class SubmitToEWS(UpdateBase):

    def _should_add_to_ews_queue(self, queue, attachment):

        # This assert() is here to make sure we're not submitting to the commit-queue.

44 # The commit-queue clients check each patch anyway, but there is not sense

44 # The commit-queue clients check each patch anyway, but there is no sense

        # in adding things to the commit-queue when they won't be processed by it.

        assert(queue.is_ews())

        latest_status = attachment.status_for_queue(queue)

Tools/QueueStatusServer/handlers/uploadattachment.py

# Copyright (C) 2018 Apple Inc. All rights reserved.

#

# Redistribution and use in source and binary forms, with or without

# modification, are permitted provided that the following conditions

# are met:

# 1.  Redistributions of source code must retain the above copyright

#     notice, this list of conditions and the following disclaimer.

# 2.  Redistributions in binary form must reproduce the above copyright

#     notice, this list of conditions and the following disclaimer in the

#     documentation and/or other materials provided with the distribution.

#

# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND

# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED

# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE

# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR

# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL

# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR

# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER

# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,

# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE

# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

from google.appengine.ext import webapp, db

from google.appengine.ext.webapp import template

from handlers.updatebase import UpdateBase

from model.attachmentdata import AttachmentData

class UploadAttachment(UpdateBase):

    def get(self):

        self.response.out.write(template.render("templates/uploadattachment.html", None))

    def post(self):

        attachment_id = self._int_from_request("attachment_id")

        attachment_metadata = self.request.get("attachment_metadata")

        attachment_data = self.request.get("attachment_data")

        AttachmentData.add_attachment_data(attachment_id, str(attachment_metadata), str(attachment_data))

        self.response.out.write(attachment_id)

Tools/QueueStatusServer/main.py

@@from google.appengine.ext import webapp

from google.appengine.ext.webapp.util import run_wsgi_app

from handlers.activebots import ActiveBots

38from handlers.fetchattachment import FetchAttachment

from handlers.gc import GC

from handlers.nextpatch import NextPatch

from handlers.patch import Patch

@@from handlers.syncqueuelogs import SyncQueueLogs

from handlers.updatestatus import UpdateStatus

from handlers.updatesvnrevision import UpdateSVNRevision

from handlers.updateworkitems import UpdateWorkItems

59from handlers.uploadattachment import UploadAttachment

webapp.template.register_template_library('filters.webkit_extras')

@@routes = [

    (r'/queue-status/(.*)', QueueStatus),

    (r'/queue-status-json/(.*)', QueueStatusJSON),

    (r'/next-patch/(.*)', NextPatch),

81 (r'/attachment/(.*)/(.*)', FetchAttachment),

    ('/release-patch', ReleasePatch),

    ('/release-lock', ReleaseLock),

    ('/update-status', UpdateStatus),

    ('/update-work-items', UpdateWorkItems),

    ('/update-svn-revision', UpdateSVNRevision),

87 ('/upload-attachment', UploadAttachment),

    ('/active-bots', ActiveBots),

    (r'/processing-times-json/(\d+)\-(\d+)\-(\d+)\-(\d+)\-(\d+)\-(\d+)\-(\d+)\-(\d+)\-(\d+)\-(\d+)\-(\d+)\-(\d+)', ProcessingTimesJSON),

]

Tools/QueueStatusServer/model/attachmentdata.py

# Copyright (C) 2018 Apple Inc. All rights reserved.

#

# Redistribution and use in source and binary forms, with or without

# modification, are permitted provided that the following conditions

# are met:

# 1.  Redistributions of source code must retain the above copyright

#     notice, this list of conditions and the following disclaimer.

# 2.  Redistributions in binary form must reproduce the above copyright

#     notice, this list of conditions and the following disclaimer in the

#     documentation and/or other materials provided with the distribution.

#

# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND

# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED

# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE

# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR

# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL

# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR

# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER

# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,

# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE

# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

from google.appengine.ext import db

class AttachmentData(db.Model):

    attachment_id = db.IntegerProperty()

    metadata = db.BlobProperty()

    data = db.BlobProperty()

    @classmethod

    def add_attachment_data(cls, attachment_id, metadata, data):

        cls.get_or_insert(str(attachment_id), attachment_id=attachment_id, metadata=db.Blob(metadata), data=db.Blob(data))

    @classmethod

    def lookup_if_exists(cls, attachment_id):

        return cls.get_by_key_name(str(attachment_id))

    @classmethod

    def remove_attachment_data(cls, attachment_id):

        attachment_data = cls.lookup_if_exists(attachment_id)

        if attachment_data:

            attachment_data.delete()

Tools/QueueStatusServer/templates/uploadattachment.html

<form name="upload_attachment" enctype="multipart/form-data" method="POST">

Attachment id: <input name="attachment_id"><br>

Metadata: <input type="file" name="attachment_metadata"><br>

Data: <input type="file" name="attachment_data"><br>

<input type="submit" value="Upload Attachment">

</form>

Tools/Scripts/webkitpy/common/net/bugzilla/attachment.py

# Copyright (c) 2010 Research In Motion Limited. All rights reserved.

#

# Redistribution and use in source and binary forms, with or without

# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE

# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

31import json

3132import logging

3233

34from datetime import datetime

3335from webkitpy.common.memoized import memoized

36from webkitpy.common.net.bugzilla.constants import BUGZILLA_DATE_FORMAT

_log = logging.getLogger(__name__)

@@class Attachment(object):

        if not self._committer:

            self._committer = self._validate_flag_value("committer")

        return self._committer

    def to_json(self):

        temp = dict(self._attachment_dictionary)

        if 'attach_date' in temp:

            temp['attach_date'] = temp['attach_date'].strftime(BUGZILLA_DATE_FORMAT)

        return json.dumps(temp)

    @classmethod

    def from_json(cls, json_string):

        temp = json.loads(json_string)

        if 'attach_date' in temp:

            temp['attach_date'] = datetime.strptime(temp['attach_date'], BUGZILLA_DATE_FORMAT)

        return Attachment(temp, None)

Tools/Scripts/webkitpy/common/net/bugzilla/attachment_unittest.py

# Copyright (C) 2018 Apple Inc. All rights reserved.

#

# Redistribution and use in source and binary forms, with or without

# modification, are permitted provided that the following conditions

# are met:

# 1.  Redistributions of source code must retain the above copyright

#     notice, this list of conditions and the following disclaimer.

# 2.  Redistributions in binary form must reproduce the above copyright

#     notice, this list of conditions and the following disclaimer in the

#     documentation and/or other materials provided with the distribution.

#

# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND

# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED

# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE

# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR

# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL

# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR

# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER

# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,

# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE

# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

import unittest

from datetime import datetime

from webkitpy.common.net.bugzilla.constants import BUGZILLA_DATE_FORMAT

from .attachment import Attachment

class AttachmentTest(unittest.TestCase):

    def test_convert_to_json_and_back(self):

        bugzilla_formatted_date_string = datetime.today().strftime(BUGZILLA_DATE_FORMAT)

        expected_date = datetime.strptime(bugzilla_formatted_date_string, BUGZILLA_DATE_FORMAT)

        attachment = Attachment({'attach_date': expected_date}, None)

        self.assertEqual(Attachment.from_json(attachment.to_json()).attach_date(), expected_date)

Tools/Scripts/webkitpy/common/net/bugzilla/bug.py

@@class Bug(object):

    def status(self):

        return self.bug_dictionary["bug_status"]

    def group(self):

        # FIXME: A bug may be in more than one group.

        return self.bug_dictionary.get('group', '')

    def is_security_sensitive(self):

        return self.group() == 'Security-Sensitive'

    # Bugzilla has many status states we don't really use in WebKit:

    # https://bugs.webkit.org/page.cgi?id=fields.html#status

    _open_states = ["UNCONFIRMED", "NEW", "ASSIGNED", "REOPENED"]

Tools/Scripts/webkitpy/common/net/bugzilla/bugzilla.py

# Copyright (c) 2010 Research In Motion Limited. All rights reserved.

# Copyright (c) 2013 University of Szeged. All rights reserved.

#

@@from .bug import Bug

from webkitpy.common.config import committers

import webkitpy.common.config.urls as config_urls

48from webkitpy.common.net.bugzilla.constants import BUGZILLA_DATE_FORMAT

from webkitpy.common.net.credentials import Credentials

from webkitpy.common.net.networktransaction import NetworkTransaction

from webkitpy.common.system.user import User

@@class BugzillaQueries(object):

    # NOTE: This is the only client of _fetch_attachment_ids_request_query

    # This method only makes one request to bugzilla.

281 def fetch_attachment_ids_from_review_queue(self, since=None):

282 def fetch_attachment_ids_from_review_queue(self, since=None, only_security_bugs=False):

282283 review_queue_url = "request.cgi?action=queue&type=review&group=type"

284 if only_security_bugs:

285 review_queue_url += '&product=Security'

        return self._fetch_attachment_ids_request_query(review_queue_url, since)

    # This only works if your account has edituser privileges.

@@class Bugzilla(object):

        # convert from NavigableString to a real unicode() object using unicode().

        return unicode(soup.string)

    # Example: 2010-01-20 14:31 PST

    # FIXME: Some bugzilla dates seem to have seconds in them?

    # Python does not support timezones out of the box.

    # Assume that bugzilla always uses PST (which is true for bugs.webkit.org)

    _bugzilla_date_format = "%Y-%m-%d %H:%M:%S"

    @classmethod

    def _parse_date(cls, date_string):

        (date, time, time_zone) = date_string.split(" ")

@@class Bugzilla(object):

            time += ':0'

        # Ignore the timezone because python doesn't understand timezones out of the box.

        date_string = "%s %s" % (date, time)

416 return datetime.strptime(date_string, ~~cls._bugzilla_date_format~~)

413 return datetime.strptime(date_string, BUGZILLA_DATE_FORMAT)

    def _date_contents(self, soup):

        return self._parse_date(self._string_contents(soup))

@@class Bugzilla(object):

    def _parse_bug_dictionary_from_xml(self, page):

        soup = BeautifulStoneSoup(page, convertEntities=BeautifulStoneSoup.XML_ENTITIES)

        bug_element = soup.find('bug')

        if bug_element and bug_element.get('error', '') == 'NotPermitted':

            _log.warning("You don't have permission to view this bug.")

            return {}

        bug = {}

        bug["id"] = int(soup.find("bug_id").string)

        bug["title"] = self._string_contents(soup.find("short_desc"))

@@class Bugzilla(object):

        bug["cc_emails"] = [self._string_contents(element) for element in soup.findAll('cc')]

        bug["attachments"] = [self._parse_attachment_element(element, bug["id"]) for element in soup.findAll('attachment')]

        bug["comments"] = [self._parse_log_descr_element(element) for element in soup.findAll('long_desc')]

        # FIXME: A bug may be in more than one group.

        group = soup.find('group')

        if group:

            bug['group'] = self._string_contents(group)

        return bug

@@class Bugzilla(object):

    # FIXME: A BugzillaCache object should provide all these fetch_ methods.

    def fetch_bug(self, bug_id):

487 return Bug(self.fetch_bug_dictionary(bug_id), self)

        bug_dictionary = self.fetch_bug_dictionary(bug_id)

        if bug_dictionary:

            return Bug(bug_dictionary, self)

        return None

    def fetch_attachment_contents(self, attachment_id):

        attachment_url = self.attachment_url_for_id(attachment_id)

@@class Bugzilla(object):

        self.authenticate()

        return self.open_url(attachment_url).read()

    def _parse_bug_title_from_attachment_page(self, page):

        return BeautifulSoup(page).find('div', attrs={'id': 'bug_title'})

    class AccessError(Exception):

        NOT_PERMITTED = 1 << 0

        OTHER = 1 << 1

        def __init__(self, attachment_id, error_code, bug_title):

            super(Bugzilla.AccessError, self).__init__('Failed to access {}'.format(attachment_id))

            self.attachment_id = attachment_id

            self.error_code = error_code

            self.bug_title = bug_title

496517 def _parse_bug_id_from_attachment_page(self, page):

497518 # The "Up" relation happens to point to the bug.

        title = BeautifulSoup(page).find('div', attrs={'id':'bug_title'})

        if not title :

            _log.warning("This attachment does not exist (or you don't have permissions to view it).")

            return None

        title = self._parse_bug_title_from_attachment_page(page)

        if not title:

            _log.warning("This attachment does not exist (or you don't have permission to view it).")

            return (None, Bugzilla.AccessError.OTHER)

        if title.getText() == 'Bug Access Denied':

            _log.warning("You don't have permission to view this attachment.")

            return (None, Bugzilla.AccessError.NOT_PERMITTED)

        match = re.search("show_bug.cgi\?id=(?P<bug_id>\d+)", str(title))

        if not match:

            _log.warning("Unable to parse bug id from attachment")

505 return None

506 return int(match.group('bug_id'))

529 return (None, None)

530 return (int(match.group('bug_id')), None)

507531

508 def bug_id_for_attachment_id(self, attachment_id):

509 return NetworkTransaction().run(lambda: self.get_bug_id_for_attachment_id(attachment_id))

532 def bug_id_for_attachment_id(self, attachment_id, throw_on_access_error=False):

533 return NetworkTransaction().run(lambda: self.get_bug_id_for_attachment_id(attachment_id, throw_on_access_error))

510534

511 def get_bug_id_for_attachment_id(self, attachment_id):

535 def get_bug_id_for_attachment_id(self, attachment_id, throw_on_access_error=False):

        self.authenticate()

        attachment_url = self.attachment_url_for_id(attachment_id, 'edit')

        _log.info("Fetching: %s" % attachment_url)

        page = self.open_url(attachment_url)

517 return self._parse_bug_id_from_attachment_page(page)

        bug_id, error_code = self._parse_bug_id_from_attachment_page(page)

        if bug_id:

            return bug_id

        if error_code is not None and throw_on_access_error:

            raise Bugzilla.AccessError(attachment_id, error_code, str(self._parse_bug_title_from_attachment_page(page)))

        return None

    # FIXME: This should just return Attachment(id), which should be able to

    # lazily fetch needed data.

522 def fetch_attachment(self, attachment_id):

551 def fetch_attachment(self, attachment_id, throw_on_access_error=False):

        # We could grab all the attachment details off of the attachment edit

        # page but we already have working code to do so off of the bugs page,

        # so re-use that.

526 bug_id = self.bug_id_for_attachment_id(attachment_id)

555 bug_id = self.bug_id_for_attachment_id(attachment_id, throw_on_access_error)

        if not bug_id:

            _log.warning("Unable to parse bug_id from attachment {}".format(attachment_id))

            return None

Tools/Scripts/webkitpy/common/net/bugzilla/bugzilla_mock.py

#

# Redistribution and use in source and binary forms, with or without

# modification, are permitted provided that the following conditions are

@@import datetime

import logging

from .bug import Bug

34from .bugzilla import Bugzilla

from .attachment import Attachment

from webkitpy.common.config.committers import CommitterList, Reviewer

@@_patch8 = { # Resolved bug, without review flag, not marked obsolete (maybe alr

    "attacher_email": "eric@webkit.org",

}

_patch9 = {

    'id': 10008,

    'bug_id': 50007,

    'url': 'http://example.com/10008',

    'name': 'Patch9',

    'is_obsolete': False,

    'is_patch': True,

    'review': '?',

    'commit-queue': '-',

    'attacher_email': 'dbates@webkit.org',

    'attach_date': datetime.datetime.today(),

}

# This matches one of Bug.unassigned_emails

_unassigned_email = "webkit-unassigned@lists.webkit.org"

# This is needed for the FlakyTestReporter to believe the bug

@@_bug7 = {

}

_bug8 = {

    'id': 50007,

    'title': 'Security bug with a patch needing review.',

    'reporter_email': 'dbates@webkit.org',

    'assigned_to_email': 'foo@foo.com',

    'cc_emails': [],

    'attachments': [_patch9],

    'bug_status': 'ASSIGNED',

    'group': 'Security-Sensitive',

    'comments': [{'comment_date':  datetime.datetime(2011, 6, 11, 9, 4, 3),

                  'comment_email': 'bar@foo.com',

                  'text': 'Message1.\nCommitted r35: <https://trac.webkit.org/changeset/35>',

                  },

                 ],

}

class MockBugzillaQueries(object):

    def __init__(self, bugzilla):

@@class MockBugzillaQueries(object):

                self._all_bugs())

        return map(lambda bug: bug.id(), bugs_with_commit_queued_patches)

296 def fetch_attachment_ids_from_review_queue(self, since=None):

328 def fetch_attachment_ids_from_review_queue(self, since=None, only_security_bugs=False):

        unreviewed_patches = sum([bug.unreviewed_patches()

                                  for bug in self._all_bugs()], [])

        if since:

            unreviewed_patches = [patch for patch in unreviewed_patches

                                        if patch.attach_date() >= since]

334 if only_security_bugs:

335 unreviewed_patches = filter(lambda patch: patch.bug().is_security_sensitive(), unreviewed_patches)

        return map(lambda patch: patch.id(), unreviewed_patches)

    def fetch_patches_from_commit_queue(self):

@@class MockBugzilla(object):

    bug_server_url = "http://example.com"

347 bug_cache = _id_to_object_dictionary(_bug1, _bug2, _bug3, _bug4, _bug5, _bug6, _bug7)

381 bug_cache = _id_to_object_dictionary(_bug1, _bug2, _bug3, _bug4, _bug5, _bug6, _bug7, _bug8)

348382

    attachment_cache = _id_to_object_dictionary(_patch1,

                                                _patch2,

                                                _patch3,

                                                _patch4,

                                                _patch5,

                                                _patch6,

                                                _patch7,

                                                _patch8)

383 attachment_cache = _id_to_object_dictionary(_patch1, _patch2, _patch3, _patch4, _patch5, _patch6,

384 _patch7, _patch8, _patch9)

    def __init__(self):

        self.queries = MockBugzillaQueries(self)

@@class MockBugzilla(object):

    def set_override_patch(self, patch):

        self._override_patch = patch

398 def fetch_attachment(self, attachment_id):

426 def fetch_attachment(self, attachment_id, throw_on_access_error=False):

        if self._override_patch:

            return self._override_patch

402 attachment_dictionary = self.attachment_cache.get(attachment_id)

430 attachment_dictionary = self.attachment_cache.get(int(attachment_id))

        if not attachment_dictionary:

            print("MOCK: fetch_attachment: %s is not a known attachment id" % attachment_id)

            return None

        bug = self.fetch_bug(attachment_dictionary["bug_id"])

435 if bug.is_security_sensitive() and throw_on_access_error:

436 raise Bugzilla.AccessError(attachment_id, Bugzilla.AccessError.NOT_PERMITTED, 'Bug Access Denied')

        for attachment in bug.attachments(include_obsolete=True):

            if attachment.id() == int(attachment_id):

                return attachment

    def fetch_attachment_contents(self, attachment_id):

        return 'Patch'

    def bug_url_for_bug_id(self, bug_id):

        return "%s/%s" % (self.bug_server_url, bug_id)

@@class MockBugzilla(object):

            _log.info("-- Begin comment --")

            _log.info(comment_text)

            _log.info("-- End comment --")

        bug = self.fetch_bug(bug_id)

        if bug.bug_dictionary:

            patches = bug.patches()

            if len(patches) == 1:

                return patches[0].id()

        return '10001'

    def add_cc_to_bug(self, bug_id, ccs):

Tools/Scripts/webkitpy/common/net/bugzilla/bugzilla_unittest.py

@@ZEZpbmlzaExvYWRXaXRoUmVhc29uOnJlYXNvbl07Cit9CisKIEBlbmQKIAogI2VuZGlmCg==

</bugzilla>

""" % _bug_xml

    _single_not_permitted_bug_xml = """

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>

<!DOCTYPE bugzilla SYSTEM "https://bugs.webkit.org/bugzilla.dtd">

<bugzilla version="3.2.3"

          urlbase="https://bugs.webkit.org/"

          maintainer="admin@webkit.org"

>

    <bug error="NotPermitted">

        <bug_id>32585</bug_id>

    </bug>

</bugzilla>

"""

    _expected_example_bug_parsing = {

        "id" : 32585,

        "title" : u"bug to test webkit-patch's and commit-queue's failures",

@@Ignore this bug. Just for testing failure modes of webkit-patch and the commit-

        bug = Bugzilla()._parse_bug_dictionary_from_xml(self._single_bug_xml)

        self._assert_dictionaries_equal(bug, self._expected_example_bug_parsing)

    def test_parse_bug_dictionary_from_xml_for_not_permitted_bug(self):

        bug = Bugzilla()._parse_bug_dictionary_from_xml(self._single_not_permitted_bug_xml)

        self.assertEqual(bug, {})

    _sample_multi_bug_xml = """

<bugzilla version="3.2.3" urlbase="https://bugs.webkit.org/" maintainer="admin@webkit.org" exporter="eric@webkit.org">

    %s

@@Ignore this bug. Just for testing failure modes of webkit-patch and the commit-

    def test_attachment_detail_bug_parsing(self):

        bugzilla = Bugzilla()

248 self.assertEqual(27314, bugzilla._parse_bug_id_from_attachment_page(self._sample_attachment_detail_page))

265 self.assertEqual((27314, None), bugzilla._parse_bug_id_from_attachment_page(self._sample_attachment_detail_page))

    def test_add_cc_to_bug(self):

        bugzilla = Bugzilla()

Tools/Scripts/webkitpy/common/net/bugzilla/constants.py

# Example: 2010-01-20 14:31 PST

# FIXME: Some bugzilla dates seem to have seconds in them?

# Python does not support timezones out of the box.

# Assume that bugzilla always uses PST (which is true for bugs.webkit.org)

BUGZILLA_DATE_FORMAT = "%Y-%m-%d %H:%M:%S"

Tools/Scripts/webkitpy/common/net/statusserver.py

#

# Redistribution and use in source and binary forms, with or without

# modification, are permitted provided that the following conditions are

#

# This the client designed to talk to Tools/QueueStatusServer.

32from webkitpy.common.config.urls import statusserver_default_host

33from webkitpy.common.net.bugzilla.attachment import Attachment

3134from webkitpy.common.net.networktransaction import NetworkTransaction

3235from webkitpy.thirdparty.BeautifulSoup import BeautifulSoup

33 from webkitpy.common.config.urls import statusserver_default_host

3436

37import StringIO

import logging

import urllib2

@@class StatusServer:

        self._browser.set_handle_robots(False)

        self.set_bot_id(bot_id)

49 def set_host(self, host):

52 def set_host(self, host, use_https=False):

5053 self.host = host

51 self.url = "http://%s" % self.host

        if use_https:

            protocol_to_use = 'https'

        else:

            protocol_to_use = 'http'

        self.url = '{}://{}'.format(protocol_to_use, self.host)

    def set_bot_id(self, bot_id):

        self.bot_id = bot_id

    def set_api_key(self, api_key):

        AUTHORIZATION_HEADER_NAME = 'Authorization'

        new_headers = filter(lambda header: header[0] != AUTHORIZATION_HEADER_NAME, self._browser.addheaders)

        if api_key:

            new_headers.append((AUTHORIZATION_HEADER_NAME, 'APIKey ' + api_key))

        self._browser.addheaders = new_headers

    def results_url_for_status(self, status_id):

        return "%s/results/%s" % (self.url, status_id)

@@class StatusServer:

        self._browser["high_priority_work_items"] = " ".join(high_priority_work_items)

        return self._browser.submit().read()

    def _upload_attachment_to_server(self, attachment_id, attachment_metadata, attachment_data):

        upload_attachment_url = '{}/upload-attachment'.format(self.url)

        self._browser.open(upload_attachment_url)

        self._browser.select_form(name='upload_attachment')

        self._browser['attachment_id'] = unicode(attachment_id)

        self._browser.add_file(StringIO.StringIO(unicode(attachment_metadata)), 'application/json', 'attachment-{}-metadata.json'.format(attachment_id), 'attachment_metadata')

        if isinstance(attachment_data, unicode):

            attachment_data = attachment_data.encode('utf-8')

        self._browser.add_file(StringIO.StringIO(attachment_data), 'text/plain', 'attachment-{}.patch'.format(attachment_id), 'attachment_data')

        self._browser.submit()

    def upload_attachment(self, attachment):

        _log.info('Uploading attachment {} to status server'.format(attachment.id()))

        return NetworkTransaction().run(lambda: self._upload_attachment_to_server(attachment.id(), attachment.to_json(), attachment.contents()))

    def _post_work_item_to_ews(self, attachment_id):

        submit_to_ews_url = "%s/submit-to-ews" % self.url

        self._browser.open(submit_to_ews_url)

@@class StatusServer:

        _log.info("SVN revision: %s broke %s" % (svn_revision_number, broken_bot))

        return NetworkTransaction().run(lambda: self._post_svn_revision_to_server(svn_revision_number, broken_bot))

    def _fetch_attachment_page(self, action, attachment_id):

        attachment_url = '{}/attachment/{}/{}'.format(self.url, action, attachment_id)

        return self._fetch_url(attachment_url)

    def fetch_attachment(self, attachment_id):

        # We will neither have metadata nor content if our API key is missing, invalid, or revoked.

        attachment_metadata = self._fetch_attachment_page('metadata', attachment_id)

        if not attachment_metadata:

            return None

        attachment_contents = self._fetch_attachment_page('data', attachment_id)

        if not attachment_contents:

            return None

        attachment = Attachment.from_json(attachment_metadata)

        attachment.contents = lambda: attachment_contents

        return attachment

    def _fetch_url(self, url):

        # FIXME: This should use NetworkTransaction's 404 handling instead.

        try:

Tools/Scripts/webkitpy/common/net/statusserver_mock.py

import logging

31from webkitpy.common.net.bugzilla.attachment import Attachment

32

_log = logging.getLogger(__name__)

@@class MockStatusServer(object):

        self._work_items = work_items

        _log.info("MOCK: update_work_items: %s %s" % (queue_name, high_priority_work_items + work_items))

    def upload_attachment(self, attachment):

        _log.info('MOCK: upload_attachment: {}'.format(attachment.id()))

    def submit_to_ews(self, patch_id):

        _log.info("MOCK: submit_to_ews: %s" % (patch_id))

@@class MockStatusServer(object):

    def results_url_for_status(self, status_id):

        return "http://dummy_url"

    def fetch_attachment(self, attachment_id):

        attachment = Attachment({'id': 10008}, None)

        attachment.content = lambda: 'Patch'

        return attachment

Tools/Scripts/webkitpy/tool/bot/feeders.py

#

# Redistribution and use in source and binary forms, with or without

# modification, are permitted provided that the following conditions are

@@class EWSFeeder(AbstractFeeder):

        AbstractFeeder.__init__(self, tool)

    def feed(self):

87 ids_needing_review = set(self._tool.bugs.queries.fetch_attachment_ids_from_review_queue(datetime.today() - timedelta(7)))

        current_time = datetime.today()

        ids_needing_review = set(self._tool.bugs.queries.fetch_attachment_ids_from_review_queue(current_time - timedelta(7)))

        security_ids_needing_review = frozenset(self._tool.bugs.queries.fetch_attachment_ids_from_review_queue(current_time - timedelta(7), only_security_bugs=True))

        new_ids = ids_needing_review.difference(self._ids_sent_to_server)

        _log.info("Feeding EWS (%s, %s new)" % (pluralize(len(ids_needing_review), "r? patch"), len(new_ids)))

        for attachment_id in new_ids:  # Order doesn't really matter for the EWS.

            # Download patches from security sensitive bugs and upload them to the status server since the

            # EWS queues do not have permission to fetch them directly from Bugzilla.

            attachment_data = None

            if attachment_id in security_ids_needing_review:

                attachment = self._tool.bugs.fetch_attachment(attachment_id)

                if not attachment:

                    _log.error('Failed to retrieve attachment {}'.format(attachment_id))

                    continue

                self._tool.status_server.upload_attachment(attachment)

91103 self._tool.status_server.submit_to_ews(attachment_id)

92104 self._ids_sent_to_server.add(attachment_id)

Tools/Scripts/webkitpy/tool/commands/download.py

@@class AbstractPatchSequencingCommand(AbstractPatchProcessingCommand):

class ProcessAttachmentsMixin(object):

    def _fetch_list_of_patches_to_process(self, options, args, tool):

220 return map(lambda patch_id: tool.bugs.fetch_attachment(patch_id), args)

220 return filter(None, map(lambda patch_id: tool.bugs.fetch_attachment(patch_id), args))

class ProcessBugsMixin(object):

    def _fetch_list_of_patches_to_process(self, options, args, tool):

        all_patches = []

        for bug_id in args:

227 patches = tool.bugs.fetch_bug(bug_id).reviewed_patches()

            bug = tool.bugs.fetch_bug(bug_id)

            if not bug:

                continue

            patches = bug.reviewed_patches()

            _log.info("%s found on bug %s." % (pluralize(len(patches), "reviewed patch"), bug_id))

            all_patches += patches

        if not all_patches:

            _log.info("No reviewed patches found, looking for unreviewed patches.")

            for bug_id in args:

233 patches = tool.bugs.fetch_bug(bug_id).patches()

                bug = tool.bugs.fetch_bug(bug_id)

                if not bug:

                    continue

                patches = bug.patches()

                _log.info("%s found on bug %s." % (pluralize(len(patches), "patch"), bug_id))

                all_patches += patches

        return all_patches

Tools/Scripts/webkitpy/tool/commands/earlywarningsystem.py

@@from optparse import make_option

from webkitpy.common.config.committers import CommitterList

from webkitpy.common.config.ports import DeprecatedPort

37from webkitpy.common.net.bugzilla import Bugzilla

from webkitpy.common.system.filesystem import FileSystem

from webkitpy.common.system.executive import ScriptError

from webkitpy.tool.bot.earlywarningsystemtask import EarlyWarningSystemTask, EarlyWarningSystemTaskDelegate

@@class AbstractEarlyWarningSystem(AbstractReviewQueue, EarlyWarningSystemTaskDele

        return self._group

    def refetch_patch(self, patch):

153 return self._tool.bugs.fetch_attachment(patch.id())

        patch_id = patch.id()

        try:

            patch = self._tool.bugs.fetch_attachment(patch_id, throw_on_access_error=True)

        except Bugzilla.AccessError as e:

            # FIXME: Need a way to ask the status server to fetch the patch again. For now

            # we return the attachment as it was when it was originally uploaded to the

            # status server.

            if e.error_code == Bugzilla.AccessError.NOT_PERMITTED:

                patch = self._tool.status_server.fetch_attachment(patch_id)

        return patch

    def report_flaky_tests(self, patch, flaky_test_results, results_archive):

        pass

Tools/Scripts/webkitpy/tool/commands/queries_unittest.py

@@class QueryCommandsTest(CommandsTest):

            "Bugs with attachments pending review:\n" \

            "http://webkit.org/b/bugid   Description (age in days)\n" \

            "http://webkit.org/b/50001   Bug with a patch needing review. (0)\n" \

121 "Total: 1\n"

121 "http://webkit.org/b/50007 Security bug with a patch needing review. (0)\n" \

122 "Total: 2\n"

        self.assert_execute_outputs(PatchesToReview(), None, expected_stdout, expected_stderr, options=options)

        options.cc_email = None

Tools/Scripts/webkitpy/tool/commands/queues.py

@@from StringIO import StringIO

from webkitpy.common.config.committervalidator import CommitterValidator

from webkitpy.common.config.ports import DeprecatedPort

43 from webkitpy.common.net.bugzilla import Attachment

43from webkitpy.common.net.bugzilla import Bugzilla, Attachment

from webkitpy.common.system.executive import ScriptError

from webkitpy.tool.bot.botinfo import BotInfo

from webkitpy.tool.bot.commitqueuetask import CommitQueueTask, CommitQueueTaskDelegate

@@class AbstractPatchQueue(AbstractQueue):

            patch_id = self._tool.status_server.next_work_item(self.name)

            if not patch_id:

                return None

220 patch = self._tool.bugs.fetch_attachment(patch_id)

            try:

                patch = self._tool.bugs.fetch_attachment(patch_id, throw_on_access_error=True)

            except Bugzilla.AccessError as e:

                if e.error_code == Bugzilla.AccessError.NOT_PERMITTED:

                    patch = self._tool.status_server.fetch_attachment(patch_id)

            if not patch:

                # FIXME: Using a fake patch because release_work_item has the wrong API.

                # We also don't really need to release the lock (although that's fine),

Tools/Scripts/webkitpy/tool/commands/queues_unittest.py

@@MOCK setting flag 'commit-queue' to '-' on attachment '10001' with comment 'Reje

- If you have committer rights please correct the error in Tools/Scripts/webkitpy/common/config/contributors.json by adding yourself to the file (no review needed).  The commit-queue restarts itself every 2 hours.  After restart the commit-queue will correctly respect your committer rights.'

Feeding commit-queue high priority items [10005], regular items [10000]

MOCK: update_work_items: commit-queue [10005, 10000]

146 Feeding EWS (1 r? patch, 1 new)

Feeding EWS (2 r? patches, 2 new)

MOCK: upload_attachment: 10008

MOCK: submit_to_ews: 10008

MOCK: submit_to_ews: 10002

""",

            "handle_unexpected_error": "Mock error message\n",

@@class AbstractPatchQueueTest(CommandsTest):

        queue._options = Mock()

        queue._options.port = None

        self.assertIsNone(queue._next_patch())

162 tool.status_server = MockStatusServer(work_items=[2, 10000, 10001])

164 tool.status_server = MockStatusServer(work_items=[2, 10000, 10001, 10008])

        expected_stdout = "MOCK: fetch_attachment: 2 is not a known attachment id\n"  # A mock-only message to prevent us from making mistakes.

        expected_logs = "MOCK: release_work_item: None 2\n"

        patch = OutputCapture().assert_outputs(self, queue._next_patch, expected_stdout=expected_stdout, expected_logs=expected_logs)

        # The patch.id() == 2 is ignored because it doesn't exist.

        self.assertEqual(patch.id(), 10000)

        self.assertEqual(queue._next_patch().id(), 10001)

171 self.assertEqual(queue._next_patch().id(), 10008)

        self.assertEqual(queue._next_patch(), None)    # When the queue is empty

Tools/Scripts/webkitpy/tool/commands/upload_unittest.py

#

# Redistribution and use in source and binary forms, with or without

# modification, are permitted provided that the following conditions are

@@MOCK: submit_to_ews: 10001

"""

        self.assert_execute_outputs(Upload(), [50000], options=options, expected_logs=expected_logs)

    def test_upload_of_security_sensitive_patch_with_no_review_and_ews(self):

        options = MockOptions()

        options.cc = None

        options.check_style = True

        options.check_style_filter = None

        options.comment = None

        options.description = 'MOCK description'

        options.non_interactive = False

        options.request_commit = False

        options.review = False

        options.ews = True

        options.sort_xcode_project = False

        options.suggest_reviewers = False

        expected_logs = """MOCK: user.open_url: file://...

Was that diff correct?

Obsoleting 1 old patch on bug 50007

MOCK add_patch_to_bug: bug_id=50007, description=MOCK description, mark_for_review=False, mark_for_commit_queue=False, mark_for_landing=False

MOCK: user.open_url: http://example.com/50007

MOCK: upload_attachment: 10008

MOCK: submit_to_ews: 10008

"""

        self.assert_execute_outputs(Upload(), [50007], options=options, expected_logs=expected_logs)

    def test_mark_bug_fixed(self):

        tool = MockTool()

        tool._scm.last_svn_commit_log = lambda: "r9876 |"

Tools/Scripts/webkitpy/tool/main.py

@@from optparse import make_option

import os

import threading

36from webkitpy.common.checkout.scm import Git

from webkitpy.common.config.ports import DeprecatedPort

from webkitpy.common.host import Host

from webkitpy.common.net.irc import ircproxy

@@class WebKitPatch(MultiCommandTool, Host):

        make_option("-v", "--verbose", action="store_true", dest="verbose", default=False, help="enable all logging"),

        make_option("-d", "--directory", action="append", dest="patch_directories", default=[], help="Directory to look at for changed files"),

        make_option("--status-host", action="store", dest="status_host", type="string", help="Hostname (e.g. localhost or commit.webkit.org) where status updates should be posted."),

50 make_option("--status-host-uses-https", action="store_true", default=False, dest="status_host_uses_https", help="Use HTTPS when querying the status host."),

        make_option("--bot-id", action="store", dest="bot_id", type="string", help="Identifier for this bot (if multiple bots are running for a queue)"),

        make_option("--irc-password", action="store", dest="irc_password", type="string", help="Password to use when communicating via IRC."),

        make_option("--seconds-to-sleep", action="store", default=120, type="int", help="Number of seconds to sleep in the task queue."),

@@class WebKitPatch(MultiCommandTool, Host):

            return self.scm().supports_local_commits()

        return True

    @staticmethod

    def _status_server_api_key_from_git():

        try:

            if not Git.in_working_directory(os.getcwd()):

                return None

            return Git.read_git_config('webkit.status_api_key')

        except OSError as e:

            # Catch and ignore OSError exceptions such as "no such file

            # or directory" (OSError errno 2), which imply that the Git

            # command cannot be found/is not installed.

            pass

        return None

    @staticmethod

    def _status_server_api_key():

        api_key = os.environ.get('WEBKIT_STATUS_API_KEY')

        if not api_key:

            api_key = WebKitPatch._status_server_api_key_from_git()

        return api_key

    # FIXME: This may be unnecessary since we pass global options to all commands during execute() as well.

    def handle_global_options(self, options):

        self.initialize_scm(options.patch_directories)

        api_key = self._status_server_api_key()

        if api_key:

            self.status_server.set_api_key(api_key)

95122 if options.status_host:

96 self.status_server.set_host(options.status_host)

123 self.status_server.set_host(options.status_host, use_https=options.status_host_uses_https)

        if options.bot_id:

            self.status_server.set_bot_id(options.bot_id)

        if options.irc_password:

Tools/Scripts/webkitpy/tool/steps/obsoletepatches.py

@@class ObsoletePatches(AbstractStep):

        if not self._options.obsolete_patches:

            return

        bug_id = state["bug_id"]

49 patches = self._tool.bugs.fetch_bug(bug_id).patches()

        bug = self._tool.bugs.fetch_bug(bug_id)

        if not bug:

            return

        patches = bug.patches()

        if not patches:

            return

        _log.info("Obsoleting %s on bug %s" % (pluralize(len(patches), "old patch"), bug_id))

Tools/Scripts/webkitpy/tool/steps/submittoews.py

@@class SubmitToEWS(AbstractStep):

    def run(self, state):

        for attachment_id in state.get('attachment_ids', []):

            attachment = self._tool.bugs.fetch_attachment(attachment_id)

            if attachment is None:

                continue  # Either Bugzilla is down or we do not have permission to view the attachment.

            if attachment.bug().is_security_sensitive():

                self._tool.status_server.upload_attachment(attachment)

3843 self._tool.status_server.submit_to_ews(attachment_id)