Attachment #364816 for bug #195755

View Details Raw Unified Return to Bug 195755

| Differences between



2019-03-15  Jonathan Bedard  <jbedard@apple.com>

        webkitpy: Upload test results
        https://bugs.webkit.org/show_bug.cgi?id=195755
        <rdar://problem/48896182>

        Reviewed by NOBODY (OOPS!).

        Establish a new format for uploading results that is not tied to layout tests, apply
        that format to webkitpy tests.

        * Scripts/webkitpy/common/checkout/scm/git.py:
        (Git.native_branch): Return what branch the current checkout is on.
        * Scripts/webkitpy/common/checkout/scm/scm_mock.py:
        * Scripts/webkitpy/common/checkout/scm/scm_unittest.py:
        * Scripts/webkitpy/common/checkout/scm/svn.py:
        (SVN.native_branch): Ditto.
        * Scripts/webkitpy/common/system/platforminfo.py:
        (PlatformInfo.build_version): Return a build version for Mac.
        * Scripts/webkitpy/common/system/platforminfo_mock.py:
        (MockPlatformInfo.__init__):
        (MockPlatformInfo.build_version):
        * Scripts/webkitpy/results: Added.
        * Scripts/webkitpy/results/__init__.py: Added.
        * Scripts/webkitpy/results/options.py: Added.
        (upload_options): OptParse list for upload options.
        * Scripts/webkitpy/results/upload.py: Added.
        (Upload): Class which enforces the upload format expected by the results server.
        (Upload.Expectations):
        (Upload.create_configuration):
        (Upload.create_commit):
        (Upload.create_details):
        (Upload.create_run_stats):
        (Upload.create_test_result):
        (Upload.__init__):
        (Upload.Encoder): Encode Upload object as json.
        (Upload.upload): Upload results to the results server, returning 'True' if the upload is successful.
        * Scripts/webkitpy/results/upload_unittest.py: Added.
        * Scripts/webkitpy/test/main.py:
        (Tester._parse_args): Add upload arguments.
        (Tester._run_tests): Allow results to be uploaded.
        * Scripts/webkitpy/test/runner.py:
        (Runner.__init__): Record which tests were run, rather than just counting them.
        (Runner.handle):
        * Scripts/webkitpy/test/runner_unittest.py:
        (RunnerTest.test_run):
        * Scripts/webkitpy/thirdparty/__init__.py:
        (AutoinstallImportHook.find_module): Add requests auto-install.
        (AutoinstallImportHook._install_requests):
        * Scripts/webkitpy/tool/commands/queues_unittest.py:
        (PatchProcessingQueueTest.test_upload_results_archive_for_patch): Update os name for testing.

2019-03-14  Fujii Hironori  <Hironori.Fujii@sony.com>

        [Win][MinBrowser][WK2] Implement createNewPage of WKPageUIClient to open a new window



# Required for Python to search this directory for module files

# Keep this file free of any code or import statements that could
# cause either an error to occur or a log message to be logged.
# This ensures that calling code can import initialization code from
# webkitpy before any errors or log messages due to code in this file.
# Initialization code can include things like version-checking code and
# logging configuration code.
#
# We do not execute any version-checking code or logging configuration
# code in this file so that callers can opt-in as they want.  This also
# allows different callers to choose different initialization code,
# as necessary.



# Copyright (C) 2019 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1.  Redistributions of source code must retain the above copyright
#     notice, this list of conditions and the following disclaimer.
# 2.  Redistributions in binary form must reproduce the above copyright
#     notice, this list of conditions and the following disclaimer in the
#     documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

import optparse


def upload_options():
    return [
        optparse.make_option('--report', action='append', dest='report_urls', help='URL (or URLs) to report test results to'),
        optparse.make_option('--buildbot-master', help='The url of the buildbot master.'),
        optparse.make_option('--builder-name', help='The name of the buildbot builder tests were run on.'),
        optparse.make_option('--build-number', help='The buildbot build number tests are associated with.'),
        optparse.make_option('--buildbot-worker', help='The buildbot worker tests were run on.'),
    ]



# Copyright (C) 2019 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1.  Redistributions of source code must retain the above copyright
#     notice, this list of conditions and the following disclaimer.
# 2.  Redistributions in binary form must reproduce the above copyright
#     notice, this list of conditions and the following disclaimer in the
#     documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

import webkitpy.thirdparty.autoinstalled.requests

import json
import requests
import sys

import platform as platform_


class Upload(object):
    UPLOAD_ENDPOINT = '/api/upload'
    BUILDBOT_DETAILS = ['buildbot-master', 'builder-name', 'build-number', 'buildbot-worker']
    VERSION = 0

    class Expectations:
        # These are ordered by priority, meaning that a test which both crashes and has
        # a warning should be considered to have crashed.
        ORDER = [
            'CRASH',
            'TIMEOUT',
            'IMAGE',   # Image-diff
            'AUDIO',   # Audio-diff
            'TEXT',    # Text-diff
            'FAIL',
            'ERROR',
            'WARNING',
            'PASS',
        ]
        CRASH, TIMEOUT, IMAGE, AUDIO, TEXT, FAIL, ERROR, WARNING, PASS = ORDER

    @staticmethod
    def create_configuration(
            platform=None,
            is_simulator=False,
            version=None,
            architecture=None,
            version_name=None,
            model=None,
            style=None,   # Debug/Production/Release
            flavor=None,  # Dumping ground suite-wide configuration changes (ie, GuardMalloc)
            sdk=None,
        ):

        # This deviates slightly from the rest of webkitpy, but it allows this file to be entirely portable.
        config = dict(
            platform=platform or (platform_.system() if platform_.system() != 'Darwin' else 'mac').lower(),
            is_simulator=is_simulator,
            version=version or (platform_.release() if platform_.system() != 'Darwin' else platform_.mac_ver()[0]),
            architecture=architecture or platform_.machine(),
        )
        for key, value in config.iteritems():
            if value is None:
                raise ValueError('{} must be defined in the configuration'.format(key))
        for key, value in dict(version_name=version_name, model=model, style=style, flavor=flavor, sdk=sdk).iteritems():
            if value is not None:
                config[key] = value
        return config

    @staticmethod
    def create_commit(repository_id=None, id=None, branch=None):
        commit = dict(repository_id=repository_id, id=id)
        for key, value in commit.iteritems():
            if value is None:
                raise ValueError('{} must be defined in a commit'.format(key))
        if branch:
            commit['branch'] = branch
        return commit

    @staticmethod
    def create_details(link=None, options=None, **kwargs):
        result = dict(**kwargs)
        if link:
            result['link'] = link

        if options:
            for element in Upload.BUILDBOT_DETAILS:
                value = getattr(options, element.replace('-', '_'), None)
                if value is not None:
                    result[element] = value
        return result

    @staticmethod
    def create_run_stats(start_time=None, end_time=None, tests_skipped=None, **kwargs):
        stats = dict(**kwargs)
        for key, value in dict(start_time=start_time, end_time=end_time, tests_skipped=tests_skipped).iteritems():
            if value is not None:
                stats[key] = value
        return stats

    @staticmethod
    def create_test_result(expected=None, actual=None, log=None, **kwargs):
        result = dict(**kwargs)

        # Tests which don't declare expectations or results are assumed to have passed.
        for key, value in dict(expected=expected, actual=actual, log=log).iteritems():
            if value is not None:
                result[key] = value
        return result

    def __init__(self, suite=None, configuration=None, commits=[], timestamp=None, details=None, run_stats=None, results={}):
        self.suite = suite
        self.configuration = configuration
        self.commits = commits
        self.timestamp = timestamp
        self.details = details
        self.run_stats = run_stats
        self.results = results

    class Encoder(json.JSONEncoder):

        def default(self, obj):
            if isinstance(obj, Upload):
                if not obj.suite:
                    raise ValueError('No suite specified to results upload')
                if not obj.commits:
                    raise ValueError('No commits specified to results upload')

                details = obj.details or obj.create_details()
                buildbot_args = [details.get(arg, None) is None for arg in obj.BUILDBOT_DETAILS]
                if any(buildbot_args) and not all(buildbot_args):
                    raise ValueError('All buildbot details must be defined for upload')

                def unpack_test(current, path_to_test, data):
                    if len(path_to_test) == 1:
                        current[path_to_test[0]] = data
                        return
                    if not current.get(path_to_test[0]):
                        current[path_to_test[0]] = {}
                    unpack_test(current[path_to_test[0]], path_to_test[1:], data)

                results = {}
                for test, data in obj.results.iteritems():
                    unpack_test(results, test.split('/'), data)

                result = dict(
                    version=obj.VERSION,
                    suite=obj.suite,
                    configuration=obj.configuration or obj.create_configuration(),
                    commits=obj.commits,
                    test_results=dict(
                        details=details,
                        run_stats=obj.run_stats or obj.create_run_stats(),
                        results=results,
                    ),
                )
                if obj.timestamp:
                    result['timestamp'] = obj.timestamp
                return result
            return super(Upload.Encoder, self).default(obj)

    def upload(self, url, log_line=lambda val: sys.stdout.write(val + '\n')):
        try:
            response = requests.post(url + self.UPLOAD_ENDPOINT, data=json.dumps(self, cls=Upload.Encoder))
        except requests.exceptions.ConnectionError:
            log_line(' ' * 4 + 'Failed to upload to {}, results server not online'.format(url))
            return False
        except ValueError as e:
            log_line(' ' * 4 + 'Failed to encode upload data: {}'.format(e))
            return False

        if response.status_code != 200:
            log_line(' ' * 4 + 'Error uploading to {}:'.format(url))
            log_line(' ' * 8 + response.json()['description'])
            return False

        log_line(' ' * 4 + 'Uploaded results to {}'.format(url))
        return True



# Copyright (C) 2019 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1.  Redistributions of source code must retain the above copyright
#     notice, this list of conditions and the following disclaimer.
# 2.  Redistributions in binary form must reproduce the above copyright
#     notice, this list of conditions and the following disclaimer in the
#     documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

import webkitpy.thirdparty.autoinstalled.requests

import collections
import json
import requests
import time
import unittest

from webkitpy.results.upload import Upload
from webkitpy.thirdparty import mock


class UploadTest(unittest.TestCase):

    class Options(object):
        def __init__(self, **kwargs):
            for key, value in kwargs.iteritems():
                setattr(self, key, value)

    class MockResponse(object):
        def __init__(self, status_code=200, text=''):
            self.status_code = status_code
            self.text = text

        def json(self):
            return json.loads(self.text)

    @staticmethod
    def normalize(data):
        if isinstance(data, basestring):
            return str(data)
        elif isinstance(data, collections.Mapping):
            return dict(map(UploadTest.normalize, data.iteritems()))
        elif isinstance(data, collections.Iterable):
            return type(data)(map(UploadTest.normalize, data))
        else:
            return data

    def test_encoding(self):
        start_time, end_time = time.time() - 3, time.time()
        upload = Upload(
            suite='webkitpy-tests',
            configuration=Upload.create_configuration(
                platform='mac',
                version='10.13.0',
                version_name='High Sierra',
                architecture='x86_64',
                sdk='17A405',
            ),
            details=Upload.create_details(link='https://webkit.org'),
            commits=[Upload.create_commit(
                repository_id='webkit',
                id='5',
                branch='trunk',
            )],
            run_stats=Upload.create_run_stats(
                start_time=start_time,
                end_time=end_time,
                tests_skipped=0,
            ),
            results={
                'webkitpy.test1': {},
                'webkitpy.test2': Upload.create_test_result(expected=Upload.Expectations.PASS, actual=Upload.Expectations.FAIL),
            },
        )
        generated_dict = self.normalize(json.loads(json.dumps(upload, cls=Upload.Encoder)))

        self.assertEqual(generated_dict['version'], 0)
        self.assertEqual(generated_dict['suite'], 'webkitpy-tests')
        self.assertEqual(generated_dict['configuration'], self.normalize(dict(
            platform='mac',
            is_simulator=False,
            version='10.13.0',
            version_name='High Sierra',
            architecture='x86_64',
            sdk='17A405',
        )))
        self.assertEqual(generated_dict['commits'], [dict(
            repository_id='webkit',
            id='5',
            branch='trunk',
        )])
        self.assertEqual(generated_dict['test_results']['details'], self.normalize(dict(link='https://webkit.org')))
        self.assertEqual(generated_dict['test_results']['run_stats'], self.normalize(dict(
            start_time=start_time,
            end_time=end_time,
            tests_skipped=0,
        )))
        self.assertEqual(generated_dict['test_results']['results'], self.normalize({
            'webkitpy.test1': {},
            'webkitpy.test2': Upload.create_test_result(expected=Upload.Expectations.PASS, actual=Upload.Expectations.FAIL),
        }))

    def test_upload(self):
        upload = Upload(
            suite='webkitpy-tests',
            commits=[Upload.create_commit(
                repository_id='webkit',
                id='5',
                branch='trunk',
            )],
        )

        with mock.patch('requests.post', new=lambda url, data: self.MockResponse()):
            self.assertTrue(upload.upload('https://webkit.org/results', log_line=lambda _: None))

        def raise_exception(url, data):
            raise requests.exceptions.ConnectionError()

        with mock.patch('requests.post', new=raise_exception):
            self.assertFalse(upload.upload('https://webkit.org/results', log_line=lambda _: None))

        with mock.patch('requests.post', new=lambda url, data: self.MockResponse(
            status_code=404,
            text=json.dumps(dict(description='No such address')),
        )):
            self.assertFalse(upload.upload('https://webkit.org/results', log_line=lambda _: None))

    def test_packed_test(self):
        upload = Upload(
            suite='webkitpy-tests',
            commits=[Upload.create_commit(
                repository_id='webkit',
                id='5',
                branch='trunk',
            )],
            results={
                'dir1/sub-dir1/test1': Upload.create_test_result(actual=Upload.Expectations.FAIL),
                'dir1/sub-dir1/test2': Upload.create_test_result(actual=Upload.Expectations.TIMEOUT),
                'dir1/sub-dir2/test3': {},
                'dir1/sub-dir2/test4': {},
                'dir2/sub-dir3/test5': {},
                'dir2/test6': {},
            }
        )
        generated_dict = self.normalize(json.loads(json.dumps(upload, cls=Upload.Encoder)))
        self.assertEqual(generated_dict['test_results']['results'], self.normalize({
            'dir1': {
                'sub-dir1': {
                    'test1': {'actual': Upload.Expectations.FAIL},
                    'test2': {'actual': Upload.Expectations.TIMEOUT},
                }, 'sub-dir2': {
                    'test3': {},
                    'test4': {},
                }
            }, 'dir2': {
                'sub-dir3': {'test5': {}},
                'test6': {},
            },
        }))

    def test_no_suite(self):
        upload = Upload(
            commits=[Upload.create_commit(
                repository_id='webkit',
                id='5',
                branch='trunk',
            )],
        )
        with self.assertRaises(ValueError):
            json.dumps(upload, cls=Upload.Encoder)

    def test_no_commits(self):
        upload = Upload(
            suite='webkitpy-tests',
        )
        with self.assertRaises(ValueError):
            json.dumps(upload, cls=Upload.Encoder)

    def test_buildbot(self):
        upload = Upload(
            suite='webkitpy-tests',
            commits=[Upload.create_commit(
                repository_id='webkit',
                id='5',
                branch='trunk',
            )],
            details=Upload.create_details(options=self.Options(
                buildbot_master='webkit.org',
                builder_name='Queue-1',
                build_number=1,
        )))
        with self.assertRaises(ValueError):
            json.dumps(upload, cls=Upload.Encoder)

        upload.details['buildbot-worker'] = 'bot123'
        generated_dict = self.normalize(json.loads(json.dumps(upload, cls=Upload.Encoder)))
        self.assertEqual(generated_dict['test_results']['details'], self.normalize({
            'buildbot-master': 'webkit.org',
            'builder-name': 'Queue-1',
            'build-number': 1,
            'buildbot-worker': 'bot123',
        }))

Lines 290-295 class Git(SCM, SVNRepository): Tools/Scripts/webkitpy/common/checkout/scm/git.py_sec1
- Tools/Scripts/webkitpy/common/checkout/scm/git.py +8 lines
290	def native_revision(self, path):	290	def native_revision(self, path):
291	return self._run_git(['-C', self.find_checkout_root(path), 'log', '-1', '--pretty=format:%H'])	291	return self._run_git(['-C', self.find_checkout_root(path), 'log', '-1', '--pretty=format:%H'])
292		292
		293	def native_branch(self, path):
		294	result = self._run_git(['-C', self.find_checkout_root(path), 'rev-parse', '--abbrev-ref', 'HEAD']).rstrip()
		295
		296	# For git-svn
		297	if result.startswith('heads'):
		298	return result[6:]
		299	return result
		300
293	def svn_url(self):	301	def svn_url(self):
294	git_command = ['svn', 'info']	302	git_command = ['svn', 'info']
295	status = self._run_git(git_command)	303	status = self._run_git(git_command)

Lines 87-92 class MockSCM(object): Tools/Scripts/webkitpy/common/checkout/scm/scm_mock.py_sec1
- Tools/Scripts/webkitpy/common/checkout/scm/scm_mock.py +3 lines
87	def native_revision(self, path):	87	def native_revision(self, path):
88	return self.svn_revision(path)	88	return self.svn_revision(path)
89		89
		90	def native_branch(self, path):
		91	return 'trunk'
		92
90	def timestamp_of_revision(self, path, revision):	93	def timestamp_of_revision(self, path, revision):
91	return '2013-02-01 08:48:05 +0000'	94	return '2013-02-01 08:48:05 +0000'
92		95

Lines 909-914 END Tools/Scripts/webkitpy/common/checkout/scm/scm_unittest.py_sec1
- Tools/Scripts/webkitpy/common/checkout/scm/scm_unittest.py +10 lines
909	self.assertEqual(self.scm.head_svn_revision(), self.scm.native_revision('.'))	909	self.assertEqual(self.scm.head_svn_revision(), self.scm.native_revision('.'))
910	self.assertEqual(self.scm.native_revision('.'), '5')	910	self.assertEqual(self.scm.native_revision('.'), '5')
911		911
		912	def test_native_branch(self):
		913	self.assertEqual(self.scm.native_branch('.'), 'trunk')
		914
912	def test_propset_propget(self):	915	def test_propset_propget(self):
913	filepath = os.path.join(self.svn_checkout_path, "test_file")	916	filepath = os.path.join(self.svn_checkout_path, "test_file")
914	expected_mime_type = "x-application/foo-bar"	917	expected_mime_type = "x-application/foo-bar"
Lines 1112-1117 class GitTest(SCMTest): Tools/Scripts/webkitpy/common/checkout/scm/scm_unittest.py_sec2
1112	command = ['git', '-C', scm.checkout_root, 'rev-parse', 'HEAD']	1115	command = ['git', '-C', scm.checkout_root, 'rev-parse', 'HEAD']
1113	self.assertEqual(scm.native_revision(scm.checkout_root), run_command(command).strip())	1116	self.assertEqual(scm.native_revision(scm.checkout_root), run_command(command).strip())
1114		1117
		1118	def test_native_branch(self):
		1119	scm = self.tracking_scm
		1120	self.assertEqual('master', scm.native_branch(scm.checkout_root))
		1121
1115	def test_rename_files(self):	1122	def test_rename_files(self):
1116	scm = self.tracking_scm	1123	scm = self.tracking_scm
1117		1124
Lines 1627-1632 class GitSVNTest(SCMTest): Tools/Scripts/webkitpy/common/checkout/scm/scm_unittest.py_sec3
1627	command = ['git', '-C', self.git_checkout_path, 'rev-parse', 'HEAD']	1634	command = ['git', '-C', self.git_checkout_path, 'rev-parse', 'HEAD']
1628	self.assertEqual(self.scm.native_revision(self.git_checkout_path), run_command(command).strip())	1635	self.assertEqual(self.scm.native_revision(self.git_checkout_path), run_command(command).strip())
1629		1636
		1637	def test_native_branch(self):
		1638	self.assertEqual('trunk', self.scm.native_branch(self.git_checkout_path))
		1639
1630	def test_to_object_name(self):	1640	def test_to_object_name(self):
1631	relpath = 'test_file_commit1'	1641	relpath = 'test_file_commit1'
1632	fullpath = os.path.realpath(os.path.join(self.git_checkout_path, relpath))	1642	fullpath = os.path.realpath(os.path.join(self.git_checkout_path, relpath))



        SCM.__init__(self, cwd, **kwargs)
        self._bogus_dir = None
        if patch_directories == []:
            raise Exception('Empty list of patch directories passed to SCM.__init__')
        elif patch_directories == None:
            self._patch_directories = [self._filesystem.relpath(cwd, self.checkout_root)]
        else:

    def native_revision(self, path):
        return self.svn_revision(path)

    def native_branch(self, path):
        relative_url = self.value_from_svn_info(path, 'Relative URL')[2:]
        if relative_url.startswith('trunk'):
            return 'trunk'
        elif relative_url.startswith('branch'):
            return relative_url.split('/')[1]
        raise Exception('{} is not a branch'.format(relative_url.split('/')[0]))

    def timestamp_of_revision(self, path, revision):
        # We use --xml to get timestamps like 2013-02-08T08:18:04.964409Z
        repository_root = self.value_from_svn_info(self.checkout_root, 'Repository Root')

Lines 146-151 class PlatformInfo(object): Tools/Scripts/webkitpy/common/system/platforminfo.py_sec1
- Tools/Scripts/webkitpy/common/system/platforminfo.py +5 lines
146	except:	146	except:
147	return sys.maxint	147	return sys.maxint
148		148
		149	def build_version(self):
		150	if self.is_mac():
		151	return self._executive.run_command(['/usr/bin/sw_vers', '-buildVersion'], return_stderr=False, ignore_errors=True).rstrip()
		152	return None
		153
149	def xcode_sdk_version(self, sdk_name):	154	def xcode_sdk_version(self, sdk_name):
150	if self.is_mac():	155	if self.is_mac():
151	# Assumes that xcrun does not write to standard output on failure (e.g. SDK does not exist).	156	# Assumes that xcrun does not write to standard output on failure (e.g. SDK does not exist).

Lines 32-38 from webkitpy.common.version_name_map im Tools/Scripts/webkitpy/common/system/platforminfo_mock.py_sec1
- Tools/Scripts/webkitpy/common/system/platforminfo_mock.py -1 / +6 lines
32		32
33		33
34	class MockPlatformInfo(object):	34	class MockPlatformInfo(object):
35	def __init__(self, os_name='mac', os_version=Version.from_name('Snow Leopard')):	35	def __init__(self, os_name='mac', os_version=Version.from_name('High Sierra')):
36	assert isinstance(os_version, Version)	36	assert isinstance(os_version, Version)
37	self.os_name = os_name	37	self.os_name = os_name
38	self.os_version = os_version	38	self.os_version = os_version
Lines 78-83 class MockPlatformInfo(object): Tools/Scripts/webkitpy/common/system/platforminfo_mock.py_sec2
78	def terminal_width(self):	78	def terminal_width(self):
79	return 80	79	return 80
80		80
		81	def build_version(self):
		82	if self.is_mac():
		83	return '17A405'
		84	return None
		85
81	def xcode_sdk_version(self, sdk_name):	86	def xcode_sdk_version(self, sdk_name):
82	return Version(8, 1)	87	return Version(8, 1)
83		88



# Copyright (C) 2012 Google, Inc.
# Copyright (C) 2010 Chris Jerdonek (cjerdonek@webkit.org)
# Copyright (C) 2018-2019 Apple Inc. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions

import unittest

from webkitpy.common.system.logutils import configure_logging
from webkitpy.common.system.executive import ScriptError
from webkitpy.common.system.filesystem import FileSystem
from webkitpy.common.host import Host
from webkitpy.port.config import Config
from webkitpy.test.finder import Finder
from webkitpy.test.printer import Printer
from webkitpy.test.runner import Runner, unit_test_name
from webkitpy.results.upload import Upload
from webkitpy.results.options import upload_options

_log = logging.getLogger(__name__)

_host = Host()
_webkit_root = None



        parser = optparse.OptionParser(usage='usage: %prog [options] [args...]')

        #  Configuration options only effect the building of lldbWebKitTester.
        for group_name, group_options in {'Upload Options': upload_options(), 'Configuration Options': [
            optparse.make_option('--debug', action='store_const', const='Debug', dest="configuration",
                                 help='Set the configuration to Debug'),
            optparse.make_option('--release', action='store_const', const='Release', dest="configuration",
                                 help='Set the configuration to Release'),
        ]}.iteritems():
            option_group = optparse.OptionGroup(parser, group_name)
            option_group.add_options(group_options)
            parser.add_option_group(option_group)

        parser.add_option('-a', '--all', action='store_true', default=False,
                          help='run all the tests')

        from webkitpy.thirdparty import autoinstall_everything
        autoinstall_everything()

        start_time = time.time()
        config = Config(_host.executive, self.finder.filesystem)
        configuration_to_use = self._options.configuration or config.default_configuration()
        if will_run_lldb_webkit_tests:
            self.printer.write_update('Building lldbWebKitTester ...')
            build_lldbwebkittester = self.finder.filesystem.join(_webkit_root, 'Tools', 'Scripts', 'build-lldbwebkittester')
            config = Config(_host.executive, self.finder.filesystem)
            configuration_to_use = self._options.configuration or config.default_configuration()
            try:
                _host.executive.run_and_throw_if_fail([build_lldbwebkittester, config.flag_for_configuration(configuration_to_use)], quiet=(not bool(self._options.verbose)))
            except ScriptError as e:

        test_runner = Runner(self.printer, loader)
        test_runner.run(parallel_tests, self._options.child_processes)
        test_runner.run(serial_tests, 1)
        end_time = time.time()

        self.printer.print_result(time.time() - start)


        if self._options.coverage:
            cov.stop()
            cov.save()

        failed_uploads = 0
        if self._options.report_urls:
            self.printer.meter.writeln('\n')
            self.printer.write_update('Preparing upload data ...')

            results = {test: {} for test in test_runner.tests_run}
            for test, errors in test_runner.errors:
                results[test] = Upload.create_test_result(actual=Upload.Expectations.ERROR, log='/n'.join(errors))
            for test, failures in test_runner.failures:
                results[test] = Upload.create_test_result(actual=Upload.Expectations.FAIL, log='/n'.join(failures))

            _host.initialize_scm()
            upload = Upload(
                suite='webkitpy-tests',
                configuration=Upload.create_configuration(
                    platform=_host.platform.os_name,
                    version=str(_host.platform.os_version),
                    version_name=_host.platform.os_version_name(),
                    style=configuration_to_use,
                    sdk=_host.platform.build_version(),
                ),
                details=Upload.create_details(options=self._options),
                commits=[Upload.create_commit(
                    repository_id='webkit',
                    id=_host.scm().native_revision(_webkit_root),
                    branch=_host.scm().native_branch(_webkit_root),
                )],
                run_stats=Upload.create_run_stats(
                    start_time=start_time,
                    end_time=end_time,
                    tests_skipped=len(test_runner.tests_run) - len(parallel_tests) - len(serial_tests),
                ),
                results=results,
            )
            for url in self._options.report_urls:
                self.printer.write_update('Uploading to {} ...'.format(url))
                failed_uploads = failed_uploads if upload.upload(url, log_line=self.printer.meter.writeln) else (failed_uploads + 1)
            self.printer.meter.writeln('Uploads completed!')

        if self._options.coverage:
            cov.report(show_missing=False)

        return not self.printer.num_errors and not self.printer.num_failures and not failed_uploads

    def _check_imports(self, names):
        for name in names:



    def __init__(self, printer, loader):
        self.printer = printer
        self.loader = loader
        self.tests_run = []
        self.errors = []
        self.failures = []
        self.worker_factory = lambda caller: _Worker(caller, self.loader)

            self.printer.print_started_test(source, test_name)
            return

        self.tests_run.append(test_name)
        if failures:
            self.failures.append((test_name, failures))
        if errors:

Lines 92-97 class RunnerTest(unittest.TestCase): Tools/Scripts/webkitpy/test/runner_unittest.py_sec1
- Tools/Scripts/webkitpy/test/runner_unittest.py -1 / +1 lines
92	('test3 (Foo)', 'E', 'test3\nerred'))	92	('test3 (Foo)', 'E', 'test3\nerred'))
93	runner = Runner(Printer(stream, options), loader)	93	runner = Runner(Printer(stream, options), loader)
94	runner.run(['Foo.test1', 'Foo.test2', 'Foo.test3'], 1)	94	runner.run(['Foo.test1', 'Foo.test2', 'Foo.test3'], 1)
95	self.assertEqual(runner.tests_run, 3)	95	self.assertEqual(len(runner.tests_run), 3)
96	self.assertEqual(len(runner.failures), 1)	96	self.assertEqual(len(runner.failures), 1)
97	self.assertEqual(len(runner.errors), 1)	97	self.assertEqual(len(runner.errors), 1)

Lines 1-4 Tools/Scripts/webkitpy/thirdparty/__init__.py_sec1
- Tools/Scripts/webkitpy/thirdparty/__init__.py +18 lines
1	# Copyright (C) 2010 Chris Jerdonek (cjerdonek@webkit.org)	1	# Copyright (C) 2010 Chris Jerdonek (cjerdonek@webkit.org)
		2	# Copyright (C) 2019 Apple Inc. All rights reserved.
2	#	3	#
3	# Redistribution and use in source and binary forms, with or without	4	# Redistribution and use in source and binary forms, with or without
4	# modification, are permitted provided that the following conditions	5	# modification, are permitted provided that the following conditions
Lines 114-119 class AutoinstallImportHook(object): Tools/Scripts/webkitpy/thirdparty/__init__.py_sec2
114	self._install_pytest_timeout()	115	self._install_pytest_timeout()
115	elif '.pytest' in fullname:	116	elif '.pytest' in fullname:
116	self._install_pytest()	117	self._install_pytest()
		118	elif '.requests' in fullname:
		119	self._install_requests()
117		120
118	def _install_mechanize(self):	121	def _install_mechanize(self):
119	self._install("https://files.pythonhosted.org/packages/source/m/mechanize/mechanize-0.2.5.tar.gz",	122	self._install("https://files.pythonhosted.org/packages/source/m/mechanize/mechanize-0.2.5.tar.gz",
Lines 161-166 class AutoinstallImportHook(object): Tools/Scripts/webkitpy/thirdparty/__init__.py_sec3
161	self._install("https://files.pythonhosted.org/packages/a2/ec/415d0cccc1ed41cd7fdf69ad989da16a8d13057996371004cab4bafc48f3/pytest-3.6.2.tar.gz",	164	self._install("https://files.pythonhosted.org/packages/a2/ec/415d0cccc1ed41cd7fdf69ad989da16a8d13057996371004cab4bafc48f3/pytest-3.6.2.tar.gz",
162	"pytest-3.6.2/src/pytest.py")	165	"pytest-3.6.2/src/pytest.py")
163		166
		167	def _install_requests(self):
		168	self._ensure_autoinstalled_dir_is_in_sys_path()
		169	self._install("https://files.pythonhosted.org/packages/06/b8/d1ea38513c22e8c906275d135818fee16ad8495985956a9b7e2bb21942a1/certifi-2019.3.9.tar.gz",
		170	"certifi-2019.3.9/certifi")
		171	self._install("https://files.pythonhosted.org/packages/fc/bb/a5768c230f9ddb03acc9ef3f0d4a3cf93462473795d18e9535498c8f929d/chardet-3.0.4.tar.gz",
		172	"chardet-3.0.4/chardet")
		173	self._install("https://files.pythonhosted.org/packages/fc/bb/a5768c230f9ddb03acc9ef3f0d4a3cf93462473795d18e9535498c8f929d/chardet-3.0.4.tar.gz",
		174	"chardet-3.0.4/chardet")
		175	self._install("https://files.pythonhosted.org/packages/ad/13/eb56951b6f7950cadb579ca166e448ba77f9d24efc03edd7e55fa57d04b7/idna-2.8.tar.gz",
		176	"idna-2.8/idna")
		177	self._install("https://files.pythonhosted.org/packages/b1/53/37d82ab391393565f2f831b8eedbffd57db5a718216f82f1a8b4d381a1c1/urllib3-1.24.1.tar.gz",
		178	"urllib3-1.24.1/src/urllib3")
		179	self._install("https://files.pythonhosted.org/packages/52/2c/514e4ac25da2b08ca5a464c50463682126385c4272c18193876e91f4bc38/requests-2.21.0.tar.gz",
		180	"requests-2.21.0/requests")
		181
164	def _install_pylint(self):	182	def _install_pylint(self):
165	self._ensure_autoinstalled_dir_is_in_sys_path()	183	self._ensure_autoinstalled_dir_is_in_sys_path()
166	if (not self._fs.exists(self._fs.join(_AUTOINSTALLED_DIR, "pylint")) or	184	if (not self._fs.exists(self._fs.join(_AUTOINSTALLED_DIR, "pylint")) or



        queue._options = Mock()
        queue._options.port = None
        patch = queue._tool.bugs.fetch_attachment(10001)
        expected_logs = """MOCK add_attachment_to_bug: bug_id=50000, description=Archive of layout-test-results from bot for mac-highsierra filename=layout-test-results.zip mimetype=None
-- Begin comment --
The attached test failures were seen while running run-webkit-tests on the mock-queue.
Port: mac-highsierra  Platform: MockPlatform 1.0
-- End comment --
"""
        OutputCapture().assert_outputs(self, queue._upload_results_archive_for_patch, [patch, Mock()], expected_logs=expected_logs)

Return to Bug 195755

Lines 1-3 Tools/ChangeLog_sec1
- Tools/ChangeLog +52 lines
		1	2019-03-15 Jonathan Bedard <jbedard@apple.com>
		2
		3	webkitpy: Upload test results
		4	https://bugs.webkit.org/show_bug.cgi?id=195755
		5	<rdar://problem/48896182>
		6
		7	Reviewed by NOBODY (OOPS!).
		8
		9	Establish a new format for uploading results that is not tied to layout tests, apply
		10	that format to webkitpy tests.
		11
		12	* Scripts/webkitpy/common/checkout/scm/git.py:
		13	(Git.native_branch): Return what branch the current checkout is on.
		14	* Scripts/webkitpy/common/checkout/scm/scm_mock.py:
		15	* Scripts/webkitpy/common/checkout/scm/scm_unittest.py:
		16	* Scripts/webkitpy/common/checkout/scm/svn.py:
		17	(SVN.native_branch): Ditto.
		18	* Scripts/webkitpy/common/system/platforminfo.py:
		19	(PlatformInfo.build_version): Return a build version for Mac.
		20	* Scripts/webkitpy/common/system/platforminfo_mock.py:
		21	(MockPlatformInfo.__init__):
		22	(MockPlatformInfo.build_version):
		23	* Scripts/webkitpy/results: Added.
		24	* Scripts/webkitpy/results/__init__.py: Added.
		25	* Scripts/webkitpy/results/options.py: Added.
		26	(upload_options): OptParse list for upload options.
		27	* Scripts/webkitpy/results/upload.py: Added.
		28	(Upload): Class which enforces the upload format expected by the results server.
		29	(Upload.Expectations):
		30	(Upload.create_configuration):
		31	(Upload.create_commit):
		32	(Upload.create_details):
		33	(Upload.create_run_stats):
		34	(Upload.create_test_result):
		35	(Upload.__init__):
		36	(Upload.Encoder): Encode Upload object as json.
		37	(Upload.upload): Upload results to the results server, returning 'True' if the upload is successful.
		38	* Scripts/webkitpy/results/upload_unittest.py: Added.
		39	* Scripts/webkitpy/test/main.py:
		40	(Tester._parse_args): Add upload arguments.
		41	(Tester._run_tests): Allow results to be uploaded.
		42	* Scripts/webkitpy/test/runner.py:
		43	(Runner.__init__): Record which tests were run, rather than just counting them.
		44	(Runner.handle):
		45	* Scripts/webkitpy/test/runner_unittest.py:
		46	(RunnerTest.test_run):
		47	* Scripts/webkitpy/thirdparty/__init__.py:
		48	(AutoinstallImportHook.find_module): Add requests auto-install.
		49	(AutoinstallImportHook._install_requests):
		50	* Scripts/webkitpy/tool/commands/queues_unittest.py:
		51	(PatchProcessingQueueTest.test_upload_results_archive_for_patch): Update os name for testing.
		52
1	2019-03-14 Fujii Hironori <Hironori.Fujii@sony.com>	53	2019-03-14 Fujii Hironori <Hironori.Fujii@sony.com>
2		54
3	[Win][MinBrowser][WK2] Implement createNewPage of WKPageUIClient to open a new window	55	[Win][MinBrowser][WK2] Implement createNewPage of WKPageUIClient to open a new window

Line 0 Tools/Scripts/webkitpy/results/__init__.py_sec1
- Tools/Scripts/webkitpy/results/__init__.py +13 lines
	1	# Required for Python to search this directory for module files
	2
	3	# Keep this file free of any code or import statements that could
	4	# cause either an error to occur or a log message to be logged.
	5	# This ensures that calling code can import initialization code from
	6	# webkitpy before any errors or log messages due to code in this file.
	7	# Initialization code can include things like version-checking code and
	8	# logging configuration code.
	9	#
	10	# We do not execute any version-checking code or logging configuration
	11	# code in this file so that callers can opt-in as they want. This also
	12	# allows different callers to choose different initialization code,
	13	# as necessary.

Line 0 Tools/Scripts/webkitpy/results/options.py_sec1
- Tools/Scripts/webkitpy/results/options.py +33 lines
	1	# Copyright (C) 2019 Apple Inc. All rights reserved.
	2	#
	3	# Redistribution and use in source and binary forms, with or without
	4	# modification, are permitted provided that the following conditions
	5	# are met:
	6	# 1. Redistributions of source code must retain the above copyright
	7	# notice, this list of conditions and the following disclaimer.
	8	# 2. Redistributions in binary form must reproduce the above copyright
	9	# notice, this list of conditions and the following disclaimer in the
	10	# documentation and/or other materials provided with the distribution.
	11	#
	12	# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
	13	# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
	14	# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
	15	# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
	16	# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
	17	# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
	18	# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
	19	# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
	20	# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
	21	# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
	22
	23	import optparse
	24
	25
	26	def upload_options():
	27	return [
	28	optparse.make_option('--report', action='append', dest='report_urls', help='URL (or URLs) to report test results to'),
	29	optparse.make_option('--buildbot-master', help='The url of the buildbot master.'),
	30	optparse.make_option('--builder-name', help='The name of the buildbot builder tests were run on.'),
	31	optparse.make_option('--build-number', help='The buildbot build number tests are associated with.'),
	32	optparse.make_option('--buildbot-worker', help='The buildbot worker tests were run on.'),
	33	]

Line 0 Tools/Scripts/webkitpy/results/upload.py_sec1
- Tools/Scripts/webkitpy/results/upload.py +189 lines
		1	# Copyright (C) 2019 Apple Inc. All rights reserved.
		2	#
		3	# Redistribution and use in source and binary forms, with or without
		4	# modification, are permitted provided that the following conditions
		5	# are met:
		6	# 1. Redistributions of source code must retain the above copyright
		7	# notice, this list of conditions and the following disclaimer.
		8	# 2. Redistributions in binary form must reproduce the above copyright
		9	# notice, this list of conditions and the following disclaimer in the
		10	# documentation and/or other materials provided with the distribution.
		11	#
		12	# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
		13	# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
		14	# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
		15	# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
		16	# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
		17	# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
		18	# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
		19	# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
		20	# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
		21	# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
		22
		23	import webkitpy.thirdparty.autoinstalled.requests
		24
		25	import json
		26	import requests
		27	import sys
		28
		29	import platform as platform_
		30
		31
		32	class Upload(object):
		33	UPLOAD_ENDPOINT = '/api/upload'
		34	BUILDBOT_DETAILS = ['buildbot-master', 'builder-name', 'build-number', 'buildbot-worker']
		35	VERSION = 0
		36
		37	class Expectations:
		38	# These are ordered by priority, meaning that a test which both crashes and has
		39	# a warning should be considered to have crashed.
		40	ORDER = [
		41	'CRASH',
		42	'TIMEOUT',
		43	'IMAGE', # Image-diff
		44	'AUDIO', # Audio-diff
		45	'TEXT', # Text-diff
		46	'FAIL',
		47	'ERROR',
		48	'WARNING',
		49	'PASS',
		50	]
		51	CRASH, TIMEOUT, IMAGE, AUDIO, TEXT, FAIL, ERROR, WARNING, PASS = ORDER
		52
		53	@staticmethod
		54	def create_configuration(
		55	platform=None,
		56	is_simulator=False,
		57	version=None,
		58	architecture=None,
		59	version_name=None,
		60	model=None,
		61	style=None, # Debug/Production/Release
		62	flavor=None, # Dumping ground suite-wide configuration changes (ie, GuardMalloc)
		63	sdk=None,
		64	):
65
66	# This deviates slightly from the rest of webkitpy, but it allows this file to be entirely portable.
67	config = dict(
68	platform=platform or (platform_.system() if platform_.system() != 'Darwin' else 'mac').lower(),
69	is_simulator=is_simulator,
70	version=version or (platform_.release() if platform_.system() != 'Darwin' else platform_.mac_ver()[0]),
71	architecture=architecture or platform_.machine(),
72	)
73	for key, value in config.iteritems():
74	if value is None:
75	raise ValueError('{} must be defined in the configuration'.format(key))
76	for key, value in dict(version_name=version_name, model=model, style=style, flavor=flavor, sdk=sdk).iteritems():
77	if value is not None:
78	config[key] = value
79	return config
80
81	@staticmethod
82	def create_commit(repository_id=None, id=None, branch=None):
83	commit = dict(repository_id=repository_id, id=id)
84	for key, value in commit.iteritems():
85	if value is None:
86	raise ValueError('{} must be defined in a commit'.format(key))
87	if branch:
88	commit['branch'] = branch
89	return commit
90
91	@staticmethod
92	def create_details(link=None, options=None, **kwargs):
93	result = dict(**kwargs)
94	if link:
95	result['link'] = link
96
97	if options:
98	for element in Upload.BUILDBOT_DETAILS:
99	value = getattr(options, element.replace('-', '_'), None)
100	if value is not None:
101	result[element] = value
102	return result
103
104	@staticmethod
105	def create_run_stats(start_time=None, end_time=None, tests_skipped=None, **kwargs):
106	stats = dict(**kwargs)
107	for key, value in dict(start_time=start_time, end_time=end_time, tests_skipped=tests_skipped).iteritems():
108	if value is not None:
109	stats[key] = value
110	return stats
111
112	@staticmethod
113	def create_test_result(expected=None, actual=None, log=None, **kwargs):
114	result = dict(**kwargs)
115
116	# Tests which don't declare expectations or results are assumed to have passed.
117	for key, value in dict(expected=expected, actual=actual, log=log).iteritems():
118	if value is not None:
119	result[key] = value
120	return result
121
122	def __init__(self, suite=None, configuration=None, commits=[], timestamp=None, details=None, run_stats=None, results={}):
123	self.suite = suite
124	self.configuration = configuration
125	self.commits = commits
126	self.timestamp = timestamp
127	self.details = details
128	self.run_stats = run_stats
129	self.results = results
130
131	class Encoder(json.JSONEncoder):
132
133	def default(self, obj):
134	if isinstance(obj, Upload):
135	if not obj.suite:
136	raise ValueError('No suite specified to results upload')
137	if not obj.commits:
138	raise ValueError('No commits specified to results upload')
139
140	details = obj.details or obj.create_details()
141	buildbot_args = [details.get(arg, None) is None for arg in obj.BUILDBOT_DETAILS]
142	if any(buildbot_args) and not all(buildbot_args):
143	raise ValueError('All buildbot details must be defined for upload')
144
145	def unpack_test(current, path_to_test, data):
146	if len(path_to_test) == 1:
147	current[path_to_test[0]] = data
148	return
149	if not current.get(path_to_test[0]):
150	current[path_to_test[0]] = {}
151	unpack_test(current[path_to_test[0]], path_to_test[1:], data)
152
153	results = {}
154	for test, data in obj.results.iteritems():
155	unpack_test(results, test.split('/'), data)
156
157	result = dict(
158	version=obj.VERSION,
159	suite=obj.suite,
160	configuration=obj.configuration or obj.create_configuration(),
161	commits=obj.commits,
162	test_results=dict(
163	details=details,
164	run_stats=obj.run_stats or obj.create_run_stats(),
165	results=results,
166	),
167	)
168	if obj.timestamp:
169	result['timestamp'] = obj.timestamp
170	return result
171	return super(Upload.Encoder, self).default(obj)
172
173	def upload(self, url, log_line=lambda val: sys.stdout.write(val + '\n')):
174	try:
175	response = requests.post(url + self.UPLOAD_ENDPOINT, data=json.dumps(self, cls=Upload.Encoder))
176	except requests.exceptions.ConnectionError:
177	log_line(' ' * 4 + 'Failed to upload to {}, results server not online'.format(url))
178	return False
179	except ValueError as e:
180	log_line(' ' * 4 + 'Failed to encode upload data: {}'.format(e))
181	return False
182
183	if response.status_code != 200:
184	log_line(' ' * 4 + 'Error uploading to {}:'.format(url))
185	log_line(' ' * 8 + response.json()['description'])
186	return False
187
188	log_line(' ' * 4 + 'Uploaded results to {}'.format(url))
189	return True

Line 0 Tools/Scripts/webkitpy/results/upload_unittest.py_sec1
- Tools/Scripts/webkitpy/results/upload_unittest.py +215 lines
		1	# Copyright (C) 2019 Apple Inc. All rights reserved.
		2	#
		3	# Redistribution and use in source and binary forms, with or without
		4	# modification, are permitted provided that the following conditions
		5	# are met:
		6	# 1. Redistributions of source code must retain the above copyright
		7	# notice, this list of conditions and the following disclaimer.
		8	# 2. Redistributions in binary form must reproduce the above copyright
		9	# notice, this list of conditions and the following disclaimer in the
		10	# documentation and/or other materials provided with the distribution.
		11	#
		12	# THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS'' AND
		13	# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
		14	# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
		15	# DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS BE LIABLE FOR
		16	# ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
		17	# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
		18	# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
		19	# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
		20	# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
		21	# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
		22
		23	import webkitpy.thirdparty.autoinstalled.requests
		24
		25	import collections
		26	import json
		27	import requests
		28	import time
		29	import unittest
		30
		31	from webkitpy.results.upload import Upload
		32	from webkitpy.thirdparty import mock
		33
		34
		35	class UploadTest(unittest.TestCase):
		36
		37	class Options(object):
		38	def __init__(self, **kwargs):
		39	for key, value in kwargs.iteritems():
		40	setattr(self, key, value)
		41
		42	class MockResponse(object):
		43	def __init__(self, status_code=200, text=''):
		44	self.status_code = status_code
		45	self.text = text
		46
		47	def json(self):
		48	return json.loads(self.text)
		49
		50	@staticmethod
		51	def normalize(data):
		52	if isinstance(data, basestring):
		53	return str(data)
		54	elif isinstance(data, collections.Mapping):
		55	return dict(map(UploadTest.normalize, data.iteritems()))
		56	elif isinstance(data, collections.Iterable):
		57	return type(data)(map(UploadTest.normalize, data))
		58	else:
		59	return data
		60
		61	def test_encoding(self):
		62	start_time, end_time = time.time() - 3, time.time()
		63	upload = Upload(
		64	suite='webkitpy-tests',
65	configuration=Upload.create_configuration(
66	platform='mac',
67	version='10.13.0',
68	version_name='High Sierra',
69	architecture='x86_64',
70	sdk='17A405',
71	),
72	details=Upload.create_details(link='https://webkit.org'),
73	commits=[Upload.create_commit(
74	repository_id='webkit',
75	id='5',
76	branch='trunk',
77	)],
78	run_stats=Upload.create_run_stats(
79	start_time=start_time,
80	end_time=end_time,
81	tests_skipped=0,
82	),
83	results={
84	'webkitpy.test1': {},
85	'webkitpy.test2': Upload.create_test_result(expected=Upload.Expectations.PASS, actual=Upload.Expectations.FAIL),
86	},
87	)
88	generated_dict = self.normalize(json.loads(json.dumps(upload, cls=Upload.Encoder)))
89
90	self.assertEqual(generated_dict['version'], 0)
91	self.assertEqual(generated_dict['suite'], 'webkitpy-tests')
92	self.assertEqual(generated_dict['configuration'], self.normalize(dict(
93	platform='mac',
94	is_simulator=False,
95	version='10.13.0',
96	version_name='High Sierra',
97	architecture='x86_64',
98	sdk='17A405',
99	)))
100	self.assertEqual(generated_dict['commits'], [dict(
101	repository_id='webkit',
102	id='5',
103	branch='trunk',
104	)])
105	self.assertEqual(generated_dict['test_results']['details'], self.normalize(dict(link='https://webkit.org')))
106	self.assertEqual(generated_dict['test_results']['run_stats'], self.normalize(dict(
107	start_time=start_time,
108	end_time=end_time,
109	tests_skipped=0,
110	)))
111	self.assertEqual(generated_dict['test_results']['results'], self.normalize({
112	'webkitpy.test1': {},
113	'webkitpy.test2': Upload.create_test_result(expected=Upload.Expectations.PASS, actual=Upload.Expectations.FAIL),
114	}))
115
116	def test_upload(self):
117	upload = Upload(
118	suite='webkitpy-tests',
119	commits=[Upload.create_commit(
120	repository_id='webkit',
121	id='5',
122	branch='trunk',
123	)],
124	)
125
126	with mock.patch('requests.post', new=lambda url, data: self.MockResponse()):
127	self.assertTrue(upload.upload('https://webkit.org/results', log_line=lambda _: None))
128
129	def raise_exception(url, data):
130	raise requests.exceptions.ConnectionError()
131
132	with mock.patch('requests.post', new=raise_exception):
133	self.assertFalse(upload.upload('https://webkit.org/results', log_line=lambda _: None))
134
135	with mock.patch('requests.post', new=lambda url, data: self.MockResponse(
136	status_code=404,
137	text=json.dumps(dict(description='No such address')),
138	)):
139	self.assertFalse(upload.upload('https://webkit.org/results', log_line=lambda _: None))
140
141	def test_packed_test(self):
142	upload = Upload(
143	suite='webkitpy-tests',
144	commits=[Upload.create_commit(
145	repository_id='webkit',
146	id='5',
147	branch='trunk',
148	)],
149	results={
150	'dir1/sub-dir1/test1': Upload.create_test_result(actual=Upload.Expectations.FAIL),
151	'dir1/sub-dir1/test2': Upload.create_test_result(actual=Upload.Expectations.TIMEOUT),
152	'dir1/sub-dir2/test3': {},
153	'dir1/sub-dir2/test4': {},
154	'dir2/sub-dir3/test5': {},
155	'dir2/test6': {},
156	}
157	)
158	generated_dict = self.normalize(json.loads(json.dumps(upload, cls=Upload.Encoder)))
159	self.assertEqual(generated_dict['test_results']['results'], self.normalize({
160	'dir1': {
161	'sub-dir1': {
162	'test1': {'actual': Upload.Expectations.FAIL},
163	'test2': {'actual': Upload.Expectations.TIMEOUT},
164	}, 'sub-dir2': {
165	'test3': {},
166	'test4': {},
167	}
168	}, 'dir2': {
169	'sub-dir3': {'test5': {}},
170	'test6': {},
171	},
172	}))
173
174	def test_no_suite(self):
175	upload = Upload(
176	commits=[Upload.create_commit(
177	repository_id='webkit',
178	id='5',
179	branch='trunk',
180	)],
181	)
182	with self.assertRaises(ValueError):
183	json.dumps(upload, cls=Upload.Encoder)
184
185	def test_no_commits(self):
186	upload = Upload(
187	suite='webkitpy-tests',
188	)
189	with self.assertRaises(ValueError):
190	json.dumps(upload, cls=Upload.Encoder)
191
192	def test_buildbot(self):
193	upload = Upload(
194	suite='webkitpy-tests',
195	commits=[Upload.create_commit(
196	repository_id='webkit',
197	id='5',
198	branch='trunk',
199	)],
200	details=Upload.create_details(options=self.Options(
201	buildbot_master='webkit.org',
202	builder_name='Queue-1',
203	build_number=1,
204	)))
205	with self.assertRaises(ValueError):
206	json.dumps(upload, cls=Upload.Encoder)
207
208	upload.details['buildbot-worker'] = 'bot123'
209	generated_dict = self.normalize(json.loads(json.dumps(upload, cls=Upload.Encoder)))
210	self.assertEqual(generated_dict['test_results']['details'], self.normalize({
211	'buildbot-master': 'webkit.org',
212	'builder-name': 'Queue-1',
213	'build-number': 1,
214	'buildbot-worker': 'bot123',
215	}))

Lines 83-89 class SVN(SCM, SVNRepository): Tools/Scripts/webkitpy/common/checkout/scm/svn.py_sec1
- Tools/Scripts/webkitpy/common/checkout/scm/svn.py -1 / +9 lines
83	SCM.__init__(self, cwd, **kwargs)	83	SCM.__init__(self, cwd, **kwargs)
84	self._bogus_dir = None	84	self._bogus_dir = None
85	if patch_directories == []:	85	if patch_directories == []:
86	raise Exception(message='Empty list of patch directories passed to SCM.__init__')	86	raise Exception('Empty list of patch directories passed to SCM.__init__')
87	elif patch_directories == None:	87	elif patch_directories == None:
88	self._patch_directories = [self._filesystem.relpath(cwd, self.checkout_root)]	88	self._patch_directories = [self._filesystem.relpath(cwd, self.checkout_root)]
89	else:	89	else:
Lines 278-283 class SVN(SCM, SVNRepository): Tools/Scripts/webkitpy/common/checkout/scm/svn.py_sec2
278	def native_revision(self, path):	278	def native_revision(self, path):
279	return self.svn_revision(path)	279	return self.svn_revision(path)
280		280
		281	def native_branch(self, path):
		282	relative_url = self.value_from_svn_info(path, 'Relative URL')[2:]
		283	if relative_url.startswith('trunk'):
		284	return 'trunk'
		285	elif relative_url.startswith('branch'):
		286	return relative_url.split('/')[1]
		287	raise Exception('{} is not a branch'.format(relative_url.split('/')[0]))
		288
281	def timestamp_of_revision(self, path, revision):	289	def timestamp_of_revision(self, path, revision):
282	# We use --xml to get timestamps like 2013-02-08T08:18:04.964409Z	290	# We use --xml to get timestamps like 2013-02-08T08:18:04.964409Z
283	repository_root = self.value_from_svn_info(self.checkout_root, 'Repository Root')	291	repository_root = self.value_from_svn_info(self.checkout_root, 'Repository Root')

Lines 1-6 Tools/Scripts/webkitpy/test/main.py_sec1
- Tools/Scripts/webkitpy/test/main.py -13 / +61 lines
1	# Copyright (C) 2012 Google, Inc.	1	# Copyright (C) 2012 Google, Inc.
2	# Copyright (C) 2010 Chris Jerdonek (cjerdonek@webkit.org)	2	# Copyright (C) 2010 Chris Jerdonek (cjerdonek@webkit.org)
3	# Copyright (C) 2018 Apple Inc. All rights reserved.	3	# Copyright (C) 2018-2019 Apple Inc. All rights reserved.
4	#	4	#
5	# Redistribution and use in source and binary forms, with or without	5	# Redistribution and use in source and binary forms, with or without
6	# modification, are permitted provided that the following conditions	6	# modification, are permitted provided that the following conditions
Lines 38-54 import traceback Tools/Scripts/webkitpy/test/main.py_sec2
38	import unittest	38	import unittest
39		39
40	from webkitpy.common.system.logutils import configure_logging	40	from webkitpy.common.system.logutils import configure_logging
41	from webkitpy.common.system.executive import Executive, ScriptError	41	from webkitpy.common.system.executive import ScriptError
42	from webkitpy.common.system.filesystem import FileSystem	42	from webkitpy.common.system.filesystem import FileSystem
43	from webkitpy.common.system.systemhost import SystemHost	43	from webkitpy.common.host import Host
44	from webkitpy.port.config import Config	44	from webkitpy.port.config import Config
45	from webkitpy.test.finder import Finder	45	from webkitpy.test.finder import Finder
46	from webkitpy.test.printer import Printer	46	from webkitpy.test.printer import Printer
47	from webkitpy.test.runner import Runner, unit_test_name	47	from webkitpy.test.runner import Runner, unit_test_name
		48	from webkitpy.results.upload import Upload
		49	from webkitpy.results.options import upload_options
48		50
49	_log = logging.getLogger(__name__)	51	_log = logging.getLogger(__name__)
50		52
51	_host = SystemHost()	53	_host = Host()
52	_webkit_root = None	54	_webkit_root = None
53		55
54		56
Lines 124-135 class Tester(object): Tools/Scripts/webkitpy/test/main.py_sec3
124	parser = optparse.OptionParser(usage='usage: %prog [options] [args...]')	126	parser = optparse.OptionParser(usage='usage: %prog [options] [args...]')
125		127
126	# Configuration options only effect the building of lldbWebKitTester.	128	# Configuration options only effect the building of lldbWebKitTester.
127	configuration_group = optparse.OptionGroup(parser, 'Configuration options')	129	for group_name, group_options in {'Upload Options': upload_options(), 'Configuration Options': [
128	configuration_group.add_option('--debug', action='store_const', const='Debug', dest="configuration",	130	optparse.make_option('--debug', action='store_const', const='Debug', dest="configuration",
129	help='Set the configuration to Debug')	131	help='Set the configuration to Debug'),
130	configuration_group.add_option('--release', action='store_const', const='Release', dest="configuration",	132	optparse.make_option('--release', action='store_const', const='Release', dest="configuration",
131	help='Set the configuration to Release')	133	help='Set the configuration to Release'),
132	parser.add_option_group(configuration_group)	134	]}.iteritems():
		135	option_group = optparse.OptionGroup(parser, group_name)
		136	option_group.add_options(group_options)
		137	parser.add_option_group(option_group)
133		138
134	parser.add_option('-a', '--all', action='store_true', default=False,	139	parser.add_option('-a', '--all', action='store_true', default=False,
135	help='run all the tests')	140	help='run all the tests')
Lines 181-191 class Tester(object): Tools/Scripts/webkitpy/test/main.py_sec4
181	from webkitpy.thirdparty import autoinstall_everything	186	from webkitpy.thirdparty import autoinstall_everything
182	autoinstall_everything()	187	autoinstall_everything()
183		188
		189	start_time = time.time()
		190	config = Config(_host.executive, self.finder.filesystem)
		191	configuration_to_use = self._options.configuration or config.default_configuration()
184	if will_run_lldb_webkit_tests:	192	if will_run_lldb_webkit_tests:
185	self.printer.write_update('Building lldbWebKitTester ...')	193	self.printer.write_update('Building lldbWebKitTester ...')
186	build_lldbwebkittester = self.finder.filesystem.join(_webkit_root, 'Tools', 'Scripts', 'build-lldbwebkittester')	194	build_lldbwebkittester = self.finder.filesystem.join(_webkit_root, 'Tools', 'Scripts', 'build-lldbwebkittester')
187	config = Config(_host.executive, self.finder.filesystem)
188	configuration_to_use = self._options.configuration or config.default_configuration()
189	try:	195	try:
190	_host.executive.run_and_throw_if_fail([build_lldbwebkittester, config.flag_for_configuration(configuration_to_use)], quiet=(not bool(self._options.verbose)))	196	_host.executive.run_and_throw_if_fail([build_lldbwebkittester, config.flag_for_configuration(configuration_to_use)], quiet=(not bool(self._options.verbose)))
191	except ScriptError as e:	197	except ScriptError as e:
Lines 218-223 class Tester(object): Tools/Scripts/webkitpy/test/main.py_sec5
218	test_runner = Runner(self.printer, loader)	224	test_runner = Runner(self.printer, loader)
219	test_runner.run(parallel_tests, self._options.child_processes)	225	test_runner.run(parallel_tests, self._options.child_processes)
220	test_runner.run(serial_tests, 1)	226	test_runner.run(serial_tests, 1)
		227	end_time = time.time()
221		228
222	self.printer.print_result(time.time() - start)	229	self.printer.print_result(time.time() - start)
223		230
Lines 232-240 class Tester(object): Tools/Scripts/webkitpy/test/main.py_sec6
232	if self._options.coverage:	239	if self._options.coverage:
233	cov.stop()	240	cov.stop()
234	cov.save()	241	cov.save()
		242
		243	failed_uploads = 0
		244	if self._options.report_urls:
		245	self.printer.meter.writeln('\n')
		246	self.printer.write_update('Preparing upload data ...')
		247
		248	results = {test: {} for test in test_runner.tests_run}
		249	for test, errors in test_runner.errors:
		250	results[test] = Upload.create_test_result(actual=Upload.Expectations.ERROR, log='/n'.join(errors))
		251	for test, failures in test_runner.failures:
		252	results[test] = Upload.create_test_result(actual=Upload.Expectations.FAIL, log='/n'.join(failures))
		253
		254	_host.initialize_scm()
		255	upload = Upload(
		256	suite='webkitpy-tests',
		257	configuration=Upload.create_configuration(
		258	platform=_host.platform.os_name,
		259	version=str(_host.platform.os_version),
		260	version_name=_host.platform.os_version_name(),
		261	style=configuration_to_use,
		262	sdk=_host.platform.build_version(),
		263	),
		264	details=Upload.create_details(options=self._options),
		265	commits=[Upload.create_commit(
		266	repository_id='webkit',
		267	id=_host.scm().native_revision(_webkit_root),
		268	branch=_host.scm().native_branch(_webkit_root),
		269	)],
		270	run_stats=Upload.create_run_stats(
		271	start_time=start_time,
		272	end_time=end_time,
		273	tests_skipped=len(test_runner.tests_run) - len(parallel_tests) - len(serial_tests),
		274	),
		275	results=results,
		276	)
		277	for url in self._options.report_urls:
		278	self.printer.write_update('Uploading to {} ...'.format(url))
		279	failed_uploads = failed_uploads if upload.upload(url, log_line=self.printer.meter.writeln) else (failed_uploads + 1)
		280	self.printer.meter.writeln('Uploads completed!')
		281
		282	if self._options.coverage:
235	cov.report(show_missing=False)	283	cov.report(show_missing=False)
236		284
237	return not self.printer.num_errors and not self.printer.num_failures	285	return not self.printer.num_errors and not self.printer.num_failures and not failed_uploads
238		286
239	def _check_imports(self, names):	287	def _check_imports(self, names):
240	for name in names:	288	for name in names:

Lines 40-46 class Runner(object): Tools/Scripts/webkitpy/test/runner.py_sec1
- Tools/Scripts/webkitpy/test/runner.py -2 / +2 lines
40	def __init__(self, printer, loader):	40	def __init__(self, printer, loader):
41	self.printer = printer	41	self.printer = printer
42	self.loader = loader	42	self.loader = loader
43	self.tests_run = 0	43	self.tests_run = []
44	self.errors = []	44	self.errors = []
45	self.failures = []	45	self.failures = []
46	self.worker_factory = lambda caller: _Worker(caller, self.loader)	46	self.worker_factory = lambda caller: _Worker(caller, self.loader)
Lines 60-66 class Runner(object): Tools/Scripts/webkitpy/test/runner.py_sec2
60	self.printer.print_started_test(source, test_name)	60	self.printer.print_started_test(source, test_name)
61	return	61	return
62		62
63	self.tests_run += 1	63	self.tests_run.append(test_name)
64	if failures:	64	if failures:
65	self.failures.append((test_name, failures))	65	self.failures.append((test_name, failures))
66	if errors:	66	if errors:

Lines 183-192 class PatchProcessingQueueTest(CommandsT Tools/Scripts/webkitpy/tool/commands/queues_unittest.py_sec1
- Tools/Scripts/webkitpy/tool/commands/queues_unittest.py -2 / +2 lines
183	queue._options = Mock()	183	queue._options = Mock()
184	queue._options.port = None	184	queue._options.port = None
185	patch = queue._tool.bugs.fetch_attachment(10001)	185	patch = queue._tool.bugs.fetch_attachment(10001)
186	expected_logs = """MOCK add_attachment_to_bug: bug_id=50000, description=Archive of layout-test-results from bot for mac-snowleopard filename=layout-test-results.zip mimetype=None	186	expected_logs = """MOCK add_attachment_to_bug: bug_id=50000, description=Archive of layout-test-results from bot for mac-highsierra filename=layout-test-results.zip mimetype=None
187	-- Begin comment --	187	-- Begin comment --
188	The attached test failures were seen while running run-webkit-tests on the mock-queue.	188	The attached test failures were seen while running run-webkit-tests on the mock-queue.
189	Port: mac-snowleopard Platform: MockPlatform 1.0	189	Port: mac-highsierra Platform: MockPlatform 1.0
190	-- End comment --	190	-- End comment --
191	"""	191	"""
192	OutputCapture().assert_outputs(self, queue._upload_results_archive_for_patch, [patch, Mock()], expected_logs=expected_logs)	192	OutputCapture().assert_outputs(self, queue._upload_results_archive_for_patch, [patch, Mock()], expected_logs=expected_logs)