Bug 188118 – [GTK] Many WebDriver tests are failing

• Status: NEW
• Component: WebKitGTK

[Michael Catanzaro]

Many WebDriver tests are failing:

Unexpected failures (11)
  imported/selenium/py/test/selenium/webdriver/common/alerts_tests.py::testShouldHandleAlertOnPageBeforeUnload[WebKitGTK]
  imported/selenium/py/test/selenium/webdriver/common/page_load_timeout_tests.py::testClickShouldTimeout[WebKitGTK]
  imported/selenium/py/test/selenium/webdriver/common/page_load_timeout_tests.py::testClickShouldTimeout[WebKitGTK]
  imported/selenium/py/test/selenium/webdriver/common/window_tests.py::test_should_fullscreen_the_current_window[WebKitGTK]
  imported/w3c/webdriver/tests/close_window/user_prompts.py::test_handle_prompt_accept
  imported/w3c/webdriver/tests/close_window/close.py::test_close_last_browsing_context
  imported/w3c/webdriver/tests/set_window_rect/set.py::test_fully_exit_fullscreen
  imported/w3c/webdriver/tests/maximize_window/maximize.py::test_fully_exit_fullscreen
  imported/w3c/webdriver/tests/new_session/response.py::test_resp_capabilites
  imported/w3c/webdriver/tests/new_session/response.py::test_resp_data
  imported/w3c/webdriver/tests/minimize_window/minimize.py::test_fully_exit_fullscreen

Unexpected timeouts (1)
  imported/w3c/webdriver/tests/actions/special_keys.py::test_webdriver_special_key_sends_keydown[F5-expected55]


Of those, page_load_timeout_tests.py, set.py, maximize.py, and minimize.py are all flaky. I think the rest are consistent failures.

This is too many failures for me to report separate bugs for each one. I'll use separate comments. Feel free to split failures into separate bugs as desired.

[Michael Catanzaro]

______________ testShouldHandleAlertOnPageBeforeUnload[WebKitGTK] ______________

driver = <selenium.webdriver.webkitgtk.webdriver.WebDriver (session="768f3a45-1b30-4b4e-9b6e-ea1ad8c3c301")>
pages = <conftest.Pages object at 0x7f6061968a10>

    def testShouldHandleAlertOnPageBeforeUnload(driver, pages):
        pages.load("pageWithOnBeforeUnloadMessage.html")
    
        element = driver.find_element(By.ID, "navigate")
        element.click()
    
>       alert = _waitForAlert(driver)

driver     = <selenium.webdriver.webkitgtk.webdriver.WebDriver (session="768f3a45-1b30-4b4e-9b6e-ea1ad8c3c301")>
element    = <selenium.webdriver.remote.webelement.WebElement (session="768f3a45-1b30-4b4e-9b6e-ea1ad8c3c301", element="node-A7AFE675-0BF1-4957-8B3B-BFB8ACE80CE1")>
pages      = <conftest.Pages object at 0x7f6061968a10>

WebDriverTests/imported/selenium/py/test/selenium/webdriver/common/alerts_tests.py:249: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
WebDriverTests/imported/selenium/py/test/selenium/webdriver/common/alerts_tests.py:318: in _waitForAlert
    return WebDriverWait(driver, 3).until(EC.alert_is_present())
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <selenium.webdriver.support.wait.WebDriverWait (session="768f3a45-1b30-4b4e-9b6e-ea1ad8c3c301")>
method = <selenium.webdriver.support.expected_conditions.alert_is_present object at 0x7f6061968210>
message = ''

    def until(self, method, message=''):
        """Calls the method provided with the driver as an argument until the \
            return value is not False."""
        screen = None
        stacktrace = None
    
        end_time = time.time() + self._timeout
        while True:
            try:
                value = method(self._driver)
                if value:
                    return value
            except self._ignored_exceptions as exc:
                screen = getattr(exc, 'screen', None)
                stacktrace = getattr(exc, 'stacktrace', None)
            time.sleep(self._poll)
            if time.time() > end_time:
                break
>       raise TimeoutException(message, screen, stacktrace)
E       TimeoutException: Message:

end_time   = 1532715605.473464
message    = ''
method     = <selenium.webdriver.support.expected_conditions.alert_is_present object at 0x7f6061968210>
screen     = None
self       = <selenium.webdriver.support.wait.WebDriverWait (session="768f3a45-1b30-4b4e-9b6e-ea1ad8c3c301")>
stacktrace = None
value      = False

WebDriverTests/imported/selenium/py/selenium/webdriver/support/wait.py:80: TimeoutException

[Michael Catanzaro]

____________ ERROR at teardown of testClickShouldTimeout[WebKitGTK] ____________

driver = <selenium.webdriver.webkitgtk.webdriver.WebDriver (session="165d5c3a-2682-4b25-9434-5672d2903e95")>

    @pytest.fixture(autouse=True)
    def reset_timeouts(driver):
        yield
>       driver.set_page_load_timeout(300)

driver     = <selenium.webdriver.webkitgtk.webdriver.WebDriver (session="165d5c3a-2682-4b25-9434-5672d2903e95")>

WebDriverTests/imported/selenium/py/test/selenium/webdriver/common/page_load_timeout_tests.py:26: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
WebDriverTests/imported/selenium/py/selenium/webdriver/remote/webdriver.py:926: in set_page_load_timeout
    'pageLoad': int(float(time_to_wait) * 1000)})
WebDriverTests/imported/selenium/py/selenium/webdriver/remote/webdriver.py:310: in execute
    response = self.command_executor.execute(driver_command, params)
WebDriverTests/imported/selenium/py/selenium/webdriver/remote/remote_connection.py:466: in execute
    return self._request(command_info[0], url, body=data)
WebDriverTests/imported/selenium/py/selenium/webdriver/remote/remote_connection.py:528: in _request
    resp = opener.open(request, timeout=self._timeout)
/usr/lib/python2.7/urllib2.py:429: in open
    response = self._open(req, data)
/usr/lib/python2.7/urllib2.py:447: in _open
    '_open', req)
/usr/lib/python2.7/urllib2.py:407: in _call_chain
    result = func(*args)
/usr/lib/python2.7/urllib2.py:1228: in http_open
    return self.do_open(httplib.HTTPConnection, req)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <urllib2.HTTPHandler instance at 0x7f6062a34878>
http_class = <class httplib.HTTPConnection at 0x7f60682bb050>
req = <selenium.webdriver.remote.remote_connection.Request instance at 0x7f6062a34e60>
http_conn_args = {}, host = '127.0.0.1:52629'
h = <httplib.HTTPConnection instance at 0x7f6062a348c0>
err = error(111, 'Connection refused')

    def do_open(self, http_class, req, **http_conn_args):
        """Return an addinfourl object for the request, using http_class.
    
            http_class must implement the HTTPConnection API from httplib.
            The addinfourl return value is a file-like object.  It also
            has methods and attributes including:
                - info(): return a mimetools.Message object for the headers
                - geturl(): return the original request URL
                - code: HTTP status code
            """
        host = req.get_host()
        if not host:
            raise URLError('no host given')
    
        # will parse host:port
        h = http_class(host, timeout=req.timeout, **http_conn_args)
        h.set_debuglevel(self._debuglevel)
    
        headers = dict(req.unredirected_hdrs)
        headers.update(dict((k, v) for k, v in req.headers.items()
                            if k not in headers))
    
        # We want to make an HTTP/1.1 request, but the addinfourl
        # class isn't prepared to deal with a persistent connection.
        # It will try to read all remaining data from the socket,
        # which will block while the server waits for the next request.
        # So make sure the connection gets closed after the (only)
        # request.
        headers["Connection"] = "close"
        headers = dict(
            (name.title(), val) for name, val in headers.items())
    
        if req._tunnel_host:
            tunnel_headers = {}
            proxy_auth_hdr = "Proxy-Authorization"
            if proxy_auth_hdr in headers:
                tunnel_headers[proxy_auth_hdr] = headers[proxy_auth_hdr]
                # Proxy-Authorization should not be sent to origin
                # server.
                del headers[proxy_auth_hdr]
            h.set_tunnel(req._tunnel_host, headers=tunnel_headers)
    
        try:
            h.request(req.get_method(), req.get_selector(), req.data, headers)
        except socket.error, err: # XXX what error?
            h.close()
>           raise URLError(err)
E           URLError: <urlopen error [Errno 111] Connection refused>

err        = error(111, 'Connection refused')
h          = <httplib.HTTPConnection instance at 0x7f6062a348c0>
headers    = {'Accept': 'application/json', 'Connection': 'close', 'Content-Length': '20', 'Content-Type': 'application/json;charset=UTF-8', ...}
host       = '127.0.0.1:52629'
http_class = <class httplib.HTTPConnection at 0x7f60682bb050>
http_conn_args = {}
req        = <selenium.webdriver.remote.remote_connection.Request instance at 0x7f6062a34e60>
self       = <urllib2.HTTPHandler instance at 0x7f6062a34878>

/usr/lib/python2.7/urllib2.py:1198: URLError

[Michael Catanzaro]

_____________ test_should_fullscreen_the_current_window[WebKitGTK] _____________

driver = <selenium.webdriver.webkitgtk.webdriver.WebDriver (session="e15545c7-7d93-476c-a5c2-d1c8aaab5d35")>

    @pytest.mark.xfail_chrome(raises=WebDriverException,
                              reason='Fullscreen command not implemented')
    @pytest.mark.xfail_firefox(raises=WebDriverException,
                               reason='Fullscreen command not implemented')
    @pytest.mark.xfail_remote(raises=WebDriverException,
                              reason='Fullscreen command not implemented')
    @pytest.mark.xfail_safari(raises=WebDriverException,
                              reason='Fullscreen command not implemented')
    @pytest.mark.skipif(os.environ.get('CI') == 'true',
                        reason='Fullscreen command causes Travis to hang')
    def test_should_fullscreen_the_current_window(driver):
        start_width = driver.execute_script('return window.innerWidth;')
        start_height = driver.execute_script('return window.innerHeight;')
    
        driver.fullscreen_window()
    
>       WebDriverWait(driver, 2).until(lambda d: driver.execute_script('return window.innerWidth;') >
                                       start_width)

driver     = <selenium.webdriver.webkitgtk.webdriver.WebDriver (session="e15545c7-7d93-476c-a5c2-d1c8aaab5d35")>
start_height = 716
start_width = 1014

WebDriverTests/imported/selenium/py/test/selenium/webdriver/common/window_tests.py:140: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <selenium.webdriver.support.wait.WebDriverWait (session="e15545c7-7d93-476c-a5c2-d1c8aaab5d35")>
method = <function <lambda> at 0x7f6062df12a8>, message = ''

    def until(self, method, message=''):
        """Calls the method provided with the driver as an argument until the \
            return value is not False."""
        screen = None
        stacktrace = None
    
        end_time = time.time() + self._timeout
        while True:
            try:
                value = method(self._driver)
                if value:
                    return value
            except self._ignored_exceptions as exc:
                screen = getattr(exc, 'screen', None)
                stacktrace = getattr(exc, 'stacktrace', None)
            time.sleep(self._poll)
            if time.time() > end_time:
                break
>       raise TimeoutException(message, screen, stacktrace)
E       TimeoutException: Message:

end_time   = 1532716091.136181
message    = ''
method     = <function <lambda> at 0x7f6062df12a8>
screen     = None
self       = <selenium.webdriver.support.wait.WebDriverWait (session="e15545c7-7d93-476c-a5c2-d1c8aaab5d35")>
stacktrace = None
value      = False

WebDriverTests/imported/selenium/py/selenium/webdriver/support/wait.py:80: TimeoutException

[Michael Catanzaro]

__________________________ test_handle_prompt_accept ___________________________

new_session = <function create_session at 0x7f60619b5d70>
add_browser_capabilites = <function update_capabilities at 0x7f60619b5b18>

    def test_handle_prompt_accept(new_session, add_browser_capabilites):
        _, session = new_session({"capabilities": {
            "alwaysMatch": add_browser_capabilites({"unhandledPromptBehavior": "accept"})}})
        session.window_handle = create_window(session)()
    
        session.url = inline("<title>WD doc title</title>")
    
        create_dialog(session)("alert", text="dismiss #1", result_var="dismiss1")
        response = close(session)
        assert response.status == 200
>       assert_dialog_handled(session, "dismiss #1")

_          = {'capabilities': {'browserName': 'MiniBrowser', 'browserVersion': '2.21.5', 'pageLoadStrategy': 'normal', 'platformName': 'linux', ...}, 'sessionId': '0ab3c417-b701-462d-bbbf-5590b1e34149'}
add_browser_capabilites = <function update_capabilities at 0x7f60619b5b18>
new_session = <function create_session at 0x7f60619b5d70>
response   = <Responsetatus=200 body={"value": ["page-3DC2B05B-A549-41AC-B31B-E91082BCBA74"]}>
session    = <Session 0ab3c417-b701-462d-bbbf-5590b1e34149>

WebDriverTests/imported/w3c/webdriver/tests/close_window/user_prompts.py:35: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

session = <Session 0ab3c417-b701-462d-bbbf-5590b1e34149>
expected_text = 'dismiss #1'

    def assert_dialog_handled(session, expected_text):
        result = session.transport.send("GET",
                                        "session/%s/alert/text" % session.session_id)
    
        # If there were any existing dialogs prior to the creation of this
        # fixture's dialog, then the "Get Alert Text" command will return
        # successfully. In that case, the text must be different than that
        # of this fixture's dialog.
        try:
            assert_error(result, "no such alert")
        except:
>           assert (result.status == 200 and
                    result.body["value"] != expected_text), (
                "Dialog with text '%s' was not handled." % expected_text)
E           AssertionError: Dialog with text 'dismiss #1' was not handled.
E           assert (404 == 200)
E            +  where 404 = <Response status=404 error=<NoSuchWindowException http_status=404>>.status

expected_text = 'dismiss #1'
result     = <Response status=404 error=<NoSuchWindowException http_status=404>>
session    = <Session 0ab3c417-b701-462d-bbbf-5590b1e34149>

WebDriverTests/imported/w3c/webdriver/tests/support/asserts.py:101: AssertionError

[Michael Catanzaro]

WebDriverTests/imported/w3c/webdriver/tests/close_window/close.py::test_no_browsing_context PASSED
WebDriverTests/imported/w3c/webdriver/tests/close_window/close.py::test_close_browsing_context PASSED
WebDriverTests/imported/w3c/webdriver/tests/close_window/close.py::test_close_last_browsing_context PASSEDIgnored exception session not created (500): Failed to connect to browser: Failed to execute child process “/usr/local/libexec/webkit2gtk-4.0/MiniBrowser” (No such file or directory)

Ignored exception session not created (500): Failed to connect to browser: Failed to execute child process “/usr/local/libexec/webkit2gtk-4.0/MiniBrowser” (No such file or directory)

Ignored exception session not created (500): Failed to connect to browser: Failed to execute child process “/usr/local/libexec/webkit2gtk-4.0/MiniBrowser” (No such file or directory)

Ignored exception session not created (500): Failed to connect to browser: Failed to execute child process “/usr/local/libexec/webkit2gtk-4.0/MiniBrowser” (No such file or directory)


WebDriverTests/imported/w3c/webdriver/tests/close_window/close.py::test_close_last_browsing_context ERROR

==================================== ERRORS ====================================
____________ ERROR at teardown of test_close_last_browsing_context _____________

>   request.addfinalizer(lambda: _switch_to_top_level_browsing_context(_current_session))


WebDriverTests/imported/w3c/webdriver/tests/support/fixtures.py:166: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
WebDriverTests/imported/w3c/webdriver/tests/support/fixtures.py:21: in inner
    return f(*args, **kwargs)
WebDriverTests/imported/w3c/webdriver/tests/support/fixtures.py:85: in _switch_to_top_level_browsing_context
    session.switch_frame(None)
WebDriverTests/imported/w3c/tools/webdriver/webdriver/client.py:535: in switch_frame
    return self.send_session_command("POST", url, body)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <Session (disconnected)>, method = 'POST', uri = 'frame'
body = {'id': None}

    def send_session_command(self, method, uri, body=None):
        """
            Send a command to an established session and validate its success.
    
            :param method: HTTP method to use in request.
            :param url: "Command part" of the HTTP request URL,
                e.g. `window/rect`.
            :param body: Optional body of the HTTP request.  Must be JSON
                serialisable.
    
            :return: `None` if the HTTP response body was empty, otherwise
                the result of parsing the body as JSON.
    
            :raises error.SessionNotCreatedException: If there is no active
                session.
            :raises error.WebDriverException: If the remote end returns
                an error.
            """
        if self.session_id is None:
>           raise error.SessionNotCreatedException()
E           TypeError: __init__() takes at least 2 arguments (1 given)

body       = {'id': None}
method     = 'POST'
self       = <Session (disconnected)>
uri        = 'frame'

[Michael Catanzaro]

__________________________ test_fully_exit_fullscreen __________________________

session = <Session 32832ce0-1f20-41d0-a140-9809e8c899c3>

    def test_fully_exit_fullscreen(session):
        """
        10. Fully exit fullscreen.
    
        [...]
    
        To fully exit fullscreen a document document, run these steps:
    
          1. If document's fullscreen element is null, terminate these steps.
    
          2. Unfullscreen elements whose fullscreen flag is set, within
          document's top layer, except for document's fullscreen element.
    
          3. Exit fullscreen document.
        """
>       session.window.fullscreen()

session    = <Session 32832ce0-1f20-41d0-a140-9809e8c899c3>

WebDriverTests/imported/w3c/webdriver/tests/set_window_rect/set.py:192: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
WebDriverTests/imported/w3c/tools/webdriver/webdriver/client.py:21: in inner
    return func(self, *args, **kwargs)
WebDriverTests/imported/w3c/tools/webdriver/webdriver/client.py:285: in fullscreen
    return self.session.send_session_command("POST", "window/fullscreen")
WebDriverTests/imported/w3c/tools/webdriver/webdriver/client.py:479: in send_session_command
    return self.send_command(method, url, body)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <Session 32832ce0-1f20-41d0-a140-9809e8c899c3>, method = 'POST'
url = 'session/32832ce0-1f20-41d0-a140-9809e8c899c3/window/fullscreen'
body = None

    def send_command(self, method, url, body=None):
        """
            Send a command to the remote end and validate its success.
    
            :param method: HTTP method to use in request.
            :param uri: "Command part" of the HTTP request URL,
                e.g. `window/rect`.
            :param body: Optional body of the HTTP request.
    
            :return: `None` if the HTTP response body was empty, otherwise
                the `value` field returned after parsing the response
                body as JSON.
    
            :raises ValueError: If the response body does not contain a
                `value` key.
            :raises error.WebDriverException: If the remote end returns
                an error.
            """
        response = self.transport.send(
            method, url, body,
            encoder=protocol.Encoder, decoder=protocol.Decoder,
            session=self)
    
        if response.status != 200:
>           raise error.from_response(response)
E           ScriptTimeoutException: script timeout (408):

body       = None
method     = 'POST'
response   = <Response status=408 error=<ScriptTimeoutException http_status=408>>
self       = <Session 32832ce0-1f20-41d0-a140-9809e8c899c3>
url        = 'session/32832ce0-1f20-41d0-a140-9809e8c899c3/window/fullscreen'

WebDriverTests/imported/w3c/tools/webdriver/webdriver/client.py:438: ScriptTimeoutException

[Michael Catanzaro]

____________________________ test_resp_capabilites _____________________________

new_session = <function create_session at 0x7f606192f230>
add_browser_capabilites = <function update_capabilities at 0x7f606192f1b8>

    def test_resp_capabilites(new_session, add_browser_capabilites):
        resp, _ = new_session({"capabilities": {"alwaysMatch": add_browser_capabilites({})}})
        assert isinstance(resp["sessionId"], unicode)
        assert isinstance(resp["capabilities"], dict)
>       assert {"browserName",
                "browserVersion",
                "platformName",
                "acceptInsecureCerts",
                "setWindowRect",
                "timeouts",
                "proxy",
                "pageLoadStrategy"}.issubset(
                    set(resp["capabilities"].keys()))
E       AssertionError: assert False
E        +  where False = <built-in method issubset of set object at 0x7f6062c7e5c0>(set(['browserName', 'browserVersion', 'pageLoadStrategy', 'platformName', 'proxy', 'setWindowRect', ...]))
E        +    where <built-in method issubset of set object at 0x7f6062c7e5c0> = set(['acceptInsecureCerts', 'browserName', 'browserVersion', 'pageLoadStrategy', 'platformName', 'proxy', ...]).issubset
E        +    and   set(['browserName', 'browserVersion', 'pageLoadStrategy', 'platformName', 'proxy', 'setWindowRect', ...]) = set(['browserVersion', 'timeouts', 'browserName', 'setWindowRect', 'proxy', 'pageLoadStrategy', ...])
E        +      where ['browserVersion', 'timeouts', 'browserName', 'setWindowRect', 'proxy', 'pageLoadStrategy', ...] = <built-in method keys of dict object at 0x7f6061957c58>()
E        +        where <built-in method keys of dict object at 0x7f6061957c58> = {'browserName': 'MiniBrowser', 'browserVersion': '2.21.5', 'pageLoadStrategy': 'normal', 'platformName': 'linux', ...}.keys

_          = <Session 2ba6e599-c519-436c-b896-4804f8947ac7>
add_browser_capabilites = <function update_capabilities at 0x7f606192f1b8>
new_session = <function create_session at 0x7f606192f230>
resp       = {'capabilities': {'browserName': 'MiniBrowser', 'browserVersion': '2.21.5', 'pageLoadStrategy': 'normal', 'platformName': 'linux', ...}, 'sessionId': '2ba6e599-c519-436c-b896-4804f8947ac7'}

WebDriverTests/imported/w3c/webdriver/tests/new_session/response.py:15: AssertionError
________________________________ test_resp_data ________________________________

new_session = <function create_session at 0x7f60630dcde8>
add_browser_capabilites = <function update_capabilities at 0x7f6062ddd2a8>
platform_name = 'linux'

    def test_resp_data(new_session, add_browser_capabilites, platform_name):
        resp, _ = new_session({"capabilities": {"alwaysMatch": add_browser_capabilites({})}})
    
        assert isinstance(resp["capabilities"]["browserName"], unicode)
        assert isinstance(resp["capabilities"]["browserVersion"], unicode)
        if platform_name:
            assert resp["capabilities"]["platformName"] == platform_name
        else:
            assert "platformName" in resp["capabilities"]
>       assert resp["capabilities"]["acceptInsecureCerts"] is False
E       KeyError: 'acceptInsecureCerts'

_          = <Session a4e36348-81ad-49d4-897c-246fc076d822>
add_browser_capabilites = <function update_capabilities at 0x7f6062ddd2a8>
new_session = <function create_session at 0x7f60630dcde8>
platform_name = 'linux'
resp       = {'capabilities': {'browserName': 'MiniBrowser', 'browserVersion': '2.21.5', 'pageLoadStrategy': 'normal', 'platformName': 'linux', ...}, 'sessionId': 'a4e36348-81ad-49d4-897c-246fc076d822'}

WebDriverTests/imported/w3c/webdriver/tests/new_session/response.py:35: KeyError

[Michael Catanzaro]

Mmmm, the WebDriver expectations are not cooperating. I'm gcurrently sitting at:


Expected to fail, but passed (3)
  imported/w3c/webdriver/tests/close_window/close.py::test_close_last_browsing_context
  imported/w3c/webdriver/tests/maximize_window/maximize.py::test_fully_exit_fullscreen
  imported/w3c/webdriver/tests/minimize_window/minimize.py::test_fully_exit_fullscreen


All of the above have flaky expectations, just like for API tests. Perhaps the test runner is not able to handle multiple expectations? The same syntax works for API tests.


Unexpected failures (4)
  imported/selenium/py/test/selenium/webdriver/common/alerts_tests.py::testShouldHandleAlertOnPageBeforeUnload[WebKitGTK]
  imported/selenium/py/test/selenium/webdriver/common/page_load_timeout_tests.py::testClickShouldTimeout[WebKitGTK]
  imported/selenium/py/test/selenium/webdriver/common/page_load_timeout_tests.py::testClickShouldTimeout[WebKitGTK]
  imported/selenium/py/test/selenium/webdriver/common/window_tests.py::test_should_fullscreen_the_current_window[WebKitGTK]


Ditto, except for that last test, which has just a FAIL expectation. I'll try removing the "[WebKitGTK]" from the end of the expectations and see if it makes any difference.


Unexpected timeouts (1)
  imported/w3c/webdriver/tests/actions/special_keys.py::test_webdriver_special_key_sends_keydown[F5-expected55]


This one is skipped, and has been for ages, so at least this one should not be my fault. :)

[Michael Catanzaro]

I think the test runner can't handle flakiness yet. So I'm skipping the following tests:

imported/selenium/py/test/selenium/webdriver/common/page_load_timeout_tests.py::testClickShouldTimeout

imported/w3c/webdriver/tests/set_window_rect/set.py::test_fully_exit_fullscreen
imported/w3c/webdriver/tests/maximize_window/maximize.py::test_fully_exit_fullscreen
imported/w3c/webdriver/tests/maximize_window/minimize.py::test_fully_exit_fullscreen

[Michael Catanzaro]

imported/w3c/webdriver/tests/close_window/close.py::test_close_last_browsing_context is now passing.

[Michael Catanzaro]

These tests are all flaky:

  imported/w3c/webdriver/tests/element_clear/clear.py::test_input_file
  imported/w3c/webdriver/tests/element_clear/clear.py::test_input_file_multiple
  imported/w3c/webdriver/tests/element_clear/clear.py::test_resettable_element_does_not_satisfy_validation_constraints[number-foo]
  imported/w3c/webdriver/tests/element_clear/clear.py::test_resettable_element_does_not_satisfy_validation_constraints[range-foo]
  imported/w3c/webdriver/tests/element_clear/clear.py::test_resettable_element_does_not_satisfy_validation_constraints[email-foo]
  imported/w3c/webdriver/tests/element_clear/clear.py::test_resettable_element_does_not_satisfy_validation_constraints[url-foo]
  imported/w3c/webdriver/tests/element_clear/clear.py::test_resettable_element_does_not_satisfy_validation_constraints[color-foo]
  imported/w3c/webdriver/tests/element_clear/clear.py::test_resettable_element_does_not_satisfy_validation_constraints[date-foo]
  imported/w3c/webdriver/tests/element_clear/clear.py::test_resettable_element_does_not_satisfy_validation_constraints[datetime-foo]
  imported/w3c/webdriver/tests/element_clear/clear.py::test_resettable_element_does_not_satisfy_validation_constraints[datetime-local-foo]
  imported/w3c/webdriver/tests/element_clear/clear.py::test_resettable_element_does_not_satisfy_validation_constraints[time-foo]
  imported/w3c/webdriver/tests/element_clear/clear.py::test_resettable_element_does_not_satisfy_validation_constraints[month-foo]
  imported/w3c/webdriver/tests/element_clear/clear.py::test_resettable_element_does_not_satisfy_validation_constraints[week-foo]


Since the test runner can't handle multiple expectations for the same test, I will skip them.

[Michael Catanzaro]

(In reply to Michael Catanzaro from comment #11)
> These tests are all flaky:
> 
>   imported/w3c/webdriver/tests/element_clear/clear.py::test_input_file
>  
> imported/w3c/webdriver/tests/element_clear/clear.py::test_input_file_multiple
>  
> imported/w3c/webdriver/tests/element_clear/clear.py::
> test_resettable_element_does_not_satisfy_validation_constraints[number-foo]
>  
> imported/w3c/webdriver/tests/element_clear/clear.py::
> test_resettable_element_does_not_satisfy_validation_constraints[range-foo]
>  
> imported/w3c/webdriver/tests/element_clear/clear.py::
> test_resettable_element_does_not_satisfy_validation_constraints[email-foo]
>  
> imported/w3c/webdriver/tests/element_clear/clear.py::
> test_resettable_element_does_not_satisfy_validation_constraints[url-foo]
>  
> imported/w3c/webdriver/tests/element_clear/clear.py::
> test_resettable_element_does_not_satisfy_validation_constraints[color-foo]
>  
> imported/w3c/webdriver/tests/element_clear/clear.py::
> test_resettable_element_does_not_satisfy_validation_constraints[date-foo]
>  
> imported/w3c/webdriver/tests/element_clear/clear.py::
> test_resettable_element_does_not_satisfy_validation_constraints[datetime-foo]
>  
> imported/w3c/webdriver/tests/element_clear/clear.py::
> test_resettable_element_does_not_satisfy_validation_constraints[datetime-
> local-foo]
>  
> imported/w3c/webdriver/tests/element_clear/clear.py::
> test_resettable_element_does_not_satisfy_validation_constraints[time-foo]
>  
> imported/w3c/webdriver/tests/element_clear/clear.py::
> test_resettable_element_does_not_satisfy_validation_constraints[month-foo]
>  
> imported/w3c/webdriver/tests/element_clear/clear.py::
> test_resettable_element_does_not_satisfy_validation_constraints[week-foo]
> 
> 
> Since the test runner can't handle multiple expectations for the same test,
> I will skip them.

Sorry, let's track these in bug #180404, not here.

[Michael Catanzaro]

New failures:

imported/w3c/webdriver/tests/element_send_keys/events.py::test_not_blurred[input]
  imported/w3c/webdriver/tests/element_send_keys/events.py::test_not_blurred[textarea]
  imported/w3c/webdriver/tests/element_click/navigate.py::test_link_unload_event
  imported/w3c/webdriver/tests/actions/control_click.py::test_control_click[\ue009-ctrlKey]
  imported/w3c/webdriver/tests/actions/control_click.py::test_control_click[\ue051-ctrlKey]
  imported/w3c/webdriver/tests/actions/control_click.py::test_release_control_click
  imported/w3c/webdriver/tests/is_element_selected/user_prompts.py::test_accept[capabilities0-prompt-]
  imported/w3c/webdriver/tests/is_element_selected/user_prompts.py::test_dismiss[capabilities0-prompt-None]

New flakes (skipped):

imported/w3c/webdriver/tests/fullscreen_window/user_prompts.py::test_accept[capabilities0-alert-None]
  imported/w3c/webdriver/tests/fullscreen_window/user_prompts.py::test_accept[capabilities0-confirm-True]
  imported/w3c/webdriver/tests/fullscreen_window/user_prompts.py::test_accept[capabilities0-prompt-]
  imported/w3c/webdriver/tests/fullscreen_window/user_prompts.py::test_dismiss[capabilities0-alert-None]
  imported/w3c/webdriver/tests/fullscreen_window/user_prompts.py::test_dismiss[capabilities0-confirm-False]
  imported/w3c/webdriver/tests/fullscreen_window/user_prompts.py::test_dismiss[capabilities0-prompt-None]
  imported/w3c/webdriver/tests/fullscreen_window/fullscreen.py::test_fullscreen
  imported/w3c/webdriver/tests/fullscreen_window/fullscreen.py::test_payload


New timeouts:

imported/w3c/webdriver/tests/element_send_keys/user_prompts.py::test_accept[capabilities0-alert-None]
  imported/w3c/webdriver/tests/element_send_keys/user_prompts.py::test_accept[capabilities0-confirm-True]
  imported/w3c/webdriver/tests/element_send_keys/user_prompts.py::test_accept[capabilities0-prompt-]
  imported/w3c/webdriver/tests/element_send_keys/user_prompts.py::test_dismiss[capabilities0-alert-None]
  imported/w3c/webdriver/tests/element_send_keys/user_prompts.py::test_dismiss[capabilities0-confirm-False]
  imported/w3c/webdriver/tests/element_send_keys/user_prompts.py::test_dismiss[capabilities0-prompt-None]
  imported/w3c/webdriver/tests/is_element_selected/user_prompts.py::test_accept[capabilities0-confirm-True]
  imported/w3c/webdriver/tests/is_element_selected/user_prompts.py::test_dismiss[capabilities0-confirm-False]

New flaky timeout:

imported/w3c/webdriver/tests/element_clear/clear.py::test_input_file

[Michael Catanzaro]

Actually these ones are flaky too. I'm just going to skip all the user_prompts.py tests.

(In reply to Michael Catanzaro from comment #13)  
> imported/w3c/webdriver/tests/is_element_selected/user_prompts.py::
> test_accept[capabilities0-prompt-]
>  
> imported/w3c/webdriver/tests/is_element_selected/user_prompts.py::
> test_dismiss[capabilities0-prompt-None]