new-run-webkit-tests fails to start HTTP server if there are proxy settings

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Xianzhu Wang]

new-run-webkit-tests script checks if the testing HTTP server has been properly started by sending testing requests to the server with urllib.urlopen(). However, in an environment with proxy set, urllib.urlopen() can't recognize the no_proxy settings, and always use proxy even to access http://127.0.0.1:8000/. This causes the checker fails to check the status of HTTP servers and reports 'failed to start httpd.'

To resolve this, we can explicitly disable proxies in urllib.urlopen().

[Xianzhu Wang]

Even sometimes urlopen can recognize no_proxy config for non-https requests, the proxy setting sometimes still causes "SSL unknown protocol" error.  Forcing disabling proxy can resolve this.

[Xianzhu Wang]

Created attachment 79708 [details]
The patch

[Eric Seidel (no email)]

How does one set a proxy so as to trigger this?

Can we unit test this?

[Xianzhu Wang]

(In reply to comment #3)
> How does one set a proxy so as to trigger this?

I reproduce the issue on an Ubuntu Lucid with the following proxy settings:
 * Manual proxy configuration
 * Use the same proxy for all protocols
 * HTTP proxy: some valid proxy server
 * Ignored Host list:
      localhost
      127.0.0.1/8
      *.local

$ python (before this, should start a web server serving https on 8443 port) 
>>> import urllib
>>> urllib.urlopen('https://127.0.0.1:8443/')
Traceback (most recent call last):
  File "/usr/lib/python2.6/urllib.py", line 86, in urlopen
    return opener.open(url)
  File "/usr/lib/python2.6/urllib.py", line 205, in open
    return getattr(self, name)(url)
  File "/usr/lib/python2.6/urllib.py", line 439, in open_https
    h.endheaders()
  File "/usr/lib/python2.6/httplib.py", line 904, in endheaders
    self._send_output()
  File "/usr/lib/python2.6/httplib.py", line 776, in _send_output
    self.send(msg)
  File "/usr/lib/python2.6/httplib.py", line 735, in send
    self.connect()
  File "/usr/lib/python2.6/httplib.py", line 1112, in connect
    self.sock = ssl.wrap_socket(sock, self.key_file, self.cert_file)
  File "/usr/lib/python2.6/ssl.py", line 350, in wrap_socket
    suppress_ragged_eofs=suppress_ragged_eofs)
  File "/usr/lib/python2.6/ssl.py", line 118, in __init__
    self.do_handshake()
  File "/usr/lib/python2.6/ssl.py", line 293, in do_handshake
    self._sslobj.do_handshake()
IOError: [Errno socket error] [Errno 1] _ssl.c:480: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol

>>> urllib.urlopen('https://127.0.0.1:8443/', proxies={})
<addinfourl at 19614032 whose fp = <socket._fileobject object at 0x129f550>>
>>> 

This seems a bug of python or some underlying library. Unsetting http_proxy, https_proxy, all_proxy environment variables doesn't help. I searched the error message and knew that many people encountered this and disabling proxy can resolve.

> 
> Can we unit test this?

I've no idea. Is it necessary?

[Eric Seidel (no email)]

Tests are generally more important than code fixes :)  As they prevent us from making the same bugs twice!  But if it's not possible to unit test this then we don't need to.  It may not be realistic to unit test this give that it depends on system settings.

[Tony Chang]

Will DRT pick up system proxy settings?  If so, can having a proxy set cause the http layout tests to fail?

Would it be better to just detect there is a proxy set and provide a good warning message to the user (e.g., what environment variables to set to disable when running the tests)?

[Xianzhu Wang]

(In reply to comment #6)
> Will DRT pick up system proxy settings?  If so, can having a proxy set cause the http layout tests to fail?
> 

A correct proxy setting should bypass localhost, and webkit can handle this correctly, so layout tests won't fail. The problem only occurs during python urllib.urlopen().

> Would it be better to just detect there is a proxy set and provide a good warning message to the user (e.g., what environment variables to set to disable when running the tests)?

Environment variables is not the only way to set proxy. Only unsetting environment variables seems not working for me. And I think disabling proxy before testing might be annoying for people who rely on proxies all the time.

[Tony Chang]

Comment on attachment 79708 [details]
The patch

I see, given that DRT should be able to handle system proxy settings and this is just a bug with python on Linux, this seems fine to me.

[WebKit Commit Bot]

Comment on attachment 79708 [details]
The patch

Clearing flags on attachment: 79708

Committed r76547: <http://trac.webkit.org/changeset/76547>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[WebKit Review Bot]

http://trac.webkit.org/changeset/76547 might have broken Qt Linux Release
The following tests are not passing:
fast/text/international/cjk-segmentation.html