Bug 64594 – test-webkitpy failing with Python 2.5

• Component: Tools / Tests
• Status: RESOLVED FIXED

[WebKit Review Bot]

webkitpy uses urlparse.parse_qs not available in Python 2.5
Requested by philn-tp on #webkit.

[Philippe Normand]

======================================================================
ERROR: test_changelog (webkitpy.tool.servers.gardeningserver_unittest.GardeningServerTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/slave/webkitgtk/gtk-linux-32-debug/build/Tools/Scripts/webkitpy/tool/servers/gardeningserver_unittest.py", line 78, in test_changelog
    self._post_to_path("/changelog?revision=2314", expected_stderr=expected_stderr, expected_stdout=expected_stdout)
  File "/home/slave/webkitgtk/gtk-linux-32-debug/build/Tools/Scripts/webkitpy/tool/servers/gardeningserver_unittest.py", line 70, in _post_to_path
    OutputCapture().assert_outputs(self, handler.do_POST, expected_stderr=expected_stderr, expected_stdout=expected_stdout)
  File "/home/slave/webkitgtk/gtk-linux-32-debug/build/Tools/Scripts/webkitpy/common/system/outputcapture.py", line 63, in assert_outputs
    return_value = function(*args, **kwargs)
  File "/home/slave/webkitgtk/gtk-linux-32-debug/build/Tools/Scripts/webkitpy/tool/servers/reflectionhandler.py", line 61, in do_POST
    self._handle_request()
  File "/home/slave/webkitgtk/gtk-linux-32-debug/build/Tools/Scripts/webkitpy/tool/servers/reflectionhandler.py", line 66, in _handle_request
    self.query = urlparse.parse_qs(query_string)
AttributeError: 'module' object has no attribute 'parse_qs'

[Philippe Normand]

In that case we should fallback to cgi.parse_qs

[Philippe Normand]

thirdparty.simplejson also needs an update:


======================================================================
FAIL: test_changelog (webkitpy.tool.servers.gardeningserver_unittest.GardeningServerTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/wk/WebKit/Tools/Scripts/webkitpy/tool/servers/gardeningserver_unittest.py", line 78, in test_changelog
    self._post_to_path("/changelog?revision=2314", expected_stderr=expected_stderr, expected_stdout=expected_stdout)
  File "/wk/WebKit/Tools/Scripts/webkitpy/tool/servers/gardeningserver_unittest.py", line 70, in _post_to_path
    OutputCapture().assert_outputs(self, handler.do_POST, expected_stderr=expected_stderr, expected_stdout=expected_stdout)
  File "/wk/WebKit/Tools/Scripts/webkitpy/common/system/outputcapture.py", line 65, in assert_outputs
    testcase.assertEqual(stdout_string, expected_stdout)
AssertionError: '== Begin JSON Response ==\n{"bug_id": 42, "author_email": "[e-mail]", "reviewer_text": "Darin Adler", "author_name": "Adam Barth", "changed_files": ["path\\/to\\/file", "another\\/file"]}\n== End JSON Response ==\n' != '== Begin JSON Response ==\n{"bug_id": 42, "author_email": "[e-mail]", "reviewer_text": "Darin Adler", "author_name": "Adam Barth", "changed_files": ["path/to/file", "another/file"]}\n== End JSON Response ==\n'

[Philippe Normand]

Created attachment 100964 [details]
proposed patch

[WebKit Review Bot]

Attachment 100964 [details] did not pass style-queue:

Failed to run "['Tools/Scripts/check-webkit-style', '--diff-files', u'Tools/ChangeLog', u'Tools/Scripts/webkitpy..." exit_code: 1

Tools/Scripts/webkitpy/thirdparty/simplejson/__init__.py:413:  trailing whitespace  [pep8/W291] [5]
Total errors found: 1 in 10 files


If any of these errors are false positives, please file a bug against check-webkit-style.

[Eric Seidel (no email)]

Comment on attachment 100964 [details]
proposed patch

View in context: https://bugs.webkit.org/attachment.cgi?id=100964&action=review

> Tools/ChangeLog:11
> +        Updated simplejson to version 2.1.6 that fixes an issue when
> +        dumping slashes. Also use cgi.parse_qs instead of the urlparse
> +        version to be compatible with Python2.5. Verified the fix with
> +        test-webkitpy on python 2.5 and python 2.6.

Ok.  So you're just updating simplejson, and then separately making changes to our code, correct?  You're not changing simplejson besides the update?

[Philippe Normand]

(In reply to comment #6)
> (From update of attachment 100964 [details])
> View in context: https://bugs.webkit.org/attachment.cgi?id=100964&action=review
> 
> > Tools/ChangeLog:11
> > +        Updated simplejson to version 2.1.6 that fixes an issue when
> > +        dumping slashes. Also use cgi.parse_qs instead of the urlparse
> > +        version to be compatible with Python2.5. Verified the fix with
> > +        test-webkitpy on python 2.5 and python 2.6.
> 
> Ok.  So you're just updating simplejson, and then separately making changes to our code, correct? 

Yes

> You're not changing simplejson besides the update?

I did change two import statements from:

from simplejson import something

to:

import something

I will land this patch later unless you have more comments. I locally checked test-webkitpy still passes with python 2.5 and 2.6.

[Daniel Bates]

What's the status on landing this patch? Can we land this by commit-queue? Both the Leopard Debug and Release bots are failing the unit tests because of this issue.

[Adam Barth]

Comment on attachment 100964 [details]
proposed patch

I mean, this should be two separate patches, but its ok.

[WebKit Review Bot]

Comment on attachment 100964 [details]
proposed patch

Clearing flags on attachment: 100964

Committed r91167: <http://trac.webkit.org/changeset/91167>

[WebKit Review Bot]

All reviewed patches have been landed.  Closing bug.

[James Robinson]

When running NRWT on chromium-linux I'm getting the following at the end of the test run, before results.html is produced:


Traceback (most recent call last):
  File "WebKit/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 438, in <module>
    sys.exit(main())
  File "WebKit/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 433, in main
    return run(port, options, args)
  File "WebKit/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 107, in run
    num_unexpected_results = manager.run(result_summary)
  File "WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 933, in run
    self._upload_json_files(summarized_results, result_summary, individual_test_timings)
  File "WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 1080, in _upload_json_files
    json_results_generator.write_json(self._fs, times_trie, times_json_path)
  File "WebKit/Tools/Scripts/webkitpy/layout_tests/layout_package/json_results_generator.py", line 65, in write_json
    json_data = simplejson.dumps(json_object, separators=(',', ':'))
  File "WebKit/Tools/Scripts/webkitpy/thirdparty/simplejson/__init__.py", line 269, in dumps
    use_decimal=use_decimal, **kw).encode(obj)
  File "WebKit/Tools/Scripts/webkitpy/thirdparty/simplejson/encoder.py", line 216, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "WebKit/Tools/Scripts/webkitpy/thirdparty/simplejson/encoder.py", line 277, in iterencode
    self.skipkeys, self.allow_nan, key_memo, self.use_decimal)
TypeError: make_encoder() takes at most 9 arguments (11 given)


Is this related?  What's the fix?

[Eric Seidel (no email)]

I suspect the recent simplejson upgrade?

[Philippe Normand]

(In reply to comment #13)
> I suspect the recent simplejson upgrade?

Yes. Bug 64757 seems to fix the issue. Is the chromium bot using python 2.6?
I wonder if with the patch of that bug and a bot running python 2.5 you'd still get the traceback mentioned above...

[Philippe Normand]

(In reply to comment #14)
> (In reply to comment #13)
> > I suspect the recent simplejson upgrade?
> 
> Yes. Bug 64757 seems to fix the issue. Is the chromium bot using python 2.6?
> I wonder if with the patch of that bug and a bot running python 2.5 you'd still get the traceback mentioned above...

I've just checked NRWT with python2.5 on GTK and we're indeed safe. Thanks Eric and Adam!