Bug 137338 – Remove workarounds for versions of Python before 2.7

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Éva Balázsfalvi]

Remove workarounds for versions of Python before 2.7

[Éva Balázsfalvi]

Created attachment 239098 [details]
Patch

[WebKit Commit Bot]

Comment on attachment 239098 [details]
Patch

Clearing flags on attachment: 239098

Committed r174264: <http://trac.webkit.org/changeset/174264>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[WebKit Commit Bot]

Re-opened since this is blocked by bug 137392

[Carlos Alberto Lopez Perez]

This seems to have broken run-webkit-tests on all the bots (except EFL it seems, not sure):

 * https://build.webkit.org/builders/Apple%20Mavericks%20Debug%20WK1%20%28Tests%29/builds/8010
 * https://build.webkit.org/builders/Apple%20Mavericks%20Debug%20WK2%20%28Tests%29/builds/7162
 * https://build.webkit.org/builders/Apple%20MountainLion%20Debug%20WK2%20%28Tests%29/builds/19455
 * https://build.webkit.org/builders/Apple%20MountainLion%20Release%20WK1%20%28Tests%29/builds/22775
 * https://build.webkit.org/builders/Apple%20MountainLion%20Release%20WK2%20%28Tests%29/builds/21216
 * https://build.webkit.org/builders/Apple%20Win%207%20Debug%20%28Tests%29/builds/62998
 * https://build.webkit.org/builders/GTK%20Linux%2064-bit%20Release%20%28Tests%29/builds/3203

On this log from the GTK bot (at the bottom) there is a python traceback that could give some hints about what is wrong: https://build.webkit.org/builders/GTK%20Linux%2064-bit%20Release%20%28Tests%29/builds/3203/steps/layout-test/logs/stdio

[Carlos Alberto Lopez Perez]

run-perf-tests seems to be also broken. Locally I get:

$ Tools/Scripts/run-perf-tests  --platform gtk --release -2 Interactive/window-resize.html
Running 1 tests
Running Interactive/window-resize.html (1 of 1)
Last character read from DRT stdout line was not a newline!  This indicates either a NRWT or DRT bug.
Last character read from DRT stdout line was not a newline!  This indicates either a NRWT or DRT bug.
Last character read from DRT stdout line was not a newline!  This indicates either a NRWT or DRT bug.
Last character read from DRT stdout line was not a newline!  This indicates either a NRWT or DRT bug.
Last character read from DRT stdout line was not a newline!  This indicates either a NRWT or DRT bug.
Last character read from DRT stdout line was not a newline!  This indicates either a NRWT or DRT bug.
Last character read from DRT stdout line was not a newline!  This indicates either a NRWT or DRT bug.
Last character read from DRT stdout line was not a newline!  This indicates either a NRWT or DRT bug.
Last character read from DRT stdout line was not a newline!  This indicates either a NRWT or DRT bug.
Last character read from DRT stdout line was not a newline!  This indicates either a NRWT or DRT bug.
Last character read from DRT stdout line was not a newline!  This indicates either a NRWT or DRT bug.
Last character read from DRT stdout line was not a newline!  This indicates either a NRWT or DRT bug.
Last character read from DRT stdout line was not a newline!  This indicates either a NRWT or DRT bug.
Last character read from DRT stdout line was not a newline!  This indicates either a NRWT or DRT bug.
Last character read from DRT stdout line was not a newline!  This indicates either a NRWT or DRT bug.
Last character read from DRT stdout line was not a newline!  This indicates either a NRWT or DRT bug.
Traceback (most recent call last):
  File "Tools/Scripts/run-perf-tests", line 40, in <module>
    sys.exit(PerfTestsRunner().run())
  File "/home/clopez/webkit/webkit/Tools/Scripts/webkitpy/performance_tests/perftestsrunner.py", line 205, in run
    unexpected = self._run_tests_set(sorted(list(tests), key=lambda test: test.test_name()))
  File "/home/clopez/webkit/webkit/Tools/Scripts/webkitpy/performance_tests/perftestsrunner.py", line 372, in _run_tests_set
    metrics = test.run(self._options.time_out_ms)
  File "/home/clopez/webkit/webkit/Tools/Scripts/webkitpy/performance_tests/perftest.py", line 132, in run
    if not self._run_with_driver(driver, time_out_ms):
  File "/home/clopez/webkit/webkit/Tools/Scripts/webkitpy/performance_tests/perftest.py", line 179, in _run_with_driver
    self._filter_output(output)
  File "/home/clopez/webkit/webkit/Tools/Scripts/webkitpy/performance_tests/perftest.py", line 273, in _filter_output
    output.text = '\n'.join([line for line in re.split('\n', output.text) if not self._should_ignore_line(self._lines_to_ignore_in_parser_result, line)])
TypeError: sequence item 0: expected string, bytearray found

[Csaba Osztrogonác]

Comment on attachment 239098 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=239098&action=review

> Tools/Scripts/webkitpy/port/driver.py:534
> -        self.content = str()  # FIXME: Should be bytearray() once we require Python 2.6.
> +        self.content = bytearray()

This one ...

> Tools/Scripts/webkitpy/port/server_process.py:99
> -        self._output = str()  # bytesarray() once we require Python 2.6
> -        self._error = str()  # bytesarray() once we require Python 2.6
> +        self._output = bytearray()
> +        self._error = bytearray()

... or this one broke the tests. Éva, could you check what happened?

[Éva Balázsfalvi]

Created attachment 239203 [details]
Patch

Took out the str() to bytearray() modification which seemingly made the layout tests fail. I'll investigate what caused the problem later.

[Carlos Alberto Lopez Perez]

(In reply to comment #8)
> Created an attachment (id=239203) [details]
> Patch
> 
> Took out the str() to bytearray() modification which seemingly made the layout tests fail. I'll investigate what caused the problem later.

I tested this patch, and both run-webkit-test and run-perf-test seems to work as expected on the GTK port.

[Csaba Osztrogonác]

Comment on attachment 239203 [details]
Patch

Let's do it without bytearray(), r=me.

[WebKit Commit Bot]

Comment on attachment 239203 [details]
Patch

Clearing flags on attachment: 239203

Committed r174266: <http://trac.webkit.org/changeset/174266>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Csaba Osztrogonác]

Comment on attachment 239203 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=239203&action=review

> Tools/Scripts/webkitpy/common/system/autoinstall.py:-248
> -    # This is a replacement for ZipFile.extractall(), which is
> -    # available in Python 2.6 but not in earlier versions.
> -    # NOTE: The version in 2.6.1 (which shipped on Snow Leopard) is broken!
> -    def _extract_all(self, zip_file, target_dir):
> -        for name in zip_file.namelist():
> -            path = os.path.join(target_dir, name)
> -            if not os.path.basename(path):
> -                # Then the path ends in a slash, so it is a directory.
> -                os.makedirs(path)
> -                continue
> -
> -            try:
> -                # We open this file w/o encoding, as we're reading/writing
> -                # the raw byte-stream from the zip file.
> -                outfile = open(path, 'wb')
> -            except IOError:
> -                # Not all zip files seem to list the directories explicitly,
> -                # so try again after creating the containing directory.
> -                _log.debug("Got IOError: retrying after creating directory...")
> -                dirname = os.path.dirname(path)
> -                os.makedirs(dirname)
> -                outfile = open(path, 'wb')
> -
> -            try:
> -                outfile.write(zip_file.read(name))
> -            finally:
> -                outfile.close()
> -

Unfortunately it broke autoinstalling python-irclib-0.4.8.zip with the stock python on 
- Ubuntu 12.04: python 2.7.3
- Ubuntu 14.04: python 2.7.6
- OS X 10.9: python 2.7.5

I think we should add this workaround back with proper comment.

> Tools/Scripts/webkitpy/common/system/autoinstall.py:235
> -            self._extract_all(zip_file, scratch_dir)
> +            zip_file.extractall(zip_file, scratch_dir)

extractall raises the following exception:

Auto-installing package: irclib.py
Traceback (most recent call last):
  File "/Users/oszi/WebKit/Tools/Scripts/webkitpy/test/main.py", line 234, in <module>
    sys.exit(main())
  File "/Users/oszi/WebKit/Tools/Scripts/webkitpy/test/main.py", line 72, in main
    return not tester.run()
  File "/Users/oszi/WebKit/Tools/Scripts/webkitpy/test/main.py", line 122, in run
    return self._run_tests(names)
  File "/Users/oszi/WebKit/Tools/Scripts/webkitpy/test/main.py", line 132, in _run_tests
    autoinstall_everything()
  File "/Users/oszi/WebKit/Tools/Scripts/webkitpy/thirdparty/__init__.py", line 166, in autoinstall_everything
    getattr(_hook, method)()
  File "/Users/oszi/WebKit/Tools/Scripts/webkitpy/thirdparty/__init__.py", line 150, in _install_irc
    url_subpath="irclib.py")
  File "/Users/oszi/WebKit/Tools/Scripts/webkitpy/common/system/autoinstall.py", line 473, in install
    raise Exception(message)
Exception: Error auto-installing the irclib.py package to:
 "/Users/oszi/WebKit/Tools/Scripts/webkitpy/thirdparty/autoinstalled/irc/irclib.py"
 --> Inner message: "There is no item named '/' in the archive"

[Csaba Osztrogonác]

added back in http://trac.webkit.org/changeset/174448