Bug 87548 – check-webkit-style cannot running in Chinese windows.

• Component: Tools / Tests
• Status: RESOLVED FIXED

[huangxueqing]

The traceback as:
$ ./check-webkit-style --help
Traceback (most recent call last):
  File "./check-webkit-style", line 48, in <module> sys.exit(CheckWebKitStyle().main())
  File "/cygdrive/e/webkit/Tools/Scripts/webkitpy/style/main.py", line 126, in main host = Host()
  File "/cygdrive/e/webkit/Tools/Scripts/webkitpy/common/host.py", line 49, in __init__    SystemHost.__init__(self)
  File "/cygdrive/e/webkit/Tools/Scripts/webkitpy/common/system/systemhost.py", line 40, in __init__  self.user = user.User()
  File "/cygdrive/e/webkit/Tools/Scripts/webkitpy/common/system/user.py", line 61, in __init__  self._platforminfo = platforminfo or PlatformInfo(sys, platform, Executive())
  File "/cygdrive/e/webkit/Tools/Scripts/webkitpy/common/system/platforminfo.py", line 56, in __init__  self.os_version = self._determine_win_version(self._win_version_tuple(sys_module))
  File "/cygdrive/e/webkit/Tools/Scripts/webkitpy/common/system/platforminfo.py", line 132, in _win_version_tuple  return self._win_version_tuple_from_cmd()
  File "/cygdrive/e/webkit/Tools/Scripts/webkitpy/common/system/platforminfo.py", line 136, in _win_version_tuple_from_cmd   ver_output = self._executive.run_command(['cmd', '/c', 'ver'])
  File "/cygdrive/e/webkit/Tools/Scripts/webkitpy/common/system/executive.py", line 402, in run_command  output = output.decode(self._child_process_encoding())
  File "/usr/lib/python2.6/encodings/utf_8.py", line 16, in decode  return codecs.utf_8_decode(input, errors, True) UnicodeDecodeError: 'utf8' codec can't decode byte 0xb0 in position 21: unexpected code byte

[huangxueqing]

Created attachment 182481 [details]
patch

The version info retrieved in Chinese Windows was "Microsoft Windows [版本 6.1.7601]", "版本"('version' in English) was encoded as "0xB0 0xE6 0xB1 0xBE" and '0xB0' was not a valid utf-8 character, thus check-webkit-style running failed.
Actually, we execute "cmd /c ver" then use regular expression to match version infomation, it's unnecessary to decode the output string of execution.

[Eric Seidel (no email)]

I thought that Executive was smart enough to use the local code page on windows?  

This line:

  File "/cygdrive/e/webkit/Tools/Scripts/webkitpy/common/system/executive.py", line 402, in run_command  output = output.decode(self._child_process_encoding())


Clearly its the wrong encoding in your case:
http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/common/system/executive.py#L444

444	    def _child_process_encoding(self):
445	        # Win32 Python 2.x uses CreateProcessA rather than CreateProcessW
446	        # to launch subprocesses, so we have to encode arguments using the
447	        # current code page.
448	        if sys.platform == 'win32' and sys.version < '3':
449	            return 'mbcs'
450	        # All other platforms use UTF-8.
451	        # FIXME: Using UTF-8 on Cygwin will confuse Windows-native commands
452	        # which will expect arguments to be encoded using the current code
453	        # page.
454	        return 'utf-8'

[Eric Seidel (no email)]

It looks like this article has some relevant help:
http://stackoverflow.com/questions/1259084/what-encoding-code-page-is-cmd-exe-using

[huangxueqing]

(In reply to comment #2)
> I thought that Executive was smart enough to use the local code page on windows?  
I execute WebKit scripts in cygwin, sys.platform == 'cygwin' and python intergraded in cygwin did not include 'mbcs' decoder, thus it's always use 'utf-8' to decode output string. Actually GBK encoding used in my system but we could not dectect it.

> This line:
>   File "/cygdrive/e/webkit/Tools/Scripts/webkitpy/common/system/executive.py", line 402, in run_command  output = output.decode(self._child_process_encoding())
> Clearly its the wrong encoding in your case:
> http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/common/system/executive.py#L444
> 444        def _child_process_encoding(self):
> 445            # Win32 Python 2.x uses CreateProcessA rather than CreateProcessW
> 446            # to launch subprocesses, so we have to encode arguments using the
> 447            # current code page.
> 448            if sys.platform == 'win32' and sys.version < '3':
> 449                return 'mbcs'
> 450            # All other platforms use UTF-8.
> 451            # FIXME: Using UTF-8 on Cygwin will confuse Windows-native commands
> 452            # which will expect arguments to be encoded using the current code
> 453            # page.
> 454            return 'utf-8'

[huangxueqing]

(In reply to comment #3)
> It looks like this article has some relevant help:
> http://stackoverflow.com/questions/1259084/what-encoding-code-page-is-cmd-exe-using

Thanks, I will take a look at that.

[Darin Adler]

Comment on attachment 182481 [details]
patch

While I am not an expert, this looks fine to me.

[WebKit Review Bot]

Comment on attachment 182481 [details]
patch

Clearing flags on attachment: 182481

Committed r139924: <http://trac.webkit.org/changeset/139924>

[WebKit Review Bot]

All reviewed patches have been landed.  Closing bug.

[Eric Seidel (no email)]

It is. :)  It solves this specific case, but there will be many more troubles if we're not able to decode output from cmd to the right code-page on windows.  We have code which should be doing this, but clearly isnt' doing it right.