87548 2012-05-25 16:34:36 -0700 check-webkit-style cannot running in Chinese windows. 2013-01-16 14:22:53 -0800 1 1 1 Unclassified WebKit Tools / Tests 528+ (Nightly build) Unspecified Unspecified RESOLVED FIXED P2 Normal --- 0 huangxueqing webkit-unassigned abarth alancutter dpranke eric levin ojan tony vestbo webkit.review.bot oldest_to_newest 634428 0 huangxueqing 2012-05-25 16:34:36 -0700 The traceback as: $ ./check-webkit-style --help Traceback (most recent call last): File "./check-webkit-style", line 48, in <module> sys.exit(CheckWebKitStyle().main()) File "/cygdrive/e/webkit/Tools/Scripts/webkitpy/style/main.py", line 126, in main host = Host() File "/cygdrive/e/webkit/Tools/Scripts/webkitpy/common/host.py", line 49, in __init__ SystemHost.__init__(self) File "/cygdrive/e/webkit/Tools/Scripts/webkitpy/common/system/systemhost.py", line 40, in __init__ self.user = user.User() File "/cygdrive/e/webkit/Tools/Scripts/webkitpy/common/system/user.py", line 61, in __init__ self._platforminfo = platforminfo or PlatformInfo(sys, platform, Executive()) File "/cygdrive/e/webkit/Tools/Scripts/webkitpy/common/system/platforminfo.py", line 56, in __init__ self.os_version = self._determine_win_version(self._win_version_tuple(sys_module)) File "/cygdrive/e/webkit/Tools/Scripts/webkitpy/common/system/platforminfo.py", line 132, in _win_version_tuple return self._win_version_tuple_from_cmd() File "/cygdrive/e/webkit/Tools/Scripts/webkitpy/common/system/platforminfo.py", line 136, in _win_version_tuple_from_cmd ver_output = self._executive.run_command(['cmd', '/c', 'ver']) File "/cygdrive/e/webkit/Tools/Scripts/webkitpy/common/system/executive.py", line 402, in run_command output = output.decode(self._child_process_encoding()) File "/usr/lib/python2.6/encodings/utf_8.py", line 16, in decode return codecs.utf_8_decode(input, errors, True) UnicodeDecodeError: 'utf8' codec can't decode byte 0xb0 in position 21: unexpected code byte 805956 1 182481 huangxueqing 2013-01-13 02:15:57 -0800 Created attachment 182481 patch The version info retrieved in Chinese Windows was "Microsoft Windows [版本 6.1.7601]", "版本"('version' in English) was encoded as "0xB0 0xE6 0xB1 0xBE" and '0xB0' was not a valid utf-8 character, thus check-webkit-style running failed. Actually, we execute "cmd /c ver" then use regular expression to match version infomation, it's unnecessary to decode the output string of execution. 806013 2 eric 2013-01-13 15:28:55 -0800 I thought that Executive was smart enough to use the local code page on windows? This line: File "/cygdrive/e/webkit/Tools/Scripts/webkitpy/common/system/executive.py", line 402, in run_command output = output.decode(self._child_process_encoding()) Clearly its the wrong encoding in your case: http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/common/system/executive.py#L444 444 def _child_process_encoding(self): 445 # Win32 Python 2.x uses CreateProcessA rather than CreateProcessW 446 # to launch subprocesses, so we have to encode arguments using the 447 # current code page. 448 if sys.platform == 'win32' and sys.version < '3': 449 return 'mbcs' 450 # All other platforms use UTF-8. 451 # FIXME: Using UTF-8 on Cygwin will confuse Windows-native commands 452 # which will expect arguments to be encoded using the current code 453 # page. 454 return 'utf-8' 806014 3 eric 2013-01-13 15:30:24 -0800 It looks like this article has some relevant help: http://stackoverflow.com/questions/1259084/what-encoding-code-page-is-cmd-exe-using 806042 4 huangxueqing 2013-01-13 18:24:55 -0800 (In reply to comment #2) > I thought that Executive was smart enough to use the local code page on windows? I execute WebKit scripts in cygwin, sys.platform == 'cygwin' and python intergraded in cygwin did not include 'mbcs' decoder, thus it's always use 'utf-8' to decode output string. Actually GBK encoding used in my system but we could not dectect it. > This line: > File "/cygdrive/e/webkit/Tools/Scripts/webkitpy/common/system/executive.py", line 402, in run_command output = output.decode(self._child_process_encoding()) > Clearly its the wrong encoding in your case: > http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/common/system/executive.py#L444 > 444 def _child_process_encoding(self): > 445 # Win32 Python 2.x uses CreateProcessA rather than CreateProcessW > 446 # to launch subprocesses, so we have to encode arguments using the > 447 # current code page. > 448 if sys.platform == 'win32' and sys.version < '3': > 449 return 'mbcs' > 450 # All other platforms use UTF-8. > 451 # FIXME: Using UTF-8 on Cygwin will confuse Windows-native commands > 452 # which will expect arguments to be encoded using the current code > 453 # page. > 454 return 'utf-8' 806043 5 huangxueqing 2013-01-13 18:27:44 -0800 (In reply to comment #3) > It looks like this article has some relevant help: > http://stackoverflow.com/questions/1259084/what-encoding-code-page-is-cmd-exe-using Thanks, I will take a look at that. 808683 6 182481 darin 2013-01-16 13:50:44 -0800 Comment on attachment 182481 patch While I am not an expert, this looks fine to me. 808703 7 182481 webkit.review.bot 2013-01-16 14:09:26 -0800 Comment on attachment 182481 patch Clearing flags on attachment: 182481 Committed r139924: <http://trac.webkit.org/changeset/139924> 808704 8 webkit.review.bot 2013-01-16 14:09:31 -0800 All reviewed patches have been landed. Closing bug. 808716 9 eric 2013-01-16 14:22:53 -0800 It is. :) It solves this specific case, but there will be many more troubles if we're not able to decode output from cmd to the right code-page on windows. We have code which should be doing this, but clearly isnt' doing it right. 182481 2013-01-13 02:15:57 -0800 2013-01-16 14:09:26 -0800 patch 87548.patch text/plain 1489 huangxueqing SW5kZXg6IFRvb2xzL0NoYW5nZUxvZw0KPT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQ0KLS0tIFRvb2xzL0NoYW5nZUxvZwko cmV2aXNpb24gMTM5NTYxKQ0KKysrIFRvb2xzL0NoYW5nZUxvZwkod29ya2luZyBjb3B5KQ0KQEAg LTEsMyArMSwxMyBAQA0KKzIwMTMtMDEtMTMgIFh1ZXFpbmcgSHVhbmcgIDxodWFuZ3h1ZXFpbmdA YmFpZHUuY29tPgorCisgICAgICAgIGNoZWNrLXdlYmtpdC1zdHlsZSBzY3JpcHQgY2Fubm90IHJ1 bm5pbmcgaW4gQ2hpbmVzZSB3aW5kb3dzLgorICAgICAgICBodHRwczovL2J1Z3Mud2Via2l0Lm9y Zy9zaG93X2J1Zy5jZ2k/aWQ9ODc1NDgKKworICAgICAgICBSZXZpZXdlZCBieSBOT0JPRFkgKE9P UFMhKS4KKworICAgICAgICAqIFNjcmlwdHMvd2Via2l0cHkvY29tbW9uL3N5c3RlbS9wbGF0Zm9y bWluZm8ucHk6CisgICAgICAgIChQbGF0Zm9ybUluZm8uX3dpbl92ZXJzaW9uX3R1cGxlX2Zyb21f Y21kKToKKwogMjAxMy0wMS0xMiAgTmljbyBXZWJlciAgPHRoYWtpc0BjaHJvbWl1bS5vcmc+CiAK ICAgICAgICAgbnJ3dC9jaHJvbWl1bTogRG9uJ3QgcHJlZmVyIGNocm9taXVtIG92ZXIgd2Via2l0 IGJ1aWxkIGRpcmVjdG9yaWVzCkluZGV4OiBUb29scy9TY3JpcHRzL3dlYmtpdHB5L2NvbW1vbi9z eXN0ZW0vcGxhdGZvcm1pbmZvLnB5DQo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09DQotLS0gVG9vbHMvU2NyaXB0cy93ZWJr aXRweS9jb21tb24vc3lzdGVtL3BsYXRmb3JtaW5mby5weQkocmV2aXNpb24gMTM5NTYwKQ0KKysr IFRvb2xzL1NjcmlwdHMvd2Via2l0cHkvY29tbW9uL3N5c3RlbS9wbGF0Zm9ybWluZm8ucHkJKHdv cmtpbmcgY29weSkNCkBAIC0xNTUsNyArMTU1LDcgQEANCiAKICAgICBkZWYgX3dpbl92ZXJzaW9u X3R1cGxlX2Zyb21fY21kKHNlbGYpOgogICAgICAgICAjIE5vdGUgdGhhdCB0aGlzIHNob3VsZCBv bmx5IGV2ZXIgYmUgY2FsbGVkIG9uIHdpbmRvd3MsIHNvIHRoaXMgc2hvdWxkIGFsd2F5cyB3b3Jr LgotICAgICAgICB2ZXJfb3V0cHV0ID0gc2VsZi5fZXhlY3V0aXZlLnJ1bl9jb21tYW5kKFsnY21k JywgJy9jJywgJ3ZlciddKQorICAgICAgICB2ZXJfb3V0cHV0ID0gc2VsZi5fZXhlY3V0aXZlLnJ1 bl9jb21tYW5kKFsnY21kJywgJy9jJywgJ3ZlciddLCBkZWNvZGVfb3V0cHV0PUZhbHNlKQogICAg ICAgICBtYXRjaF9vYmplY3QgPSByZS5zZWFyY2gocicoP1A8bWFqb3I+XGQpXC4oP1A8bWlub3I+ XGQpXC4oP1A8YnVpbGQ+XGQrKScsIHZlcl9vdXRwdXQpCiAgICAgICAgIGFzc2VydCBtYXRjaF9v YmplY3QsICdjbWQgcmV0dXJuZWQgYW4gdW5leHBlY3RlZCB2ZXJzaW9uIHN0cmluZzogJyArIHZl cl9vdXRwdXQKICAgICAgICAgcmV0dXJuIHR1cGxlKG1hcChpbnQsIG1hdGNoX29iamVjdC5ncm91 cHMoKSkpCg==