50843 2010-12-10 13:57:47 -0800 gtk-ews having trouble with non-ascii characters 2011-06-27 10:16:45 -0700 1 1 1 Unclassified WebKit Tools / Tests 528+ (Nightly build) All All NEW P2 Normal --- 63452 1 abarth webkit-unassigned eric leandro mrobinson oldest_to_newest 320392 0 abarth 2010-12-10 13:57:47 -0800 ../../JavaScriptCore/wtf/TCPageMap.h: In function ‘size_t WTF::fastMallocSize(const void*)’: Traceback (most recent call last): File "/mnt/git/webkit-gtk-ews/WebKitTools/Scripts/webkitpy/tool/bot/queueengine.py", line 108, in run if not self._delegate.process_work_item(work_item): File "/mnt/git/webkit-gtk-ews/WebKitTools/Scripts/webkitpy/tool/commands/queues.py", line 362, in process_work_item if not self.review_patch(patch): File "/mnt/git/webkit-gtk-ews/WebKitTools/Scripts/webkitpy/tool/commands/earlywarningsystem.py", line 92, in review_patch if not self._can_build(): File "/mnt/git/webkit-gtk-ews/WebKitTools/Scripts/webkitpy/tool/commands/earlywarningsystem.py", line 53, in _can_build "--no-update"]) File "/mnt/git/webkit-gtk-ews/WebKitTools/Scripts/webkitpy/tool/commands/queues.py", line 96, in run_webkit_patch return self._tool.executive.run_and_throw_if_fail(webkit_patch_args) File "/mnt/git/webkit-gtk-ews/WebKitTools/Scripts/webkitpy/common/system/executive.py", line 141, in run_and_throw_if_fail exit_code = self._run_command_with_teed_output(args, child_stdout) File "/mnt/git/webkit-gtk-ews/WebKitTools/Scripts/webkitpy/common/system/executive.py", line 126, in _run_command_with_teed_output teed_output.write(output_line) File "/mnt/git/webkit-gtk-ews/WebKitTools/Scripts/webkitpy/common/system/deprecated_logging.py", line 55, in write file.write(bytes) File "/mnt/git/webkit-gtk-ews/WebKitTools/Scripts/webkitpy/common/system/deprecated_logging.py", line 55, in write file.write(bytes) File "/usr/lib/python2.6/codecs.py", line 691, in write return self.writer.write(data) File "/usr/lib/python2.6/codecs.py", line 351, in write data, consumed = self.encode(object, self.errors) UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 50: ordinal not in range(128) 320400 1 eric 2010-12-10 14:05:28 -0800 I'm not sure how to solve this. I remember explicitly moving tee() to operate on bytes instead of unicode strings long ago. This seems to suggest that the logging module is using a codecs.open'd log file and trying to decode the byte stream we're sending to it. 320402 2 eric 2010-12-10 14:07:39 -0800 Maybe it doesn't make sense to write bytes to std out? 322018 3 leandro 2010-12-14 09:49:44 -0800 (In reply to comment #0) > UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 50: > ordinal not in range(128) BTW, having the same problem with EFL-EWS. (In reply to comment #2) > Maybe it doesn't make sense to write bytes to std out? Maybe only saving to a log file and printing the file name would help? Less noise on EWS output, and debuggable whenever needed. 322085 4 eric 2010-12-14 11:53:45 -0800 We haven't yet been able to produce a minimal reduction. However we used: setenv LANG en_US.US-ASCII to work around the issue on the gtk-ews for the moment. 322086 5 eric 2010-12-14 11:54:07 -0800 gcc seems to like to print fancy quotes in recent versions 322114 6 leandro 2010-12-14 12:24:07 -0800 (In reply to comment #4) > However we used: > setenv LANG en_US.US-ASCII > to work around the issue on the gtk-ews for the moment. Using the same workaround on EFL-EWS. Seems it's working. 424104 7 eric 2011-06-20 17:12:34 -0700 I'm struggling to reproduce this with a minimal example. I'm not sure how we're hitting this. I could see we might hit a decoding error with run_and_throw_if_fail(cmd, silent=True), because /dev/null is opened w/o any encoding. But I don't see how we hit this case. What stream are we opening with encoding of ascii? The logging stream? Maybe the python on that system doesn't correctly default to utf8? What's the lang value before we override it to US-ASCII? 424217 8 mrobinson 2011-06-20 20:18:31 -0700 (In reply to comment #7) > But I don't see how we hit this case. What stream are we opening with encoding of ascii? The logging stream? Maybe the python on that system doesn't correctly default to utf8? Looks like you can figure out the default encoding in Python by running this in the REPL: import sys sys.getdefaultencoding() 424271 9 eric 2011-06-20 22:47:52 -0700 On both my mac and on linux, sys.getdefaultencoding() returns 'ascii'. It does seem like that must be the encoding we're hitting. The question is what file is opened with default encoding? I assume it must be stderr/stdout. But why? 428045 10 eric 2011-06-27 10:16:45 -0700 If I were able to reproduce this this would be easy to fix. But I failed to make a reduced python script on our EC2 bots. I could probably just edit a webkit file and wait for a whole build to fail, but I was too lazy to try that.