128971 2014-02-18 07:04:36 -0800 [GTK] webkit 2.3.5 build failure with python 3.3.4 2016-01-02 01:40:46 -0800 1 1 1 Unclassified WebKit WebCore Misc. 528+ (Nightly build) PC Other RESOLVED FIXED P3 Normal --- 1 prlw1 webkit-unassigned berto cgarcia darin mcatanzaro vincent.riera zan oldest_to_newest 981912 0 prlw1 2014-02-18 07:04:36 -0800 Trying to build webkit 2.3.5 with python 3.3.4, I see GEN DerivedSources/WebCore/XMLViewerJS.h Traceback (most recent call last): File "./Source/JavaScriptCore/inspector/scripts/cssmin.py", line 44, in <module> sys.stdout.write(cssminify(sys.stdin.read())) File "/usr/pkg/lib/python3.3/encodings/ascii.py", line 26, in decode return codecs.ascii_decode(input, self.errors)[0] UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 642: ordinal not in range(128) GNUmakefile:81815: recipe for target 'DerivedSources/WebCore/XMLViewerCSS.h' failed (Also, GNU bison 3.0.2 isn't happy with DerivedSources/WebCore/CSSGrammar.y:62.1-12: warning: deprecated directive, use '%pure-parser' [-Wdeprecated] %pure_parser ^^^^^^^^^^^^ ) 982570 1 prlw1 2014-02-19 15:07:57 -0800 Changing component to WebCore as that is were the build fault happens. It seems as though webkitgtk 2.3.5 contains m4_define([webkit_user_agent_major_version], [538]) m4_define([webkit_user_agent_minor_version], [15]) version 538 of webkit? 985096 2 berto 2014-02-27 04:02:08 -0800 Probably a duplicate of bug 128927, can you try with the patch attached there? 985141 3 prlw1 2014-02-27 06:28:15 -0800 It might be similar, but it isn't the same: you had the bug building documentation, so the python program involved was gtkdoc.py, here I am building WebCore, and the python program involved is cssmin.py. sys.stdout.write(cssminify(sys.stdin.read())) has to be imbued with UTF-8 somehow? 985142 4 berto 2014-02-27 06:33:30 -0800 (In reply to comment #3) > sys.stdout.write(cssminify(sys.stdin.read())) > > has to be imbued with UTF-8 somehow? I think so, see this: http://bugs.python.org/issue4947 985367 5 prlw1 2014-02-27 15:05:57 -0800 Interesting! That bug seems to revolve around Python 2.7. I am surprised by: That bug: >>> type(sys.stdout) <type 'file'> >>> sys.stdout.encoding 'UTF-8' My Python 3.3.4: >>> type(sys.stdout) <class '_io.TextIOWrapper'> >>> sys.stdout.encoding '646' Presumably 646 is the ISO successor to ASCII which would explain the message... It seems that Python 2.7 had a problem even when UTF-8 was set. 985374 6 prlw1 2014-02-27 15:16:29 -0800 P.S. on my Python 3.3.4 installation >>> sys.getdefaultencoding() 'utf-8' 985399 7 prlw1 2014-02-27 16:06:09 -0800 Actually, it makes far more sense not to change the python, but to change the file that it processes. It seems that the only unicode characters in XMLViewer.css which cssmin.py processes are there because someone thought it would look pretty to change " to unicode double quotes in the licence. I think I can live with the ugly but practical ascii ". 985413 8 225428 prlw1 2014-02-27 16:37:03 -0800 Created attachment 225428 build fix Pretty quotes in licence break Python stdin. 986766 9 berto 2014-03-04 04:29:32 -0800 (In reply to comment #8) > Created an attachment (id=225428) [details] > build fix > > Pretty quotes in licence break Python stdin. Although that fixes the build issue, I think I'd rather fix the python code so that it will not happen again in the future. 987010 10 prlw1 2014-03-04 15:21:46 -0800 Why not do both? Apply this trivial patch now, so that webkitgtk 2.4 won't have the problem. Then ponder the better solution. 1124611 11 vincent.riera 2015-09-09 06:30:24 -0700 Ping. Is there any strong reason to not change those unicode double quotes by the ascii ones in the upstream code? 1125926 12 berto 2015-09-15 03:49:01 -0700 (In reply to comment #11) > Is there any strong reason to not change those unicode double quotes > by the ascii ones in the upstream code? I'm not a big fan of sweeping bugs under the rug rather than fixing them for good, but this has been certainly open for too long and the change is really not important. So I'm not against changing the quotes. The patch would need a ChangeLog entry (you can use 'Tools/Scripts/prepare-ChangeLog'). Still I would like to know what's the problem exactly. How do you reproduce this problem and how come it doesn't break for everyone? 1125939 13 mcatanzaro 2015-09-15 05:44:33 -0700 CC Darin: question for you at the bottom of this comment. It's because cssmin.py uses the following shebang: #!/usr/bin/python For about four years now, this has declared that the script will run correctly under both python2 and python3 [1], but our script is only compatible with python2. (Yes, this was a compatibility break requiring changing all the shebangs of all python files everywhere. Note that /usr/bin/python is a symlink to /usr/bin/python3 on Arch, but to /usr/bin/python2 on other distros, so the bug presumably only occurs for Arch users. I have no comment on whether any of this was a good idea. ;) The script must be modified to work with python3 in addition to python2 (the worst way to do this would be to ensure that no files it process include Unicode characters; the best way would be for someone who groks python to just fix the broken input stream), or the shebang must be changed to use /usr/bin/python2 instead of /usr/bin/python. I'm quite impressed that this is apparently the only such compatibility issue in our build. Warning: OS X used to not provide /usr/bin/python2. Presumably they do nowadays, since otherwise it will be a pain to fix the shebang, but maybe they don't. (I've CCed Darin to find out if we still need to support this case.) Without a /usr/bin/python2, you'd have to find the right python2 binary at the CMake step, then generate the python script using a CMake template file to substitute the right python binary into the shebang. In Autotools land you would do that using AM_PATH_PYTHON. Probably easier just to fix the script to work with python3. [1] https://www.python.org/dev/peps/pep-0394/ 1125940 14 mcatanzaro 2015-09-15 05:45:05 -0700 Forgot to CC Darin: question for you in the comment above. 1125942 15 vincent.riera 2015-09-15 05:50:59 -0700 (In reply to comment #13) > the best way would be for someone who groks python to > just fix the broken input stream), or the shebang must be changed to use > /usr/bin/python2 instead of /usr/bin/python. I'd suggest #!/usr/bin/env python2 1125959 16 mcatanzaro 2015-09-15 08:03:26 -0700 (In reply to comment #15) > (In reply to comment #13) > > the best way would be for someone who groks python to > > just fix the broken input stream), or the shebang must be changed to use > > /usr/bin/python2 instead of /usr/bin/python. > > I'd suggest #!/usr/bin/env python2 Note that various distributions (e.g. Debian) prohibit using env for installed scripts, as the script could then be run in an unexpected interpreter. For scripts only needed for the build, it's probably fine. But again, we can only do that if all supported versions of OS X provide a python2 in $PATH. 1125962 17 berto 2015-09-15 08:17:53 -0700 (In reply to comment #13) > It's because cssmin.py uses the following shebang: > > #!/usr/bin/python > > For about four years now, this has declared that the script will run > correctly under both python2 and python3 [1], but our script is only > compatible with python2. Is it really like that? These work fine in my computer: $ python2 --version Python 2.7.10 $ python2 ./Source/WebInspectorUI/Scripts/cssmin.py < ./Source/WebCore/xml/XMLViewer.css > XMLViewer.min.css $ python3 --version Python 3.4.3+ $ python3 ./Source/WebInspectorUI/Scripts/cssmin.py < ./Source/WebCore/xml/XMLViewer.css > XMLViewer.min.css 1125987 18 mcatanzaro 2015-09-15 09:36:02 -0700 Good find Berto. I tested Python 2.7.10 and 3.4.2. sys.stdout.encoding returns UTF-8 in both versions. But in Patrick's report above, it returns 646 in Python 3.3.4. I am presuming the version number is responsible for this difference, and not some other difference in configuration, but I can't find any documentation about this change, not even in the Python 3.4 release notes, so I'm not sure. Anyway, the script *appears* to be incompatible with earlier versions of Python 3. I would therefore suggest RESOLVED -> OBSOLETE for this bug. If you're using older Python 3 you can continue to replace the quotes like you're already doing. 1126085 19 261222 berto 2015-09-15 12:06:56 -0700 Created attachment 261222 Patch (In reply to comment #18) > Good find Berto. I tested Python 2.7.10 and > 3.4.2. sys.stdout.encoding returns UTF-8 in both versions. But in > Patrick's report above, it returns 646 in Python 3.3.4. I am > presuming the version number is responsible for this difference Nah, it's not a problem with the Python version, it's the locale of the build environment: $ LC_ALL=C python2 ./Source/WebInspectorUI/Scripts/cssmin.py < ./Source/WebCore/xml/XMLViewer.css > /dev/null $ LC_ALL=C python3 ./Source/WebInspectorUI/Scripts/cssmin.py < ./Source/WebCore/xml/XMLViewer.css > /dev/null Traceback (most recent call last): File "./Source/WebInspectorUI/Scripts/cssmin.py", line 46, in <module> sys.stdout.write(cssminify(sys.stdin.read())) File "/usr/lib/python3.4/encodings/ascii.py", line 26, in decode return codecs.ascii_decode(input, self.errors)[0] UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 642: ordinal not in range(128) So if the locale is set to ascii, then python 3 expects the standard input to be ascii, and otherwise it complains. We can tell people to use UTF-8 for building WebKit, but we can also force the stdin enconding using this patch. 1151949 20 berto 2016-01-02 01:40:46 -0800 Committed r194498: <http://trac.webkit.org/changeset/194498> 225428 2014-02-27 16:37:03 -0800 2015-09-15 12:06:56 -0700 build fix asis.diff text/plain 1256 prlw1 UHJldHR5IHF1b3RlcyBpbiBsaWNlbmNlIGJyZWFrIFB5dGhvbiBzdGRpbi4KCmh0dHBzOi8vYnVn cy53ZWJraXQub3JnL3Nob3dfYnVnLmNnaT9pZD0xMjg5NzEKCi0tLSBTb3VyY2UvV2ViQ29yZS94 bWwvWE1MVmlld2VyLmNzcy5vcmlnCTIwMTQtMDEtMjMgMTY6NDk6NTguMDAwMDAwMDAwICswMDAw CisrKyBTb3VyY2UvV2ViQ29yZS94bWwvWE1MVmlld2VyLmNzcwpAQCAtMTQsNyArMTQsNyBAQAog ICogZGlzdHJpYnV0aW9uLgogICoKICAqIFRISVMgU09GVFdBUkUgSVMgUFJPVklERUQgQlkgR09P R0xFIElOQy4gQU5EIElUUyBDT05UUklCVVRPUlMKLSAqIOKAnEFTIElT4oCdIEFORCBBTlkgRVhQ UkVTUyBPUiBJTVBMSUVEIFdBUlJBTlRJRVMsIElOQ0xVRElORywgQlVUIE5PVAorICogIkFTIElT IiBBTkQgQU5ZIEVYUFJFU1MgT1IgSU1QTElFRCBXQVJSQU5USUVTLCBJTkNMVURJTkcsIEJVVCBO T1QKICAqIExJTUlURUQgVE8sIFRIRSBJTVBMSUVEIFdBUlJBTlRJRVMgT0YgTUVSQ0hBTlRBQklM SVRZIEFORCBGSVRORVNTIEZPUgogICogQSBQQVJUSUNVTEFSIFBVUlBPU0UgQVJFIERJU0NMQUlN RUQuIElOIE5PIEVWRU5UIFNIQUxMIEdPT0dMRSBJTkMuCiAgKiBPUiBJVFMgQ09OVFJJQlVUT1JT IEJFIExJQUJMRSBGT1IgQU5ZIERJUkVDVCwgSU5ESVJFQ1QsIElOQ0lERU5UQUwsCi0tLSBTb3Vy Y2UvV2ViQ29yZS94bWwvWE1MVmlld2VyLmpzLm9yaWcJMjAxNC0wMS0yMyAxNjo0OTo1OC4wMDAw MDAwMDAgKzAwMDAKKysrIFNvdXJjZS9XZWJDb3JlL3htbC9YTUxWaWV3ZXIuanMKQEAgLTE1LDcg KzE1LDcgQEAKICAqIGRpc3RyaWJ1dGlvbi4KICAqCiAgKiBUSElTIFNPRlRXQVJFIElTIFBST1ZJ REVEIEJZIEdPT0dMRSBJTkMuIEFORCBJVFMgQ09OVFJJQlVUT1JTCi0gKiDigJxBUyBJU+KAnSBB TkQgQU5ZIEVYUFJFU1MgT1IgSU1QTElFRCBXQVJSQU5USUVTLCBJTkNMVURJTkcsIEJVVCBOT1QK KyAqICJBUyBJUyIgQU5EIEFOWSBFWFBSRVNTIE9SIElNUExJRUQgV0FSUkFOVElFUywgSU5DTFVE SU5HLCBCVVQgTk9UCiAgKiBMSU1JVEVEIFRPLCBUSEUgSU1QTElFRCBXQVJSQU5USUVTIE9GIE1F UkNIQU5UQUJJTElUWSBBTkQgRklUTkVTUyBGT1IKICAqIEEgUEFSVElDVUxBUiBQVVJQT1NFIEFS RSBESVNDTEFJTUVELiBJTiBOTyBFVkVOVCBTSEFMTCBHT09HTEUgSU5DLgogICogT1IgSVRTIENP TlRSSUJVVE9SUyBCRSBMSUFCTEUgRk9SIEFOWSBESVJFQ1QsIElORElSRUNULCBJTkNJREVOVEFM LAo= 261222 2015-09-15 12:06:56 -0700 2015-12-31 14:22:29 -0800 Patch 128971.diff text/plain 1200 berto ZGlmZiAtLWdpdCBhL1NvdXJjZS9XZWJJbnNwZWN0b3JVSS9DaGFuZ2VMb2cgYi9Tb3VyY2UvV2Vi SW5zcGVjdG9yVUkvQ2hhbmdlTG9nCmluZGV4IGU3YzZlZDMuLmY0YmE4MjcgMTAwNjQ0Ci0tLSBh L1NvdXJjZS9XZWJJbnNwZWN0b3JVSS9DaGFuZ2VMb2cKKysrIGIvU291cmNlL1dlYkluc3BlY3Rv clVJL0NoYW5nZUxvZwpAQCAtMSwzICsxLDE1IEBACisyMDE1LTA5LTE1ICBBbGJlcnRvIEdhcmNp YSAgPGJlcnRvQGlnYWxpYS5jb20+CisKKyAgICAgICAgW0dUS10gd2Via2l0IDIuMy41IGJ1aWxk IGZhaWx1cmUgd2l0aCBweXRob24gMy4zLjQKKyAgICAgICAgaHR0cHM6Ly9idWdzLndlYmtpdC5v cmcvc2hvd19idWcuY2dpP2lkPTEyODk3MQorCisgICAgICAgIFJldmlld2VkIGJ5IE5PQk9EWSAo T09QUyEpLgorCisgICAgICAgIE1ha2Ugc3VyZSB0aGF0IHRoZSBzdGRpbiBlbmNvZGluZyBpcyAn VVRGLTgnIGlmIHVzaW5nIFB5dGhvbiAzLAorICAgICAgICBvdGhlcndpc2UgdGhlIGJ1aWxkIHdp bGwgYnJlYWsuCisKKyAgICAgICAgKiBTY3JpcHRzL2Nzc21pbi5weToKKwogMjAxNS0wOS0xNSAg Sm9zZXBoIFBlY29yYXJvICA8cGVjb3Jhcm9AYXBwbGUuY29tPgogCiAgICAgICAgIFdlYiBJbnNw ZWN0b3I6IFdhdGNoIEV4cHJlc3Npb25zIHVuY2F1Z2h0IGV4Y2VwdGlvbnMgb24gcGFnZSByZWxv YWQKZGlmZiAtLWdpdCBhL1NvdXJjZS9XZWJJbnNwZWN0b3JVSS9TY3JpcHRzL2Nzc21pbi5weSBi L1NvdXJjZS9XZWJJbnNwZWN0b3JVSS9TY3JpcHRzL2Nzc21pbi5weQppbmRleCA1NDhjMjAwLi5j ZTBkNWEyIDEwMDc1NQotLS0gYS9Tb3VyY2UvV2ViSW5zcGVjdG9yVUkvU2NyaXB0cy9jc3NtaW4u cHkKKysrIGIvU291cmNlL1dlYkluc3BlY3RvclVJL1NjcmlwdHMvY3NzbWluLnB5CkBAIC00Myw0 ICs0Myw3IEBAIGRlZiBjc3NtaW5pZnkoY3NzKToKIAogaWYgX19uYW1lX18gPT0gIl9fbWFpbl9f IjoKICAgICBpbXBvcnQgc3lzCisgICAgaWYgc3lzLnZlcnNpb25faW5mb1swXSA9PSAzIGFuZCBz eXMuc3RkaW4uZW5jb2RpbmcgIT0gJ1VURi04JzoKKyAgICAgICAgaW1wb3J0IGlvCisgICAgICAg IHN5cy5zdGRpbiA9IGlvLlRleHRJT1dyYXBwZXIoc3lzLnN0ZGluLmJ1ZmZlciwgZW5jb2Rpbmc9 J1VURi04JykKICAgICBzeXMuc3Rkb3V0LndyaXRlKGNzc21pbmlmeShzeXMuc3RkaW4ucmVhZCgp KSkK