Bug 176761 – The EWS build system is forcing a non UTF-8 locale and breaks meson.

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Carlos Alberto Lopez Perez]

The WPE EWS has been failing lately a lot when trying to build the JHBuild, always with this error:

Traceback (most recent call last):
  File "/home/ews/igalia-wpe-ews/WebKit/WebKitBuild/DependenciesWPE/Root/lib/python3.5/site-packages/mesonbuild/mesonmain.py", line 338, in run
    app.generate()
  File "/home/ews/igalia-wpe-ews/WebKit/WebKitBuild/DependenciesWPE/Root/lib/python3.5/site-packages/mesonbuild/mesonmain.py", line 181, in generate
    intr.run()
  File "/home/ews/igalia-wpe-ews/WebKit/WebKitBuild/DependenciesWPE/Root/lib/python3.5/site-packages/mesonbuild/interpreter.py", line 2728, in run
    super().run()
  File "/home/ews/igalia-wpe-ews/WebKit/WebKitBuild/DependenciesWPE/Root/lib/python3.5/site-packages/mesonbuild/interpreterbase.py", line 148, in run
    self.evaluate_codeblock(self.ast, start=1)
  File "/home/ews/igalia-wpe-ews/WebKit/WebKitBuild/DependenciesWPE/Root/lib/python3.5/site-packages/mesonbuild/interpreterbase.py", line 170, in evaluate_codeblock
    raise e
  File "/home/ews/igalia-wpe-ews/WebKit/WebKitBuild/DependenciesWPE/Root/lib/python3.5/site-packages/mesonbuild/interpreterbase.py", line 164, in evaluate_codeblock
    self.evaluate_statement(cur)
  File "/home/ews/igalia-wpe-ews/WebKit/WebKitBuild/DependenciesWPE/Root/lib/python3.5/site-packages/mesonbuild/interpreterbase.py", line 175, in evaluate_statement
    return self.function_call(cur)
  File "/home/ews/igalia-wpe-ews/WebKit/WebKitBuild/DependenciesWPE/Root/lib/python3.5/site-packages/mesonbuild/interpreterbase.py", line 385, in function_call
    return self.funcs[func_name](node, self.flatten(posargs), kwargs)
  File "/home/ews/igalia-wpe-ews/WebKit/WebKitBuild/DependenciesWPE/Root/lib/python3.5/site-packages/mesonbuild/interpreterbase.py", line 47, in wrapped
    return f(self, node, args, kwargs)
  File "/home/ews/igalia-wpe-ews/WebKit/WebKitBuild/DependenciesWPE/Root/lib/python3.5/site-packages/mesonbuild/interpreter.py", line 2401, in func_subdir
    self.evaluate_codeblock(codeblock)
  File "/home/ews/igalia-wpe-ews/WebKit/WebKitBuild/DependenciesWPE/Root/lib/python3.5/site-packages/mesonbuild/interpreterbase.py", line 170, in evaluate_codeblock
    raise e
  File "/home/ews/igalia-wpe-ews/WebKit/WebKitBuild/DependenciesWPE/Root/lib/python3.5/site-packages/mesonbuild/interpreterbase.py", line 164, in evaluate_codeblock
    self.evaluate_statement(cur)
  File "/home/ews/igalia-wpe-ews/WebKit/WebKitBuild/DependenciesWPE/Root/lib/python3.5/site-packages/mesonbuild/interpreterbase.py", line 175, in evaluate_statement
    return self.function_call(cur)
  File "/home/ews/igalia-wpe-ews/WebKit/WebKitBuild/DependenciesWPE/Root/lib/python3.5/site-packages/mesonbuild/interpreterbase.py", line 385, in function_call
    return self.funcs[func_name](node, self.flatten(posargs), kwargs)
  File "/home/ews/igalia-wpe-ews/WebKit/WebKitBuild/DependenciesWPE/Root/lib/python3.5/site-packages/mesonbuild/interpreterbase.py", line 77, in wrapped
    return f(s, node_or_state, args, kwargs)
  File "/home/ews/igalia-wpe-ews/WebKit/WebKitBuild/DependenciesWPE/Root/lib/python3.5/site-packages/mesonbuild/interpreter.py", line 2546, in func_configure_file
    conf.held_object)
  File "/home/ews/igalia-wpe-ews/WebKit/WebKitBuild/DependenciesWPE/Root/lib/python3.5/site-packages/mesonbuild/mesonlib.py", line 440, in do_conf_file
    replace_if_different(dst, dst_tmp)
  File "/home/ews/igalia-wpe-ews/WebKit/WebKitBuild/DependenciesWPE/Root/lib/python3.5/site-packages/mesonbuild/mesonlib.py", line 473, in replace_if_different
    if f1.read() == f2.read():
  File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 52: ordinal not in range(128)


Upon further investigation, it seems the Meson build system throws this error when the system locale is not an UTF8 one.

Check https://github.com/mesonbuild/meson/issues/2268

$ env -i python3 -c "import locale; print(locale.getpreferredencoding())"
ANSI_X3.4-1968

$ env -i LC_ALL=C.UTF-8 python3 -c "import locale; print(locale.getpreferredencoding())"
UTF-8

$ env -i LANG=C.UTF-8 python3 -c "import locale; print(locale.getpreferredencoding())"
ANSI_X3.4-1968

$ env -i LC_ALL=C LANG=en_US.UTF8 python3 -c "import locale; print(locale.getpreferredencoding())"
ANSI_X3.4-1968


So, passing "C.UTF-8" is fine, but not passing just "C".

And I have been forcing this on the WPE EWS bot one way or the another until I realized its the WebKit build system itself the one overriding this and setting LC_ALL=C in order to disable GCC "smartquotes"! https://trac.webkit.org/browser/webkit/trunk/Tools/Scripts/webkitpy/common/system/environment.py?rev=221898#L37

[Michael Catanzaro]

So, we definitely need to use a UTF-8 locale always. That should go without saying. Our platform is not non-UTF-8 safe anymore and I expect there is no shortage of odd bugs caused by using a weird locale.

Anyway, apparently commit-queue can't handle UTF-8. Or couldn't in the past, see bug #71983. So we'll need to test to see if commit queue breaks when we change this.

P.S. Passing C.UTF-8 is a bad idea, because I'm pretty sure that locale is a downstream Debian/Fedora patch. We should be nice to distros that don't have C.UTF-8 by not using it. en_US.UTF-8 is the standard locale nowadays.

[Carlos Alberto Lopez Perez]

(In reply to Michael Catanzaro from comment #1)
> So, we definitely need to use a UTF-8 locale always. That should go without
> saying. Our platform is not non-UTF-8 safe anymore and I expect there is no
> shortage of odd bugs caused by using a weird locale.
> 
> Anyway, apparently commit-queue can't handle UTF-8. Or couldn't in the past,
> see bug #71983. So we'll need to test to see if commit queue breaks when we
> change this.
> 


I think this is not longer an issue.

1) I have checked and all the GTK/WPE bots are right now producing text with UTF8 strings. This is because disable_gcc_smartquotes() is not called on the normal build done by the buildbots, only for the EWS bots.

Example: https://build.webkit.org/builders/GTK%20Linux%20ARM%20Release/builds/1051/steps/compile-webkit/logs/stdio/text


2) Both the build.webkit.org as the EWS log server are now setting proper mime-type:

* EWS:
$ curl -i https://webkit-queues.webkit.org/results/4489980 2>&1 |grep ^Content-Type:
Content-Type: text/plain; charset=utf-8

* build.webkit.org
$ curl -i https://build.webkit.org/builders/GTK%20Linux%20ARM%20Release/builds/1051/steps/compile-webkit/logs/stdio/text 2>&1 |grep ^Content-
Content-Type: text/plain; charset=utf-8


And any browser that is not able to understand text/plain with charset=utf-8 (or confuses that with binary data) in 2017 has bigger problems that go beyond this issue.

So the issue raised in https://lists.webkit.org/pipermail/webkit-dev/2014-January/026140.html (not sure if was the same) should not be an issue anymore. 


And regarding commit-queue itself, I expect it to be handle UTF-8 nowadays. But even in that remote case this should not be an issue since the ports that commit-queue tests are the Mac/iOS ones (not the GTK/WPE ones). And the Apple port has long ago switched to Clang. So the GCC smartquotes thing should not be an issue anymore




> P.S. Passing C.UTF-8 is a bad idea, because I'm pretty sure that locale is a
> downstream Debian/Fedora patch. We should be nice to distros that don't have
> C.UTF-8 by not using it. en_US.UTF-8 is the standard locale nowadays.

Also passing C.UTF-8 will not disable GCC smartquotes (because GCC will enable that on any UTF8 locale). So we may as well just not pass anything.

I think the best thing we can do is to completely remove this environment.disable_gcc_smartquotes() thing. Forcing the locale its a gunfoot.

If the commit-queue server has problems with its own locale it should fix that locally (in the script that starts it locally) rather than doing that on WebKit IMHO.

[Carlos Alberto Lopez Perez]

(In reply to Carlos Alberto Lopez Perez from comment #2)
> (In reply to Michael Catanzaro from comment #1)
> > So, we definitely need to use a UTF-8 locale always. That should go without
> > saying. Our platform is not non-UTF-8 safe anymore and I expect there is no
> > shortage of odd bugs caused by using a weird locale.
> > 
> > Anyway, apparently commit-queue can't handle UTF-8. Or couldn't in the past,
> > see bug #71983. So we'll need to test to see if commit queue breaks when we
> > change this.
> > 
> 
> 
> I think this is not longer an issue.
> 
> 1) I have checked and all the GTK/WPE bots are right now producing text with
> UTF8 strings. This is because disable_gcc_smartquotes() is not called on the
> normal build done by the buildbots, only for the EWS bots.
> 
> Example:
> https://build.webkit.org/builders/GTK%20Linux%20ARM%20Release/builds/1051/
> steps/compile-webkit/logs/stdio/text
> 
> 
> 2) Both the build.webkit.org as the EWS log server are now setting proper
> mime-type:
> 
> * EWS:
> $ curl -i https://webkit-queues.webkit.org/results/4489980 2>&1 |grep
> ^Content-Type:
> Content-Type: text/plain; charset=utf-8
> 
> * build.webkit.org
> $ curl -i
> https://build.webkit.org/builders/GTK%20Linux%20ARM%20Release/builds/1051/
> steps/compile-webkit/logs/stdio/text 2>&1 |grep ^Content-
> Content-Type: text/plain; charset=utf-8
> 
> 
> And any browser that is not able to understand text/plain with charset=utf-8
> (or confuses that with binary data) in 2017 has bigger problems that go
> beyond this issue.
> 
> So the issue raised in
> https://lists.webkit.org/pipermail/webkit-dev/2014-January/026140.html (not
> sure if was the same) should not be an issue anymore. 
> 

I think the issue raised on the mailing list was related to the VT100 characters that JHBuild generates that are not UTF8. For that we are setting TERM=none with environment.disable_jhbuild_VT100_output(). We can keep that as this one makes sense. I also set on the bots the TERM variable to disable the terminal control codes from JHBuild


So, taken everything into account I think is safe now to remove the environment.disable_gcc_smartquotes() thing

[Carlos Alberto Lopez Perez]

Created attachment 320545 [details]
Patch

[Michael Catanzaro]

Let's wait for EWS, since this could maybe break Apple ports.

[Carlos Alberto Lopez Perez]

(In reply to Michael Catanzaro from comment #5)
> Let's wait for EWS, since this could maybe break Apple ports.

EWS are green (except WPE which has a long queue because its broken due to this).
In any case I think restarting the EWS process is needed to the changes to take effect.

[Michael Catanzaro]

Comment on attachment 320545 [details]
Patch

I'm surprised EWS passed. :P

Still think it's liable to break something unexpected, but we need to try it, and now seems as good a time as any.

[WebKit Commit Bot]

Comment on attachment 320545 [details]
Patch

Clearing flags on attachment: 320545

Committed r221926: <http://trac.webkit.org/changeset/221926>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Radar WebKit Bug Importer]

<rdar://problem/34693416>