212459 2020-05-28 03:23:27 -0700 [Flatpak SDK] Missing locale warnings 2020-06-23 08:48:26 -0700 1 1 1 Unclassified WebKit WebKitGTK WebKit Nightly Build Unspecified Unspecified RESOLVED FIXED P2 Normal --- 1 aboya aboya bugs-noreply pnormand oldest_to_newest 1656803 0 aboya 2020-05-28 03:23:27 -0700 Running any perl script under flatpak spams lots of locale warnings. We should have a locale set inside flatpak. Probably en_US.UTF-8 for consistency. $ build-webkit --gtk --debug Building flatpak based environment perl: warning: Setting locale failed. perl: warning: Please check that your locale settings: LANGUAGE = (unset), LC_ALL = (unset), LC_MONETARY = "es_ES.UTF-8", LC_PAPER = "es_ES.UTF-8", LC_MEASUREMENT = "es_ES.UTF-8", LC_TIME = "es_ES.UTF-8", LC_NUMERIC = "es_ES.UTF-8", LANG = "en_US.UTF-8" are supported and installed on your system. perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). + cmake --build /app/webkit/WebKitBuild/Debug --config Debug -- [1/377] Generate bindings (WebKitTestRunnerInjectedBundleBindings) perl: warning: Setting locale failed. perl: warning: Please check that your locale settings: LANGUAGE = (unset), LC_ALL = (unset), LC_MONETARY = "es_ES.UTF-8", LC_PAPER = "es_ES.UTF-8", LC_MEASUREMENT = "es_ES.UTF-8", LC_TIME = "es_ES.UTF-8", LC_NUMERIC = "es_ES.UTF-8", LANG = "en_US.UTF-8" are supported and installed on your system. perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). [...] ==================================================================== WebKit is now built (00m:02s). To run MiniBrowser with this newly-built code, use the "Tools/Scripts/run-minibrowser" script. ==================================================================== 1657224 1 pnormand 2020-05-29 05:26:23 -0700 LANG is set already, to "en_US.UTF-8". I don't see any Perl warning on the bots. Seems like you have "es_ES.UTF-8" configured in your shell profile, that seems to conflict with en_US... Can you try to set LANGUAGE from flatpakutils.py, look for en_EN and add the LANGUAGE variable in `sandbox_environment`. 1657225 2 pnormand 2020-05-29 05:28:25 -0700 or set LC_ALL ? There's too many locale env vars. :) 1661899 3 aboya 2020-06-12 02:56:52 -0700 The problem is not that the locale is not being set, the problem is the locale I'm using in my computer (es_ES.UTF-8) is not built inside the flatpak environment. Setting these settings to en_US.UTF-8 in the shell avoids the warning. I think it's reasonable that development versions of WebKit, especially the tests, run on the same locale for everybody, but we can probably agree telling people not to set their computer locale to something else would not be an appropriate solution. What could we do? Some helpful info: I found this doc explaining the different locale variables. https://help.ubuntu.com/community/Locale The priority goes as follows: LC_ALL > LC_[specific locale aspect] > LANG LANGUAGE is used by gettext, and in the case of translations using that library, it takes priority over the other variables including LC_ALL. https://www.gnu.org/software/gettext/manual/html_node/The-LANGUAGE-variable.html#The-LANGUAGE-variable 1661904 4 pnormand 2020-06-12 04:21:25 -0700 (In reply to Alicia Boya García from comment #3) > The problem is not that the locale is not being set, the problem is the > locale I'm using in my computer (es_ES.UTF-8) is not built inside the > flatpak environment. > > Setting these settings to en_US.UTF-8 in the shell avoids the warning. > > I think it's reasonable that development versions of WebKit, especially the > tests, run on the same locale for everybody, but we can probably agree > telling people not to set their computer locale to something else would not > be an appropriate solution. > For the record this isn't what I was suggesting either. We agree. > What could we do? > Currently LANG is forwarded from the host to the sandbox. Can you check if things work better if we don't do that? 1661940 5 pnormand 2020-06-12 07:38:56 -0700 I could reproduce the bug and came up with this, might need more work, but can you check/test? diff --git a/Tools/flatpak/webkit-bwrap b/Tools/flatpak/webkit-bwrap index b342a1da7e29..970b671cad6a 100755 --- a/Tools/flatpak/webkit-bwrap +++ b/Tools/flatpak/webkit-bwrap @@ -58,6 +58,10 @@ def main(args: list) -> int: for dst, src in try_bind_mounts.items(): bwrap_args.extend(["--bind-try", src, dst]) + for env in os.environ.keys(): + if env.startswith("LC_"): + bwrap_args.extend(["--unsetenv", env]) + command_line = ' '.join(shlex.quote(a) for a in bwrap_args + args) # os.system return code behaves like os.wait. A 16 bit number with the # signal in the lower byte and, if the signal is zero, the exit code in 1664834 6 aboya 2020-06-21 02:53:07 -0700 The proposed change almost works! It's just missing a LANG and LANGUAGE, that should also be resetted. Otherwise you would get the warnings if you have any other locale set. diff --git a/Tools/flatpak/webkit-bwrap b/Tools/flatpak/webkit-bwrap index b342a1da7e29..a286dd1377a3 100755 --- a/Tools/flatpak/webkit-bwrap +++ b/Tools/flatpak/webkit-bwrap @@ -58,6 +58,10 @@ def main(args: list) -> int: for dst, src in try_bind_mounts.items(): bwrap_args.extend(["--bind-try", src, dst]) + for env in os.environ.keys(): + if env.startswith("LC_") or env in {"LANG", "LANGUAGE"}: + bwrap_args.extend(["--unsetenv", env]) + command_line = ' '.join(shlex.quote(a) for a in bwrap_args + args) # os.system return code behaves like os.wait. A 16 bit number with the # signal in the lower byte and, if the signal is zero, the exit code in 1664857 7 pnormand 2020-06-21 08:46:01 -0700 (In reply to Alicia Boya García from comment #6) > The proposed change almost works! It's just missing a LANG and LANGUAGE, > that should also be resetted. Otherwise you would get the warnings if you > have any other locale set. > The en_US locale needs to be set for the layout tests and jsc tests though. Otherwise some intl-related tests will fail. 1664867 8 aboya 2020-06-21 12:00:42 -0700 (In reply to Philippe Normand from comment #7) > (In reply to Alicia Boya García from comment #6) > > The proposed change almost works! It's just missing a LANG and LANGUAGE, > > that should also be resetted. Otherwise you would get the warnings if you > > have any other locale set. > > > > The en_US locale needs to be set for the layout tests and jsc tests though. > Otherwise some intl-related tests will fail. Oh, right. It's not being set in all cases though and it should. If you remove LANG from the list and run `LANG=es_ES.UTF-8 webkit-flatpak --gtk --debug` and inside that, call `perl`, you'll see the locale warnings since the LANG from outside got inside. 1664868 9 pnormand 2020-06-21 12:26:20 -0700 Yeah I suspect this is a regression introduced when we moved from `flatpak build` to `flatpak run`. Would it be OK if our flatpak-bwrap script unconditionally sets $LANG? 1664870 10 aboya 2020-06-21 12:45:07 -0700 I hope so. What do you think about this? diff --git a/Tools/flatpak/webkit-bwrap b/Tools/flatpak/webkit-bwrap index b342a1da7e29..31d3894e04c2 100755 --- a/Tools/flatpak/webkit-bwrap +++ b/Tools/flatpak/webkit-bwrap @@ -58,6 +58,11 @@ def main(args: list) -> int: for dst, src in try_bind_mounts.items(): bwrap_args.extend(["--bind-try", src, dst]) + for env in os.environ.keys(): + if env.startswith("LC_") or env == "LANGUAGE": + bwrap_args.extend(["--unsetenv", env]) + bwrap_args.extend(["--setenv", "LANG", "en_US.UTF-8"]) + command_line = ' '.join(shlex.quote(a) for a in bwrap_args + args) # os.system return code behaves like os.wait. A 16 bit number with the # signal in the lower byte and, if the signal is zero, the exit code in 1664871 11 pnormand 2020-06-21 12:49:16 -0700 LGTM, if it doesn't break tests :) r=me also if you remove if from flatpakutils as it would now be un-needed there. 1665341 12 402547 aboya 2020-06-23 05:28:31 -0700 Created attachment 402547 Patch 1665349 13 402547 pnormand 2020-06-23 05:54:06 -0700 Comment on attachment 402547 Patch Thanks! Can you remove the LANG reference from flatpakutils too please? 1665390 14 402560 aboya 2020-06-23 08:24:58 -0700 Created attachment 402560 Patch 1665404 15 ews-feeder 2020-06-23 08:48:25 -0700 Committed r263397: <https://trac.webkit.org/changeset/263397> All reviewed patches have been landed. Closing bug and clearing flags on attachment 402560. 402547 2020-06-23 05:28:31 -0700 2020-06-23 08:24:37 -0700 Patch bug-212459-20200623142825.patch text/plain 1949 aboya U3VidmVyc2lvbiBSZXZpc2lvbjogMjYzMjU4CmRpZmYgLS1naXQgYS9Ub29scy9DaGFuZ2VMb2cg Yi9Ub29scy9DaGFuZ2VMb2cKaW5kZXggNzAyZjU3OWMzYjE4ZjUzZDgyZTE0OTkxODAyN2Q2ZmI5 MGU1NDMxYi4uZmM2MDY3ZGZlMTY4M2ZhNDBjNjFmNmE4NjMwODhiMDk1NzBhYTg2MCAxMDA2NDQK LS0tIGEvVG9vbHMvQ2hhbmdlTG9nCisrKyBiL1Rvb2xzL0NoYW5nZUxvZwpAQCAtMSwzICsxLDIz IEBACisyMDIwLTA2LTIzICBBbGljaWEgQm95YSBHYXJjw61hICA8YWJveWFAaWdhbGlhLmNvbT4K KworICAgICAgICBbZmxhdHBha10gRW5zdXJlIGVuX1VTLlVURi04IGxvY2FsZQorICAgICAgICBo dHRwczovL2J1Z3Mud2Via2l0Lm9yZy9zaG93X2J1Zy5jZ2k/aWQ9MjEyNDU5CisKKyAgICAgICAg UmV2aWV3ZWQgYnkgTk9CT0RZIChPT1BTISkuCisKKyAgICAgICAgV2hlbiBkZXZlbG9waW5nIFdl YktpdCB3ZSB3YW50IGFzIGZldyBzeXN0ZW0gZGlmZmVyZW5jZXMgYXMgcG9zc2libGUKKyAgICAg ICAgdG8gbWFrZSB0ZXN0cyByZXByb2R1Y2libGUuCisKKyAgICAgICAgZW5fVVMuVVRGLTggaXMg aW5kZWVkIHRoZSBvbmx5IGxvY2FsZSBpbnN0YWxsZWQgaW4gb3VyIGRldmVsb3BtZW50CisgICAg ICAgIGZsYXRwYWssIGJ1dCBvdXIgc2NyaXB0cyB3ZXJlIG5vdCBjb3JyZWN0bHkgY2xlYXJpbmcg bW9zdCBsb2NhbGUKKyAgICAgICAgZW52aXJvbm1lbnQgdmFyaWFibGVzLCB3aGljaCByZXN1bHRl ZCBpbiB3YXJuaW5ncyB3aGVuIHJ1bm5pbmcgc29tZQorICAgICAgICB0b29scyAobm90YWJseSBw ZXJsKSB3aGVuIHRoZSB1c2VyIE9TIGhhcyBhIGRpZmZlcmVudCBsb2NhbGUuCisKKyAgICAgICAg VGhpcyBwYXRjaCBlbnN1cmVzIGFsbCBsb2NhbGUgZW52aXJvbm1lbnQgdmFyaWFibGVzIGFyZSBj bGVhcmVkIGFuZAorICAgICAgICBMQU5HIGlzIHNldCB0byBlbl9VUy5VVEYtOC4KKworICAgICAg ICAqIGZsYXRwYWsvd2Via2l0LWJ3cmFwOgorCiAyMDIwLTA2LTE5ICBDYXJsb3MgR2FyY2lhIENh bXBvcyAgPGNnYXJjaWFAaWdhbGlhLmNvbT4KIAogICAgICAgICBBZGQgc3VwcG9ydCBmb3IgZmV0 Y2hpbmcgcmVnaXN0cmFibGUgZG9tYWlucyB3aXRoIHJlc291cmNlIGxvYWQgc3RhdGlzdGljcwpk aWZmIC0tZ2l0IGEvVG9vbHMvZmxhdHBhay93ZWJraXQtYndyYXAgYi9Ub29scy9mbGF0cGFrL3dl YmtpdC1id3JhcAppbmRleCBiMzQyYTFkYTdlMjk5MWU0ODY0ZTUyZTczYzExZDc2MDliMDgwOGNi Li4zMWQzODk0ZTA0YzJhNDQwYTUxZWU1ZThhZThhZDU1NjY3MjM0Y2ViIDEwMDc1NQotLS0gYS9U b29scy9mbGF0cGFrL3dlYmtpdC1id3JhcAorKysgYi9Ub29scy9mbGF0cGFrL3dlYmtpdC1id3Jh cApAQCAtNTgsNiArNTgsMTEgQEAgZGVmIG1haW4oYXJnczogbGlzdCkgLT4gaW50OgogICAgIGZv ciBkc3QsIHNyYyBpbiB0cnlfYmluZF9tb3VudHMuaXRlbXMoKToKICAgICAgICAgYndyYXBfYXJn cy5leHRlbmQoWyItLWJpbmQtdHJ5Iiwgc3JjLCBkc3RdKQogCisgICAgZm9yIGVudiBpbiBvcy5l bnZpcm9uLmtleXMoKToKKyAgICAgICAgaWYgZW52LnN0YXJ0c3dpdGgoIkxDXyIpIG9yIGVudiA9 PSAiTEFOR1VBR0UiOgorICAgICAgICAgICAgYndyYXBfYXJncy5leHRlbmQoWyItLXVuc2V0ZW52 IiwgZW52XSkKKyAgICBid3JhcF9hcmdzLmV4dGVuZChbIi0tc2V0ZW52IiwgIkxBTkciLCAiZW5f VVMuVVRGLTgiXSkKKwogICAgIGNvbW1hbmRfbGluZSA9ICcgJy5qb2luKHNobGV4LnF1b3RlKGEp IGZvciBhIGluIGJ3cmFwX2FyZ3MgKyBhcmdzKQogICAgICMgb3Muc3lzdGVtIHJldHVybiBjb2Rl IGJlaGF2ZXMgbGlrZSBvcy53YWl0LiBBIDE2IGJpdCBudW1iZXIgd2l0aCB0aGUKICAgICAjIHNp Z25hbCBpbiB0aGUgbG93ZXIgYnl0ZSBhbmQsIGlmIHRoZSBzaWduYWwgaXMgemVybywgdGhlIGV4 aXQgY29kZSBpbgo= 402560 2020-06-23 08:24:58 -0700 2020-06-23 08:48:25 -0700 Patch bug-212459-20200623172451.patch text/plain 2782 aboya U3VidmVyc2lvbiBSZXZpc2lvbjogMjYzMjU4CmRpZmYgLS1naXQgYS9Ub29scy9DaGFuZ2VMb2cg Yi9Ub29scy9DaGFuZ2VMb2cKaW5kZXggNzAyZjU3OWMzYjE4ZjUzZDgyZTE0OTkxODAyN2Q2ZmI5 MGU1NDMxYi4uYTAwZTU0MzJjNzVkNjM4ZDE5YzY3Njg2ZmUwOThmODEwNGE3Y2U5MSAxMDA2NDQK LS0tIGEvVG9vbHMvQ2hhbmdlTG9nCisrKyBiL1Rvb2xzL0NoYW5nZUxvZwpAQCAtMSwzICsxLDI2 IEBACisyMDIwLTA2LTIzICBBbGljaWEgQm95YSBHYXJjw61hICA8YWJveWFAaWdhbGlhLmNvbT4K KworICAgICAgICBbZmxhdHBha10gRW5zdXJlIGVuX1VTLlVURi04IGxvY2FsZQorICAgICAgICBo dHRwczovL2J1Z3Mud2Via2l0Lm9yZy9zaG93X2J1Zy5jZ2k/aWQ9MjEyNDU5CisKKyAgICAgICAg UmV2aWV3ZWQgYnkgTk9CT0RZIChPT1BTISkuCisKKyAgICAgICAgV2hlbiBkZXZlbG9waW5nIFdl YktpdCB3ZSB3YW50IGFzIGZldyBzeXN0ZW0gZGlmZmVyZW5jZXMgYXMgcG9zc2libGUKKyAgICAg ICAgdG8gbWFrZSB0ZXN0cyByZXByb2R1Y2libGUuCisKKyAgICAgICAgZW5fVVMuVVRGLTggaXMg aW5kZWVkIHRoZSBvbmx5IGxvY2FsZSBpbnN0YWxsZWQgaW4gb3VyIGRldmVsb3BtZW50CisgICAg ICAgIGZsYXRwYWssIGJ1dCBvdXIgc2NyaXB0cyB3ZXJlIG5vdCBjb3JyZWN0bHkgY2xlYXJpbmcg bW9zdCBsb2NhbGUKKyAgICAgICAgZW52aXJvbm1lbnQgdmFyaWFibGVzLCB3aGljaCByZXN1bHRl ZCBpbiB3YXJuaW5ncyB3aGVuIHJ1bm5pbmcgc29tZQorICAgICAgICB0b29scyAobm90YWJseSBw ZXJsKSB3aGVuIHRoZSB1c2VyIE9TIGhhcyBhIGRpZmZlcmVudCBsb2NhbGUuCisKKyAgICAgICAg VGhpcyBwYXRjaCBlbnN1cmVzIGFsbCBsb2NhbGUgZW52aXJvbm1lbnQgdmFyaWFibGVzIGFyZSBj bGVhcmVkIGFuZAorICAgICAgICBMQU5HIGlzIHNldCB0byBlbl9VUy5VVEYtOC4gSXQgYWxzbyBy ZW1vdmVzIG5vdyByZWR1bmRhbnQgY29kZSBmcm9tCisgICAgICAgIGZsYXRwYWt1dGlscy5weS4K KworICAgICAgICAqIGZsYXRwYWsvZmxhdHBha3V0aWxzLnB5OgorICAgICAgICAoV2Via2l0Rmxh dHBhay5ydW5faW5fc2FuZGJveCk6CisgICAgICAgICogZmxhdHBhay93ZWJraXQtYndyYXA6CisK IDIwMjAtMDYtMTkgIENhcmxvcyBHYXJjaWEgQ2FtcG9zICA8Y2dhcmNpYUBpZ2FsaWEuY29tPgog CiAgICAgICAgIEFkZCBzdXBwb3J0IGZvciBmZXRjaGluZyByZWdpc3RyYWJsZSBkb21haW5zIHdp dGggcmVzb3VyY2UgbG9hZCBzdGF0aXN0aWNzCmRpZmYgLS1naXQgYS9Ub29scy9mbGF0cGFrL2Zs YXRwYWt1dGlscy5weSBiL1Rvb2xzL2ZsYXRwYWsvZmxhdHBha3V0aWxzLnB5CmluZGV4IGIyMGVj ZWFjMDVmYjkwMDc1ZDFmNzk5MjNlOGE4OWRiMmU1OTVmOGUuLmM0MWU4YWI4NWYxMGNjZGQ5NmY2 OGM1NGVhN2FhZDliM2U1NmM2NmMgMTAwNjQ0Ci0tLSBhL1Rvb2xzL2ZsYXRwYWsvZmxhdHBha3V0 aWxzLnB5CisrKyBiL1Rvb2xzL2ZsYXRwYWsvZmxhdHBha3V0aWxzLnB5CkBAIC03MDgsNyArNzA4 LDYgQEAgY2xhc3MgV2Via2l0RmxhdHBhazoKIAogICAgICAgICAgICAgc2FuZGJveF9lbnZpcm9u bWVudC51cGRhdGUoewogICAgICAgICAgICAgICAgICJUWiI6ICJQU1Q4UERUIiwKLSAgICAgICAg ICAgICAgICAiTEFORyI6ICJlbl9VUy5VVEYtOCIsCiAgICAgICAgICAgICB9KQogCiAgICAgICAg IGVudl92YXJfcHJlZml4ZXNfdG9fa2VlcCA9IFsKQEAgLTc0MCw3ICs3MzksNiBAQCBjbGFzcyBX ZWJraXRGbGF0cGFrOgogICAgICAgICAgICAgIkNYWEZMQUdTIiwKICAgICAgICAgICAgICJESVNQ TEFZIiwKICAgICAgICAgICAgICJKYXZhU2NyaXB0Q29yZVVzZUpJVCIsCi0gICAgICAgICAgICAi TEFORyIsCiAgICAgICAgICAgICAiTERGTEFHUyIsCiAgICAgICAgICAgICAiTWFsbG9jIiwKICAg ICAgICAgICAgICJOVU1CRVJfT0ZfUFJPQ0VTU09SUyIsCmRpZmYgLS1naXQgYS9Ub29scy9mbGF0 cGFrL3dlYmtpdC1id3JhcCBiL1Rvb2xzL2ZsYXRwYWsvd2Via2l0LWJ3cmFwCmluZGV4IGIzNDJh MWRhN2UyOTkxZTQ4NjRlNTJlNzNjMTFkNzYwOWIwODA4Y2IuLjMxZDM4OTRlMDRjMmE0NDBhNTFl ZTVlOGFlOGFkNTU2NjcyMzRjZWIgMTAwNzU1Ci0tLSBhL1Rvb2xzL2ZsYXRwYWsvd2Via2l0LWJ3 cmFwCisrKyBiL1Rvb2xzL2ZsYXRwYWsvd2Via2l0LWJ3cmFwCkBAIC01OCw2ICs1OCwxMSBAQCBk ZWYgbWFpbihhcmdzOiBsaXN0KSAtPiBpbnQ6CiAgICAgZm9yIGRzdCwgc3JjIGluIHRyeV9iaW5k X21vdW50cy5pdGVtcygpOgogICAgICAgICBid3JhcF9hcmdzLmV4dGVuZChbIi0tYmluZC10cnki LCBzcmMsIGRzdF0pCiAKKyAgICBmb3IgZW52IGluIG9zLmVudmlyb24ua2V5cygpOgorICAgICAg ICBpZiBlbnYuc3RhcnRzd2l0aCgiTENfIikgb3IgZW52ID09ICJMQU5HVUFHRSI6CisgICAgICAg ICAgICBid3JhcF9hcmdzLmV4dGVuZChbIi0tdW5zZXRlbnYiLCBlbnZdKQorICAgIGJ3cmFwX2Fy Z3MuZXh0ZW5kKFsiLS1zZXRlbnYiLCAiTEFORyIsICJlbl9VUy5VVEYtOCJdKQorCiAgICAgY29t bWFuZF9saW5lID0gJyAnLmpvaW4oc2hsZXgucXVvdGUoYSkgZm9yIGEgaW4gYndyYXBfYXJncyAr IGFyZ3MpCiAgICAgIyBvcy5zeXN0ZW0gcmV0dXJuIGNvZGUgYmVoYXZlcyBsaWtlIG9zLndhaXQu IEEgMTYgYml0IG51bWJlciB3aXRoIHRoZQogICAgICMgc2lnbmFsIGluIHRoZSBsb3dlciBieXRl IGFuZCwgaWYgdGhlIHNpZ25hbCBpcyB6ZXJvLCB0aGUgZXhpdCBjb2RlIGluCg==