Bug 273060 - [Win] UnicodeDecodeError on run-webkit-tests on non-English System Locale Windows 11
Summary: [Win] UnicodeDecodeError on run-webkit-tests on non-English System Locale Win...
Status: NEW
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: WebKit Local Build
Hardware: PC Other
: P2 Trivial
Assignee: Nobody
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2024-04-22 01:45 PDT by Kohei Asano
Modified: 2024-05-16 00:06 PDT (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Kohei Asano 2024-04-22 01:45:18 PDT
I tried to run `python Tools/Scripts/run-webkit-tests --release` on Windows 11, Japanese PowerShell. I got many UnicodeDecodeError like a following message


```
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x90 in position 0: invalid start byte
[16242/21282] storage/indexeddb/mozilla/key-requirements.html passed
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Python311\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "C:\Python311\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\0300093582\WebKit\Tools\Scripts\libraries\webkitcorepy\webkitcorepy\task_pool.py", line 308, in main
    typ, typ('{} (from {})'.format(str(exception), name)), traceback,
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: function takes exactly 5 arguments (1 given)
```

This can be resolved by changing, system language to English via
Start Menu -> Control Panel -> Region -> Admistrative -> Change system locale...
Comment 1 Kohei Asano 2024-04-22 17:31:04 PDT
This error wouldn't depend on PowerShell, would cause Command Prompt as well. 
This is longer log, we suspect Shift_JIS from taskkill.exe is problematic.

```
[16175/21282] webgl/2.0.0/conformance2/textures/image_bitmap_from_blob/tex-2d-rgba4-rgba-unsigned_short_4_4_4_4.html pas                                                                                                                        Process Process-31:
worker/30 stopping
Traceback (most recent call last):
  File "C:\Users\\WebKit\Tools\Scripts\libraries\webkitcorepy\webkitcorepy\task_pool.py", line 303, in main
    queue.send(_Result(value=task(None), id=task.id))
                             ^^^^^^^^^^
  File "C:\Users\\WebKit\Tools\Scripts\libraries\webkitcorepy\webkitcorepy\task_pool.py", line 56, in __call__
    return self.function(*self.args, **self.kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\\WebKit\Tools\Scripts\webkitpy\layout_tests\controllers\layout_test_runner.py", line 77, in run_shard
    return Worker.instance.run_tests(shard)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\\WebKit\Tools\Scripts\webkitpy\layout_tests\controllers\layout_test_runner.py", line 325, in run_tests
    Worker.instance.run_test(input, shard.name)
  File "C:\Users\\WebKit\Tools\Scripts\webkitpy\layout_tests\controllers\layout_test_runner.py", line 353, in run_test
    result = self._run_test_with_or_without_timeout(test_input, test_timeout_sec, stop_when_done)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\\WebKit\Tools\Scripts\webkitpy\layout_tests\controllers\layout_test_runner.py", line 410, in _run_test_with_or_without_timeout
    return self._run_test_in_this_thread(test_input, stop_when_done)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\\WebKit\Tools\Scripts\webkitpy\layout_tests\controllers\layout_test_runner.py", line 496, in _run_test_in_this_thread
    return self._run_single_test(self._driver, test_input, stop_when_done)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\\WebKit\Tools\Scripts\webkitpy\layout_tests\controllers\layout_test_runner.py", line 499, in _run_single_test
    return single_test_runner.run_single_test(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\\WebKit\Tools\Scripts\webkitpy\layout_tests\controllers\single_test_runner.py", line 48, in run_single_test
    return runner.run()
           ^^^^^^^^^^^^
  File "C:\Users\\WebKit\Tools\Scripts\webkitpy\layout_tests\controllers\single_test_runner.py", line 133, in run
    return self._run_compare_test()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\\WebKit\Tools\Scripts\webkitpy\layout_tests\controllers\single_test_runner.py", line 136, in _run_compare_test
    driver_output = self._driver.run_test(self._driver_input(), self._stop_when_done)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\\WebKit\Tools\Scripts\webkitpy\port\driver.py", line 888, in run_test
    return self._driver.run_test(driver_input, stop_when_done)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\\WebKit\Tools\Scripts\webkitpy\port\driver.py", line 251, in run_test
    out, err = self._server_process.stop(self._port.driver_stop_timeout() if stop_when_done else 0.0)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\\WebKit\Tools\Scripts\webkitpy\port\server_process.py", line 409, in stop
    return self._wait_for_stop(timeout_secs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\\WebKit\Tools\Scripts\webkitpy\port\server_process.py", line 422, in _wait_for_stop
    self._kill()
  File "C:\Users\\WebKit\Tools\Scripts\webkitpy\port\server_process.py", line 440, in _kill
    self._target_host.executive.kill_process(self._proc.pid)
  File "C:\Users\\WebKit\Tools\Scripts\webkitpy\common\system\executive.py", line 199, in kill_process
    self.run_command(command, ignore_errors=True)
  File "C:\Users\\WebKit\Tools\Scripts\webkitpy\common\system\executive.py", line 424, in run_command
    output = string_utils.decode(output, encoding=self._child_process_encoding()).replace('\r\n', '\n')
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\\WebKit\Tools\Scripts\libraries\webkitcorepy\webkitcorepy\string_utils.py", line 46, in decode
    return data.decode(encoding, errors=errors)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x90 in position 0: invalid start byte
...
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x90 in position 0: invalid start byte
[16242/21282] storage/indexeddb/mozilla/key-requirements.html passed
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Python311\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "C:\Python311\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\\WebKit\Tools\Scripts\libraries\webkitcorepy\webkitcorepy\task_pool.py", line 308, in main
    typ, typ('{} (from {})'.format(str(exception), name)), traceback,
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: function takes exactly 5 arguments (1 given)
...
[34/35] js/dom/navigator-language.html failed unexpectedly (text diff)
Stopping WebSocket server ...
UnicodeDecodeError raised: 'utf-8' codec can't decode byte 0x90 in position 0: invalid start byte
Traceback (most recent call last):
  File "C:\Users\\WebKit\Tools\Scripts\webkitpy\layout_tests\run_webkit_tests.py", line 94, in main
    run_details = run(port, options, args, stderr)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\\WebKit\Tools\Scripts\webkitpy\layout_tests\run_webkit_tests.py", line 566, in run
    run_details = manager.run(args)
                  ^^^^^^^^^^^^^^^^^
  File "C:\Users\\WebKit\Tools\Scripts\webkitpy\layout_tests\controllers\manager.py", line 470, in run
    self._runner.stop_servers()
  File "C:\Users\\WebKit\Tools\Scripts\webkitpy\layout_tests\controllers\layout_test_runner.py", line 291, in stop_servers
    self._port.stop_websocket_server()
  File "C:\Users\\WebKit\Tools\Scripts\webkitpy\port\base.py", line 1008, in stop_websocket_server
    Port._websocket_server.stop()
  File "C:\Users\\WebKit\Tools\Scripts\webkitpy\layout_tests\servers\http_server_base.py", line 140, in stop
    self._stop_running_server()
  File "C:\Users\\WebKit\Tools\Scripts\webkitpy\layout_tests\servers\websocket_server.py", line 178, in _stop_running_server
    self._wait_for_action(self._check_and_kill)
  File "C:\Users\\WebKit\Tools\Scripts\webkitpy\layout_tests\servers\http_server_base.py", line 199, in _wait_for_action
    if action():
       ^^^^^^^^
  File "C:\Users\\WebKit\Tools\Scripts\webkitpy\layout_tests\servers\websocket_server.py", line 200, in _check_and_kill
    self._executive.run_command(["taskkill.exe", "/f", "/t", "/pid", self._pid], ignore_errors=True)
  File "C:\Users\\WebKit\Tools\Scripts\webkitpy\common\system\executive.py", line 424, in run_command
    output = string_utils.decode(output, encoding=self._child_process_encoding()).replace('\r\n', '\n')
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\\WebKit\Tools\Scripts\libraries\webkitcorepy\webkitcorepy\string_utils.py", line 46, in decode
    return data.decode(encoding, errors=errors)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x90 in position 0: invalid start byte
Stopping WebSocket server ...Failed to stop pywebsocket: pid file is missing
Exception ignored in atexit callback: <function LayoutTestRunner.__init__.<locals>.<lambda> at 0x0000016806AC59E0>
Traceback (most recent call last):
  File "C:\Users\\WebKit\Tools\Scripts\webkitpy\layout_tests\controllers\layout_test_runner.py", line 125, in <lambda>
    atexit.register(lambda: self.stop_servers())
                            ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\\WebKit\Tools\Scripts\webkitpy\layout_tests\controllers\layout_test_runner.py", line 291, in stop_servers
    self._port.stop_websocket_server()
  File "C:\Users\\WebKit\Tools\Scripts\webkitpy\port\base.py", line 1011, in stop_websocket_server
    Port._websocket_secure_server.stop()
  File "C:\Users\\WebKit\Tools\Scripts\webkitpy\layout_tests\servers\http_server_base.py", line 140, in stop
    self._stop_running_server()
  File "C:\Users\\WebKit\Tools\Scripts\webkitpy\layout_tests\servers\websocket_server.py", line 178, in _stop_running_server
    self._wait_for_action(self._check_and_kill)
  File "C:\Users\\WebKit\Tools\Scripts\webkitpy\layout_tests\servers\http_server_base.py", line 199, in _wait_for_action
    if action():
       ^^^^^^^^
  File "C:\Users\\WebKit\Tools\Scripts\webkitpy\layout_tests\servers\websocket_server.py", line 200, in _check_and_kill
    self._executive.run_command(["taskkill.exe", "/f", "/t", "/pid", self._pid], ignore_errors=True)
  File "C:\Users\\WebKit\Tools\Scripts\webkitpy\common\system\executive.py", line 424, in run_command
    output = string_utils.decode(output, encoding=self._child_process_encoding()).replace('\r\n', '\n')
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\\WebKit\Tools\Scripts\libraries\webkitcorepy\webkitcorepy\string_utils.py", line 46, in decode
    return data.decode(encoding, errors=errors)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x90 in position 0: invalid start byte
```
Comment 2 Fujii Hironori 2024-04-23 00:36:27 PDT
How about an idea of using pywin32 rather than taskkill.exe?

from win32.win32process import TerminateProcess
from win32.win32api import OpenProcess
import win32con

handle = OpenProcess(win32con.PROCESS_TERMINATE, False, 12345)
TerminateProcess(handle, 0)
Comment 3 Fujii Hironori 2024-05-16 00:06:10 PDT
python - subprocess: deleting child processes in Windows - Stack Overflow
https://stackoverflow.com/q/1230669