Windows Python 3 fails to run run-builtins-generator-tests : ModuleNotFoundError: No module named 'builtins_model' r282424 (Bug 229879) switched to python3 for running run-builtins-generator-tests, after this change dipatched to Buildbot, WinCairo testing Buidbots are failing. https://build.webkit.org/#/builders/60/builds/2751/steps/20/logs/stdio > Traceback (most recent call last): > File "C:\BW\wincairo-wkl-release-tests\build\Source\JavaScriptCore\Scripts\generate-js-builtins.py", line 41, in <module> > from wkbuiltins import * > File "C:\BW\wincairo-wkl-release-tests\build\Source\JavaScriptCore\Scripts\wkbuiltins\__init__.py", line 3, in <module> > from .wkbuiltins import * > File "C:\BW\wincairo-wkl-release-tests\build\Source\JavaScriptCore\Scripts\wkbuiltins\wkbuiltins.py", line 4, in <module> > from builtins_model import * > ModuleNotFoundError: No module named 'builtins_model' > Traceback (most recent call last): > File "C:\BW\wincairo-wkl-release-tests\build\Source\JavaScriptCore\Scripts\generate-js-builtins.py", line 41, in <module> > from wkbuiltins import * > File "C:\BW\wincairo-wkl-release-tests\build\Source\JavaScriptCore\Scripts\wkbuiltins\__init__.py", line 3, in <module> > from .wkbuiltins import * > File "C:\BW\wincairo-wkl-release-tests\build\Source\JavaScriptCore\Scripts\wkbuiltins\wkbuiltins.py", line 4, in <module> > from builtins_model import * > ModuleNotFoundError: No module named 'builtins_model' > (...)
Created attachment 439560 [details] WIP patch This isn't a Windows Python3 specific issue. webkitpy/codegen/main.py explicitly invoke 'python'. It expects this is Python 2. https://github.com/WebKit/WebKit/blob/b8f82ead5a81881d83127c9b5592c1b30e0797c9/Tools/Scripts/webkitpy/codegen/main.py#L41 I confirmed the same error happens if I replace it with sys.executable on Linux.
With this WIP patch and Python3, one test reports ParseException. PASS: WebCore-ArbitraryConditionalGuard-Separate.js-result Traceback (most recent call last): File "JavaScriptCore/Scripts/generate-js-builtins.py", line 178, in <module> generate_bindings_for_builtins_files(builtins_files=input_filepaths, **options) File "JavaScriptCore/Scripts/generate-js-builtins.py", line 78, in generate_bindings_for_builtins_files model.parse_builtins_file(file_name, file_text) File "/mnt/c/home/webkit/gc/Source/JavaScriptCore/Scripts/wkbuiltins/builtins_model.py", line 192, in parse_builtins_file object_annotations = self._parse_annotations(text) File "/mnt/c/home/webkit/gc/Source/JavaScriptCore/Scripts/wkbuiltins/builtins_model.py", line 290, in _parse_annotations raise ParseException("Duplicate annotation found: %s" % key) wkbuiltins.builtins_model.ParseException: Duplicate annotation found: internal During handling of the above exception, another exception occurred: Traceback (most recent call last): File "JavaScriptCore/Scripts/generate-js-builtins.py", line 181, in <module> log.error(e.message) AttributeError: 'ParseException' object has no attribute 'message' Traceback (most recent call last): File "JavaScriptCore/Scripts/generate-js-builtins.py", line 178, in <module> generate_bindings_for_builtins_files(builtins_files=input_filepaths, **options) File "JavaScriptCore/Scripts/generate-js-builtins.py", line 78, in generate_bindings_for_builtins_files model.parse_builtins_file(file_name, file_text) File "/mnt/c/home/webkit/gc/Source/JavaScriptCore/Scripts/wkbuiltins/builtins_model.py", line 192, in parse_builtins_file object_annotations = self._parse_annotations(text) File "/mnt/c/home/webkit/gc/Source/JavaScriptCore/Scripts/wkbuiltins/builtins_model.py", line 282, in _parse_annotations raise ParseException("Duplicate annotation found: %s" % key) wkbuiltins.builtins_model.ParseException: Duplicate annotation found: conditional During handling of the above exception, another exception occurred: Traceback (most recent call last): File "JavaScriptCore/Scripts/generate-js-builtins.py", line 181, in <module> log.error(e.message) AttributeError: 'ParseException' object has no attribute 'message'
With this WIP patch, generate-js-builtins.py fails with an ImportError. FAILED: JavaScriptCore/DerivedSources/JSCBuiltins.cpp JavaScriptCore/DerivedSources/JSCBuiltins.h C:/home/webkit/gc/WebKitBuild/Debug/JavaScriptCore/DerivedSources/JSCBuiltins.cpp C:/home/webkit/gc/WebKitBuild/Debug/JavaScriptCore/DerivedSources/JSCBuiltins.h cmd.exe /C "cd /D C:\home\webkit\gc\WebKitBuild\Debug\Source\JavaScriptCore && C:\Python39\python3.exe C:/home/webkit/gc/WebKitBuild/Debug/JavaScriptCore/Scripts/generate-js-builtins.py --framework JavaScriptCore --output-directory C:/home/webkit/gc/WebKitBuild/Debug/JavaScriptCore/DerivedSources --combined C:/home/webkit/gc/Source/JavaScriptCore/builtins/AsyncFromSyncIteratorPrototype.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/ArrayConstructor.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/ArrayIteratorPrototype.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/ArrayPrototype.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/AsyncFunctionPrototype.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/AsyncIteratorPrototype.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/AsyncGeneratorPrototype.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/DatePrototype.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/FunctionPrototype.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/GeneratorPrototype.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/GlobalObject.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/GlobalOperations.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/InternalPromiseConstructor.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/IteratorHelpers.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/IteratorPrototype.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/MapIteratorPrototype.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/MapPrototype.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/ModuleLoader.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/NumberConstructor.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/ObjectConstructor.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/PromiseConstructor.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/PromiseOperations.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/PromisePrototype.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/ReflectObject.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/RegExpPrototype.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/RegExpStringIteratorPrototype.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/SetIteratorPrototype.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/SetPrototype.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/StringConstructor.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/StringIteratorPrototype.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/StringPrototype.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/TypedArrayConstructor.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/TypedArrayPrototype.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/WebAssembly.js" Traceback (most recent call last): File "C:\home\webkit\gc\WebKitBuild\Debug\JavaScriptCore\Scripts\generate-js-builtins.py", line 41, in <module> from wkbuiltins import * File "C:\home\webkit\gc\WebKitBuild\Debug\JavaScriptCore\Scripts\wkbuiltins.py", line 4, in <module> from .builtins_model import * ImportError: attempted relative import with no known parent package
So, there are four problems here 1. ModuleNotFoundError (comment#0) Use relative imports 2. 'python' expects Python 2 (comment#1) Use sys.executable 3. ParseException (comment#2) 4. copied generate-js-builtins.py raises ImportError (comment#3)
Created attachment 439571 [details] Patch
This patch modifies the JS builtins code generator. Please ensure that you have rebaselined any generator test results (i.e., by running `Tools/Scripts/run-builtins-generator-tests --reset-results`)
Comment on attachment 439571 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=439571&action=review > Source/JavaScriptCore/Scripts/generate-js-builtins.py:36 > +sys.path.append(os.path.join(os.path.dirname(__file__), 'wkbuiltins')) Are we sure the relative/absolute import problem is our only problem in this script? How have we verified this with Python 3?
Comment on attachment 439571 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=439571&action=review >> Source/JavaScriptCore/Scripts/generate-js-builtins.py:36 >> +sys.path.append(os.path.join(os.path.dirname(__file__), 'wkbuiltins')) > > Are we sure the relative/absolute import problem is our only problem in this script? How have we verified this with Python 3? By grepping "'python'", InspectorGeneratorTests is also explicitly invoking 'python'. It is Python 2. I haven't tested the code yet. But, it can have the Python 3 issues. I replaced "'python'" with 'sys.executable' in the patch. If run-builtins-generator-tests is invoked with Python 3, it runs generate-js-builtins.py with the same python, namely Python 3.
Comment on attachment 439571 [details] Patch Clearing flags on attachment: 439571 Committed r283329 (242351@main): <https://commits.webkit.org/242351@main>
All reviewed patches have been landed. Closing bug.
<rdar://problem/83734306>