Bug 230870 – Python 3 fails to run run-builtins-generator-tests : ModuleNotFoundError: No module named 'builtins_model'

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Fujii Hironori]

Windows Python 3 fails to run run-builtins-generator-tests : ModuleNotFoundError: No module named 'builtins_model'

r282424 (Bug 229879) switched to python3 for running run-builtins-generator-tests,
after this change dipatched to Buildbot, WinCairo testing Buidbots are failing.

https://build.webkit.org/#/builders/60/builds/2751/steps/20/logs/stdio

> Traceback (most recent call last):
>   File "C:\BW\wincairo-wkl-release-tests\build\Source\JavaScriptCore\Scripts\generate-js-builtins.py", line 41, in <module>
>     from wkbuiltins import *
>   File "C:\BW\wincairo-wkl-release-tests\build\Source\JavaScriptCore\Scripts\wkbuiltins\__init__.py", line 3, in <module>
>     from .wkbuiltins import *
>   File "C:\BW\wincairo-wkl-release-tests\build\Source\JavaScriptCore\Scripts\wkbuiltins\wkbuiltins.py", line 4, in <module>
>     from builtins_model import *
> ModuleNotFoundError: No module named 'builtins_model'
> Traceback (most recent call last):
>   File "C:\BW\wincairo-wkl-release-tests\build\Source\JavaScriptCore\Scripts\generate-js-builtins.py", line 41, in <module>
>     from wkbuiltins import *
>   File "C:\BW\wincairo-wkl-release-tests\build\Source\JavaScriptCore\Scripts\wkbuiltins\__init__.py", line 3, in <module>
>     from .wkbuiltins import *
>   File "C:\BW\wincairo-wkl-release-tests\build\Source\JavaScriptCore\Scripts\wkbuiltins\wkbuiltins.py", line 4, in <module>
>     from builtins_model import *
> ModuleNotFoundError: No module named 'builtins_model'
> (...)

[Fujii Hironori]

Created attachment 439560 [details]
WIP patch

This isn't a Windows Python3 specific issue.
webkitpy/codegen/main.py explicitly invoke 'python'. It expects this is Python 2.
https://github.com/WebKit/WebKit/blob/b8f82ead5a81881d83127c9b5592c1b30e0797c9/Tools/Scripts/webkitpy/codegen/main.py#L41

I confirmed the same error happens if I replace it with sys.executable on Linux.

[Fujii Hironori]

With this WIP patch and Python3, one test reports ParseException.

PASS: WebCore-ArbitraryConditionalGuard-Separate.js-result
Traceback (most recent call last):
  File "JavaScriptCore/Scripts/generate-js-builtins.py", line 178, in <module>
    generate_bindings_for_builtins_files(builtins_files=input_filepaths, **options)
  File "JavaScriptCore/Scripts/generate-js-builtins.py", line 78, in generate_bindings_for_builtins_files
    model.parse_builtins_file(file_name, file_text)
  File "/mnt/c/home/webkit/gc/Source/JavaScriptCore/Scripts/wkbuiltins/builtins_model.py", line 192, in parse_builtins_file
    object_annotations = self._parse_annotations(text)
  File "/mnt/c/home/webkit/gc/Source/JavaScriptCore/Scripts/wkbuiltins/builtins_model.py", line 290, in _parse_annotations
    raise ParseException("Duplicate annotation found: %s" % key)
wkbuiltins.builtins_model.ParseException: Duplicate annotation found: internal

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "JavaScriptCore/Scripts/generate-js-builtins.py", line 181, in <module>
    log.error(e.message)
AttributeError: 'ParseException' object has no attribute 'message'

Traceback (most recent call last):
  File "JavaScriptCore/Scripts/generate-js-builtins.py", line 178, in <module>
    generate_bindings_for_builtins_files(builtins_files=input_filepaths, **options)
  File "JavaScriptCore/Scripts/generate-js-builtins.py", line 78, in generate_bindings_for_builtins_files
    model.parse_builtins_file(file_name, file_text)
  File "/mnt/c/home/webkit/gc/Source/JavaScriptCore/Scripts/wkbuiltins/builtins_model.py", line 192, in parse_builtins_file
    object_annotations = self._parse_annotations(text)
  File "/mnt/c/home/webkit/gc/Source/JavaScriptCore/Scripts/wkbuiltins/builtins_model.py", line 282, in _parse_annotations
    raise ParseException("Duplicate annotation found: %s" % key)
wkbuiltins.builtins_model.ParseException: Duplicate annotation found: conditional

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "JavaScriptCore/Scripts/generate-js-builtins.py", line 181, in <module>
    log.error(e.message)
AttributeError: 'ParseException' object has no attribute 'message'

[Fujii Hironori]

With this WIP patch, generate-js-builtins.py fails with an ImportError.

FAILED: JavaScriptCore/DerivedSources/JSCBuiltins.cpp JavaScriptCore/DerivedSources/JSCBuiltins.h C:/home/webkit/gc/WebKitBuild/Debug/JavaScriptCore/DerivedSources/JSCBuiltins.cpp C:/home/webkit/gc/WebKitBuild/Debug/JavaScriptCore/DerivedSources/JSCBuiltins.h
cmd.exe /C "cd /D C:\home\webkit\gc\WebKitBuild\Debug\Source\JavaScriptCore && C:\Python39\python3.exe C:/home/webkit/gc/WebKitBuild/Debug/JavaScriptCore/Scripts/generate-js-builtins.py --framework JavaScriptCore --output-directory C:/home/webkit/gc/WebKitBuild/Debug/JavaScriptCore/DerivedSources --combined C:/home/webkit/gc/Source/JavaScriptCore/builtins/AsyncFromSyncIteratorPrototype.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/ArrayConstructor.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/ArrayIteratorPrototype.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/ArrayPrototype.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/AsyncFunctionPrototype.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/AsyncIteratorPrototype.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/AsyncGeneratorPrototype.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/DatePrototype.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/FunctionPrototype.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/GeneratorPrototype.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/GlobalObject.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/GlobalOperations.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/InternalPromiseConstructor.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/IteratorHelpers.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/IteratorPrototype.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/MapIteratorPrototype.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/MapPrototype.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/ModuleLoader.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/NumberConstructor.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/ObjectConstructor.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/PromiseConstructor.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/PromiseOperations.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/PromisePrototype.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/ReflectObject.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/RegExpPrototype.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/RegExpStringIteratorPrototype.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/SetIteratorPrototype.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/SetPrototype.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/StringConstructor.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/StringIteratorPrototype.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/StringPrototype.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/TypedArrayConstructor.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/TypedArrayPrototype.js C:/home/webkit/gc/Source/JavaScriptCore/builtins/WebAssembly.js"
Traceback (most recent call last):

  File "C:\home\webkit\gc\WebKitBuild\Debug\JavaScriptCore\Scripts\generate-js-builtins.py", line 41, in <module>

    from wkbuiltins import *

  File "C:\home\webkit\gc\WebKitBuild\Debug\JavaScriptCore\Scripts\wkbuiltins.py", line 4, in <module>

    from .builtins_model import *

ImportError: attempted relative import with no known parent package

[Fujii Hironori]

So, there are four problems here

1. ModuleNotFoundError (comment#0)
   Use relative imports
2. 'python' expects Python 2 (comment#1)
   Use sys.executable
3. ParseException (comment#2)
4. copied generate-js-builtins.py raises ImportError (comment#3)

[Fujii Hironori]

Created attachment 439571 [details]
Patch

[EWS Watchlist]

This patch modifies the JS builtins code generator. Please ensure that you have rebaselined any generator test results (i.e., by running `Tools/Scripts/run-builtins-generator-tests --reset-results`)

[Jonathan Bedard]

Comment on attachment 439571 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=439571&action=review

> Source/JavaScriptCore/Scripts/generate-js-builtins.py:36
> +sys.path.append(os.path.join(os.path.dirname(__file__), 'wkbuiltins'))

Are we sure the relative/absolute import problem is our only problem in this script? How have we verified this with Python 3?

[Fujii Hironori]

Comment on attachment 439571 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=439571&action=review

>> Source/JavaScriptCore/Scripts/generate-js-builtins.py:36
>> +sys.path.append(os.path.join(os.path.dirname(__file__), 'wkbuiltins'))
> 
> Are we sure the relative/absolute import problem is our only problem in this script? How have we verified this with Python 3?

By grepping "'python'", InspectorGeneratorTests is also explicitly invoking 'python'. It is Python 2.
I haven't tested the code yet. But, it can have the Python 3 issues.

I replaced "'python'" with 'sys.executable' in the patch.
If run-builtins-generator-tests is invoked with Python 3, it runs generate-js-builtins.py with the same python, namely Python 3.

[Fujii Hironori]

Comment on attachment 439571 [details]
Patch

Clearing flags on attachment: 439571

Committed r283329 (242351@main): <https://commits.webkit.org/242351@main>

[Fujii Hironori]

All reviewed patches have been landed.  Closing bug.

[Radar WebKit Bug Importer]

<rdar://problem/83734306>