Bug 202830 – New html5lib requires newer version of six.py than that shipped in macOS 10.14.6

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Kenneth Russell]

In https://bugs.webkit.org/show_bug.cgi?id=201955 some third-party packages used by webkitpy were upgraded, html5lib among them. The new version of html5lib requires a version of six.py which exports the symbol "viewkeys", which /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six.py on macOS 10.14.6 does not export.

[Kenneth Russell]

Oops, didn't mean to file this yet.

The error that's encountered during "webkit-patch upload" is:

Traceback (most recent call last):
  File "./Tools/Scripts/webkit-patch", line 84, in <module>
    main()
  File "./Tools/Scripts/webkit-patch", line 79, in main
    WebKitPatch(os.path.abspath(__file__)).main()
  File "/Users/kbr/src/WebKit/Tools/Scripts/webkitpy/tool/multicommandtool.py", line 305, in main
    result = command.check_arguments_and_execute(options, args, self)
  File "/Users/kbr/src/WebKit/Tools/Scripts/webkitpy/tool/multicommandtool.py", line 123, in check_arguments_and_execute
    return self.execute(options, args, tool) or 0
  File "/Users/kbr/src/WebKit/Tools/Scripts/webkitpy/tool/commands/abstractsequencedcommand.py", line 55, in execute
    self._sequence.run_and_handle_errors(tool, options, state)
  File "/Users/kbr/src/WebKit/Tools/Scripts/webkitpy/tool/commands/stepsequence.py", line 73, in run_and_handle_errors
    self._run(tool, options, state)
  File "/Users/kbr/src/WebKit/Tools/Scripts/webkitpy/tool/commands/stepsequence.py", line 67, in _run
    step(tool, options).run(state)
  File "/Users/kbr/src/WebKit/Tools/Scripts/webkitpy/tool/steps/preparechangelog.py", line 113, in run
    args.append("--description=%s" % self.cached_lookup(state, 'bug_title'))
  File "/Users/kbr/src/WebKit/Tools/Scripts/webkitpy/tool/steps/abstractstep.py", line 62, in cached_lookup
    state[key] = promise(self, state)
  File "/Users/kbr/src/WebKit/Tools/Scripts/webkitpy/tool/steps/abstractstep.py", line 50, in <lambda>
    "bug_title": lambda self, state: self.cached_lookup(state, 'bug').title(),
  File "/Users/kbr/src/WebKit/Tools/Scripts/webkitpy/tool/steps/abstractstep.py", line 62, in cached_lookup
    state[key] = promise(self, state)
  File "/Users/kbr/src/WebKit/Tools/Scripts/webkitpy/tool/steps/abstractstep.py", line 48, in <lambda>
    "bug": lambda self, state: self._tool.bugs.fetch_bug(state["bug_id"]),
  File "/Users/kbr/src/WebKit/Tools/Scripts/webkitpy/common/net/bugzilla/bugzilla.py", line 489, in fetch_bug
    bug_dictionary = self.fetch_bug_dictionary(bug_id)
  File "/Users/kbr/src/WebKit/Tools/Scripts/webkitpy/common/net/bugzilla/bugzilla.py", line 481, in fetch_bug_dictionary
    self.authenticate()
  File "/Users/kbr/src/WebKit/Tools/Scripts/webkitpy/common/net/bugzilla/bugzilla.py", line 577, in authenticate
    self.browser.select_form(name="login")
  File "/Users/kbr/src/WebKit/Tools/Scripts/webkitpy/thirdparty/autoinstalled/mechanize/_mechanize.py", line 620, in select_form
    global_form = self._factory.global_form
  File "/Users/kbr/src/WebKit/Tools/Scripts/webkitpy/thirdparty/autoinstalled/mechanize/_html.py", line 313, in global_form
    self.forms()
  File "/Users/kbr/src/WebKit/Tools/Scripts/webkitpy/thirdparty/autoinstalled/mechanize/_html.py", line 320, in forms
    self._current_forms, self._current_global_form = self._get_forms()
  File "/Users/kbr/src/WebKit/Tools/Scripts/webkitpy/thirdparty/autoinstalled/mechanize/_html.py", line 335, in _get_forms
    if self.root is None:
  File "/Users/kbr/src/WebKit/Tools/Scripts/webkitpy/thirdparty/autoinstalled/mechanize/_html.py", line 300, in root
    transport_encoding=transport_encoding)
  File "/Users/kbr/src/WebKit/Tools/Scripts/webkitpy/thirdparty/autoinstalled/mechanize/_html.py", line 191, in content_parser
    from html5lib import parse
  File "/Users/kbr/src/WebKit/Tools/Scripts/webkitpy/thirdparty/autoinstalled/html5lib/__init__.py", line 25, in <module>
    from .html5parser import HTMLParser, parse, parseFragment
  File "/Users/kbr/src/WebKit/Tools/Scripts/webkitpy/thirdparty/autoinstalled/html5lib/html5parser.py", line 2, in <module>
    from six import with_metaclass, viewkeys
ImportError: cannot import name viewkeys



This problem can be resolved by always autoinstalling the version of six.py which would ordinarily only be autoinstalled along with pytest.

[Kenneth Russell]

Created attachment 380700 [details]
Patch

[Jonathan Bedard]

Comment on attachment 380700 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=380700&action=review

> Tools/Scripts/webkitpy/thirdparty/__init__.py:-169
> -        self._install("https://files.pythonhosted.org/packages/16/d8/bc6316cf98419719bd59c91742194c111b6f2e85abac88e496adefaf7afe/six-1.11.0.tar.gz",

I think pytest needs it too, no?

Probably want a _install_six() function that both _install_pytest and _install_mechanize call.

[Kenneth Russell]

Comment on attachment 380700 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=380700&action=review

>> Tools/Scripts/webkitpy/thirdparty/__init__.py:-169
>> -        self._install("https://files.pythonhosted.org/packages/16/d8/bc6316cf98419719bd59c91742194c111b6f2e85abac88e496adefaf7afe/six-1.11.0.tar.gz",
> 
> I think pytest needs it too, no?
> 
> Probably want a _install_six() function that both _install_pytest and _install_mechanize call.

Ah, I was assuming that mechanize was always installed. I'll refactor this as you suggested.

[Kenneth Russell]

Created attachment 380707 [details]
Patch

[EWS]

Comment on attachment 380707 [details]
Patch

Rejecting attachment 380707 [details] from commit-queue.

[e-mail] does not have committer permissions according to https://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/common/config/contributors.json.

- If you do not have committer rights please read http://webkit.org/coding/contributing.html for instructions on how to use bugzilla flags.

- If you have committer rights please correct the error in Tools/Scripts/webkitpy/common/config/contributors.json by adding yourself to the file (no review needed).  The commit-queue restarts itself every 2 hours.  After restart the commit-queue will correctly respect your committer rights.

[Kenneth Russell]

Oops, I'm not a committer any more. jbedard@ would you mind cq'ing this? Thanks.

[Jonathan Bedard]

(In reply to Kenneth Russell from comment #7)
> Oops, I'm not a committer any more. jbedard@ would you mind cq'ing this?
> Thanks.

cq+ed!

Just a heads up, if you mark a patch as cq?, a committer (or your reviewer) will usually cq+ it for you.

[WebKit Commit Bot]

Comment on attachment 380707 [details]
Patch

Clearing flags on attachment: 380707

Committed r250998: <https://trac.webkit.org/changeset/250998>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Radar WebKit Bug Importer]

<rdar://problem/56177626>

[Kenneth Russell]

(In reply to Jonathan Bedard from comment #8)
> (In reply to Kenneth Russell from comment #7)
> > Oops, I'm not a committer any more. jbedard@ would you mind cq'ing this?
> > Thanks.
> 
> cq+ed!
> 
> Just a heads up, if you mark a patch as cq?, a committer (or your reviewer)
> will usually cq+ it for you.

Thanks, had forgotten about that!