Bug 184986 – Make all python scripts compatible with both python2 and python3

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Michael Catanzaro]

python2 is going away on Linux, so we need to convert our scripts to be compatible with python3. But macOS does not have python2, so we need to be careful to maintain python2 compatibility. This means we should keep the existing #!/usr/bin/env python shebangs and just test all our scripts to ensure they work with /usr/bin/python symlinked to python3.

Let's do this in five stages, using separate bugs:

 (1) Ensure it's possible to build a tarball, using CMake directly. Bug #156674
 (2) Ensure it's possible to build using build-webkit.
 (3) Ensure the most important developer scripts work (webkit-patch, prepare-ChangeLog)
 (4) Ensure test-webkitpy passes.
 (5) Port other scripts on an as-needed basis.

(1) Is the most urgent, as python3 patches are already required to build WebKit on the next SUSE and RHEL distros. This should be pretty easy.

(2) Is where things will get difficult.

(4) At this point, we should be able to switch our Linux bots to using python3. This will be a longer-term effort, but we should still try to finish in advance of 2020.

(5) is optional work that we can do as-needed. We don't need to port anything that's not tested by webkitpy and which we are not actively using ourselves.

Somebody pointed me towards https://portingguide.readthedocs.io/en/latest/, which I haven't read yet, but which might be helpful.

Also, Konstantin suggested run 'pylint --py3k' in the style checker, to make it a bit less likely that developers using python2 break developers using python3. (It's inevitable, but we're going to be stuck with that until macOS gets python3.) Would be nice if we could find a way to check for python3-isms as well, to prevent developers using python3 from breaking developers using python2 (which is also inevitable).

[Carlos Alberto Lopez Perez]

(In reply to Michael Catanzaro from comment #0)
> (4) At this point, we should be able to switch our Linux bots to using
> python3. This will be a longer-term effort, but we should still try to
> finish in advance of 2020.
> 

Our bots run Debian, which already has plans to maintain python2 beyond 2020.
https://lists.debian.org/debian-devel/2018/04/msg00508.html

On top of that there is always the possibility of building python2 on the JHBuild so developers using more bleeding edge distros can still keep using it for WebKit.

Certainly, moving to python3 seems a job we will have to do at some point.
But I don't see a hard deadline in 2020 like you seem to do.

Maybe if we wait a bit more we can do this work together with the Mac developers. I hope the new MacOS (not sure when its released) will already have python3 in the base system.

[Michael Catanzaro]

(In reply to Carlos Alberto Lopez Perez from comment #1)
> (In reply to Michael Catanzaro from comment #0)
> > (4) At this point, we should be able to switch our Linux bots to using
> > python3. This will be a longer-term effort, but we should still try to
> > finish in advance of 2020.
> > 
> 
> Our bots run Debian, which already has plans to maintain python2 beyond 2020.
> https://lists.debian.org/debian-devel/2018/04/msg00508.html
> 
> On top of that there is always the possibility of building python2 on the
> JHBuild so developers using more bleeding edge distros can still keep using
> it for WebKit.

I *think* it would be possible. We'll have to first port jhbuild itself to python3, of course, because jhbuild is not itself going to be able to use the python built in the jhbuild environment. But we'd have to do that anyway. (My plan is actually to stop using jhbuild entirely, so we don't have to port it to python3, but that's an issue for another bug.)

> Certainly, moving to python3 seems a job we will have to do at some point.
> But I don't see a hard deadline in 2020 like you seem to do.

I don't use Debian, I use Fedora, and python2 has been orphaned there. It's unclear whether someone will step forward to maintain it, but if not, it might be retired later *this* year. And for our developers using Arch playing with sed to be able to run our scripts, like Adrian and Alicia, the deadline was half a decade ago. And point (1) is *already* urgent for RHEL and SUSE.

Really, we should have ported all our python long, long ago... let's stop delaying. I've volunteered to do this work (eventually), even if it means accomplishing little else for a long time; I'm not asking you to do it yourself.

> Maybe if we wait a bit more we can do this work together with the Mac
> developers. I hope the new MacOS (not sure when its released) will already
> have python3 in the base system.

Ryosuke already said on webkit-dev that this is not going to happen.

[Michael Catanzaro]

(In reply to Carlos Alberto Lopez Perez from comment #1)
> Our bots run Debian, which already has plans to maintain python2 beyond 2020.
> https://lists.debian.org/debian-devel/2018/04/msg00508.html

BTW we don't need to switch all the bots, it'd probably be sufficient to have one python3 bot running test-webkitpy.

[Carlos Alberto Lopez Perez]

(In reply to Michael Catanzaro from comment #2)
> > Maybe if we wait a bit more we can do this work together with the Mac
> > developers. I hope the new MacOS (not sure when its released) will already
> > have python3 in the base system.
> 
> Ryosuke already said on webkit-dev that this is not going to happen.

Really?

[Carlos Alberto Lopez Perez]

(In reply to Carlos Alberto Lopez Perez from comment #4)
> (In reply to Michael Catanzaro from comment #2)
> > > Maybe if we wait a bit more we can do this work together with the Mac
> > > developers. I hope the new MacOS (not sure when its released) will already
> > > have python3 in the base system.
> > 
> > Ryosuke already said on webkit-dev that this is not going to happen.
> 
> Really?

I mean.. he said that this is not going to happen now, ok.

But, my point was "Maybe if we wait " .. until .. " the NEW MacOS ..

And I think he didn't said anything about that.

[Michael Catanzaro]

If there are any plans at Apple to ship python3 in the near future, that would be very good to know, as it could make the task of porting much simpler: we would not have to worry about maintaining python2 support. Based on the comments on webkit-dev, that seems unlikely.

[Carlos Alberto Lopez Perez]

(In reply to Michael Catanzaro from comment #6)
> If there are any plans at Apple to ship python3 in the near future, that
> would be very good to know, as it could make the task of porting much
> simpler: we would not have to worry about maintaining python2 support. Based
> on the comments on webkit-dev, that seems unlikely.

Well, call me naive.. but I don't expect Mac will continue to ship python2 only after 2020.  How are they going to continue doing that? makes zero sense to me

I think the issue here is you are maybe planning too much ahead.  2020 is far from in the future still for me.

Anyway, not my battle.. as always.. patches welcomed :)

[Michael Catanzaro]

Nowadays distros (at least openSUSE and Fedora) are patching WebKit to build without python3, so we should retake this soon. Of the steps listed in my first comment, we only need to fix (1) now. (2-5) will be a lot of work and can come later.

(In reply to Michael Catanzaro from comment #0)
> python2 is going away on Linux, so we need to convert our scripts to be
> compatible with python3. But macOS does not have python2, so we need to be
> careful to maintain python2 compatibility. This means we should keep the
> existing #!/usr/bin/env python shebangs and just test all our scripts to
> ensure they work with /usr/bin/python symlinked to python3

This is no longer allowed for scripts used by the CMake build. We need to ensure all scripts called by the CMake build explicitly use the shebang #!/usr/bin/python3.

#!/usr/bin python is now explicitly banned in Fedora for being ambiguous.

#!/usr/bin/python2 is also banned for being python2.

And #!/usr/bin/env (what we mostly do now) is also banned since it could resolve to something other than the system interpreter.

The only allowed shebang is #!/usr/bin/python3. Of course, this won't work for Apple, nor for FreeBSD.

So for any scripts used by the CMake build that are also used by Apple, we need to either (a) *generate* the script using a CMake-time configure replacement, e.g. from a .py.cmake file, or (b) just remove the shebang and explicitly invoke the python3 interpreter when executing it from CMake. (We can add CMake code to detect whether to use python3 or python2.) Unless Apple magically gains python3 support and it happens to be installed as /usr/bin/python3, I believe those are the only two possible solution.

[Michael Catanzaro]

(In reply to Michael Catanzaro from comment #8)
> So for any scripts used by the CMake build that are also used by Apple, we
> need to either (a) *generate* the script using a CMake-time configure
> replacement, e.g. from a .py.cmake file, or (b) just remove the shebang and
> explicitly invoke the python3 interpreter when executing it from CMake. (We
> can add CMake code to detect whether to use python3 or python2.) Unless
> Apple magically gains python3 support and it happens to be installed as
> /usr/bin/python3, I believe those are the only two possible solution.

Of course that only applies to scripts executed by the CMake build. That obviously won't work for the developer scripts to be modified in (2-5). For those scripts, the best approach is still to use #!/usr/bin/env python like we do now, and just ensure the scripts work with both versions of python, until such time that python3 is available for Apple.

[Michael Catanzaro]

Looking at Tom's Fedora patch, it looks like there are only two scripts remaining to finish (1):

Source/WebInspectorUI/Scripts/copy-user-interface-resources.pl

(Executes a python interpreter)

Tools/gtk/generate-gtkdoc

(Not used by Apple, so we should just require python3 here.)

[Michael Catanzaro]

(In reply to Michael Catanzaro from comment #10)
> Looking at Tom's Fedora patch, it looks like there are only two scripts
> remaining to finish (1):
> 
> Source/WebInspectorUI/Scripts/copy-user-interface-resources.pl
> 
> (Executes a python interpreter)

Maybe no more changes needed here. It does this:

my $python = ($OSNAME =~ /cygwin/) ? "/usr/bin/python" : "python";
// ...
$python = $ENV{"PYTHON"} if defined($ENV{"PYTHON"});

If CMake always sets $PYTHON, then that's sufficient, and the only remaining task is to port generate-gtkdoc.

[Abderrahim Kitouni]

This is what I needed to change for it to build with python3

https://gitlab.gnome.org/GNOME/gnome-build-meta/blob/51f9cf572e93a57de916c42a3b9e66bda934df8c/files/webkitgtk/python3.patch

[Adrian Perez]

(In reply to Abderrahim Kitouni from comment #12)
> This is what I needed to change for it to build with python3
> 
> https://gitlab.gnome.org/GNOME/gnome-build-meta/blob/
> 51f9cf572e93a57de916c42a3b9e66bda934df8c/files/webkitgtk/python3.patch

Abderrahim, would you be so kind to send a patch in a new bug, following
the WebKit contribution guidelines? In particular this would mainly just
need to make the change and write ChangeLog entries, see:

  https://webkit.org/contributing-code/#changelog-files

If you don't have time, would you mind if I take the patch and go through
the process to have it merger in “trunk”? -- with properly crediting you,
of course

[Michael Catanzaro]

Too much effort for a one-line patch. I'll handle it.

[Michael Catanzaro]

Bug #194218

[Michael Catanzaro]

(In reply to Michael Catanzaro from comment #0)
>  (1) Ensure it's possible to build a tarball, using CMake directly. Bug
> #156674
>  (2) Ensure it's possible to build using build-webkit.
>  (3) Ensure the most important developer scripts work (webkit-patch,
> prepare-ChangeLog)
>  (4) Ensure test-webkitpy passes.
>  (5) Port other scripts on an as-needed basis.

So stage (1) is now complete, yay!

[Don Olmstead]

Are we using this as a meta bug for tracking stuff around this?

[Jonathan Bedard]

(In reply to Don Olmstead from comment #17)
> Are we using this as a meta bug for tracking stuff around this?

I think it's unreasonable to actually have patches for every single Python 3 compatibility change list this bug, for now, let's just relate them in bugzilla.

[Michael Catanzaro]

I noticed today I can't use webkit-patch anymore:

$ webkit-patch upload
Unsupported Python version: WebKit only supports 2.7.x, and you're running 3.7.4.

$ python --version
Python 3.7.4

So I'll upload my patch manually I guess.

python is python3 in Fedora 31. The future is now....

[Adrian Perez]

(In reply to Michael Catanzaro from comment #19)
> I noticed today I can't use webkit-patch anymore:
> 
> $ webkit-patch upload
> Unsupported Python version: WebKit only supports 2.7.x, and you're running
> 3.7.4.
> 
> $ python --version
> Python 3.7.4
> 
> So I'll upload my patch manually I guess.
> 
> python is python3 in Fedora 31. The future is now....

Arch Linux has had “python” being 3.x for a long, long time.
Welcome to the club!

The easiest (and more tractable) way of dealing with this is
using a virtualenv. Quick recipe:

  % cd ~/devel/WebKit/
  % virtualenv .pyenv -p $(which python2)
  ...
  % source .pyenv/bin/activate
  % python -V
  Python 2.7.16
  % deactivate
  % python -V
  Python 3.7.4
  %

Done, now “python” is version 2.7 and all the WebKit tooling will
work just fine. Remember to source the “activate” script when needed.
Or let something like https://github.com/inishchith/autoenv automate
that for you =)

[Jonathan Bedard]

I'm going to use this bug as an umbrella bug, linking all Python3 work in WebKit back to this bug. I'll be also linking a bunch of open bugs tracking the conversion of various directories in webkitpy.

[Radar WebKit Bug Importer]

<rdar://problem/57323396>

[Jonathan Bedard]

(In reply to Michael Catanzaro from comment #19)
> I noticed today I can't use webkit-patch anymore:
> 
> $ webkit-patch upload
> Unsupported Python version: WebKit only supports 2.7.x, and you're running
> 3.7.4.
> 
> $ python --version
> Python 3.7.4
> 
> So I'll upload my patch manually I guess.
> 
> python is python3 in Fedora 31. The future is now....

As of r253128, you should be able to use webkit-patch with Python 3. There are probably still some residual compatibility problems, but it mostly works.

[Michael Catanzaro]

Amazing :)

[Jonathan Bedard]

I'm going to close this bug.

test-webkitpy is being run with Python 3 in EWS. There are more things that could be done, and likely will in the future. However, the central request in this bug, to make all our Python 2 scripts Python 3 compatible, has been completed.