Bug 184986 - Make all python scripts compatible with both python2 and python3
Summary: Make all python scripts compatible with both python2 and python3
Status: NEW
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: WebKit Nightly Build
Hardware: PC All
: P2 Normal
Assignee: Nobody
URL: https://lists.webkit.org/pipermail/we...
Keywords:
Depends on: 156674 194218
Blocks: 192293
  Show dependency treegraph
 
Reported: 2018-04-25 11:34 PDT by Michael Catanzaro
Modified: 2019-09-18 17:03 PDT (History)
15 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Michael Catanzaro 2018-04-25 11:34:54 PDT
python2 is going away on Linux, so we need to convert our scripts to be compatible with python3. But macOS does not have python2, so we need to be careful to maintain python2 compatibility. This means we should keep the existing #!/usr/bin/env python shebangs and just test all our scripts to ensure they work with /usr/bin/python symlinked to python3.

Let's do this in five stages, using separate bugs:

 (1) Ensure it's possible to build a tarball, using CMake directly. Bug #156674
 (2) Ensure it's possible to build using build-webkit.
 (3) Ensure the most important developer scripts work (webkit-patch, prepare-ChangeLog)
 (4) Ensure test-webkitpy passes.
 (5) Port other scripts on an as-needed basis.

(1) Is the most urgent, as python3 patches are already required to build WebKit on the next SUSE and RHEL distros. This should be pretty easy.

(2) Is where things will get difficult.

(4) At this point, we should be able to switch our Linux bots to using python3. This will be a longer-term effort, but we should still try to finish in advance of 2020.

(5) is optional work that we can do as-needed. We don't need to port anything that's not tested by webkitpy and which we are not actively using ourselves.

Somebody pointed me towards https://portingguide.readthedocs.io/en/latest/, which I haven't read yet, but which might be helpful.

Also, Konstantin suggested run 'pylint --py3k' in the style checker, to make it a bit less likely that developers using python2 break developers using python3. (It's inevitable, but we're going to be stuck with that until macOS gets python3.) Would be nice if we could find a way to check for python3-isms as well, to prevent developers using python3 from breaking developers using python2 (which is also inevitable).
Comment 1 Carlos Alberto Lopez Perez 2018-04-26 04:16:36 PDT
(In reply to Michael Catanzaro from comment #0)
> (4) At this point, we should be able to switch our Linux bots to using
> python3. This will be a longer-term effort, but we should still try to
> finish in advance of 2020.
> 

Our bots run Debian, which already has plans to maintain python2 beyond 2020.
https://lists.debian.org/debian-devel/2018/04/msg00508.html

On top of that there is always the possibility of building python2 on the JHBuild so developers using more bleeding edge distros can still keep using it for WebKit.

Certainly, moving to python3 seems a job we will have to do at some point.
But I don't see a hard deadline in 2020 like you seem to do.

Maybe if we wait a bit more we can do this work together with the Mac developers. I hope the new MacOS (not sure when its released) will already have python3 in the base system.
Comment 2 Michael Catanzaro 2018-04-26 09:33:20 PDT
(In reply to Carlos Alberto Lopez Perez from comment #1)
> (In reply to Michael Catanzaro from comment #0)
> > (4) At this point, we should be able to switch our Linux bots to using
> > python3. This will be a longer-term effort, but we should still try to
> > finish in advance of 2020.
> > 
> 
> Our bots run Debian, which already has plans to maintain python2 beyond 2020.
> https://lists.debian.org/debian-devel/2018/04/msg00508.html
> 
> On top of that there is always the possibility of building python2 on the
> JHBuild so developers using more bleeding edge distros can still keep using
> it for WebKit.

I *think* it would be possible. We'll have to first port jhbuild itself to python3, of course, because jhbuild is not itself going to be able to use the python built in the jhbuild environment. But we'd have to do that anyway. (My plan is actually to stop using jhbuild entirely, so we don't have to port it to python3, but that's an issue for another bug.)

> Certainly, moving to python3 seems a job we will have to do at some point.
> But I don't see a hard deadline in 2020 like you seem to do.

I don't use Debian, I use Fedora, and python2 has been orphaned there. It's unclear whether someone will step forward to maintain it, but if not, it might be retired later *this* year. And for our developers using Arch playing with sed to be able to run our scripts, like Adrian and Alicia, the deadline was half a decade ago. And point (1) is *already* urgent for RHEL and SUSE.

Really, we should have ported all our python long, long ago... let's stop delaying. I've volunteered to do this work (eventually), even if it means accomplishing little else for a long time; I'm not asking you to do it yourself.

> Maybe if we wait a bit more we can do this work together with the Mac
> developers. I hope the new MacOS (not sure when its released) will already
> have python3 in the base system.

Ryosuke already said on webkit-dev that this is not going to happen.
Comment 3 Michael Catanzaro 2018-04-26 09:38:18 PDT
(In reply to Carlos Alberto Lopez Perez from comment #1)
> Our bots run Debian, which already has plans to maintain python2 beyond 2020.
> https://lists.debian.org/debian-devel/2018/04/msg00508.html

BTW we don't need to switch all the bots, it'd probably be sufficient to have one python3 bot running test-webkitpy.
Comment 4 Carlos Alberto Lopez Perez 2018-04-26 09:53:36 PDT
(In reply to Michael Catanzaro from comment #2)
> > Maybe if we wait a bit more we can do this work together with the Mac
> > developers. I hope the new MacOS (not sure when its released) will already
> > have python3 in the base system.
> 
> Ryosuke already said on webkit-dev that this is not going to happen.

Really?
Comment 5 Carlos Alberto Lopez Perez 2018-04-26 09:55:56 PDT
(In reply to Carlos Alberto Lopez Perez from comment #4)
> (In reply to Michael Catanzaro from comment #2)
> > > Maybe if we wait a bit more we can do this work together with the Mac
> > > developers. I hope the new MacOS (not sure when its released) will already
> > > have python3 in the base system.
> > 
> > Ryosuke already said on webkit-dev that this is not going to happen.
> 
> Really?

I mean.. he said that this is not going to happen now, ok.

But, my point was "Maybe if we wait " .. until .. " the NEW MacOS ..

And I think he didn't said anything about that.
Comment 6 Michael Catanzaro 2018-04-26 09:59:27 PDT
If there are any plans at Apple to ship python3 in the near future, that would be very good to know, as it could make the task of porting much simpler: we would not have to worry about maintaining python2 support. Based on the comments on webkit-dev, that seems unlikely.
Comment 7 Carlos Alberto Lopez Perez 2018-04-26 10:03:20 PDT
(In reply to Michael Catanzaro from comment #6)
> If there are any plans at Apple to ship python3 in the near future, that
> would be very good to know, as it could make the task of porting much
> simpler: we would not have to worry about maintaining python2 support. Based
> on the comments on webkit-dev, that seems unlikely.

Well, call me naive.. but I don't expect Mac will continue to ship python2 only after 2020.  How are they going to continue doing that? makes zero sense to me

I think the issue here is you are maybe planning too much ahead.  2020 is far from in the future still for me.

Anyway, not my battle.. as always.. patches welcomed :)
Comment 8 Michael Catanzaro 2018-12-03 09:54:06 PST
Nowadays distros (at least openSUSE and Fedora) are patching WebKit to build without python3, so we should retake this soon. Of the steps listed in my first comment, we only need to fix (1) now. (2-5) will be a lot of work and can come later.

(In reply to Michael Catanzaro from comment #0)
> python2 is going away on Linux, so we need to convert our scripts to be
> compatible with python3. But macOS does not have python2, so we need to be
> careful to maintain python2 compatibility. This means we should keep the
> existing #!/usr/bin/env python shebangs and just test all our scripts to
> ensure they work with /usr/bin/python symlinked to python3

This is no longer allowed for scripts used by the CMake build. We need to ensure all scripts called by the CMake build explicitly use the shebang #!/usr/bin/python3.

#!/usr/bin python is now explicitly banned in Fedora for being ambiguous.

#!/usr/bin/python2 is also banned for being python2.

And #!/usr/bin/env (what we mostly do now) is also banned since it could resolve to something other than the system interpreter.

The only allowed shebang is #!/usr/bin/python3. Of course, this won't work for Apple, nor for FreeBSD.

So for any scripts used by the CMake build that are also used by Apple, we need to either (a) *generate* the script using a CMake-time configure replacement, e.g. from a .py.cmake file, or (b) just remove the shebang and explicitly invoke the python3 interpreter when executing it from CMake. (We can add CMake code to detect whether to use python3 or python2.) Unless Apple magically gains python3 support and it happens to be installed as /usr/bin/python3, I believe those are the only two possible solution.
Comment 9 Michael Catanzaro 2018-12-03 09:56:30 PST
(In reply to Michael Catanzaro from comment #8)
> So for any scripts used by the CMake build that are also used by Apple, we
> need to either (a) *generate* the script using a CMake-time configure
> replacement, e.g. from a .py.cmake file, or (b) just remove the shebang and
> explicitly invoke the python3 interpreter when executing it from CMake. (We
> can add CMake code to detect whether to use python3 or python2.) Unless
> Apple magically gains python3 support and it happens to be installed as
> /usr/bin/python3, I believe those are the only two possible solution.

Of course that only applies to scripts executed by the CMake build. That obviously won't work for the developer scripts to be modified in (2-5). For those scripts, the best approach is still to use #!/usr/bin/env python like we do now, and just ensure the scripts work with both versions of python, until such time that python3 is available for Apple.
Comment 10 Michael Catanzaro 2018-12-03 11:44:18 PST
Looking at Tom's Fedora patch, it looks like there are only two scripts remaining to finish (1):

Source/WebInspectorUI/Scripts/copy-user-interface-resources.pl

(Executes a python interpreter)

Tools/gtk/generate-gtkdoc

(Not used by Apple, so we should just require python3 here.)
Comment 11 Michael Catanzaro 2019-01-17 07:51:38 PST
(In reply to Michael Catanzaro from comment #10)
> Looking at Tom's Fedora patch, it looks like there are only two scripts
> remaining to finish (1):
> 
> Source/WebInspectorUI/Scripts/copy-user-interface-resources.pl
> 
> (Executes a python interpreter)

Maybe no more changes needed here. It does this:

my $python = ($OSNAME =~ /cygwin/) ? "/usr/bin/python" : "python";
// ...
$python = $ENV{"PYTHON"} if defined($ENV{"PYTHON"});

If CMake always sets $PYTHON, then that's sufficient, and the only remaining task is to port generate-gtkdoc.
Comment 12 Abderrahim Kitouni 2019-01-31 12:34:52 PST
This is what I needed to change for it to build with python3

https://gitlab.gnome.org/GNOME/gnome-build-meta/blob/51f9cf572e93a57de916c42a3b9e66bda934df8c/files/webkitgtk/python3.patch
Comment 13 Adrian Perez 2019-02-04 02:06:11 PST
(In reply to Abderrahim Kitouni from comment #12)
> This is what I needed to change for it to build with python3
> 
> https://gitlab.gnome.org/GNOME/gnome-build-meta/blob/
> 51f9cf572e93a57de916c42a3b9e66bda934df8c/files/webkitgtk/python3.patch

Abderrahim, would you be so kind to send a patch in a new bug, following
the WebKit contribution guidelines? In particular this would mainly just
need to make the change and write ChangeLog entries, see:

  https://webkit.org/contributing-code/#changelog-files

If you don't have time, would you mind if I take the patch and go through
the process to have it merger in “trunk”? -- with properly crediting you,
of course
Comment 14 Michael Catanzaro 2019-02-04 06:37:59 PST
Too much effort for a one-line patch. I'll handle it.
Comment 15 Michael Catanzaro 2019-02-04 06:40:32 PST
Bug #194218
Comment 16 Michael Catanzaro 2019-02-04 07:07:19 PST
(In reply to Michael Catanzaro from comment #0)
>  (1) Ensure it's possible to build a tarball, using CMake directly. Bug
> #156674
>  (2) Ensure it's possible to build using build-webkit.
>  (3) Ensure the most important developer scripts work (webkit-patch,
> prepare-ChangeLog)
>  (4) Ensure test-webkitpy passes.
>  (5) Port other scripts on an as-needed basis.

So stage (1) is now complete, yay!
Comment 17 Don Olmstead 2019-08-26 10:31:47 PDT
Are we using this as a meta bug for tracking stuff around this?
Comment 18 Jonathan Bedard 2019-08-26 10:38:52 PDT
(In reply to Don Olmstead from comment #17)
> Are we using this as a meta bug for tracking stuff around this?

I think it's unreasonable to actually have patches for every single Python 3 compatibility change list this bug, for now, let's just relate them in bugzilla.