|
Description
Michael Catanzaro
2018-04-25 11:34:54 PDT
(In reply to Michael Catanzaro from comment #0) > (4) At this point, we should be able to switch our Linux bots to using > python3. This will be a longer-term effort, but we should still try to > finish in advance of 2020. > Our bots run Debian, which already has plans to maintain python2 beyond 2020. https://lists.debian.org/debian-devel/2018/04/msg00508.html On top of that there is always the possibility of building python2 on the JHBuild so developers using more bleeding edge distros can still keep using it for WebKit. Certainly, moving to python3 seems a job we will have to do at some point. But I don't see a hard deadline in 2020 like you seem to do. Maybe if we wait a bit more we can do this work together with the Mac developers. I hope the new MacOS (not sure when its released) will already have python3 in the base system. (In reply to Carlos Alberto Lopez Perez from comment #1) > (In reply to Michael Catanzaro from comment #0) > > (4) At this point, we should be able to switch our Linux bots to using > > python3. This will be a longer-term effort, but we should still try to > > finish in advance of 2020. > > > > Our bots run Debian, which already has plans to maintain python2 beyond 2020. > https://lists.debian.org/debian-devel/2018/04/msg00508.html > > On top of that there is always the possibility of building python2 on the > JHBuild so developers using more bleeding edge distros can still keep using > it for WebKit. I *think* it would be possible. We'll have to first port jhbuild itself to python3, of course, because jhbuild is not itself going to be able to use the python built in the jhbuild environment. But we'd have to do that anyway. (My plan is actually to stop using jhbuild entirely, so we don't have to port it to python3, but that's an issue for another bug.) > Certainly, moving to python3 seems a job we will have to do at some point. > But I don't see a hard deadline in 2020 like you seem to do. I don't use Debian, I use Fedora, and python2 has been orphaned there. It's unclear whether someone will step forward to maintain it, but if not, it might be retired later *this* year. And for our developers using Arch playing with sed to be able to run our scripts, like Adrian and Alicia, the deadline was half a decade ago. And point (1) is *already* urgent for RHEL and SUSE. Really, we should have ported all our python long, long ago... let's stop delaying. I've volunteered to do this work (eventually), even if it means accomplishing little else for a long time; I'm not asking you to do it yourself. > Maybe if we wait a bit more we can do this work together with the Mac > developers. I hope the new MacOS (not sure when its released) will already > have python3 in the base system. Ryosuke already said on webkit-dev that this is not going to happen. (In reply to Carlos Alberto Lopez Perez from comment #1) > Our bots run Debian, which already has plans to maintain python2 beyond 2020. > https://lists.debian.org/debian-devel/2018/04/msg00508.html BTW we don't need to switch all the bots, it'd probably be sufficient to have one python3 bot running test-webkitpy. (In reply to Michael Catanzaro from comment #2) > > Maybe if we wait a bit more we can do this work together with the Mac > > developers. I hope the new MacOS (not sure when its released) will already > > have python3 in the base system. > > Ryosuke already said on webkit-dev that this is not going to happen. Really? (In reply to Carlos Alberto Lopez Perez from comment #4) > (In reply to Michael Catanzaro from comment #2) > > > Maybe if we wait a bit more we can do this work together with the Mac > > > developers. I hope the new MacOS (not sure when its released) will already > > > have python3 in the base system. > > > > Ryosuke already said on webkit-dev that this is not going to happen. > > Really? I mean.. he said that this is not going to happen now, ok. But, my point was "Maybe if we wait " .. until .. " the NEW MacOS .. And I think he didn't said anything about that. If there are any plans at Apple to ship python3 in the near future, that would be very good to know, as it could make the task of porting much simpler: we would not have to worry about maintaining python2 support. Based on the comments on webkit-dev, that seems unlikely. (In reply to Michael Catanzaro from comment #6) > If there are any plans at Apple to ship python3 in the near future, that > would be very good to know, as it could make the task of porting much > simpler: we would not have to worry about maintaining python2 support. Based > on the comments on webkit-dev, that seems unlikely. Well, call me naive.. but I don't expect Mac will continue to ship python2 only after 2020. How are they going to continue doing that? makes zero sense to me I think the issue here is you are maybe planning too much ahead. 2020 is far from in the future still for me. Anyway, not my battle.. as always.. patches welcomed :) Nowadays distros (at least openSUSE and Fedora) are patching WebKit to build without python3, so we should retake this soon. Of the steps listed in my first comment, we only need to fix (1) now. (2-5) will be a lot of work and can come later. (In reply to Michael Catanzaro from comment #0) > python2 is going away on Linux, so we need to convert our scripts to be > compatible with python3. But macOS does not have python2, so we need to be > careful to maintain python2 compatibility. This means we should keep the > existing #!/usr/bin/env python shebangs and just test all our scripts to > ensure they work with /usr/bin/python symlinked to python3 This is no longer allowed for scripts used by the CMake build. We need to ensure all scripts called by the CMake build explicitly use the shebang #!/usr/bin/python3. #!/usr/bin python is now explicitly banned in Fedora for being ambiguous. #!/usr/bin/python2 is also banned for being python2. And #!/usr/bin/env (what we mostly do now) is also banned since it could resolve to something other than the system interpreter. The only allowed shebang is #!/usr/bin/python3. Of course, this won't work for Apple, nor for FreeBSD. So for any scripts used by the CMake build that are also used by Apple, we need to either (a) *generate* the script using a CMake-time configure replacement, e.g. from a .py.cmake file, or (b) just remove the shebang and explicitly invoke the python3 interpreter when executing it from CMake. (We can add CMake code to detect whether to use python3 or python2.) Unless Apple magically gains python3 support and it happens to be installed as /usr/bin/python3, I believe those are the only two possible solution. (In reply to Michael Catanzaro from comment #8) > So for any scripts used by the CMake build that are also used by Apple, we > need to either (a) *generate* the script using a CMake-time configure > replacement, e.g. from a .py.cmake file, or (b) just remove the shebang and > explicitly invoke the python3 interpreter when executing it from CMake. (We > can add CMake code to detect whether to use python3 or python2.) Unless > Apple magically gains python3 support and it happens to be installed as > /usr/bin/python3, I believe those are the only two possible solution. Of course that only applies to scripts executed by the CMake build. That obviously won't work for the developer scripts to be modified in (2-5). For those scripts, the best approach is still to use #!/usr/bin/env python like we do now, and just ensure the scripts work with both versions of python, until such time that python3 is available for Apple. Looking at Tom's Fedora patch, it looks like there are only two scripts remaining to finish (1): Source/WebInspectorUI/Scripts/copy-user-interface-resources.pl (Executes a python interpreter) Tools/gtk/generate-gtkdoc (Not used by Apple, so we should just require python3 here.) (In reply to Michael Catanzaro from comment #10) > Looking at Tom's Fedora patch, it looks like there are only two scripts > remaining to finish (1): > > Source/WebInspectorUI/Scripts/copy-user-interface-resources.pl > > (Executes a python interpreter) Maybe no more changes needed here. It does this: my $python = ($OSNAME =~ /cygwin/) ? "/usr/bin/python" : "python"; // ... $python = $ENV{"PYTHON"} if defined($ENV{"PYTHON"}); If CMake always sets $PYTHON, then that's sufficient, and the only remaining task is to port generate-gtkdoc. This is what I needed to change for it to build with python3 https://gitlab.gnome.org/GNOME/gnome-build-meta/blob/51f9cf572e93a57de916c42a3b9e66bda934df8c/files/webkitgtk/python3.patch (In reply to Abderrahim Kitouni from comment #12) > This is what I needed to change for it to build with python3 > > https://gitlab.gnome.org/GNOME/gnome-build-meta/blob/ > 51f9cf572e93a57de916c42a3b9e66bda934df8c/files/webkitgtk/python3.patch Abderrahim, would you be so kind to send a patch in a new bug, following the WebKit contribution guidelines? In particular this would mainly just need to make the change and write ChangeLog entries, see: https://webkit.org/contributing-code/#changelog-files If you don't have time, would you mind if I take the patch and go through the process to have it merger in “trunk”? -- with properly crediting you, of course Too much effort for a one-line patch. I'll handle it. (In reply to Michael Catanzaro from comment #0) > (1) Ensure it's possible to build a tarball, using CMake directly. Bug > #156674 > (2) Ensure it's possible to build using build-webkit. > (3) Ensure the most important developer scripts work (webkit-patch, > prepare-ChangeLog) > (4) Ensure test-webkitpy passes. > (5) Port other scripts on an as-needed basis. So stage (1) is now complete, yay! Are we using this as a meta bug for tracking stuff around this? (In reply to Don Olmstead from comment #17) > Are we using this as a meta bug for tracking stuff around this? I think it's unreasonable to actually have patches for every single Python 3 compatibility change list this bug, for now, let's just relate them in bugzilla. I noticed today I can't use webkit-patch anymore: $ webkit-patch upload Unsupported Python version: WebKit only supports 2.7.x, and you're running 3.7.4. $ python --version Python 3.7.4 So I'll upload my patch manually I guess. python is python3 in Fedora 31. The future is now.... (In reply to Michael Catanzaro from comment #19) > I noticed today I can't use webkit-patch anymore: > > $ webkit-patch upload > Unsupported Python version: WebKit only supports 2.7.x, and you're running > 3.7.4. > > $ python --version > Python 3.7.4 > > So I'll upload my patch manually I guess. > > python is python3 in Fedora 31. The future is now.... Arch Linux has had “python” being 3.x for a long, long time. Welcome to the club! The easiest (and more tractable) way of dealing with this is using a virtualenv. Quick recipe: % cd ~/devel/WebKit/ % virtualenv .pyenv -p $(which python2) ... % source .pyenv/bin/activate % python -V Python 2.7.16 % deactivate % python -V Python 3.7.4 % Done, now “python” is version 2.7 and all the WebKit tooling will work just fine. Remember to source the “activate” script when needed. Or let something like https://github.com/inishchith/autoenv automate that for you =) I'm going to use this bug as an umbrella bug, linking all Python3 work in WebKit back to this bug. I'll be also linking a bunch of open bugs tracking the conversion of various directories in webkitpy. (In reply to Michael Catanzaro from comment #19) > I noticed today I can't use webkit-patch anymore: > > $ webkit-patch upload > Unsupported Python version: WebKit only supports 2.7.x, and you're running > 3.7.4. > > $ python --version > Python 3.7.4 > > So I'll upload my patch manually I guess. > > python is python3 in Fedora 31. The future is now.... As of r253128, you should be able to use webkit-patch with Python 3. There are probably still some residual compatibility problems, but it mostly works. Amazing :) I'm going to close this bug. test-webkitpy is being run with Python 3 in EWS. There are more things that could be done, and likely will in the future. However, the central request in this bug, to make all our Python 2 scripts Python 3 compatible, has been completed. |