Bug 35584 – test-webkitpy: Get it working with Python 2.4

• Status: RESOLVED
• Resolution: WONTFIX
• Component: Tools / Tests

[Chris Jerdonek]

I'd like to do this before fixing the autoinstall patch below to work with Python 2.4:

https://bugs.webkit.org/show_bug.cgi?id=35163

I've already done this, and it wasn't too hard.  The ternary operator and relative imports were the two main things.

Should it be official that our stuff work with Python 2.4 -- or just 2.5?  There is this, for instance:

https://bugs.webkit.org/show_bug.cgi?id=35316#c11

Note that on Macs it takes a tiny bit more work to use Python 2.4, since Snow Leopard doesn't officially support 2.4.  For example, the Python man page on Mac OS X 10.6 says this:

"The current supported versions are 2.5 (provided for backward-compatibility with the Python 2.5 family), 2.6 and 3.0, with the default being 2.6."

[Adam Barth]

We have a bunch of developers still on Leopard.  My understanding is that they all have 2.4.  Can we have our scripts work out-of-the-box on Leopard and Snow Leopard?

[Chris Jerdonek]

(In reply to comment #1)
> We have a bunch of developers still on Leopard.  My understanding is that they
> all have 2.4.  Can we have our scripts work out-of-the-box on Leopard and Snow
> Leopard?

Yes, I think so.

However, after doing some work on this, there were two unit tests that aren't passing with 2.4.  Maybe you can take a look at those after I land a patch for this report.  The two tests are in multicommandtool_unittest and use outputcapture.

Here is another thread with people supporting support for Python 2.4:

https://bugs.webkit.org/show_bug.cgi?id=31533

There is also this report to get the build.webkit.org builders running the python/perl unit tests:

https://bugs.webkit.org/show_bug.cgi?id=33432

I imagine some of those run 2.4.

[Adam Barth]

OutputCapture is Eric's neck of the woods.

[Chris Jerdonek]

Created attachment 49817 [details]
Proposed patch

[Chris Jerdonek]

Created attachment 49827 [details]
Proposed patch 2

Update after rebasing.

[Eric Seidel (no email)]

Tiger is 2.4 I believe (or maybe it doesn't have python?)
Leopard is 2.5 (according to this post: http://mail.python.org/pipermail/pythonmac-sig/2008-August/020361.html)
Snow Leopard is 2.6

[Chris Jerdonek]

Created attachment 49952 [details]
Proposed patch 3

Switched things around so that the version warning will display if an error occurs while importing webkitpy.

Also added code to configure logging in test-webkitpy.

[Eric Seidel (no email)]

Who uses Python 2.4?  What operating systems.

I think we should decide if we want to support 2.4 before we do this.  I think 2.5 (and even more 2.6) have language additions which make code cleaner.

[Dirk Pranke]

(In reply to comment #8)
> Who uses Python 2.4?  What operating systems.
> 
> I think we should decide if we want to support 2.4 before we do this.  I think
> 2.5 (and even more 2.6) have language additions which make code cleaner.

IIRC Mac Tiger and Chromium Win use 2.4 (and apparently the Google Mac default config, although 2.5 is on the machine, not that that should unduly influence us).

[Chris Jerdonek]

(In reply to comment #8)
> Who uses Python 2.4?  What operating systems.

Personally, I don't really know.  I know of these comments though:

https://bugs.webkit.org/show_bug.cgi?id=35316#c11
https://bugs.webkit.org/show_bug.cgi?id=31533#c1
https://bugs.webkit.org/show_bug.cgi?id=35584#c1

I will send an e-mail to webkit-dev asking if anyone is using 2.4.  I don't have a strong opinion on this issue.  I do think it will be worthwhile to clarify the minimum version we support.

> I think we should decide if we want to support 2.4 before we do this.  I think
> 2.5 (and even more 2.6) have language additions which make code cleaner.

[Eric Seidel (no email)]

Google Mac installs do seem to change the default python, yes.

Python versions on Mac:
10.4.11 is Python 2.3.5:
http://opensource.apple.com/source/python/python-16.1.3/

10.5.8 is Python 2.5.1 (I believe by default):
http://opensource.apple.com/source/python/python-30.1.3/

10.6.2 is Python 2.6.1:
http://opensource.apple.com/source/python/python-44/

[Eric Seidel (no email)]

(In reply to comment #10)
> I will send an e-mail to webkit-dev asking if anyone is using 2.4.  I don't
> have a strong opinion on this issue.  I do think it will be worthwhile to
> clarify the minimum version we support.

Thank you.  I agree we need to standardize.  I just don't want to support anything older than we absolutely have to. :)

[Eric Seidel (no email)]

In #webkit, _wms, proton and alexmilowski confirmed those Python versions on 10.5.8 and 10.6.2.  I have not gotten real life confirmation of 2.3.5 on 10.4.11, but given that I can't find any other python source for 10.4.11 I'm quite certain it must be 2.3.5.

With that information, supporting 2.4 wouldn't even get our scripts running on Tiger if that was part of the goal.

[Eric Seidel (no email)]

Comment on attachment 49952 [details]
Proposed patch 3

Based on on-list, and off-list discussions sounds like Chromium is the only 2.4 client and they're in the process of upgrading to 2.6 so we do not need to support 2.4 (and I would argue we should not support 2.4 due to loss of code-cleanliness features).

[Eric Seidel (no email)]

Closing as WONTFIX please re-open if you disagree.

[Dirk Pranke]

I'm not sure where you got the 'in the process of moving to 2.6' from. Nobody in Chromium is currently working on this and there is no ETA.

As long as we don't need to run these scripts on python 2.4, we're fine (and I don't think we do at the moment but it would be nice to be able to run whatever unit tests we do have on 2.4), but at the very least new-run-webkit-tests and the stuff under the layout_test directory needs to continue to run on 2.4 or the Chromium tree will actively break.

[Eric Seidel (no email)]

Perhaps I misinterpreted the thread.

Nirnimesh:
"I would say stick to a common version of python across all platforms. Since py2.5 is present on mac and linux, I was going to upgrade the python in {src|deps}/third_party/ to py2.5. This shouldn't be too involved."

Nicolas:
"Nothing more I think.  We should update it too. (And at the same time try to figure out why python is checked in twice)."

Marc-Antoine:
"So in short, we could remove python24 from the tree and upgrade the
one in depot_tools to 2.6."

You're right.  No one raised their hand.  Seems like there was just general agreement that Chromium windows should move to 2.6.

I guess plus Tony's sentiment:
"That said, I don't think windows chromium should hold back webkit-patch.  It's possible to install a local copy of python (maybe update-webkit should do this on windows?) and have that live along side the python installed by depot_tools."

led me to believe we shouldn't worry about Chromium here.

[Dirk Pranke]

It would be possible for us to use one version of Python for development (for webkit-patch, etc.) and another to actually run the tests and match what the buildbots use. It would probably be awkward, however, and I would prefer not to do it.

You are correct in that Marc-Antoine and Tony both said it would be possible to do these things but neither volunteered to do so, and neither am I (too many other projects on my plate).

I personally don't use webkit-patch from Windows, and so if we update webkit-patch to be 2.5 or higher, you're probably still safe. But, like I said, it would be nice to be writing unit tests for run-webkit-tests, and it would be nice to be able to run those on 2.4. As long as we can do that, I'm happy. If I can't do that, I'm grumpy but okay. If you break run-webkit-tests itself, we have a problem.

Separately, I am concerned in that you are getting at least some feedback that there are people and use cases where 2.5 isn't available, at least right now, and you seem to be deciding that we don't need to worry about it. Have you compared how hard it will be to get this stuff to work in 2.4? Or figured out alternatives that do work in 2.4?

[Eric Seidel (no email)]

I'm *totally* happy to discuss this further.  I don't wish to act unilaterally without concensus here.

However, the only feedback I've received is that Tiger wants 2.3.5 (which I deem unreasonable and mostly irrelevant given the limited future of Tiger) and that Chromium wants 2.4 for the time-being, but at least in that email thread seemed to think they should move Windows to 2.5+ to match mac/linux which already have 2.5 plus.

So yes, we don't want to break 2.4 for the time being.  But I also don't think we really want to disuade people from using conditional expressions, or having to worry about the POpen bugs in 2.4, or avoiding with statements (although we have been mostly avoiding them for now).

I think we should break the version-check bits out of this patch and get those landed, checking for 2.5 for the moment, and look at the changes to make us 2.4 compatible separately.

The current perceived danger is that we could end up breaking new-run-webkit-tests for the Chromium windows bots because those depend on 2.4 at the moment?

[Dirk Pranke]

(In reply to comment #19)
> However, the only feedback I've received is that Tiger wants 2.3.5 (which I
> deem unreasonable and mostly irrelevant given the limited future of Tiger) and
> that Chromium wants 2.4 for the time-being, but at least in that email thread
> seemed to think they should move Windows to 2.5+ to match mac/linux which
> already have 2.5 plus.

Agreed re: Chromium but the fact that there's no ETA and no staffing makes this equivalent to hoping for the Django pony.

> So yes, we don't want to break 2.4 for the time being.  But I also don't think
> we really want to disuade people from using conditional expressions, or having
> to worry about the POpen bugs in 2.4, or avoiding with statements (although we
> have been mostly avoiding them for now).

Having written a fair amount of python code in the past year, I can't say I miss either feature, and the POpen() bugs can be worked around in a straightforward way assuming we're reusing code enough anyway.

> The current perceived danger is that we could end up breaking
> new-run-webkit-tests for the Chromium windows bots because those depend on 2.4
> at the moment?

That is the primary danger. Lesser concerns -- but still important -- are that we'd write tests I can't run on one of my primary platforms, and develop tools that I can't use on one of my primary platforms. I am sympathetic to the Tiger concerns here as well, but I don't have a Tiger box nor access to one, so I am not directly affected :)

I don't have a good sense of how much work it is to get whatever it is you and Chris are trying to do w/ autoinstall and the other third party things to work with 2.4, although frankly the autoinstall approach seems much kludgier than doing something like gclient and DEPS files. I also don't have a sense of how much work it will be to upgrade Chromium Win to 2.5.

It would be nice to get a better sense of these things before we decided for sure that this was the choice we wanted to make.

[Chris Jerdonek]

(In reply to comment #20)
> I don't have a good sense of how much work it is to get whatever it is you and
> Chris are trying to do w/ autoinstall and the other third party things to work
> with 2.4, although frankly the autoinstall approach seems much kludgier than
> doing something like gclient and DEPS files.

Just to be clear, I don't have a strong opinion on this issue since I can work with either in my dev environment (getting things to work is not really the issue in my mind -- it's the below).

The only thing I'd say is that if we are going to support one version of Python for one part of the code and another for another part of the code, I'd like for us to consider having some kind of clear-cut boundary.  For example--

webkitpy/
    python24/
        <Python 2.4 modules>
    python25/
        <Python 2.5 modules>

Or at least initially--

webkitpy/
    python24/
        <Python 2.4 modules>
    <Python 2.5 modules>

But that may take a fair amount of time to set up and so shouldn't be entered into lightly.

[Chris Jerdonek]

(In reply to comment #21)
> The only thing I'd say is that if we are going to support one version of Python
> for one part of the code and another for another part of the code, I'd like for
> us to consider having some kind of clear-cut boundary.  For example--
> 
> webkitpy/
>     python24/
>         <Python 2.4 modules>
>     python25/
>         <Python 2.5 modules>

Just to clarify, I don't mean to suggest we'd have two versions of any code.  It's simply that if we have certain code, it would go into one of the two bins.  Code in python25 would be able to import from both, but python24 could only import from itself.

[Chris Jerdonek]

A couple questions:

What other WebKit scripts do the Chromium Windows bots use aside from new-run-webkit-tests (non-Python scripts included)?

What are the possible options on the table right now?  I see at least--

(1) All Python 2.4
(2) All Python 2.5
(3) new-run-webkit-tests 2.4, everything else 2.5

Have we ruled out (2) because of the bots?

I think it would be helpful if our decision makes it so that we don't need to revisit this issue for each script we'd like to write in Python (including any scripts currently in Perl).  For that reason, options (1) and (2) seem preferable.