Bug 141946

Summary: setup-git-clone fails due to the failure to download python-irclib-0.4.8
Product: WebKit Reporter: Ryosuke Niwa <rniwa>
Component: Tools / TestsAssignee: Ryosuke Niwa <rniwa>
Status: RESOLVED FIXED    
Severity: Normal CC: ap, benjamin, commit-queue, darin, ddkilzer, glenn, kling, lforschler, ossy, rniwa
Priority: P2    
Version: 528+ (Nightly build)   
Hardware: Unspecified   
OS: Unspecified   
See Also: https://bugs.webkit.org/show_bug.cgi?id=142400
Attachments:
Description Flags
Fixes the bug
none
Patch
none
Patch none

Description Ryosuke Niwa 2015-02-23 19:22:50 PST
We can't use this command due to an autoinstall failure.
Comment 1 Ryosuke Niwa 2015-02-23 19:23:54 PST
Created attachment 247187 [details]
Fixes the bug
Comment 2 Csaba Osztrogonác 2015-02-24 05:02:52 PST
Comment on attachment 247187 [details]
Fixes the bug

I don't agree with modifying urls on each short network error.

You can also use SOURCEFORGE_MIRRORS and LOCAL_AUTOINSTALL_CACHE
environment variables to avoid broken webkit-patch on network error.
(See autoinstall.py for details.)

Of course it is possible to improce this mirror and local cache mechanism.
Comment 3 Darin Adler 2015-02-24 19:09:37 PST
(In reply to comment #2)
> I don't agree with modifying urls on each short network error.

OK, but long term is there some reason to prefer sourceforge.net over launchpad.net?
Comment 4 Csaba Osztrogonác 2015-02-25 03:11:45 PST
(In reply to comment #3)
> (In reply to comment #2)
> > I don't agree with modifying urls on each short network error.
> 
> OK, but long term is there some reason to prefer sourceforge.net over
> launchpad.net?

I don't prefer sourceforge or launchpad. But if we 
change, we should change the mirroring mechanism too.

But is there any reason why can't we check in irclib.py, 
ircbot.py and eliza.py instead of autoinstalling? 
In this case we would depend only on pypi.
Comment 5 Ryosuke Niwa 2015-02-25 03:16:44 PST
(In reply to comment #4)
>
> But is there any reason why can't we check in irclib.py, 
> ircbot.py and eliza.py instead of autoinstalling? 
> In this case we would depend only on pypi.

I suppose we can since the licensing is MIT. It would take a while to do it because I have to go through the corporate approval process if I were to do it though :(
Comment 6 Csaba Osztrogonác 2015-02-25 03:21:33 PST
irclib.py and ircbot.py have LGPL license,
but I can't find any license for the eliza.py.
Comment 7 Ryosuke Niwa 2015-02-25 03:24:43 PST
(In reply to comment #6)
> irclib.py and ircbot.py have LGPL license,
> but I can't find any license for the eliza.py.

Oh, no :(
Comment 8 Csaba Osztrogonác 2015-02-25 04:05:27 PST
Created attachment 247329 [details]
Patch

Let's check in ircbot.py and irclib.py and don't touch eliza
Comment 9 WebKit Commit Bot 2015-02-25 04:07:30 PST
Attachment 247329 [details] did not pass style-queue:

Traceback (most recent call last):
  File "/Volumes/Data/StyleQueue/Webkit/Tools/Scripts/webkit-patch", line 44, in <module>
    from webkitpy.tool.main import WebKitPatch
  File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/tool/main.py", line 38, in <module>
    from webkitpy.common.net.irc import ircproxy
  File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/common/net/irc/ircproxy.py", line 32, in <module>
    from webkitpy.common.net.irc.ircbot import IRCBot
  File "/Volumes/Data/StyleQueue/WebKit/Tools/Scripts/webkitpy/common/net/irc/ircbot.py", line 32, in <module>
    from webkitpy.thirdparty.irc import ircbot
ImportError: No module named irc


If any of these errors are false positives, please file a bug against check-webkit-style.
Comment 10 Csaba Osztrogonác 2015-02-25 04:10:06 PST
Comment on attachment 247329 [details]
Patch

remove flags to unbreak EWS bots
Comment 11 Csaba Osztrogonác 2015-02-25 04:24:03 PST
Created attachment 247330 [details]
Patch

Let's see if EWS bots like non empty __init__.py
Comment 12 Csaba Osztrogonác 2015-02-25 04:28:13 PST
Comment on attachment 247330 [details]
Patch

EWS is crazy ... it can't handle adding new module.
Comment 13 Csaba Osztrogonác 2015-03-05 01:53:35 PST
Any objection against check in ircbot.py and irclib.py ?
Comment 14 WebKit Commit Bot 2015-03-06 01:23:43 PST
Comment on attachment 247330 [details]
Patch

Clearing flags on attachment: 247330

Committed r181145: <http://trac.webkit.org/changeset/181145>
Comment 15 WebKit Commit Bot 2015-03-06 01:23:49 PST
All reviewed patches have been landed.  Closing bug.
Comment 16 David Kilzer (:ddkilzer) 2015-03-06 07:57:59 PST
(In reply to comment #13)
> Any objection against check in ircbot.py and irclib.py ?

This is causing failures on test bots, not just EWS:

<https://build.webkit.org/builders/Apple%20Mavericks%20Release%20WK2%20(Tests)?numbuilds=50>

Is that because we're trying to replace a locally downloaded file with one from the svn repository?
Comment 17 Csaba Osztrogonác 2015-03-06 07:59:22 PST
(In reply to comment #16)
> (In reply to comment #13)
> > Any objection against check in ircbot.py and irclib.py ?
> 
> This is causing failures on test bots, not just EWS:
> 
> <https://build.webkit.org/builders/
> Apple%20Mavericks%20Release%20WK2%20(Tests)?numbuilds=50>
> 
> Is that because we're trying to replace a locally downloaded file with one
> from the svn repository?

Ihanks for noticing, I'll investigate it right now.
Comment 18 David Kilzer (:ddkilzer) 2015-03-06 08:01:01 PST
(In reply to comment #17)
> (In reply to comment #16)
> > (In reply to comment #13)
> > > Any objection against check in ircbot.py and irclib.py ?
> > 
> > This is causing failures on test bots, not just EWS:
> > 
> > <https://build.webkit.org/builders/
> > Apple%20Mavericks%20Release%20WK2%20(Tests)?numbuilds=50>
> > 
> > Is that because we're trying to replace a locally downloaded file with one
> > from the svn repository?
> 
> Ihanks for noticing, I'll investigate it right now.

Or maybe we just need to blow away all *.pyc files before running the tests on the bots?
Comment 19 Csaba Osztrogonác 2015-03-06 08:15:43 PST
I have no idea what's happening ... I can't reproduce this issue on Mavericks.
I tried to run tests on 181144 and then on 181145 and again on 181144, ...
and didn't get any failure.

But something is very strange with the Mac bots:
https://build.webkit.org/buildslaves/bot199 - good revision order, no issue
https://build.webkit.org/buildslaves/bot151 - 181145, 181143, 181149 - BANG!
Comment 20 Csaba Osztrogonác 2015-03-06 08:25:26 PST
Something is very buggy with these bots ...

https://build.webkit.org/builders/Apple%20Mavericks%20Release%20WK1%20%28Tests%29/builds/12738/steps/svn/logs/stdio

   C Tools/Scripts/webkitpy/thirdparty/irc ----> what???????
   A Tools/Scripts/webkitpy/thirdparty/irc/__init__.py
   A Tools/Scripts/webkitpy/thirdparty/irc/ircbot.py
   A Tools/Scripts/webkitpy/thirdparty/irc/irclib.py

Let me check it with pure svn checkout, I only checked in git.
Comment 21 Csaba Osztrogonác 2015-03-06 08:31:26 PST
It would be great if somebody has access to these bots 
could check what is happening?
Comment 22 Csaba Osztrogonác 2015-03-06 08:38:37 PST
I kicked the buggy mavericks bots, let's see the results.
( force build + stop during svn update to remove everything + force build )
Comment 23 David Kilzer (:ddkilzer) 2015-03-06 09:19:25 PST
(In reply to comment #21)
> It would be great if somebody has access to these bots 
> could check what is happening?

This change and Bug 142083 has left bots in this state:

$ svn stat .
A  +  C LayoutTests/imported/w3c/web-platform-tests/_certs
      >   local edit, incoming delete upon update
C  +    LayoutTests/imported/w3c/web-platform-tests/_certs/01.pem
C  +    LayoutTests/imported/w3c/web-platform-tests/_certs/02.pem
C  +    LayoutTests/imported/w3c/web-platform-tests/_certs/cacert.pem
C  +    LayoutTests/imported/w3c/web-platform-tests/_certs/cakey.pem
C  +    LayoutTests/imported/w3c/web-platform-tests/_certs/index.txt.old
C  +    LayoutTests/imported/w3c/web-platform-tests/_certs/localhost.key
C  +    LayoutTests/imported/w3c/web-platform-tests/_certs/localhost.pem
D     C LayoutTests/imported/w3c/web-platform-tests/tools/scripts/__init__.py
      >   local unversioned, incoming add upon update
D     C LayoutTests/imported/w3c/web-platform-tests/tools/webdriver/webdriver/__init__.py
      >   local unversioned, incoming add upon update
D     C Tools/Scripts/webkitpy/thirdparty/irc
      >   local unversioned, incoming add upon update
D       Tools/Scripts/webkitpy/thirdparty/irc/__init__.py
D       Tools/Scripts/webkitpy/thirdparty/irc/ircbot.py
D       Tools/Scripts/webkitpy/thirdparty/irc/irclib.py
?       layout-test-results.zip
Summary of conflicts:
  Text conflicts: 7
  Tree conflicts: 4
Comment 24 Csaba Osztrogonác 2015-03-06 09:28:36 PST
filed a new bug report to handle this kind of issues in the future - bug142400
Comment 25 David Kilzer (:ddkilzer) 2015-03-06 09:35:11 PST
(In reply to comment #24)
> filed a new bug report to handle this kind of issues in the future -
> bug142400

Log of attempted cleanup of one bot.

svn-resolve doesn't do what you'd hope:

$ svn resolve --accept base --recursive Tools/Scripts/webkitpy/thirdparty
svn: warning: W155027: Tree conflicts can only be resolved to 'working' state; '/Volumes/Data/slave/gala-production-wk2-tests/build/OpenSource/Tools/Scripts/webkitpy/thirdparty/irc' not resolved
bot172:OpenSource buildbot$ svn stat .
D     C Tools/Scripts/webkitpy/thirdparty/irc
      >   local unversioned, incoming add upon update
D       Tools/Scripts/webkitpy/thirdparty/irc/__init__.py
D       Tools/Scripts/webkitpy/thirdparty/irc/ircbot.py
D       Tools/Scripts/webkitpy/thirdparty/irc/irclib.py
?       layout-test-results.zip
?       local
Summary of conflicts:
  Tree conflicts: 1

So you have to use svn-revert:

$ svn revert --recursive Tools/Scripts/webkitpy/thirdparty/
Reverted 'Tools/Scripts/webkitpy/thirdparty/irc'
Reverted 'Tools/Scripts/webkitpy/thirdparty/irc/__init__.py'
Reverted 'Tools/Scripts/webkitpy/thirdparty/irc/ircbot.py'
Reverted 'Tools/Scripts/webkitpy/thirdparty/irc/irclib.py'
bot172:OpenSource buildbot$ svn stat .
?       layout-test-results.zip
?       local

After that, you're done.