Bug 97647 – A buggy python patch can kill all EWS bots

• Component: New Bugs
• Status: NEW

[Csaba Osztrogonác]

https://bugs.webkit.org/attachment.cgi?id=165713 in https://bugs.webkit.org/show_bug.cgi?id=97623
killed all EWS bots, they stucked in an infinite loop with the following error:

Traceback (most recent call last):
  File "/mnt/git/webkit-style-queue/Tools/Scripts/webkit-patch", line 84, in <module>
    main()
  File "/mnt/git/webkit-style-queue/Tools/Scripts/webkit-patch", line 79, in main
    WebKitPatch(os.path.abspath(__file__)).main()
  File "/mnt/git/webkit-style-queue/Tools/Scripts/webkitpy/tool/main.py", line 55, in __init__
    MultiCommandTool.__init__(self)
  File "/mnt/git/webkit-style-queue/Tools/Scripts/webkitpy/tool/multicommandtool.py", line 221, in __init__
    self.commands = commands or [cls() for cls in self._find_all_commands() if cls.name]
  File "/mnt/git/webkit-style-queue/Tools/Scripts/webkitpy/tool/commands/rebaseline.py", line 185, in __init__
    AbstractRebaseliningCommand.__init__(self, options=port_options(platform='port/platform to use. Use glob-style wildcards for multiple ports'))
NameError: global name 'port_options' is not defined


We need a proper way to avoid this kind of problems in the future.

[Eric Seidel (no email)]

All the bots Adam and I run use a wrapper script which cleans the repository every N runs.

http://trac.webkit.org/browser/trunk/Tools/EWSTools/start-queue.sh

Note the git clean and the test-webkit-py invocations.

Are those not working?

[Csaba Osztrogonác]

I use this script of course. But the problem was that EWS stucked in an infinite
loop and didn't exit to let this script to clean the repository. 

Here is the latest log I got:
------------------------------
Fetching: https://bugs.webkit.org/attachment.cgi?id=165715&action=edit
Fetching: https://bugs.webkit.org/show_bug.cgi?id=97535&ctype=xml&excludefield=attachmentdata
Fetching: https://bugs.webkit.org/attachment.cgi?id=165715&action=edit
Fetching: https://bugs.webkit.org/show_bug.cgi?id=97535&ctype=xml&excludefield=attachmentdata
Traceback (most recent call last):
  File "/storage/WebKit-qt-ews/Tools/Scripts/webkit-patch", line 84, in <module>
    main()
  File "/storage/WebKit-qt-ews/Tools/Scripts/webkit-patch", line 79, in main
    WebKitPatch(os.path.abspath(__file__)).main()
  File "/storage/WebKit-qt-ews/Tools/Scripts/webkitpy/tool/main.py", line 55, in __init__
    MultiCommandTool.__init__(self)
  File "/storage/WebKit-qt-ews/Tools/Scripts/webkitpy/tool/multicommandtool.py", line 221, in __init__
    self.commands = commands or [cls() for cls in self._find_all_commands() if cls.name]
  File "/storage/WebKit-qt-ews/Tools/Scripts/webkitpy/tool/commands/rebaseline.py", line 185, in __init__
    AbstractRebaseliningCommand.__init__(self, options=port_options(platform='port/platform to use. Use glob-style wildcards for multiple ports'))
NameError: global name 'port_options' is not defined
Unable to clean working directory
Unable to process work item.
Fetching: https://bugs.webkit.org/attachment.cgi?id=165715&action=edit
Fetching: https://bugs.webkit.org/show_bug.cgi?id=97535&ctype=xml&excludefield=attachmentdata
Error: qt-ews did not process patch.
Unable to process work item.
Fetching: https://bugs.webkit.org/attachment.cgi?id=165717&action=edit
Fetching: https://bugs.webkit.org/show_bug.cgi?id=97535&ctype=xml&excludefield=attachmentdata
Error: qt-ews did not process patch.
Unable to process work item.


And then EWS stucked in the infinite loop until I r- -ed the patch and restarted it manually.

[Csaba Osztrogonác]

I don't know exactly how should we fix this bug ... 

My idea is we should detect somehow if the EWS stucked in an infinite loop.
Maybe with an external script with a given timeout ... EWS should signal to
this script if it finished a build and started the next one. If it didn't
finish in the timeout, the script can restart it with a clean env.

Additionally we should remove the buggy patch from the EWS queue 
after 2-3 retries to save the life of the EWS bots.

[Adam Barth]

> Additionally we should remove the buggy patch from the EWS queue 
> after 2-3 retries to save the life of the EWS bots.

Yeah, that sounds like a good approach.  At least that will stop us from spinning on a patch forever.

[Csaba Osztrogonác]

One more problem because of this bug, see https://bugs.webkit.org/show_bug.cgi?id=114668 for details.

[Ryosuke Niwa]

I think EWS should create a copy of Tools/Scripts and use that instead.