https://bugs.webkit.org/attachment.cgi?id=165713 in https://bugs.webkit.org/show_bug.cgi?id=97623 killed all EWS bots, they stucked in an infinite loop with the following error: Traceback (most recent call last): File "/mnt/git/webkit-style-queue/Tools/Scripts/webkit-patch", line 84, in <module> main() File "/mnt/git/webkit-style-queue/Tools/Scripts/webkit-patch", line 79, in main WebKitPatch(os.path.abspath(__file__)).main() File "/mnt/git/webkit-style-queue/Tools/Scripts/webkitpy/tool/main.py", line 55, in __init__ MultiCommandTool.__init__(self) File "/mnt/git/webkit-style-queue/Tools/Scripts/webkitpy/tool/multicommandtool.py", line 221, in __init__ self.commands = commands or [cls() for cls in self._find_all_commands() if cls.name] File "/mnt/git/webkit-style-queue/Tools/Scripts/webkitpy/tool/commands/rebaseline.py", line 185, in __init__ AbstractRebaseliningCommand.__init__(self, options=port_options(platform='port/platform to use. Use glob-style wildcards for multiple ports')) NameError: global name 'port_options' is not defined We need a proper way to avoid this kind of problems in the future.
All the bots Adam and I run use a wrapper script which cleans the repository every N runs. http://trac.webkit.org/browser/trunk/Tools/EWSTools/start-queue.sh Note the git clean and the test-webkit-py invocations. Are those not working?
I use this script of course. But the problem was that EWS stucked in an infinite loop and didn't exit to let this script to clean the repository. Here is the latest log I got: ------------------------------ Fetching: https://bugs.webkit.org/attachment.cgi?id=165715&action=edit Fetching: https://bugs.webkit.org/show_bug.cgi?id=97535&ctype=xml&excludefield=attachmentdata Fetching: https://bugs.webkit.org/attachment.cgi?id=165715&action=edit Fetching: https://bugs.webkit.org/show_bug.cgi?id=97535&ctype=xml&excludefield=attachmentdata Traceback (most recent call last): File "/storage/WebKit-qt-ews/Tools/Scripts/webkit-patch", line 84, in <module> main() File "/storage/WebKit-qt-ews/Tools/Scripts/webkit-patch", line 79, in main WebKitPatch(os.path.abspath(__file__)).main() File "/storage/WebKit-qt-ews/Tools/Scripts/webkitpy/tool/main.py", line 55, in __init__ MultiCommandTool.__init__(self) File "/storage/WebKit-qt-ews/Tools/Scripts/webkitpy/tool/multicommandtool.py", line 221, in __init__ self.commands = commands or [cls() for cls in self._find_all_commands() if cls.name] File "/storage/WebKit-qt-ews/Tools/Scripts/webkitpy/tool/commands/rebaseline.py", line 185, in __init__ AbstractRebaseliningCommand.__init__(self, options=port_options(platform='port/platform to use. Use glob-style wildcards for multiple ports')) NameError: global name 'port_options' is not defined Unable to clean working directory Unable to process work item. Fetching: https://bugs.webkit.org/attachment.cgi?id=165715&action=edit Fetching: https://bugs.webkit.org/show_bug.cgi?id=97535&ctype=xml&excludefield=attachmentdata Error: qt-ews did not process patch. Unable to process work item. Fetching: https://bugs.webkit.org/attachment.cgi?id=165717&action=edit Fetching: https://bugs.webkit.org/show_bug.cgi?id=97535&ctype=xml&excludefield=attachmentdata Error: qt-ews did not process patch. Unable to process work item. And then EWS stucked in the infinite loop until I r- -ed the patch and restarted it manually.
I don't know exactly how should we fix this bug ... My idea is we should detect somehow if the EWS stucked in an infinite loop. Maybe with an external script with a given timeout ... EWS should signal to this script if it finished a build and started the next one. If it didn't finish in the timeout, the script can restart it with a clean env. Additionally we should remove the buggy patch from the EWS queue after 2-3 retries to save the life of the EWS bots.
> Additionally we should remove the buggy patch from the EWS queue > after 2-3 retries to save the life of the EWS bots. Yeah, that sounds like a good approach. At least that will stop us from spinning on a patch forever.
One more problem because of this bug, see https://bugs.webkit.org/show_bug.cgi?id=114668 for details.
I think EWS should create a copy of Tools/Scripts and use that instead.