97647 2012-09-26 02:10:47 -0700 A buggy python patch can kill all EWS bots 2025-04-17 23:42:35 -0700 1 1 1 Unclassified WebKit New Bugs 528+ (Nightly build) All All NEW P1 Critical --- 1 ossy webkit-unassigned abarth aqeelalamdp dpranke eric jbadics kadam ossy peter rniwa Sean.Roshon tony zarvai oldest_to_newest 728335 0 ossy 2012-09-26 02:10:47 -0700 https://bugs.webkit.org/attachment.cgi?id=165713 in https://bugs.webkit.org/show_bug.cgi?id=97623 killed all EWS bots, they stucked in an infinite loop with the following error: Traceback (most recent call last): File "/mnt/git/webkit-style-queue/Tools/Scripts/webkit-patch", line 84, in <module> main() File "/mnt/git/webkit-style-queue/Tools/Scripts/webkit-patch", line 79, in main WebKitPatch(os.path.abspath(__file__)).main() File "/mnt/git/webkit-style-queue/Tools/Scripts/webkitpy/tool/main.py", line 55, in __init__ MultiCommandTool.__init__(self) File "/mnt/git/webkit-style-queue/Tools/Scripts/webkitpy/tool/multicommandtool.py", line 221, in __init__ self.commands = commands or [cls() for cls in self._find_all_commands() if cls.name] File "/mnt/git/webkit-style-queue/Tools/Scripts/webkitpy/tool/commands/rebaseline.py", line 185, in __init__ AbstractRebaseliningCommand.__init__(self, options=port_options(platform='port/platform to use. Use glob-style wildcards for multiple ports')) NameError: global name 'port_options' is not defined We need a proper way to avoid this kind of problems in the future. 728573 1 eric 2012-09-26 08:10:36 -0700 All the bots Adam and I run use a wrapper script which cleans the repository every N runs. http://trac.webkit.org/browser/trunk/Tools/EWSTools/start-queue.sh Note the git clean and the test-webkit-py invocations. Are those not working? 728582 2 ossy 2012-09-26 08:23:39 -0700 I use this script of course. But the problem was that EWS stucked in an infinite loop and didn't exit to let this script to clean the repository. Here is the latest log I got: ------------------------------ Fetching: https://bugs.webkit.org/attachment.cgi?id=165715&action=edit Fetching: https://bugs.webkit.org/show_bug.cgi?id=97535&ctype=xml&excludefield=attachmentdata Fetching: https://bugs.webkit.org/attachment.cgi?id=165715&action=edit Fetching: https://bugs.webkit.org/show_bug.cgi?id=97535&ctype=xml&excludefield=attachmentdata Traceback (most recent call last): File "/storage/WebKit-qt-ews/Tools/Scripts/webkit-patch", line 84, in <module> main() File "/storage/WebKit-qt-ews/Tools/Scripts/webkit-patch", line 79, in main WebKitPatch(os.path.abspath(__file__)).main() File "/storage/WebKit-qt-ews/Tools/Scripts/webkitpy/tool/main.py", line 55, in __init__ MultiCommandTool.__init__(self) File "/storage/WebKit-qt-ews/Tools/Scripts/webkitpy/tool/multicommandtool.py", line 221, in __init__ self.commands = commands or [cls() for cls in self._find_all_commands() if cls.name] File "/storage/WebKit-qt-ews/Tools/Scripts/webkitpy/tool/commands/rebaseline.py", line 185, in __init__ AbstractRebaseliningCommand.__init__(self, options=port_options(platform='port/platform to use. Use glob-style wildcards for multiple ports')) NameError: global name 'port_options' is not defined Unable to clean working directory Unable to process work item. Fetching: https://bugs.webkit.org/attachment.cgi?id=165715&action=edit Fetching: https://bugs.webkit.org/show_bug.cgi?id=97535&ctype=xml&excludefield=attachmentdata Error: qt-ews did not process patch. Unable to process work item. Fetching: https://bugs.webkit.org/attachment.cgi?id=165717&action=edit Fetching: https://bugs.webkit.org/show_bug.cgi?id=97535&ctype=xml&excludefield=attachmentdata Error: qt-ews did not process patch. Unable to process work item. And then EWS stucked in the infinite loop until I r- -ed the patch and restarted it manually. 728595 3 ossy 2012-09-26 08:33:13 -0700 I don't know exactly how should we fix this bug ... My idea is we should detect somehow if the EWS stucked in an infinite loop. Maybe with an external script with a given timeout ... EWS should signal to this script if it finished a build and started the next one. If it didn't finish in the timeout, the script can restart it with a clean env. Additionally we should remove the buggy patch from the EWS queue after 2-3 retries to save the life of the EWS bots. 728624 4 abarth 2012-09-26 08:59:30 -0700 > Additionally we should remove the buggy patch from the EWS queue > after 2-3 retries to save the life of the EWS bots. Yeah, that sounds like a good approach. At least that will stop us from spinning on a patch forever. 876407 5 ossy 2013-04-16 02:08:05 -0700 One more problem because of this bug, see https://bugs.webkit.org/show_bug.cgi?id=114668 for details. 876409 6 rniwa 2013-04-16 02:10:25 -0700 I think EWS should create a copy of Tools/Scripts and use that instead.