Bug 190662 – [EWS] Patches that break *many* tests seem to never get out of the queue

• Component: Tools / Tests
• Status: RESOLVED INVALID

[Guillaume Emont]

To test the jsc armv7 EWS, I made a patch that would make all the tests fail on arm. This patch has already been tested 9x2 times on the bot, as can be seen in https://webkit-queues.webkit.org/patch/352475/jsc-armv7-ews. For now, I've removed the "r?" from the patch, hoping that it will remove it from the list, but I think this might be revealing an issue with reporting a very high number of test failures.

[Alexey Proskuryakov]

This is definitely a case that's supposed to work in general. It may be that JSC support has a bug in this code path, or some edge case that gets triggered for this specific result set.

Could you please check logs on the bot for any signs of trouble? Those are not uploaded anywhere, so we cannot see them.

[Guillaume Emont]

Finally found some time to start investigating that. Looking through my 26GB log file, I found this: after the bot has processed the patch (running twice with the patch + once without it to confirm that tests pass without the patch) and got failures, it seems to fail in the step where it tries to post a message on the bug. Here is an extract from the log, it was listing the tests that failed (most tests as I made a patch specifically for this on #190428), and then this:

stress/put-by-id-direct-strict-transition.js.no-cjit-validate-phases
stress/sink-to-impossible-multi-get-by-offset-on-prototypes.js.dfg-maximal-flush-validate-no-cjit
stress/arrowfunction-others.js.no-cjit-collect-continuously
sunspider-1.0/access-nbody.js.no-llint
apiTests
Traceback (most recent call last):
  File "/home/buildbot/ews/WebKit/Tools/Scripts/webkitpy/tool/bot/queueengine.py", line 103, in run
    if not self._delegate.process_work_item(work_item):
  File "/home/buildbot/ews/WebKit/Tools/Scripts/webkitpy/tool/commands/queues.py", line 479, in process_work_item
    passed = self.review_patch(patch)
  File "/home/buildbot/ews/WebKit/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem.py", line 120, in review_patch
    self._post_reject_message_on_bug(self._tool, patch, task.failure_status_id, self._failing_tests_message(task, patch))
  File "/home/buildbot/ews/WebKit/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem.py", line 95, in _post_reject_message_on_bug
    tool.bugs.set_flag_on_attachment(patch.id(), "commit-queue", "-", message)
  File "/home/buildbot/ews/WebKit/Tools/Scripts/webkitpy/common/net/bugzilla/bugzilla.py", line 822, in set_flag_on_attachment
    self._find_select_element_for_flag(flag_name).value = (flag_value,)
  File "/home/buildbot/ews/WebKit/Tools/Scripts/webkitpy/common/net/bugzilla/bugzilla.py", line 786, in _find_select_element_for_flag
    return self.browser.find_control(type='select', nr=1)
  File "/home/buildbot/ews/WebKit/Tools/Scripts/webkitpy/thirdparty/autoinstalled/mechanize/_form.py", line 3101, in find_control
    return self._find_control(name, type, kind, id, label, predicate, nr)
  File "/home/buildbot/ews/WebKit/Tools/Scripts/webkitpy/thirdparty/autoinstalled/mechanize/_form.py", line 3185, in _find_control
    raise ControlNotFoundError("no control matching "+description)
ControlNotFoundError: no control matching type 'select', nr 1
2019-04-08 06:51:20,519 - Exception while preparing queue Sleeping until 2019-04-08 06:53:20 (120 seconds).
2019-04-08 06:53:20,520 - Fetching next work item for jsc-armv7-ews

After that it goes onto the next patch in the queue, but it seems that the faulty patch gets added to the end of the queue.

[Guillaume Emont]

The issue seems to stem from the fact that the ews bugzilla user ([e-mail]) does not have commit rights in bugzilla, therefore the box to set the commit queue status does not exist on the page that it gets.

[Guillaume Emont]

Can confirm thanks to #190428 that adding the editbugs privilege to the EWS bugzilla user solved the issue.

[Guillaume Emont]

(In reply to Guillaume Emont from comment #4)
> Can confirm thanks to #190428 that adding the editbugs privilege to the EWS
> bugzilla user solved the issue.

In other conditions, I'd file a bug on the EWS for not providing a better error message/checking for that specific issue, but since it's being replaced, that's probably not worth it.