Bug 208003 – EWS fails to retry build in case of kill-old-processes exception

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Aakash Jain]

EWS retries the build in case of kill-old-processes failure (since https://trac.webkit.org/changeset/252324). However, EWS fails to retry build in case of kill-old-processes Exception. Exception in case of kill-old-processes is noticeable only on WinCairo queue.

e.g.:
https://ews-build.webkit.org/#/builders/12/builds/17905
https://ews-build.webkit.org/#/builders/12/builds/17860
https://ews-build.webkit.org/#/builders/12/builds/17858

It would be nice if EWS can retry the build in case of kill-old-processes exception as well.

[Aakash Jain]

Created attachment 391283 [details]
Patch

[Aakash Jain]

(In reply to Aakash Jain from comment #1)
> Created attachment 391283 [details]
> Patch
Speculative fix.

[Jonathan Bedard]

(In reply to Aakash Jain from comment #0)
> EWS retries the build in case of kill-old-processes failure (since
> https://trac.webkit.org/changeset/252324). However, EWS fails to retry build
> in case of kill-old-processes Exception. Exception in case of
> kill-old-processes is noticeable only on WinCairo queue.
> 
> e.g.:
> https://ews-build.webkit.org/#/builders/12/builds/17905
> https://ews-build.webkit.org/#/builders/12/builds/17860
> https://ews-build.webkit.org/#/builders/12/builds/17858
> 
> It would be nice if EWS can retry the build in case of kill-old-processes
> exception as well.

What even causes a kill-old-processes exception? If that script is having issues, I would bet that the entire machine is in trouble.

[Aakash Jain]

(In reply to Jonathan Bedard from comment #3)
> What even causes a kill-old-processes exception? 
Probably the machine is in a very bad state.
> If that script is having issues, I would bet that the entire machine is in trouble.
Yes. The idea is to retry the build so that the patch gets re-tested on some other machine. If we don't retry the status-bubble for that patch will remain purple, until someone manually retries the build.

Even though that particular machine will keep failing at kill-old-processes, the patch being tested will eventually be tested on some other machine because of this RETRY. Hopefully someone would notice that the machine is in bad state and fix it eventually.

For example: https://ews-build.webkit.org/#/builders/22/builds/10002 failed with kill-old-processed, however the build was retried and picked up by another bot (in https://ews-build.webkit.org/#/builders/22/builds/10034) and passed. The status-bubble for corresponding patch 390315 (in Bug 207516) became green without any manual intervention (of bot-watcher or patch author). 

Similar thing in https://ews-build.webkit.org/#/builders/3/builds/16167 and https://ews-build.webkit.org/#/builders/3/builds/16168

Retry made the system work fine even though one bot was in bad state.

[Jonathan Bedard]

Comment on attachment 391283 [details]
Patch

Do we actually have a practical example of an exception?

[Aakash Jain]

> Do we actually have a practical example of an exception?
https://ews-build.webkit.org/#/builders/12/builds/17858

Although, this is speculative fix. I am not sure if it would really work in case of exception. It's hard to reproduce and I don't have machine to test it out.

[Jonathan Bedard]

(In reply to Aakash Jain from comment #6)
> > Do we actually have a practical example of an exception?
> https://ews-build.webkit.org/#/builders/12/builds/17858
> 
> Although, this is speculative fix. I am not sure if it would really work in
> case of exception. It's hard to reproduce and I don't have machine to test
> it out.

Fair enough.

As I said in the other bug, if a machine fails kill-old-processes, it probably needs to be rebooted.

[WebKit Commit Bot]

Comment on attachment 391283 [details]
Patch

Clearing flags on attachment: 391283

Committed r257654: <https://trac.webkit.org/changeset/257654>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Radar WebKit Bug Importer]

<rdar://problem/59899567>