Bug 106596 - Make Mac EWS bots run API tests
Summary: Make Mac EWS bots run API tests
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: 528+ (Nightly build)
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Nobody
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2013-01-10 13:39 PST by Ryosuke Niwa
Modified: 2019-04-06 20:48 PDT (History)
12 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Ryosuke Niwa 2013-01-10 13:39:53 PST
It'll be nice if EWS bots run API tests in addition to layout tests especially once we add Mac WK2 EWS bots (webkit.org/b/106590). I've seen unit tests getting broken by some checkins.
Comment 1 Eric Seidel (no email) 2013-01-10 13:59:58 PST
They will need to understand how to parse the results and track failing tests.  Otherwise the EWS bot will stop being able to process any patches while the tree is red.
Comment 2 Radar WebKit Bug Importer 2013-04-18 19:46:55 PDT
<rdar://problem/13690874>
Comment 3 Aakash Jain 2019-04-05 09:43:17 PDT
We launched EWS for API tests (for iOS and macOS) along-with new EWS yesterday.

https://lists.webkit.org/pipermail/webkit-dev/2019-April/030592.html
Comment 4 Daniel Bates 2019-04-05 14:28:03 PDT
I tend to relish the archaeology of code. In reality it tends to not matter in the end, but sometimes you can find a gold nugget in the old code. So, for myself and for others  that enjoy this kind of archaeology I am duping this bug to bug #170028. That bug, and the bugs/patches by the same author Srinivasan Vijayaraghavan implemented EWS support for running API tests and analyzing the results. I could have sweared that we even turned it on at one point in time and had a Mac API EWS bot (that is an archaeology dig left for others :)). Sometime before 04/03/2019, it was taken down. I seem to recall hearing that the reason it was taken down was due to the fact that the API tester had to build WebKit just to run the tests (there was no separation of builders and testers as in the New EWS mentioned in comment #3) AND it was never backed by sufficient hardware or thought of as being as important as the layout test bots at the time. It would often fall behind and couldn't process patches fast enough. So, the bot was taken down. (BTW the same problem existed for layout test EWS bots, but sufficient hardware was allocated to these bots). Or maybe there was some showstopping issue that required all such Mac API bot to be taken down.

Anyway, as Aakash wrote in comment #3 we now have API EWS testers (again!). This time for iOS as well.

*** This bug has been marked as a duplicate of bug 170028 ***
Comment 5 Alexey Proskuryakov 2019-04-05 18:06:27 PDT
This is not a duplicate of bug 170028. That one was tracking adding support for API tests in webkitpy, which has very little to do with this bug.

> I could have sweared that we even turned it on at one point in time and had a Mac API EWS bot (that is an archaeology dig left for others :))

This is incorrect, we did not have bots.
Comment 6 Daniel Bates 2019-04-05 20:31:21 PDT
(In reply to Alexey Proskuryakov from comment #5)
> This is not a duplicate of bug 170028. That one was tracking adding support
> for API tests in webkitpy, which has very little to do with this bug.
> 

Try again 😀 I think I know where the confusion comes from. You are putting way too much emphasis into the word “run” in my opinion 🙃 Not much value in having a pure “run” (or flip the light switch) bug when you don’t even have a light switch to flip. That is, when this bug was filed we didn’t even have the infrastructure to run API tests (why you ask? The answer is in comment 1) let alone “flip a switch” and run an EWS server to run them. So, fixing this bug means adding such infra. and “flipping the switch” from my understanding. You claim we never did the “flip”, but I think we did. I don’t care about that argument it’s tangential and in my opinion it would be asinine to think the purpose of this 6 year old bug to be the “flip the switch bug” (Completely illogical to think Ryosuke meant for it to be that, but maybe he did). If so, I am wrong 🙂

So, I can see how the bug title and description for bug 170028 were a little off, but the actual purpose of that bug comes ***straight from your mouth***, bug 170028, comment 3 (including comment 2 just for context), emphasis mine:

Sam Weinig 2017-03-24 05:37:15 PDT
I don't understand why this is necessary? The api tests script is perl script, why involve another language?

Comment 3 Alexey Proskuryakov 2017-03-24 09:00:32 PDT
****It's to run API tests in EWS***, which is all written in Python.

Anyway, I would dup this bug to bug 170028, that’s what taught the EWS to run them. That’s the bug that you even said above would make the EWS run the tests and that is even what Ryosuke wishes for in the first sentence in comment 0. Just didn’t flip the switch or did and flipped it back off.
Comment 7 Alexey Proskuryakov 2019-04-06 19:45:00 PDT
Bug 170028 was a tiny step towards running API tests in the old EWS. This was the plan, but there was a lot of work ahead, which was never completed.

If you are still confused, I can explain in person.
Comment 8 Ryosuke Niwa 2019-04-06 20:48:14 PDT
(In reply to Daniel Bates from comment #6)
>
> Try again 😀 I think I know where the confusion comes from. You are putting
> way too much emphasis into the word “run” in my opinion 🙃 Not much value in
> having a pure “run” (or flip the light switch) bug when you don’t even have
> a light switch to flip. That is, when this bug was filed we didn’t even have
> the infrastructure to run API tests (why you ask? The answer is in comment
> 1) let alone “flip a switch” and run an EWS server to run them.

This bug was filed to run API tests in EWS. If anything, this bug should be dup'ed to https://bugs.webkit.org/show_bug.cgi?id=188156, or some kind of an umbrella bug thereof. Whether there is webkitpy code to do that or not is irrelevant as long as there is no bot doing it.