Bug 106596
| Summary: | Make Mac EWS bots run API tests | ||
|---|---|---|---|
| Product: | WebKit | Reporter: | Ryosuke Niwa <rniwa> |
| Component: | Tools / Tests | Assignee: | Nobody <webkit-unassigned> |
| Status: | RESOLVED FIXED | ||
| Severity: | Normal | CC: | aakash_jain, abarth, alancutter, ap, dbates, dpranke, eric, jake.nielsen.webkit, jberlin, lforschler, sam, webkit-bug-importer |
| Priority: | P2 | Keywords: | InRadar |
| Version: | 528+ (Nightly build) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| See Also: |
https://bugs.webkit.org/show_bug.cgi?id=172607 https://bugs.webkit.org/show_bug.cgi?id=172558 https://bugs.webkit.org/show_bug.cgi?id=179506 https://bugs.webkit.org/show_bug.cgi?id=170028 |
||
Ryosuke Niwa
It'll be nice if EWS bots run API tests in addition to layout tests especially once we add Mac WK2 EWS bots (webkit.org/b/106590). I've seen unit tests getting broken by some checkins.
| Attachments | ||
|---|---|---|
| Add attachment proposed patch, testcase, etc. |
Eric Seidel (no email)
They will need to understand how to parse the results and track failing tests. Otherwise the EWS bot will stop being able to process any patches while the tree is red.
Radar WebKit Bug Importer
<rdar://problem/13690874>
Aakash Jain
We launched EWS for API tests (for iOS and macOS) along-with new EWS yesterday.
https://lists.webkit.org/pipermail/webkit-dev/2019-April/030592.html
Daniel Bates
I tend to relish the archaeology of code. In reality it tends to not matter in the end, but sometimes you can find a gold nugget in the old code. So, for myself and for others that enjoy this kind of archaeology I am duping this bug to bug #170028. That bug, and the bugs/patches by the same author Srinivasan Vijayaraghavan implemented EWS support for running API tests and analyzing the results. I could have sweared that we even turned it on at one point in time and had a Mac API EWS bot (that is an archaeology dig left for others :)). Sometime before 04/03/2019, it was taken down. I seem to recall hearing that the reason it was taken down was due to the fact that the API tester had to build WebKit just to run the tests (there was no separation of builders and testers as in the New EWS mentioned in comment #3) AND it was never backed by sufficient hardware or thought of as being as important as the layout test bots at the time. It would often fall behind and couldn't process patches fast enough. So, the bot was taken down. (BTW the same problem existed for layout test EWS bots, but sufficient hardware was allocated to these bots). Or maybe there was some showstopping issue that required all such Mac API bot to be taken down.
Anyway, as Aakash wrote in comment #3 we now have API EWS testers (again!). This time for iOS as well.
*** This bug has been marked as a duplicate of bug 170028 ***
Alexey Proskuryakov
This is not a duplicate of bug 170028. That one was tracking adding support for API tests in webkitpy, which has very little to do with this bug.
> I could have sweared that we even turned it on at one point in time and had a Mac API EWS bot (that is an archaeology dig left for others :))
This is incorrect, we did not have bots.
Daniel Bates
(In reply to Alexey Proskuryakov from comment #5)
> This is not a duplicate of bug 170028. That one was tracking adding support
> for API tests in webkitpy, which has very little to do with this bug.
>
Try again đ I think I know where the confusion comes from. You are putting way too much emphasis into the word ârunâ in my opinion đ Not much value in having a pure ârunâ (or flip the light switch) bug when you donât even have a light switch to flip. That is, when this bug was filed we didnât even have the infrastructure to run API tests (why you ask? The answer is in comment 1) let alone âflip a switchâ and run an EWS server to run them. So, fixing this bug means adding such infra. and âflipping the switchâ from my understanding. You claim we never did the âflipâ, but I think we did. I donât care about that argument itâs tangential and in my opinion it would be asinine to think the purpose of this 6 year old bug to be the âflip the switch bugâ (Completely illogical to think Ryosuke meant for it to be that, but maybe he did). If so, I am wrong đ
So, I can see how the bug title and description for bug 170028 were a little off, but the actual purpose of that bug comes ***straight from your mouth***, bug 170028, comment 3 (including comment 2 just for context), emphasis mine:
Sam Weinig 2017-03-24 05:37:15 PDT
I don't understand why this is necessary? The api tests script is perl script, why involve another language?
Comment 3 Alexey Proskuryakov 2017-03-24 09:00:32 PDT
****It's to run API tests in EWS***, which is all written in Python.
Anyway, I would dup this bug to bug 170028, thatâs what taught the EWS to run them. Thatâs the bug that you even said above would make the EWS run the tests and that is even what Ryosuke wishes for in the first sentence in comment 0. Just didnât flip the switch or did and flipped it back off.
Alexey Proskuryakov
Bug 170028 was a tiny step towards running API tests in the old EWS. This was the plan, but there was a lot of work ahead, which was never completed.
If you are still confused, I can explain in person.
Ryosuke Niwa
(In reply to Daniel Bates from comment #6)
>
> Try again đ I think I know where the confusion comes from. You are putting
> way too much emphasis into the word ârunâ in my opinion đ Not much value in
> having a pure ârunâ (or flip the light switch) bug when you donât even have
> a light switch to flip. That is, when this bug was filed we didnât even have
> the infrastructure to run API tests (why you ask? The answer is in comment
> 1) let alone âflip a switchâ and run an EWS server to run them.
This bug was filed to run API tests in EWS. If anything, this bug should be dup'ed to https://bugs.webkit.org/show_bug.cgi?id=188156, or some kind of an umbrella bug thereof. Whether there is webkitpy code to do that or not is irrelevant as long as there is no bot doing it.