Bug 70236 - [Qt][WK2][meta] Fix failing API tests
Summary: [Qt][WK2][meta] Fix failing API tests
Status: RESOLVED INVALID
Alias: None
Product: WebKit
Classification: Unclassified
Component: WebKit Qt (show other bugs)
Version: 528+ (Nightly build)
Hardware: All All
: P2 Normal
Assignee: Nobody
URL:
Keywords: Qt, QtTriaged
Depends on: 96928 66664 70237 70238 70525 70715 71311 72612 72633 72916 73292 73994 74176 74180 74923 76906 77554 80247 81143 81771 82167 82483 82700 85049 87236 88870 89871 90372 91888 91900 94949 95968 96531 97561 97907 98045 98296 100224 100247 100661 100758 101167 101168 103875 103889 104574 114680 118129 118134 118138 118357 118360 118384 118389
Blocks:
  Show dependency treegraph
 
Reported: 2011-10-17 07:27 PDT by Csaba Osztrogonác
Modified: 2014-02-03 03:19 PST (History)
7 users (show)

See Also:


Attachments
Patch (1.49 KB, patch)
2011-10-17 08:21 PDT, Alexis Menard (darktears)
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Csaba Osztrogonác 2011-10-17 07:27:13 PDT
It is a meta bug for fixing failing API tests.
Comment 1 Alexis Menard (darktears) 2011-10-17 08:21:51 PDT
Created attachment 111267 [details]
Patch
Comment 2 WebKit Review Bot 2011-10-17 08:26:16 PDT
Attachment 111267 [details] did not pass style-queue:

Failed to run "['Tools/Scripts/update-webkit', '--chromium']" exit_code: 2

Updating OpenSource
Current branch master is up to date.
Updating chromium port dependencies using gclient...
Error: Can't switch the checkout to http://v8.googlecode.com/svn/branches/3﷒0﷓; UUID don't match and there is local changes in /mnt/git/webkit-style-queue/Source/WebKit/chromium/v8. Delete the directory and try again.
Re-trying 'depot_tools/gclient sync'
Error: Can't switch the checkout to http://v8.googlecode.com/svn/branches/3﷒1﷓; UUID don't match and there is local changes in /mnt/git/webkit-style-queue/Source/WebKit/chromium/v8. Delete the directory and try again.
Re-trying 'depot_tools/gclient sync'
Error: Can't switch the checkout to http://v8.googlecode.com/svn/branches/3﷒2﷓; UUID don't match and there is local changes in /mnt/git/webkit-style-queue/Source/WebKit/chromium/v8. Delete the directory and try again.
Error: 'depot_tools/gclient sync' failed 3 tries and returned 256 at Tools/Scripts/update-webkit-chromium line 107.
Re-trying 'depot_tools/gclient sync'
No such file or directory at Tools/Scripts/update-webkit line 104.


If any of these errors are false positives, please file a bug against check-webkit-style.
Comment 3 Csaba Osztrogonác 2011-10-17 08:28:00 PDT
Comment on attachment 111267 [details]
Patch

Clearing flags on attachment: 111267

Committed r97623: <http://trac.webkit.org/changeset/97623>
Comment 4 Csaba Osztrogonác 2011-10-17 08:28:07 PDT
All reviewed patches have been landed.  Closing bug.
Comment 5 Csaba Osztrogonác 2011-10-17 08:28:49 PDT
.
Comment 6 Csaba Osztrogonác 2011-10-26 09:54:55 PDT
There are too many failing API tests for long time, so API tests aren't ready for making the bot red if a test fails. From now a failing API test doesn't make the WK2 bot red, but its build step is orange.
Comment 7 Alexis Menard (darktears) 2011-10-26 10:20:21 PDT
(In reply to comment #6)
> There are too many failing API tests for long time, so API tests aren't ready for making the bot red if a test fails. From now a failing API test doesn't make the WK2 bot red, but its build step is orange.

NOOOOO please....

It's a real issue. Almost all the QtWebKit community is in dev days so we don't have time to look at it *right now*. But we will. Let's not go back to the past and let it broken. Any API breakage is as important as LayoutTests breakage.

By any chance did you catch the faulty revision or the import of Qt5 that it broke? I recall they were properly green last week.
Comment 8 Csaba Osztrogonác 2011-11-01 13:08:30 PDT
(In reply to comment #7)
> (In reply to comment #6)
> > There are too many failing API tests for long time, so API tests aren't ready for making the bot red if a test fails. From now a failing API test doesn't make the WK2 bot red, but its build step is orange.
> 
> NOOOOO please....
> 
> It's a real issue. Almost all the QtWebKit community is in dev days so we don't have time to look at it *right now*. But we will. Let's not go back to the past and let it broken. Any API breakage is as important as LayoutTests breakage.
> 
> By any chance did you catch the faulty revision or the import of Qt5 that it broke? I recall they were properly green last week.

There are 10 failing API tests now, I filed bug reports for all fails which block this bug. I don't think if always red bot is a good idea. Always red bot _can't_ get regressions and _can't_ send email if there is a new regression. We can catch new regressions easily if the bot is green almost always. In this case the green/red change means a possible new regression. I can't fix failing API tests myself, but API maintainer can.
Comment 9 Alexis Menard (darktears) 2011-11-01 13:58:40 PDT
(In reply to comment #8)
> (In reply to comment #7)
> > (In reply to comment #6)
> > > There are too many failing API tests for long time, so API tests aren't ready for making the bot red if a test fails. From now a failing API test doesn't make the WK2 bot red, but its build step is orange.
> > 
> > NOOOOO please....
> > 
> > It's a real issue. Almost all the QtWebKit community is in dev days so we don't have time to look at it *right now*. But we will. Let's not go back to the past and let it broken. Any API breakage is as important as LayoutTests breakage.
> > 
> > By any chance did you catch the faulty revision or the import of Qt5 that it broke? I recall they were properly green last week.
> 
> There are 10 failing API tests now, I filed bug reports for all fails which block this bug. I don't think if always red bot is a good idea. Always red bot _can't_ get regressions and _can't_ send email if there is a new regression. We can catch new regressions easily if the bot is green almost always. In this case the green/red change means a possible new regression. I can't fix failing API tests myself, but API maintainer can.

But szeged build bots doesn't send email/add comments on bugs even if you are the first guilty revision and I think that's the problem. Of course I do see why it doesn't do it because our bots there are not that stable or not meant to be used by other contributors. But would it be possible that they comment on bugs that matters for QtWebKit like the ones with [Qt] because today people don't care about API tests because they don't get warn on their original bug id. At least the WK2 could do so that would force people to care about them. It seems that except 

FAIL!  : qmltests::DesktopWebViewNavigationPolicyForUrl::test_ignorePolicy() window not shown

they seems to be easily fixable. The latter seems to be a race triggered in V8 with a recent change. Hopefully we can get that fixed in the next Qt update.
Comment 10 Jesus Sanchez-Palencia 2011-11-11 21:23:14 PST
Alexis, all the API tests are currently passing, right?!
Comment 11 Csaba Osztrogonác 2011-11-17 08:00:31 PST
The answer was yes, but now we have 2 new fails: https://bugs.webkit.org/show_bug.cgi?id=72612
Comment 12 Csaba Osztrogonác 2011-11-17 08:12:30 PST
(In reply to comment #9)
> But szeged build bots doesn't send email/add comments on bugs even if you are the first guilty revision and I think that's the problem. Of course I do see why it doesn't do it because our bots there are not that stable or not meant to be used by other contributors. But would it be possible that they comment on bugs that matters for QtWebKit like the ones with [Qt] because today people don't care about API tests because they don't get warn on their original bug id. At least the WK2 could do so that would force people to care about them.

I remember that sheriffbot commented bugs regularly if somebody caused a regression. But it was disabled, because folks didn't want to get SPAM.
Buildbot on its own can't send e-mails if a commit increases the number of
failing layout or API tests. Buildbot can send e-mail when its color changes 
from green to red. (or from all red revision)

Of course it is possible to implement a sophisticated script which polls buildbot page regularly, parse its output, find who break which layout or API tests and send e-mail if the bug title has [Qt] prefix, etc. Is there any volunteer for it?
Comment 13 Alexis Menard (darktears) 2011-11-22 06:19:24 PST
(In reply to comment #12)
> (In reply to comment #9)
> > But szeged build bots doesn't send email/add comments on bugs even if you are the first guilty revision and I think that's the problem. Of course I do see why it doesn't do it because our bots there are not that stable or not meant to be used by other contributors. But would it be possible that they comment on bugs that matters for QtWebKit like the ones with [Qt] because today people don't care about API tests because they don't get warn on their original bug id. At least the WK2 could do so that would force people to care about them.
> 
> I remember that sheriffbot commented bugs regularly if somebody caused a regression. But it was disabled, because folks didn't want to get SPAM.
> Buildbot on its own can't send e-mails if a commit increases the number of
> failing layout or API tests. Buildbot can send e-mail when its color changes 
> from green to red. (or from all red revision)
> 
> Of course it is possible to implement a sophisticated script which polls buildbot page regularly, parse its output, find who break which layout or API tests and send e-mail if the bug title has [Qt] prefix, etc. Is there any volunteer for it?

All green now.
Comment 14 Jocelyn Turcotte 2014-02-03 03:19:04 PST
=== Bulk closing of Qt bugs ===

If you believe that this bug report is still relevant for a non-Qt port of webkit.org, please re-open it and remove [Qt] from the summary.

If you believe that this is still an important QtWebKit bug, please fill a new report at https://bugreports.qt-project.org and add a link to this issue. See http://qt-project.org/wiki/ReportingBugsInQt for additional guidelines.