Bug 70236 – [Qt][WK2][meta] Fix failing API tests

• Status: RESOLVED
• Resolution: INVALID
• Component: WebKit Qt

[Csaba Osztrogonác]

It is a meta bug for fixing failing API tests.

[Alexis Menard (darktears)]

Created attachment 111267 [details]
Patch

[WebKit Review Bot]

Attachment 111267 [details] did not pass style-queue:

Failed to run "['Tools/Scripts/update-webkit', '--chromium']" exit_code: 2

Updating OpenSource
Current branch master is up to date.
Updating chromium port dependencies using gclient...
Error: Can't switch the checkout to http://v8.googlecode.com/svn/branches/3.6@9637; UUID don't match and there is local changes in /mnt/git/webkit-style-queue/Source/WebKit/chromium/v8. Delete the directory and try again.
Re-trying 'depot_tools/gclient sync'
Error: Can't switch the checkout to http://v8.googlecode.com/svn/branches/3.6@9637; UUID don't match and there is local changes in /mnt/git/webkit-style-queue/Source/WebKit/chromium/v8. Delete the directory and try again.
Re-trying 'depot_tools/gclient sync'
Error: Can't switch the checkout to http://v8.googlecode.com/svn/branches/3.6@9637; UUID don't match and there is local changes in /mnt/git/webkit-style-queue/Source/WebKit/chromium/v8. Delete the directory and try again.
Error: 'depot_tools/gclient sync' failed 3 tries and returned 256 at Tools/Scripts/update-webkit-chromium line 107.
Re-trying 'depot_tools/gclient sync'
No such file or directory at Tools/Scripts/update-webkit line 104.


If any of these errors are false positives, please file a bug against check-webkit-style.

[Csaba Osztrogonác]

Comment on attachment 111267 [details]
Patch

Clearing flags on attachment: 111267

Committed r97623: <http://trac.webkit.org/changeset/97623>

[Csaba Osztrogonác]

All reviewed patches have been landed.  Closing bug.

[Csaba Osztrogonác]

.

[Csaba Osztrogonác]

There are too many failing API tests for long time, so API tests aren't ready for making the bot red if a test fails. From now a failing API test doesn't make the WK2 bot red, but its build step is orange.

[Alexis Menard (darktears)]

(In reply to comment #6)
> There are too many failing API tests for long time, so API tests aren't ready for making the bot red if a test fails. From now a failing API test doesn't make the WK2 bot red, but its build step is orange.

NOOOOO please....

It's a real issue. Almost all the QtWebKit community is in dev days so we don't have time to look at it *right now*. But we will. Let's not go back to the past and let it broken. Any API breakage is as important as LayoutTests breakage.

By any chance did you catch the faulty revision or the import of Qt5 that it broke? I recall they were properly green last week.

[Csaba Osztrogonác]

(In reply to comment #7)
> (In reply to comment #6)
> > There are too many failing API tests for long time, so API tests aren't ready for making the bot red if a test fails. From now a failing API test doesn't make the WK2 bot red, but its build step is orange.
> 
> NOOOOO please....
> 
> It's a real issue. Almost all the QtWebKit community is in dev days so we don't have time to look at it *right now*. But we will. Let's not go back to the past and let it broken. Any API breakage is as important as LayoutTests breakage.
> 
> By any chance did you catch the faulty revision or the import of Qt5 that it broke? I recall they were properly green last week.

There are 10 failing API tests now, I filed bug reports for all fails which block this bug. I don't think if always red bot is a good idea. Always red bot _can't_ get regressions and _can't_ send email if there is a new regression. We can catch new regressions easily if the bot is green almost always. In this case the green/red change means a possible new regression. I can't fix failing API tests myself, but API maintainer can.

[Alexis Menard (darktears)]

(In reply to comment #8)
> (In reply to comment #7)
> > (In reply to comment #6)
> > > There are too many failing API tests for long time, so API tests aren't ready for making the bot red if a test fails. From now a failing API test doesn't make the WK2 bot red, but its build step is orange.
> > 
> > NOOOOO please....
> > 
> > It's a real issue. Almost all the QtWebKit community is in dev days so we don't have time to look at it *right now*. But we will. Let's not go back to the past and let it broken. Any API breakage is as important as LayoutTests breakage.
> > 
> > By any chance did you catch the faulty revision or the import of Qt5 that it broke? I recall they were properly green last week.
> 
> There are 10 failing API tests now, I filed bug reports for all fails which block this bug. I don't think if always red bot is a good idea. Always red bot _can't_ get regressions and _can't_ send email if there is a new regression. We can catch new regressions easily if the bot is green almost always. In this case the green/red change means a possible new regression. I can't fix failing API tests myself, but API maintainer can.

But szeged build bots doesn't send email/add comments on bugs even if you are the first guilty revision and I think that's the problem. Of course I do see why it doesn't do it because our bots there are not that stable or not meant to be used by other contributors. But would it be possible that they comment on bugs that matters for QtWebKit like the ones with [Qt] because today people don't care about API tests because they don't get warn on their original bug id. At least the WK2 could do so that would force people to care about them. It seems that except 

FAIL!  : qmltests::DesktopWebViewNavigationPolicyForUrl::test_ignorePolicy() window not shown

they seems to be easily fixable. The latter seems to be a race triggered in V8 with a recent change. Hopefully we can get that fixed in the next Qt update.

[Jesus Sanchez-Palencia]

Alexis, all the API tests are currently passing, right?!

[Csaba Osztrogonác]

The answer was yes, but now we have 2 new fails: https://bugs.webkit.org/show_bug.cgi?id=72612

[Csaba Osztrogonác]

(In reply to comment #9)
> But szeged build bots doesn't send email/add comments on bugs even if you are the first guilty revision and I think that's the problem. Of course I do see why it doesn't do it because our bots there are not that stable or not meant to be used by other contributors. But would it be possible that they comment on bugs that matters for QtWebKit like the ones with [Qt] because today people don't care about API tests because they don't get warn on their original bug id. At least the WK2 could do so that would force people to care about them.

I remember that sheriffbot commented bugs regularly if somebody caused a regression. But it was disabled, because folks didn't want to get SPAM.
Buildbot on its own can't send e-mails if a commit increases the number of
failing layout or API tests. Buildbot can send e-mail when its color changes 
from green to red. (or from all red revision)

Of course it is possible to implement a sophisticated script which polls buildbot page regularly, parse its output, find who break which layout or API tests and send e-mail if the bug title has [Qt] prefix, etc. Is there any volunteer for it?

[Alexis Menard (darktears)]

(In reply to comment #12)
> (In reply to comment #9)
> > But szeged build bots doesn't send email/add comments on bugs even if you are the first guilty revision and I think that's the problem. Of course I do see why it doesn't do it because our bots there are not that stable or not meant to be used by other contributors. But would it be possible that they comment on bugs that matters for QtWebKit like the ones with [Qt] because today people don't care about API tests because they don't get warn on their original bug id. At least the WK2 could do so that would force people to care about them.
> 
> I remember that sheriffbot commented bugs regularly if somebody caused a regression. But it was disabled, because folks didn't want to get SPAM.
> Buildbot on its own can't send e-mails if a commit increases the number of
> failing layout or API tests. Buildbot can send e-mail when its color changes 
> from green to red. (or from all red revision)
> 
> Of course it is possible to implement a sophisticated script which polls buildbot page regularly, parse its output, find who break which layout or API tests and send e-mail if the bug title has [Qt] prefix, etc. Is there any volunteer for it?

All green now.

[Jocelyn Turcotte]

=== Bulk closing of Qt bugs ===

If you believe that this bug report is still relevant for a non-Qt port of webkit.org, please re-open it and remove [Qt] from the summary.

If you believe that this is still an important QtWebKit bug, please fill a new report at https://bugreports.qt-project.org and add a link to this issue. See http://qt-project.org/wiki/ReportingBugsInQt for additional guidelines.