[Qt][WK2] REGRESSION(r123583): Removing a test made 88 tests flakey

• Component: Tools / Tests
• Status: RESOLVED DUPLICATE

[Csaba Osztrogonác]

I have no idead how is it possible, but it must be a serious bug somewhere in Qt-WK2 port.

r123582 -  22 flakes - http://build.webkit.sed.hu/builders/x86-64%20Linux%20Qt%20Release%20WebKit2%20%28Amazon%20EC2%29/builds/6482
r123584 - 110 flakes - http://build.webkit.sed.hu/builders/x86-64%20Linux%20Qt%20Release%20WebKit2%20%28Amazon%20EC2%29/builds/6483

[Zan Dobersek]

From little experience gained from poking around the Gtk's implementation, I think the EventSenderProxy lacks a method that would reset all the members to their default values before each test is run. Currently the EventSenderProxy is created at TestController initialization and just reused for every test without being reset.

Looking through the Qt's implementation I can see a problem occurring if two subsequent tests happen to click with the same button in the exact same location with not enough timespan between them - this would generate a double click in the second test instead of a simple click. Other similar problems can occur due to no resetting of the EventSenderProxy state.

I'm not saying that's exactly the problem nor do I have any evidence of this actually happening and causing increase in flaky tests. It just seems to me this might be the most possible explanation to the problem due to the flakiness and the fact this started occurring after a test utilizing the EventSender was removed from the test run.

[Csaba Osztrogonác]

Created attachment 154378 [details]
Patch

Thanks for the idea, I tried to reinstantiate EventSenderProxy between each tests, and it works for me. Maybe there is simpler solution, but I don't know ...

[Gyuyoung Kim]

Comment on attachment 154378 [details]
Patch

Attachment 154378 [details] did not pass efl-ews (efl):
Output: http://queues.webkit.org/results/13339346

[Gustavo Noronha (kov)]

Comment on attachment 154378 [details]
Patch

Attachment 154378 [details] did not pass gtk-ews (gtk):
Output: http://queues.webkit.org/results/13339395

[Simon Hausmann]

Comment on attachment 154378 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=154378&action=review

> Tools/WebKitTestRunner/TestController.cpp:432
> +    if (m_eventSenderProxy)
> +        delete m_eventSenderProxy;

You don't need to check m_eventSenderProxy for being non-null before deleting it. delete 0 is fine :)

However I don't think this approach is quite right. It _seems_ the cause is the event queues in the event sender proxy, which is something only non-mac ports "suffer" from.

Either our approach of leapForward should be imlemented differently (can we specify time stamps in the events like mac?) or perhaps we are missing some sort of reset() call here in resetStateToConsistentValues() that clears the queue and has a comment explaining why. Deleting and re-creating the event sender proxy is just hiding the real problem IMHO.

[Csaba Osztrogonác]

There are many notifydone-timeout test (and pass after retry) because of 
this bug. Unfortunately https://trac.webkit.org/changeset/128395 increased
the notifydone timeout from 6 seconds to 30 seconds. It slowed down the
testing time from 15.5 minutes to 23.5 minutes.

I'm back to fix this bug, runtime is good for me, but there are still
4-5 fails (not flakey, fail), so it seems reseting of some members of
EventSenderProxy are still missing.

[Csaba Osztrogonác]

http://trac.webkit.org/changeset/129486 unskipped a bunch of tests,
and this bug became hidden again. :-/ But I'm going to fix once I
have a day when folks don't break everything at least 5-10 times.

[Csaba Osztrogonác]

*** This bug has been marked as a duplicate of bug 101353 ***