Bug 93434 – [NRWT] REGRESSION(r124967): New tests without expected results handled as failures

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Csaba Osztrogonác]

Before r124967 new tests without expected files were handled as new tests,
not as failures. Which means only orange buildbot, not red buildbot. But
after r124967, new tests make the bot red, which is not the expected behaviour.

Let's see the NRWT's output:

before r124967:
----------------
Unexpected flakiness: text diff mismatch (4)
  fast/forms/range/slider-mouse-events.html = TEXT PASS
  fast/forms/range/slider-onchange-event.html = TEXT PASS
  http/tests/security/contentSecurityPolicy/object-src-url-allowed.html = TEXT PASS
  http/tests/security/cross-frame-access-put.html = TEXT PASS


Regressions: Unexpected no expected results found : (2)
  fast/table/table-row-outline-paint.html = MISSING
  svg/as-image/animated-svg-repaints-completely-in-hidpi.html = MISSING


and the buildbot reports:  4 flakes 2 missing results 

after r124967:
---------------
Unexpected flakiness: text failures (4)
  fast/forms/range/slider-mouse-events.html = TEXT PASS
  fast/forms/range/slider-onchange-event.html = TEXT PASS
  http/tests/security/contentSecurityPolicy/object-src-url-allowed.html = TEXT PASS
  http/tests/security/cross-frame-access-put.html = TEXT PASS


Regressions: Unexpected missing results : (2)
  fast/table/table-row-outline-paint.html = MISSING
  svg/as-image/animated-svg-repaints-completely-in-hidpi.html = MISSING


and the buildbot reports:  2 failures 4 flakes


It seems it is a similar bug as https://bugs.webkit.org/show_bug.cgi?id=93346

[Csaba Osztrogonác]

Yes, it is so similar bug ...
The new "Regressions: Unexpected missing results" match for the regexp of failures ..


Tools/BuildSlaveSupport/build.webkit.org-config/master.cfg
-----------------------------------------------------------
...

    def _parseNewRunWebKitTestsOutput(self, logText):
        incorrectLayoutLines = []
        expressions = [
            ('flakes', re.compile(r'Unexpected flakiness.+:?\s*\((\d+)\)')),
            ('new passes', re.compile(r'Expected to .+, but passed:\s+\((\d+)\)')),
            ('missing results', re.compile(r'no expected results found\s*:\s+\((\d+)\)')),
            ('failures', re.compile(r'Regressions: Unexpected.+:?\s*\((\d+)\)')),
        ]
        testFailures = {}
...

[Csaba Osztrogonác]

Created attachment 157115 [details]
Patch

[Csaba Osztrogonác]

If we really want to change the output of NRWT, please make it sync with master.cfg changes. And please contact with Lucas and/or Bill to restart
the master after the change to ensure correct behaviour.

And please ping me, because I have to make similar changes in master.cfg
of build.webkit.sed.hu. (Maybe you have to make similar changes in
master.cfg of build.chromium.org too)

[Csaba Osztrogonác]

Comment on attachment 157115 [details]
Patch

Clearing flags on attachment: 157115

Committed r124988: <http://trac.webkit.org/changeset/124988>

[Csaba Osztrogonác]

All reviewed patches have been landed.  Closing bug.

[Dirk Pranke]

Comment on attachment 157115 [details]
Patch

Sorry! (again).

[Dirk Pranke]

(In reply to comment #6)
> (From update of attachment 157115 [details])
> Sorry! (again).

To be clear(er), it was not actually my intention to change the output that the bot would be seeing :(. I did not mean to break the parser (or require changes to the parser).

[Csaba Osztrogonác]

not problem ;)

I didn't notice that the order of two messages changed
(printed for each test, printed at the end), so my fix
wasn't correct. I fixed it - https://trac.webkit.org/changeset/124994