Bug 104435
| Summary: | TestExpectations knows tests fail but NRWT reports a regression | ||
|---|---|---|---|
| Product: | WebKit | Reporter: | noel gordon <noel.gordon> |
| Component: | Tools / Tests | Assignee: | Dirk Pranke <dpranke> |
| Status: | RESOLVED DUPLICATE | ||
| Severity: | Normal | CC: | dpranke, eae, ojan, podivilov, senorblanco, tkent |
| Priority: | P2 | ||
| Version: | 528+ (Nightly build) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
noel gordon
http://build.webkit.org/builders/Chromium%20Win%20Release%20%28Tests%29/builds/31815/steps/layout-test/logs/stdio
Regressions: Unexpected audio failures (4)
webaudio/audiobuffersource-loop-points.html [ Failure ]
webaudio/codec-tests/vorbis/vbr-128kbps-44khz.html [ Failure ]
webaudio/codec-tests/vorbis/vbr-70kbps-44khz.html [ Failure ]
webaudio/codec-tests/vorbis/vbr-128kbps-44khz.html [ Failure ]
However, chromium TestExpectations contains:
% ack "buffersource-loop" LayoutTests/platform/chromium/TestExpectations
webkit.org/b/100647 [ Win ] webaudio/audiobuffersource-loop-points.html [ Failure ]
% ack "vorbis/vbr" LayoutTests/platform/chromium/TestExpectations
Bug(dpranke) [ Win ] webaudio/codec-tests/vorbis/vbr-70kbps-44khz.html [ Failure ]
Bug(dpranke) [ Win ] webaudio/codec-tests/vorbis/vbr-96kbps-44khz.html [ Failure ]
Bug(dpranke) [ Win ] webaudio/codec-tests/vorbis/vbr-128kbps-44khz.html [ Failure ]
Seems these are expected failures, not regressions?
| Attachments | ||
|---|---|---|
| Add attachment proposed patch, testcase, etc. |
Dirk Pranke
Ah, yeah, looks like I broke this case (mapping AUDIO onto FAIL) in my recent refactoring.
noel gordon
Currently on chrome linux.
http://build.webkit.org/builders/Chromium%20Linux%20Release%20%28Tests%29/builds/42173/steps/layout-test/logs/stdio
Regressions: Unexpected image and text failures (5)
fast/text/hyphenate-character.html [ Failure ]
fast/text/hyphenate-first-word.html [ Failure ]
fast/text/hyphenate-limit-lines.html [ Failure ]
fast/text/hyphenate-locale.html [ Failure ]
fast/text/hyphens.html [ Failure ]
% ack "text/hyphen" LayoutTests/platform/chromium/TestExpectations
crbug.com/47083 fast/text/hyphens.html [ Failure ImageOnlyFailure ]
crbug.com/47083 fast/text/hyphenate-character.html [ Failure ImageOnlyFailure ]
crbug.com/47083 fast/text/hyphenate-first-word.html [ Failure ImageOnlyFailure ]
crbug.com/47083 fast/text/hyphenate-limit-lines.html [ Failure ImageOnlyFailure ]
crbug.com/47083 fast/text/hyphenate-limit-before-after.html [ Failure ImageOnlyFailure ]
crbug.com/47083 fast/text/hyphenate-locale.html [ Failure ImageOnlyFailure ]
Not just mapping AUDIO into FAIL it seems.
Dirk Pranke
Hm. That's puzzling ... if mapping IMAGE+TEXT onto Failure was broken, I'd expect a lot more than 5 failures. Looking further ...
Dirk Pranke
(In reply to comment #3)
> Hm. That's puzzling ... if mapping IMAGE+TEXT onto Failure was broken, I'd expect a lot more than 5 failures. Looking further ...
I can't reproduce this now ... I think maybe you happened to see a failure on the bots after r137009 but hadn't updated your local checkout yet, and tkent added the suppressions back in r137144.
Stephen White
(In reply to comment #4)
> (In reply to comment #3)
> > Hm. That's puzzling ... if mapping IMAGE+TEXT onto Failure was broken, I'd expect a lot more than 5 failures. Looking further ...
>
> I can't reproduce this now ... I think maybe you happened to see a failure on the bots after r137009 but hadn't updated your local checkout yet, and tkent added the suppressions back in r137144.
What's weird is, AFAICT they're only failing on the build.webkit.org bots. They're passing on the canaries.
Might need a Chromium DEPS roll into WebKit; I'm not sure.
Dirk Pranke
bug fix posted in bug 104597 (I forgot that this was a dedicated bug for this issue). I'm going to close this as a dup of 104597 for now, but if we find other problems with other results showing up incorrectly, we can re-open this to reuse it.
*** This bug has been marked as a duplicate of bug 104597 ***