Bug 206545

Summary:	PerformanceTests should not treat stderr output as a test failure
Product:	WebKit	Reporter:	Brent Fulgham <bfulgham>
Component:	Tools / Tests	Assignee:	Nobody <webkit-unassigned>
Status:	NEW ---
Severity:	Normal	CC:	ap, rniwa, webkit-bug-importer
Priority:	P2	Keywords:	InRadar
Version:	WebKit Nightly Build
Hardware:	Unspecified
OS:	Unspecified
See Also:	https://bugs.webkit.org/show_bug.cgi?id=206544

Description Brent Fulgham 2020-01-21 11:41:56 PST

The PerformanceTests test harness treats any stderr output as a test failure. Instead, it should treat it like the normal test system, where stderr is captured and noted, but does not contribute to the pass or fail status of a test.

Comment 1 Radar WebKit Bug Importer 2020-01-21 11:42:09 PST

<rdar://problem/58767355>

Comment 2 Alexey Proskuryakov 2020-01-21 12:39:20 PST

IIRC Ryosuke found it very important to catch errors on this way. Perhaps this is less relevant after redoing much of the logging system?

Comment 3 Ryosuke Niwa 2020-01-21 23:38:17 PST

(In reply to Brent Fulgham from comment #0)
> The PerformanceTests test harness treats any stderr output as a test
> failure. Instead, it should treat it like the normal test system, where
> stderr is captured and noted, but does not contribute to the pass or fail
> status of a test.

I don't think we want to do this. Some of our internal benchmarks get broken all the time because we ignore all error messages.