Bug 84917 - [chromium] use "drt-style" output, not "test-shell-style" output, on mac and linux DRT
Summary: [chromium] use "drt-style" output, not "test-shell-style" output, on mac and ...
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: New Bugs (show other bugs)
Version: 528+ (Nightly build)
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Dirk Pranke
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2012-04-25 17:48 PDT by Dirk Pranke
Modified: 2012-04-27 16:32 PDT (History)
7 users (show)

See Also:


Attachments
Patch (11.45 KB, patch)
2012-04-25 17:51 PDT, Dirk Pranke
no flags Details | Formatted Diff | Diff
fix unit tests (11.45 KB, patch)
2012-04-25 19:01 PDT, Dirk Pranke
no flags Details | Formatted Diff | Diff
Archive of layout-test-results from ec2-cr-linux-03 (2.66 MB, application/zip)
2012-04-26 14:57 PDT, WebKit Review Bot
no flags Details
update w/ review feedback (10.62 KB, patch)
2012-04-26 15:18 PDT, Dirk Pranke
no flags Details | Formatted Diff | Diff
Archive of layout-test-results from ec2-cr-linux-03 (6.30 MB, application/zip)
2012-04-27 05:38 PDT, WebKit Review Bot
no flags Details
fix win brokenness (12.19 KB, patch)
2012-04-27 14:00 PDT, Dirk Pranke
ojan: review+
Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Dirk Pranke 2012-04-25 17:48:14 PDT
[chromium] use "drt-style" output, not "test-shell-style" output, on mac and linux DRT
Comment 1 Dirk Pranke 2012-04-25 17:51:56 PDT
Created attachment 138910 [details]
Patch
Comment 2 Dirk Pranke 2012-04-25 19:01:07 PDT
Created attachment 138924 [details]
fix unit tests
Comment 3 Ojan Vafai 2012-04-25 19:06:25 PDT
Comment on attachment 138924 [details]
fix unit tests

View in context: https://bugs.webkit.org/attachment.cgi?id=138924&action=review

> Tools/Scripts/webkitpy/layout_tests/port/chromium.py:668
> +            time.sleep(0.01)

I"m fine with squeezing this into this patchm, but please include a line in the ChangeLog.
Comment 4 Dirk Pranke 2012-04-25 19:11:44 PDT
Comment on attachment 138924 [details]
fix unit tests

View in context: https://bugs.webkit.org/attachment.cgi?id=138924&action=review

>> Tools/Scripts/webkitpy/layout_tests/port/chromium.py:668
>> +            time.sleep(0.01)
> 
> I"m fine with squeezing this into this patchm, but please include a line in the ChangeLog.

Oh, this wasn't supposed to be in this patch at all, I don't think.
Comment 5 WebKit Review Bot 2012-04-26 14:57:45 PDT
Comment on attachment 138924 [details]
fix unit tests

Attachment 138924 [details] did not pass chromium-ews (chromium-xvfb):
Output: http://queues.webkit.org/results/12560004

New failing tests:
platform/chromium/virtual/gpu/fast/canvas/canvas-text-baseline.html
platform/chromium/virtual/gpu/fast/canvas/fillrect_gradient.html
platform/chromium/virtual/gpu/fast/canvas/canvas-bg.html
platform/chromium/virtual/gpu/fast/canvas/canvas-resize-after-paint-without-layout.html
platform/chromium/virtual/gpu/canvas/philip/tests/2d.gradient.radial.cone.cylinder.html
platform/chromium/virtual/gpu/fast/canvas/patternfill-repeat.html
platform/chromium/virtual/gpu/fast/canvas/canvas-shadow.html
http/tests/security/sandboxed-iframe-modify-self.html
platform/chromium/virtual/gpu/fast/canvas/canvas-as-image-incremental-repaint.html
platform/chromium/virtual/gpu/fast/canvas/gradient-add-second-start-end-stop.html
accessibility/aria-disabled.html
fast/frames/lots-of-objects.html
platform/chromium/virtual/gpu/fast/canvas/set-colors.html
fast/loader/text-document-wrapping.html
platform/chromium/virtual/gpu/fast/canvas/canvas-text-alignment.html
platform/chromium/virtual/gpu/fast/canvas/setWidthResetAfterForcedRender.html
platform/chromium/virtual/gpu/fast/canvas/canvas-composite-fill-repaint.html
fast/canvas/webgl/shader-precision-format.html
platform/chromium/virtual/gpu/fast/canvas/canvas-transforms-during-path.html
platform/chromium/virtual/gpu/fast/canvas/canvas-as-image.html
platform/chromium/virtual/gpu/fast/canvas/canvas-bg-zoom.html
platform/chromium/virtual/gpu/fast/canvas/fillrect-gradient-zero-stops.html
http/tests/xmlhttprequest/xmlhttprequest-unsafe-redirect.html
platform/chromium/virtual/gpu/fast/canvas/canvas-currentColor.html
platform/chromium/virtual/gpu/canvas/philip/tests/2d.line.width.basic.html
platform/chromium/virtual/gpu/canvas/philip/tests/2d.path.arcTo.shape.curve2.html
platform/chromium/virtual/gpu/canvas/philip/tests/2d.line.width.transformed.html
platform/chromium/virtual/gpu/canvas/philip/tests/2d.path.arcTo.shape.curve1.html
Comment 6 WebKit Review Bot 2012-04-26 14:57:49 PDT
Created attachment 139077 [details]
Archive of layout-test-results from ec2-cr-linux-03

The attached test failures were seen while running run-webkit-tests on the chromium-ews.
Bot: ec2-cr-linux-03  Port: <class 'webkitpy.common.config.ports.ChromiumXVFBPort'>  Platform: Linux-2.6.35-28-virtual-x86_64-with-Ubuntu-10.10-maverick
Comment 7 Dirk Pranke 2012-04-26 15:18:27 PDT
Created attachment 139081 [details]
update w/ review feedback
Comment 8 Dirk Pranke 2012-04-26 15:32:00 PDT
the ews bots results are due to this running without bug 84910 having been fixed first ...
Comment 9 WebKit Review Bot 2012-04-27 05:37:59 PDT
Comment on attachment 139081 [details]
update w/ review feedback

Attachment 139081 [details] did not pass chromium-ews (chromium-xvfb):
Output: http://queues.webkit.org/results/12556185

New failing tests:
fast/images/gif-large-checkerboard.html
svg/custom/clip-path-referencing-use2.svg
Comment 10 WebKit Review Bot 2012-04-27 05:38:06 PDT
Created attachment 139176 [details]
Archive of layout-test-results from ec2-cr-linux-03

The attached test failures were seen while running run-webkit-tests on the chromium-ews.
Bot: ec2-cr-linux-03  Port: <class 'webkitpy.common.config.ports.ChromiumXVFBPort'>  Platform: Linux-2.6.35-28-virtual-x86_64-with-Ubuntu-10.10-maverick
Comment 11 Dirk Pranke 2012-04-27 11:13:40 PDT
(In reply to comment #9)
> (From update of attachment 139081 [details])
> Attachment 139081 [details] did not pass chromium-ews (chromium-xvfb):
> Output: http://queues.webkit.org/results/12556185
> 
> New failing tests:
> fast/images/gif-large-checkerboard.html

I think this is an unrelated failure.

> svg/custom/clip-path-referencing-use2.svg

I've reproduced this one and am totally befuddled by it. My best guess is that this has something to do with "test shell mode" getting file:/// URLs and "drt mode" getting absolute paths and this triggering a different code path in DRT. I think this is actually fixing an issue, but I know so little about SVG I don't know. I'm gonna file a separate bug for this and suppress the failure until someone more knowledgeable can look into it.
Comment 12 Dirk Pranke 2012-04-27 11:23:56 PDT
Committed r115453: <http://trac.webkit.org/changeset/115453>
Comment 13 Dirk Pranke 2012-04-27 13:54:45 PDT
This was rolled out in r115469 because the chromium win bots were busted; the logic for handling windows in ChromiumDriver caused DRT to get launched without the --test-shell flag.
Comment 14 Dirk Pranke 2012-04-27 14:00:02 PDT
Created attachment 139268 [details]
fix win brokenness
Comment 15 Dirk Pranke 2012-04-27 14:02:39 PDT
Committed r115490: <http://trac.webkit.org/changeset/115490>
Comment 16 Tony Chang 2012-04-27 15:44:14 PDT
Out of curiosity, how did you test this change?  Previously, DRT mode had lots of flakiness on Mac/Linux when we used it (sometimes a bunch of tests would start timing out).  E.g., were the failures in comment 5 and comment 9 expected and accounted for?
Comment 17 Dirk Pranke 2012-04-27 16:20:12 PDT
(In reply to comment #16)
> Out of curiosity, how did you test this change?  Previously, DRT mode had lots of flakiness on Mac/Linux when we used it (sometimes a bunch of tests would start timing out).  

Sure, good question. I spent about a week doing a combination of eliminating general flakiness for my local mac and linux bots (in regular --test-shell mode) and then making sure that the tests ran reliably w/o flakiness for both modes, fixing a several bugs in the process in both webkitpy and DRT (see bugs 84904 and 84910) 

In addition, the webkit.py / server_process model has had a lot of bugs beaten out of it (on the apple bots and others) since we first tried drt mode.

> E.g., were the failures in comment 5 and comment 9 expected and accounted for?

the failures in comment #5 I addressed in earlier comments; the one test that actually seems to fail only in DRT mode was filed as a separate bug, bug 85085, which I would be curious to see your thoughts on.

I am hoping that this'll stay stable on the bots, but the only way to be sure would be to either land it and turn it on, or set up an experimental bot. I was confident enough in the change (and it's easy enough to revert) that setting up another bot seemed like way overkill.
Comment 18 Tony Chang 2012-04-27 16:28:07 PDT
I think it's fine to turn this on. I agree it's the only way to test.

I just want to make sure that if flakiness increases, it's something that a gardener will be able to detect (rather than just marking a bunch of tests as flaky in test_expectations.txt).

When you were testing, did you see lots of timeouts?  What should a gardener be watching for to know if things are broken?
Comment 19 Dirk Pranke 2012-04-27 16:32:34 PDT
(In reply to comment #18)
> I think it's fine to turn this on. I agree it's the only way to test.
> 
> I just want to make sure that if flakiness increases, it's something that a gardener will be able to detect (rather than just marking a bunch of tests as flaky in test_expectations.txt).
> 
> When you were testing, did you see lots of timeouts?  What should a gardener be watching for to know if things are broken?

after I fixed the bugs, things ran about the same (there was no increase in timeouts). I would say if anything seemed unusual at all they should let me know. I'll send a note to webkit-gardening@ to that effect.