Bug 84917 – [chromium] use "drt-style" output, not "test-shell-style" output, on mac and linux DRT

• Component: New Bugs
• Status: RESOLVED FIXED

[Dirk Pranke]

[chromium] use "drt-style" output, not "test-shell-style" output, on mac and linux DRT

[Dirk Pranke]

Created attachment 138910 [details]
Patch

[Dirk Pranke]

Created attachment 138924 [details]
fix unit tests

[Ojan Vafai]

Comment on attachment 138924 [details]
fix unit tests

View in context: https://bugs.webkit.org/attachment.cgi?id=138924&action=review

> Tools/Scripts/webkitpy/layout_tests/port/chromium.py:668
> +            time.sleep(0.01)

I"m fine with squeezing this into this patchm, but please include a line in the ChangeLog.

[Dirk Pranke]

Comment on attachment 138924 [details]
fix unit tests

View in context: https://bugs.webkit.org/attachment.cgi?id=138924&action=review

>> Tools/Scripts/webkitpy/layout_tests/port/chromium.py:668
>> +            time.sleep(0.01)
> 
> I"m fine with squeezing this into this patchm, but please include a line in the ChangeLog.

Oh, this wasn't supposed to be in this patch at all, I don't think.

[WebKit Review Bot]

Comment on attachment 138924 [details]
fix unit tests

Attachment 138924 [details] did not pass chromium-ews (chromium-xvfb):
Output: http://queues.webkit.org/results/12560004

New failing tests:
platform/chromium/virtual/gpu/fast/canvas/canvas-text-baseline.html
platform/chromium/virtual/gpu/fast/canvas/fillrect_gradient.html
platform/chromium/virtual/gpu/fast/canvas/canvas-bg.html
platform/chromium/virtual/gpu/fast/canvas/canvas-resize-after-paint-without-layout.html
platform/chromium/virtual/gpu/canvas/philip/tests/2d.gradient.radial.cone.cylinder.html
platform/chromium/virtual/gpu/fast/canvas/patternfill-repeat.html
platform/chromium/virtual/gpu/fast/canvas/canvas-shadow.html
http/tests/security/sandboxed-iframe-modify-self.html
platform/chromium/virtual/gpu/fast/canvas/canvas-as-image-incremental-repaint.html
platform/chromium/virtual/gpu/fast/canvas/gradient-add-second-start-end-stop.html
accessibility/aria-disabled.html
fast/frames/lots-of-objects.html
platform/chromium/virtual/gpu/fast/canvas/set-colors.html
fast/loader/text-document-wrapping.html
platform/chromium/virtual/gpu/fast/canvas/canvas-text-alignment.html
platform/chromium/virtual/gpu/fast/canvas/setWidthResetAfterForcedRender.html
platform/chromium/virtual/gpu/fast/canvas/canvas-composite-fill-repaint.html
fast/canvas/webgl/shader-precision-format.html
platform/chromium/virtual/gpu/fast/canvas/canvas-transforms-during-path.html
platform/chromium/virtual/gpu/fast/canvas/canvas-as-image.html
platform/chromium/virtual/gpu/fast/canvas/canvas-bg-zoom.html
platform/chromium/virtual/gpu/fast/canvas/fillrect-gradient-zero-stops.html
http/tests/xmlhttprequest/xmlhttprequest-unsafe-redirect.html
platform/chromium/virtual/gpu/fast/canvas/canvas-currentColor.html
platform/chromium/virtual/gpu/canvas/philip/tests/2d.line.width.basic.html
platform/chromium/virtual/gpu/canvas/philip/tests/2d.path.arcTo.shape.curve2.html
platform/chromium/virtual/gpu/canvas/philip/tests/2d.line.width.transformed.html
platform/chromium/virtual/gpu/canvas/philip/tests/2d.path.arcTo.shape.curve1.html

[WebKit Review Bot]

Created attachment 139077 [details]
Archive of layout-test-results from ec2-cr-linux-03

The attached test failures were seen while running run-webkit-tests on the chromium-ews.
Bot: ec2-cr-linux-03  Port: <class 'webkitpy.common.config.ports.ChromiumXVFBPort'>  Platform: Linux-2.6.35-28-virtual-x86_64-with-Ubuntu-10.10-maverick

[Dirk Pranke]

Created attachment 139081 [details]
update w/ review feedback

[Dirk Pranke]

the ews bots results are due to this running without bug 84910 having been fixed first ...

[WebKit Review Bot]

Comment on attachment 139081 [details]
update w/ review feedback

Attachment 139081 [details] did not pass chromium-ews (chromium-xvfb):
Output: http://queues.webkit.org/results/12556185

New failing tests:
fast/images/gif-large-checkerboard.html
svg/custom/clip-path-referencing-use2.svg

[WebKit Review Bot]

Created attachment 139176 [details]
Archive of layout-test-results from ec2-cr-linux-03

The attached test failures were seen while running run-webkit-tests on the chromium-ews.
Bot: ec2-cr-linux-03  Port: <class 'webkitpy.common.config.ports.ChromiumXVFBPort'>  Platform: Linux-2.6.35-28-virtual-x86_64-with-Ubuntu-10.10-maverick

[Dirk Pranke]

(In reply to comment #9)
> (From update of attachment 139081 [details])
> Attachment 139081 [details] did not pass chromium-ews (chromium-xvfb):
> Output: http://queues.webkit.org/results/12556185
> 
> New failing tests:
> fast/images/gif-large-checkerboard.html

I think this is an unrelated failure.

> svg/custom/clip-path-referencing-use2.svg

I've reproduced this one and am totally befuddled by it. My best guess is that this has something to do with "test shell mode" getting file:/// URLs and "drt mode" getting absolute paths and this triggering a different code path in DRT. I think this is actually fixing an issue, but I know so little about SVG I don't know. I'm gonna file a separate bug for this and suppress the failure until someone more knowledgeable can look into it.

[Dirk Pranke]

Committed r115453: <http://trac.webkit.org/changeset/115453>

[Dirk Pranke]

This was rolled out in r115469 because the chromium win bots were busted; the logic for handling windows in ChromiumDriver caused DRT to get launched without the --test-shell flag.

[Dirk Pranke]

Created attachment 139268 [details]
fix win brokenness

[Dirk Pranke]

Committed r115490: <http://trac.webkit.org/changeset/115490>

[Tony Chang]

Out of curiosity, how did you test this change?  Previously, DRT mode had lots of flakiness on Mac/Linux when we used it (sometimes a bunch of tests would start timing out).  E.g., were the failures in comment 5 and comment 9 expected and accounted for?

[Dirk Pranke]

(In reply to comment #16)
> Out of curiosity, how did you test this change?  Previously, DRT mode had lots of flakiness on Mac/Linux when we used it (sometimes a bunch of tests would start timing out).  

Sure, good question. I spent about a week doing a combination of eliminating general flakiness for my local mac and linux bots (in regular --test-shell mode) and then making sure that the tests ran reliably w/o flakiness for both modes, fixing a several bugs in the process in both webkitpy and DRT (see bugs 84904 and 84910) 

In addition, the webkit.py / server_process model has had a lot of bugs beaten out of it (on the apple bots and others) since we first tried drt mode.

> E.g., were the failures in comment 5 and comment 9 expected and accounted for?

the failures in comment #5 I addressed in earlier comments; the one test that actually seems to fail only in DRT mode was filed as a separate bug, bug 85085, which I would be curious to see your thoughts on.

I am hoping that this'll stay stable on the bots, but the only way to be sure would be to either land it and turn it on, or set up an experimental bot. I was confident enough in the change (and it's easy enough to revert) that setting up another bot seemed like way overkill.

[Tony Chang]

I think it's fine to turn this on. I agree it's the only way to test.

I just want to make sure that if flakiness increases, it's something that a gardener will be able to detect (rather than just marking a bunch of tests as flaky in test_expectations.txt).

When you were testing, did you see lots of timeouts?  What should a gardener be watching for to know if things are broken?

[Dirk Pranke]

(In reply to comment #18)
> I think it's fine to turn this on. I agree it's the only way to test.
> 
> I just want to make sure that if flakiness increases, it's something that a gardener will be able to detect (rather than just marking a bunch of tests as flaky in test_expectations.txt).
> 
> When you were testing, did you see lots of timeouts?  What should a gardener be watching for to know if things are broken?

after I fixed the bugs, things ran about the same (there was no increase in timeouts). I would say if anything seemed unusual at all they should let me know. I'll send a note to webkit-gardening@ to that effect.