Bug 110734 - Layout Test Multiple html5lib/ tests are too slow and should be split.
Summary: Layout Test Multiple html5lib/ tests are too slow and should be split.
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: 528+ (Nightly build)
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Eric Seidel (no email)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-02-25 01:09 PST by Vsevolod Vlasov
Modified: 2013-02-26 23:51 PST (History)
7 users (show)

See Also:


Attachments
Patch (151.14 KB, patch)
2013-02-25 15:50 PST, Eric Seidel (no email)
no flags Details | Formatted Diff | Diff
Patch for landing (151.84 KB, patch)
2013-02-25 16:28 PST, Eric Seidel (no email)
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Vsevolod Vlasov 2013-02-25 01:09:35 PST
Similar bug for the main test: https://bugs.webkit.org/show_bug.cgi?id=110642

The following layout tests are too slow:

html5lib/run-test9.html
html5lib/run-test6.html
html5lib/run-template.html
html5lib/run-test20.html
html5lib/run-test1.html
html5lib/run-test7.html
html5lib/run-test10.html
html5lib/run-test16.html
html5lib/run-test19.html
html5lib/run-test2.html

http://test-results.appspot.com/dashboards/flakiness_dashboard.html#tests=html5lib%2Frun-test9.html%2Chtml5lib%2Frun-test6.html%2Chtml5lib%2Frun-template.html%2Chtml5lib%2Frun-test20.html%2Chtml5lib%2Frun-test1.html%2Chtml5lib%2Frun-test10.html%2Chtml5lib%2Frun-test16.html%2Chtml5lib%2Frun-test19.html%2Chtml5lib%2Frun-test2.html
Comment 1 Eric Seidel (no email) 2013-02-25 01:14:50 PST
Thank you for filing.  I'm not yet sure what we should do about these, besides mark them as slow, but I'll look in the morning.
Comment 2 Vsevolod Vlasov 2013-02-25 01:32:03 PST
Committed r143891: <http://trac.webkit.org/changeset/143891>
Comment 3 Eric Seidel (no email) 2013-02-25 15:50:30 PST
Created attachment 190143 [details]
Patch
Comment 4 Ojan Vafai 2013-02-25 16:12:18 PST
Comment on attachment 190143 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=190143&action=review

> LayoutTests/ChangeLog:42
> +        * html5lib/generated/run-adoption01-data-expected.txt: Added.
> +        * html5lib/generated/run-adoption01-data.html: Copied from LayoutTests/html5lib/generated/run-tests1.html.
> +        * html5lib/generated/run-adoption01-write-expected.txt: Added.
> +        * html5lib/generated/run-adoption01-write.html: Renamed from LayoutTests/html5lib/generated/run-template.html.
> +        * html5lib/generated/run-adoption02-data-expected.txt: Added.
> +        * html5lib/generated/run-adoption02-data.html: Copied from LayoutTests/html5lib/generated/run-tests1.html.
> +        * html5lib/generated/run-adoption02-write-expected.txt: Added.
> +        * html5lib/generated/run-adoption02-write.html: Renamed from LayoutTests/html5lib/generated/run-tests11.html.
> +        * html5lib/generated/run-comments01-data-expected.txt: Added.
> +        * html5lib/generated/run-comments01-data.html: Copied from LayoutTests/html5lib/generated/run-tests1.html.

Standard practice when there's a very long file list like this is to delete it.

> LayoutTests/html5lib/generate-test-wrappers.py:30
> +# Used for generating LayoutTest-compatible html files to run html5lib *.dat files.

This should either have a usage thing in the comment or when you run on the commandline (e.g. with --help).

> LayoutTests/html5lib/generate-test-wrappers.py:79
> +                os.remove(path)

Should this not be scm.remove or some equivalent?
Comment 5 Adam Barth 2013-02-25 16:24:12 PST
Comment on attachment 190143 [details]
Patch

I would add a README
Comment 6 Eric Seidel (no email) 2013-02-25 16:28:11 PST
Created attachment 190152 [details]
Patch for landing
Comment 7 WebKit Review Bot 2013-02-26 02:33:31 PST
Comment on attachment 190152 [details]
Patch for landing

Rejecting attachment 190152 [details] from commit-queue.

Failed to run "['/mnt/git/webkit-commit-queue/Tools/Scripts/webkit-patch', '--status-host=webkit-commit-queue.appspot.com', '--bot-id=gce-cq-02', 'land-attachment', '--force-clean', '--non-interactive', '--parent-command=commit-queue', 190152, '--port=chromium-xvfb']" exit_code: 2 cwd: /mnt/git/webkit-commit-queue

Last 500 characters of output:
WebKit/chromium/v8 --revision 13692 --non-interactive --force --accept theirs-conflict --ignore-externals' in '/mnt/git/webkit-commit-queue/Source/WebKit/chromium'
54>At revision 13692.

________ running '/usr/bin/python tools/clang/scripts/update.py --mac-only' in '/mnt/git/webkit-commit-queue/Source/WebKit/chromium'

________ running '/usr/bin/python gyp_webkit' in '/mnt/git/webkit-commit-queue/Source/WebKit/chromium'
Updating webkit projects from gyp files...
Total errors found: 0 in 0 files

Full output: http://webkit-commit-queue.appspot.com/results/16747951
Comment 8 WebKit Review Bot 2013-02-26 03:07:11 PST
Comment on attachment 190152 [details]
Patch for landing

Clearing flags on attachment: 190152

Committed r144032: <http://trac.webkit.org/changeset/144032>
Comment 9 WebKit Review Bot 2013-02-26 03:07:14 PST
All reviewed patches have been landed.  Closing bug.
Comment 10 Csaba Osztrogonác 2013-02-26 03:57:22 PST
(In reply to comment #8)
> (From update of attachment 190152 [details])
> Clearing flags on attachment: 190152
> 
> Committed r144032: <http://trac.webkit.org/changeset/144032>

FYI: There are 8 failures after this patch everywhere:
- Qt - http://build.webkit.sed.hu/results/x86-32%20Linux%20Qt%20Release%20NRWT/r144032%20%2829232%29/results.html
- GTK - http://build.webkit.org/results/GTK%20Linux%2064-bit%20Release/r144033%20%2834924%29/results.html
...
Comment 11 Vsevolod Vlasov 2013-02-26 05:49:30 PST
Some html5lib tests are still timing out https://bugs.webkit.org/show_bug.cgi?id=110876
Comment 12 Csaba Osztrogonác 2013-02-26 06:02:22 PST
(In reply to comment #10)
> (In reply to comment #8)
> > (From update of attachment 190152 [details] [details])
> > Clearing flags on attachment: 190152
> > 
> > Committed r144032: <http://trac.webkit.org/changeset/144032>
> 
> FYI: There are 8 failures after this patch everywhere:
> - Qt - http://build.webkit.sed.hu/results/x86-32%20Linux%20Qt%20Release%20NRWT/r144032%20%2829232%29/results.html
> - GTK - http://build.webkit.org/results/GTK%20Linux%2064-bit%20Release/r144033%20%2834924%29/results.html
> ...

Updated landed due to JSC/V8 differences in https://trac.webkit.org/changeset/144051

But unfortunately there are 2 really failing tests everywhere:
--- /Volumes/Data/slave/lion-debug-tests-wk1/build/layout-test-results/html5lib/generated/run-template-data-expected.txt
+++ /Volumes/Data/slave/lion-debug-tests-wk1/build/layout-test-results/html5lib/generated/run-template-data-actual.txt
@@ -1,1 +1,3 @@
-../resources/template.dat: PASS
+CONSOLE MESSAGE: line 149: TypeError: 'undefined' is not an object (evaluating 'node.nodeType')
+FAIL: Timed out waiting for notifyDone to be called
+Running test 1 of 96 in ../resources/template.dat

--- /Volumes/Data/slave/lion-debug-tests-wk1/build/layout-test-results/html5lib/generated/run-template-write-expected.txt
+++ /Volumes/Data/slave/lion-debug-tests-wk1/build/layout-test-results/html5lib/generated/run-template-write-actual.txt
@@ -1,1 +1,3 @@
-../resources/template.dat: PASS
+CONSOLE MESSAGE: line 149: TypeError: 'undefined' is not an object (evaluating 'node.nodeType')
+FAIL: Timed out waiting for notifyDone to be called
+Running test 1 of 96 in ../resources/template.dat


I marked them as failure in the general TestExpectations, 
please check and fix this regression.
Comment 13 Csaba Osztrogonác 2013-02-26 06:10:32 PST
I checked TestExpectations files, run-template.html was skipped on Qt/GTK/Mac,
because ENABLE(TEMPLATE_ELEMENT) is disabled. Are the new run-template-data.html and run-template-write.html originated from this test? If yes,
we should simple skip them on these ports.
Comment 14 Eric Seidel (no email) 2013-02-26 11:29:29 PST
Thank you ossy.  I was expecting the JSC errors, but didn't have a JSC port from my machine.  Thanks for taking care of it.
Comment 15 Eric Seidel (no email) 2013-02-26 11:30:20 PST
Yes, run-template* is skipped on most ports, but not Chromium.
Comment 16 Eric Seidel (no email) 2013-02-26 16:13:40 PST
This change is very wrong:
http://trac.webkit.org/changeset/144051/trunk/LayoutTests/html5lib/generated/run-template-write.html

All the generated files shoudl not be hand edited.  And run-template should run template.dat... not tests3. :)
Comment 17 Csaba Osztrogonác 2013-02-26 23:15:40 PST
(In reply to comment #16)
> This change is very wrong:
> http://trac.webkit.org/changeset/144051/trunk/LayoutTests/html5lib/generated/run-template-write.html
> 
> All the generated files shoudl not be hand edited.  And run-template should run template.dat... not tests3. :)

Ooops, it was an accidental change, will fix immediately.
Comment 18 Csaba Osztrogonác 2013-02-26 23:51:24 PST
Thanks for the notification and sorry for the noise. I fixed the accidental
change and the TestExpectations files - https://trac.webkit.org/changeset/144149