Bug 110734 – Layout Test Multiple html5lib/ tests are too slow and should be split.

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Vsevolod Vlasov]

Similar bug for the main test: https://bugs.webkit.org/show_bug.cgi?id=110642

The following layout tests are too slow:

html5lib/run-test9.html
html5lib/run-test6.html
html5lib/run-template.html
html5lib/run-test20.html
html5lib/run-test1.html
html5lib/run-test7.html
html5lib/run-test10.html
html5lib/run-test16.html
html5lib/run-test19.html
html5lib/run-test2.html

http://test-results.appspot.com/dashboards/flakiness_dashboard.html#tests=html5lib%2Frun-test9.html%2Chtml5lib%2Frun-test6.html%2Chtml5lib%2Frun-template.html%2Chtml5lib%2Frun-test20.html%2Chtml5lib%2Frun-test1.html%2Chtml5lib%2Frun-test10.html%2Chtml5lib%2Frun-test16.html%2Chtml5lib%2Frun-test19.html%2Chtml5lib%2Frun-test2.html

[Eric Seidel (no email)]

Thank you for filing.  I'm not yet sure what we should do about these, besides mark them as slow, but I'll look in the morning.

[Vsevolod Vlasov]

Committed r143891: <http://trac.webkit.org/changeset/143891>

[Eric Seidel (no email)]

Created attachment 190143 [details]
Patch

[Ojan Vafai]

Comment on attachment 190143 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=190143&action=review

> LayoutTests/ChangeLog:42
> +        * html5lib/generated/run-adoption01-data-expected.txt: Added.
> +        * html5lib/generated/run-adoption01-data.html: Copied from LayoutTests/html5lib/generated/run-tests1.html.
> +        * html5lib/generated/run-adoption01-write-expected.txt: Added.
> +        * html5lib/generated/run-adoption01-write.html: Renamed from LayoutTests/html5lib/generated/run-template.html.
> +        * html5lib/generated/run-adoption02-data-expected.txt: Added.
> +        * html5lib/generated/run-adoption02-data.html: Copied from LayoutTests/html5lib/generated/run-tests1.html.
> +        * html5lib/generated/run-adoption02-write-expected.txt: Added.
> +        * html5lib/generated/run-adoption02-write.html: Renamed from LayoutTests/html5lib/generated/run-tests11.html.
> +        * html5lib/generated/run-comments01-data-expected.txt: Added.
> +        * html5lib/generated/run-comments01-data.html: Copied from LayoutTests/html5lib/generated/run-tests1.html.

Standard practice when there's a very long file list like this is to delete it.

> LayoutTests/html5lib/generate-test-wrappers.py:30
> +# Used for generating LayoutTest-compatible html files to run html5lib *.dat files.

This should either have a usage thing in the comment or when you run on the commandline (e.g. with --help).

> LayoutTests/html5lib/generate-test-wrappers.py:79
> +                os.remove(path)

Should this not be scm.remove or some equivalent?

[Adam Barth]

Comment on attachment 190143 [details]
Patch

I would add a README

[Eric Seidel (no email)]

Created attachment 190152 [details]
Patch for landing

[WebKit Review Bot]

Comment on attachment 190152 [details]
Patch for landing

Rejecting attachment 190152 [details] from commit-queue.

Failed to run "['/mnt/git/webkit-commit-queue/Tools/Scripts/webkit-patch', '--status-host=webkit-commit-queue.appspot.com', '--bot-id=gce-cq-02', 'land-attachment', '--force-clean', '--non-interactive', '--parent-command=commit-queue', 190152, '--port=chromium-xvfb']" exit_code: 2 cwd: /mnt/git/webkit-commit-queue

Last 500 characters of output:
WebKit/chromium/v8 --revision 13692 --non-interactive --force --accept theirs-conflict --ignore-externals' in '/mnt/git/webkit-commit-queue/Source/WebKit/chromium'
54>At revision 13692.

________ running '/usr/bin/python tools/clang/scripts/update.py --mac-only' in '/mnt/git/webkit-commit-queue/Source/WebKit/chromium'

________ running '/usr/bin/python gyp_webkit' in '/mnt/git/webkit-commit-queue/Source/WebKit/chromium'
Updating webkit projects from gyp files...
Total errors found: 0 in 0 files

Full output: http://webkit-commit-queue.appspot.com/results/16747951

[WebKit Review Bot]

Comment on attachment 190152 [details]
Patch for landing

Clearing flags on attachment: 190152

Committed r144032: <http://trac.webkit.org/changeset/144032>

[WebKit Review Bot]

All reviewed patches have been landed.  Closing bug.

[Csaba Osztrogonác]

(In reply to comment #8)
> (From update of attachment 190152 [details])
> Clearing flags on attachment: 190152
> 
> Committed r144032: <http://trac.webkit.org/changeset/144032>

FYI: There are 8 failures after this patch everywhere:
- Qt - http://build.webkit.sed.hu/results/x86-32%20Linux%20Qt%20Release%20NRWT/r144032%20%2829232%29/results.html
- GTK - http://build.webkit.org/results/GTK%20Linux%2064-bit%20Release/r144033%20%2834924%29/results.html
...

[Vsevolod Vlasov]

Some html5lib tests are still timing out https://bugs.webkit.org/show_bug.cgi?id=110876

[Csaba Osztrogonác]

(In reply to comment #10)
> (In reply to comment #8)
> > (From update of attachment 190152 [details] [details])
> > Clearing flags on attachment: 190152
> > 
> > Committed r144032: <http://trac.webkit.org/changeset/144032>
> 
> FYI: There are 8 failures after this patch everywhere:
> - Qt - http://build.webkit.sed.hu/results/x86-32%20Linux%20Qt%20Release%20NRWT/r144032%20%2829232%29/results.html
> - GTK - http://build.webkit.org/results/GTK%20Linux%2064-bit%20Release/r144033%20%2834924%29/results.html
> ...

Updated landed due to JSC/V8 differences in https://trac.webkit.org/changeset/144051

But unfortunately there are 2 really failing tests everywhere:
--- /Volumes/Data/slave/lion-debug-tests-wk1/build/layout-test-results/html5lib/generated/run-template-data-expected.txt
+++ /Volumes/Data/slave/lion-debug-tests-wk1/build/layout-test-results/html5lib/generated/run-template-data-actual.txt
@@ -1,1 +1,3 @@
-../resources/template.dat: PASS
+CONSOLE MESSAGE: line 149: TypeError: 'undefined' is not an object (evaluating 'node.nodeType')
+FAIL: Timed out waiting for notifyDone to be called
+Running test 1 of 96 in ../resources/template.dat

--- /Volumes/Data/slave/lion-debug-tests-wk1/build/layout-test-results/html5lib/generated/run-template-write-expected.txt
+++ /Volumes/Data/slave/lion-debug-tests-wk1/build/layout-test-results/html5lib/generated/run-template-write-actual.txt
@@ -1,1 +1,3 @@
-../resources/template.dat: PASS
+CONSOLE MESSAGE: line 149: TypeError: 'undefined' is not an object (evaluating 'node.nodeType')
+FAIL: Timed out waiting for notifyDone to be called
+Running test 1 of 96 in ../resources/template.dat


I marked them as failure in the general TestExpectations, 
please check and fix this regression.

[Csaba Osztrogonác]

I checked TestExpectations files, run-template.html was skipped on Qt/GTK/Mac,
because ENABLE(TEMPLATE_ELEMENT) is disabled. Are the new run-template-data.html and run-template-write.html originated from this test? If yes,
we should simple skip them on these ports.

[Eric Seidel (no email)]

Thank you ossy.  I was expecting the JSC errors, but didn't have a JSC port from my machine.  Thanks for taking care of it.

[Eric Seidel (no email)]

Yes, run-template* is skipped on most ports, but not Chromium.

[Eric Seidel (no email)]

This change is very wrong:
http://trac.webkit.org/changeset/144051/trunk/LayoutTests/html5lib/generated/run-template-write.html

All the generated files shoudl not be hand edited.  And run-template should run template.dat... not tests3. :)

[Csaba Osztrogonác]

(In reply to comment #16)
> This change is very wrong:
> http://trac.webkit.org/changeset/144051/trunk/LayoutTests/html5lib/generated/run-template-write.html
> 
> All the generated files shoudl not be hand edited.  And run-template should run template.dat... not tests3. :)

Ooops, it was an accidental change, will fix immediately.

[Csaba Osztrogonác]

Thanks for the notification and sorry for the noise. I fixed the accidental
change and the TestExpectations files - https://trac.webkit.org/changeset/144149