Bug 110876 – Multiple html5lib Layout Tests are slow since r144032

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Vsevolod Vlasov]

Multiple html5lib Layout Tests are timing out on chromium since r144032

html5lib/generated/run-doctype01-data.html
html5lib/generated/run-domjs-unsafe-data.html
html5lib/generated/run-entities01-data.html
html5lib/generated/run-plain-text-unsafe-data.html
html5lib/generated/run-scriptdata01-data.html
html5lib/generated/run-template-data.html
html5lib/generated/run-tests1-data.html
html5lib/generated/run-tests10-data.html
html5lib/generated/run-tests16-data.html
html5lib/generated/run-tests19-data.html
html5lib/generated/run-tests2-data.html
html5lib/generated/run-tests20-data.html
html5lib/generated/run-tests6-data.html
html5lib/generated/run-tests7-data.html
html5lib/generated/run-tests9-data.html
html5lib/generated/run-webkit01-data.html

Caused by http://trac.webkit.org/changeset/144032/

[Vsevolod Vlasov]

Marked as flaky: http://trac.webkit.org/changeset/144050

[Eric Seidel (no email)]

These should just be marked as slow.  They complete fine for me on my linux machine.

[Eric Seidel (no email)]

I'm sure we could look at ways to improve the data: url harness for this test to not be so slow.  But it's likely these tests will just be marked as "slow" for the forseeable future.   What they test is extremely valuable. :)

[Vsevolod Vlasov]

Marked as slow http://trac.webkit.org/changeset/144156

[Vsevolod Vlasov]

Some of them still fail when marked as slow, some html5lib/ started failing when others were made slow.
I am going to add more information in this bug.

[Vsevolod Vlasov]

These tests are timing out even when marked as slow: 

html5lib/generated/run-entities01-data.html
html5lib/generated/run-template-data.html
html5lib/generated/run-tests1-data.html
html5lib/generated/run-tests10-data.html
html5lib/generated/run-tests16-data.html
html5lib/generated/run-tests19-data.html

http://test-results.appspot.com/dashboards/flakiness_dashboard.html#tests=html5lib%2Fgenerated%2Frun-entities01-data.html%2Chtml5lib%2Fgenerated%2Frun-template-data.html%2Chtml5lib%2Fgenerated%2Frun-tests1-data.html%2Chtml5lib%2Fgenerated%2Frun-tests10-data.html%2Chtml5lib%2Fgenerated%2Frun-tests16-data.html%2Chtml5lib%2Fgenerated%2Frun-tests19-data.html

[Vsevolod Vlasov]

Updated expectations: http://trac.webkit.org/changeset/144171

[Vsevolod Vlasov]

These are also failing:

html5lib/generated/run-tests10-write.html
html5lib/generated/run-tests16-write.html
html5lib/generated/run-tests19-write.html

http://test-results.appspot.com/dashboards/flakiness_dashboard.html#tests=html5lib%2Fgenerated%2Frun-tests10-write.html%2Chtml5lib%2Fgenerated%2Frun-tests16-write.html%2Chtml5lib%2Fgenerated%2Frun-tests19-write.html

Updated expectations: http://trac.webkit.org/changeset/144172

[Rafael Weinstein]

Marked all html5lib/generated as crashing here:

http://trac.webkit.org/changeset/144692

Note that I removed all the more specific expectations in this patch.

[Eric Seidel (no email)]

Profiling an individual test, you can see parts of WebCore working.  But profiling all the tests in aggregate, seems to show all our time being spent in V8 creating the window object each time we call document.open(). :)

+   6.69%  DumpRenderTree  [kernel.kallsyms]            [k] 0xffffffff8103b51a
+   4.09%  DumpRenderTree  DumpRenderTree               [.] v8::internal::Scanner::ScanIdentifierOrKeyword()
+   2.33%  DumpRenderTree  DumpRenderTree               [.] v8::internal::Scanner::Scan()
+   1.63%  DumpRenderTree  DumpRenderTree               [.] v8::internal::Scanner::Next()
+   1.54%  DumpRenderTree  DumpRenderTree               [.] v8::internal::RelocIterator::next()
+   1.48%  DumpRenderTree  DumpRenderTree               [.] v8::internal::TemplateHashMapImpl<v8::internal::ZoneAllocationPolicy>::Lookup(void*, unsigne
+   1.38%  DumpRenderTree  DumpRenderTree               [.] v8::internal::DescriptorArray::Append(v8::internal::Descriptor*)
+   1.33%  DumpRenderTree  DumpRenderTree               [.] v8::internal::Parser::ParseBinaryExpression(int, bool, bool*)
+   1.25%  DumpRenderTree  DumpRenderTree               [.] v8::internal::Parser::ParseAssignmentExpression(bool, bool*)
+   1.23%  DumpRenderTree  perf-16889.map               [.] 0xf48aab3592e
+   1.23%  DumpRenderTree  DumpRenderTree               [.] v8::internal::SymbolTable::LookupOneByteSymbol(v8::internal::Vector<unsigned char const>, v8
+   1.17%  DumpRenderTree  DumpRenderTree               [.] v8::internal::Zone::New(int)
+   1.09%  DumpRenderTree  DumpRenderTree               [.] v8::internal::Parser::ParsePrimaryExpression(bool*)
+   1.04%  DumpRenderTree  DumpRenderTree               [.] v8::internal::FlexibleBodyVisitor<v8::internal::MarkCompactMarkingVisitor, v8::internal::Fix
+   0.99%  DumpRenderTree  [unknown]                    [.] 0xf48aaa5ecb2

[Eric Seidel (no email)]

I'm using bug 111384 to check the perf of these tests.

[Eric Seidel (no email)]

(In reply to comment #10)
> Profiling an individual test, you can see parts of WebCore working.  But profiling all the tests in aggregate, seems to show all our time being spent in V8 creating the window object each time we call document.open(). :)

After further investigation, this V8 start-up time is dominant only in the data url versions of the tests.

[Eric Seidel (no email)]

But it is *so* dominant, that it pushes the top symbol from the write versions down to < 1% of total time. :)

The write() versions only create/load 1 iframe (repeatedly calling open(), write(), end() on said iframe), the data versions set the same iframe to a different src=data: url for each subtest.

[Eric Seidel (no email)]

The crashes which rafael is seeing only seem to occur on mac/win.  I'm not able to repro them on linux.

[Eric Seidel (no email)]

(In reply to comment #9)
> Marked all html5lib/generated as crashing here:
> 
> http://trac.webkit.org/changeset/144692
> 
> Note that I removed all the more specific expectations in this patch.

Thanks.  I think the crashes are likely some side-effect of the way we're timing out.  Since we're only seeing these troubles on Debug, I am going to post a patch to Skip all of these tests on Debug until the V8 per-document boot time is faster.

[Eric Seidel (no email)]

Created attachment 191465 [details]
Patch

[Rafael Weinstein]

lgtm

[Eric Seidel (no email)]

I filed a v8 bug as well https://code.google.com/p/v8/issues/detail?id=2567

[WebKit Review Bot]

Comment on attachment 191465 [details]
Patch

Clearing flags on attachment: 191465

Committed r144807: <http://trac.webkit.org/changeset/144807>

[WebKit Review Bot]

All reviewed patches have been landed.  Closing bug.

[Alan Cutter]

The run-tests16-data.html test has been timing out on the Chromium Linux release build now.
http://build.webkit.org/results/Chromium%20Linux%20Release%20(Tests)/r144809%20(45627)/results.html

Should this test be marked as flaky in release as well given a V8 issue exists for it?

[Eric Seidel (no email)]

(In reply to comment #21)
> The run-tests16-data.html test has been timing out on the Chromium Linux release build now.
> http://build.webkit.org/results/Chromium%20Linux%20Release%20(Tests)/r144809%20(45627)/results.html
> 
> Should this test be marked as flaky in release as well given a V8 issue exists for it?

That's unfortunate.  Yes, I expect some of the data- tests with lots of subtests will hit the timeout.