Summary: | [GTK] WebKitWebProcess at 100% CPU loading hyphenation dictionaries | ||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Product: | WebKit | Reporter: | Milan Crha <mcrha> | ||||||||||||
Component: | WebKitGTK | Assignee: | Zan Dobersek <zan> | ||||||||||||
Status: | RESOLVED FIXED | ||||||||||||||
Severity: | Normal | CC: | bugs-noreply, buildbot, cgarcia, commit-queue, mcatanzaro, mmaxfield, mrobinson, rniwa, tpopela, wk, zan | ||||||||||||
Priority: | P2 | ||||||||||||||
Version: | WebKit Local Build | ||||||||||||||
Hardware: | Unspecified | ||||||||||||||
OS: | Unspecified | ||||||||||||||
See Also: |
https://bugzilla.redhat.com/show_bug.cgi?id=1402586 https://bugzilla.redhat.com/show_bug.cgi?id=1405791 https://bugs.webkit.org/show_bug.cgi?id=166841 |
||||||||||||||
Attachments: |
|
Description
Milan Crha
2016-12-08 07:51:27 PST
Note in particular that, based on the backtraces in the downstream report, it's clear that WebKit is creating multiple HyphenationDictionary objects in some unexpectedly-long loop. This issue is reproducible on https://www.python.org/dev/peps/pep-0484/. WebKit spends 5-7 seconds at a time locked up in the hyphenation code. There is an assertion failure: ASSERTION FAILED: xPos + prefixWidth <= availableWidth ../../Source/WebCore/rendering/line/BreakingContext.h(724) : void WebCore::tryHyphenating(WebCore::RenderText&, const WebCore::FontCascade&, const WTF::AtomicString&, unsigned int, int, int, int, unsigned int, unsigned int, float, int, bool, bool, int, WebCore::InlineIterator&, WTF::Optional<unsigned int>, bool&) 1 0x7f4cbe3b4c28 /home/mcatanzaro/src/jhbuild/install/lib/libjavascriptcoregtk-4.0.so.18(WTFCrash+0x1e) [0x7f4cbe3b4c28] 2 0x7f4cc5fb9fc5 /home/mcatanzaro/src/jhbuild/install/lib/libwebkit2gtk-4.0.so.37(_ZN7WebCore14tryHyphenatingERNS_10RenderTextERKNS_11FontCascadeERKN3WTF12AtomicStringEjiiijjfibbiRNS_14InlineIteratorENS5_8OptionalIjEERb+0x4a6) [0x7f4cc5fb9fc5] 3 0x7f4cc5fba5a3 /home/mcatanzaro/src/jhbuild/install/lib/libwebkit2gtk-4.0.so.37(_ZZN7WebCore15BreakingContext10handleTextERN3WTF6VectorINS_15WordMeasurementELm64ENS1_15CrashOnOverflowELm16EEERbRjENKUlRNS_14InlineIteratorEE0_clESA_+0x191) [0x7f4cc5fba5a3] 4 0x7f4cc5fc0a5f /home/mcatanzaro/src/jhbuild/install/lib/libwebkit2gtk-4.0.so.37(_ZNSt17_Function_handlerIFvRN7WebCore14InlineIteratorEEZNS0_15BreakingContext10handleTextERN3WTF6VectorINS0_15WordMeasurementELm64ENS5_15CrashOnOverflowELm16EEERbRjEUlS2_E0_E9_M_invokeERKSt9_Any_dataS2_+0x37) [0x7f4cc5fc0a5f] 5 0x7f4cc5fbed05 /home/mcatanzaro/src/jhbuild/install/lib/libwebkit2gtk-4.0.so.37(_ZNKSt8functionIFvRN7WebCore14InlineIteratorEEEclES2_+0x49) [0x7f4cc5fbed05] 6 0x7f4cc5fb7afc /home/mcatanzaro/src/jhbuild/install/lib/libwebkit2gtk-4.0.so.37(_ZN7WebCore15BreakingContext21InlineIteratorHistory4pushESt8functionIFvRNS_14InlineIteratorEEE+0xba) [0x7f4cc5fb7afc] 7 0x7f4cc5fbc188 /home/mcatanzaro/src/jhbuild/install/lib/libwebkit2gtk-4.0.so.37(_ZN7WebCore15BreakingContext10handleTextERN3WTF6VectorINS_15WordMeasurementELm64ENS1_15CrashOnOverflowELm16EEERbRj+0x19de) [0x7f4cc5fbc188] 8 0x7f4cc5fb5ce7 /home/mcatanzaro/src/jhbuild/install/lib/libwebkit2gtk-4.0.so.37(_ZN7WebCore11LineBreaker13nextLineBreakERNS_23BidiResolverWithIsolateINS_14InlineIteratorENS_7BidiRunENS_15BidiIsolatedRunEEERNS_8LineInfoERNS_15LineLayoutStateERNS_14RenderTextInfoEPNS_14FloatingObjectEjRN3WTF6VectorINS_15WordMeasurementELm64ENSF_15CrashOnOverflowELm16EEE+0x35b) [0x7f4cc5fb5ce7] 9 0x7f4cc5da050a /home/mcatanzaro/src/jhbuild/install/lib/libwebkit2gtk-4.0.so.37(_ZN7WebCore15RenderBlockFlow26layoutRunsAndFloatsInRangeERNS_15LineLayoutStateERNS_23BidiResolverWithIsolateINS_14InlineIteratorENS_7BidiRunENS_15BidiIsolatedRunEEERKS4_RKNS_10BidiStatusEj+0x358) [0x7f4cc5da050a] 10 0x7f4cc5da0109 /home/mcatanzaro/src/jhbuild/install/lib/libwebkit2gtk-4.0.so.37(_ZN7WebCore15RenderBlockFlow19layoutRunsAndFloatsERNS_15LineLayoutStateEb+0x44b) [0x7f4cc5da0109] 11 0x7f4cc5da2881 /home/mcatanzaro/src/jhbuild/install/lib/libwebkit2gtk-4.0.so.37(_ZN7WebCore15RenderBlockFlow15layoutLineBoxesEbRNS_10LayoutUnitES2_+0x6b1) [0x7f4cc5da2881] 12 0x7f4cc5d802c1 /home/mcatanzaro/src/jhbuild/install/lib/libwebkit2gtk-4.0.so.37(_ZN7WebCore15RenderBlockFlow20layoutInlineChildrenEbRNS_10LayoutUnitES2_+0xb7) [0x7f4cc5d802c1] 13 0x7f4cc5d7f648 /home/mcatanzaro/src/jhbuild/install/lib/libwebkit2gtk-4.0.so.37(_ZN7WebCore15RenderBlockFlow11layoutBlockEbNS_10LayoutUnitE+0x3b6) [0x7f4cc5d7f648] 14 0x7f4cc5d4d032 /home/mcatanzaro/src/jhbuild/install/lib/libwebkit2gtk-4.0.so.37(_ZN7WebCore11RenderBlock6layoutEv+0x7c) [0x7f4cc5d4d032] 15 0x7f4cc5d80682 /home/mcatanzaro/src/jhbuild/install/lib/libwebkit2gtk-4.0.so.37(_ZN7WebCore15RenderBlockFlow16layoutBlockChildERNS_9RenderBoxERNS0_10MarginInfoERNS_10LayoutUnitES6_+0x3be) [0x7f4cc5d80682] 16 0x7f4cc5d801db /home/mcatanzaro/src/jhbuild/install/lib/libwebkit2gtk-4.0.so.37(_ZN7WebCore15RenderBlockFlow19layoutBlockChildrenEbRNS_10LayoutUnitE+0x247) [0x7f4cc5d801db] 17 0x7f4cc5d7f66c /home/mcatanzaro/src/jhbuild/install/lib/libwebkit2gtk-4.0.so.37(_ZN7WebCore15RenderBlockFlow11layoutBlockEbNS_10LayoutUnitE+0x3da) [0x7f4cc5d7f66c] 18 0x7f4cc5d4d032 /home/mcatanzaro/src/jhbuild/install/lib/libwebkit2gtk-4.0.so.37(_ZN7WebCore11RenderBlock6layoutEv+0x7c) [0x7f4cc5d4d032] 19 0x7f4cc5d80682 /home/mcatanzaro/src/jhbuild/install/lib/libwebkit2gtk-4.0.so.37(_ZN7WebCore15RenderBlockFlow16layoutBlockChildERNS_9RenderBoxERNS0_10MarginInfoERNS_10LayoutUnitES6_+0x3be) [0x7f4cc5d80682] 20 0x7f4cc5d801db /home/mcatanzaro/src/jhbuild/install/lib/libwebkit2gtk-4.0.so.37(_ZN7WebCore15RenderBlockFlow19layoutBlockChildrenEbRNS_10LayoutUnitE+0x247) [0x7f4cc5d801db] 21 0x7f4cc5d7f66c /home/mcatanzaro/src/jhbuild/install/lib/libwebkit2gtk-4.0.so.37(_ZN7WebCore15RenderBlockFlow11layoutBlockEbNS_10LayoutUnitE+0x3da) [0x7f4cc5d7f66c] 22 0x7f4cc5d4d032 /home/mcatanzaro/src/jhbuild/install/lib/libwebkit2gtk-4.0.so.37(_ZN7WebCore11RenderBlock6layoutEv+0x7c) [0x7f4cc5d4d032] 23 0x7f4cc5d80682 /home/mcatanzaro/src/jhbuild/install/lib/libwebkit2gtk-4.0.so.37(_ZN7WebCore15RenderBlockFlow16layoutBlockChildERNS_9RenderBoxERNS0_10MarginInfoERNS_10LayoutUnitES6_+0x3be) [0x7f4cc5d80682] 24 0x7f4cc5d801db /home/mcatanzaro/src/jhbuild/install/lib/libwebkit2gtk-4.0.so.37(_ZN7WebCore15RenderBlockFlow19layoutBlockChildrenEbRNS_10LayoutUnitE+0x247) [0x7f4cc5d801db] 25 0x7f4cc5d7f66c /home/mcatanzaro/src/jhbuild/install/lib/libwebkit2gtk-4.0.so.37(_ZN7WebCore15RenderBlockFlow11layoutBlockEbNS_10LayoutUnitE+0x3da) [0x7f4cc5d7f66c] 26 0x7f4cc5d4d032 /home/mcatanzaro/src/jhbuild/install/lib/libwebkit2gtk-4.0.so.37(_ZN7WebCore11RenderBlock6layoutEv+0x7c) [0x7f4cc5d4d032] 27 0x7f4cc5d1a223 /home/mcatanzaro/src/jhbuild/install/lib/libwebkit2gtk-4.0.so.37(_ZN7WebCore13RenderElement14layoutIfNeededEv+0x35) [0x7f4cc5d1a223] 28 0x7f4cc5d87e23 /home/mcatanzaro/src/jhbuild/install/lib/libwebkit2gtk-4.0.so.37(_ZN7WebCore15RenderBlockFlow20insertFloatingObjectERNS_9RenderBoxE+0x219) [0x7f4cc5d87e23] 29 0x7f4cc5d8019e /home/mcatanzaro/src/jhbuild/install/lib/libwebkit2gtk-4.0.so.37(_ZN7WebCore15RenderBlockFlow19layoutBlockChildrenEbRNS_10LayoutUnitE+0x20a) [0x7f4cc5d8019e] 30 0x7f4cc5d7f66c /home/mcatanzaro/src/jhbuild/install/lib/libwebkit2gtk-4.0.so.37(_ZN7WebCore15RenderBlockFlow11layoutBlockEbNS_10LayoutUnitE+0x3da) [0x7f4cc5d7f66c] 31 0x7f4cc5d4d032 /home/mcatanzaro/src/jhbuild/install/lib/libwebkit2gtk-4.0.so.37(_ZN7WebCore11RenderBlock6layoutEv+0x7c) [0x7f4cc5d4d032] Could it be a bug in the FreeType backend? I guess textWidth must be returning something wrong. Possibly from FontCascade::width? I'm using ToT on Debian stretch. What locale are you using? Specifically, what locale is being passed (via `localeIdentifier`) to lastHyphenLocation? In relation to that locale, how many dictionaries for that locale are stored at /usr/share/hyphen or /usr/local/share/hyphen? In my case, I'm using the 'en' locale, meaning that related locales will be mapped via the availableLocales() HashMap to three dictionary files: hyph_en_GB.dic, hyph_en_AU.dic and hyph_en_ZA.dic, as provided by the hyphen-en-gb package. We're storing these in a TinyLRUCache with a default capacity of 4, meaning that they will fit in just fine. If I overwrite the locale to be 'es', the Spanish locale, then related locales will be mapped via the availableLocales() HashMap to 21 dictionary files. In lastHyphenLocation(), we iterate over these locales and retrieve a HyphenDictionary object from the TinyLRUCache for each one, until we find a locale that works for us. But because of using the TinyLRUCache with the default capacity of 4, the dictionary objects stored in the TinyLRUCache keep getting on evicted and being replaced with new ones. With the 'en' locale, loading the Python PEP page, only three HyphenDictionary objects are created, with the underlying hyph_en_*.dic file opened and processed. With the 'es' locale, loading the same page, 1723 HyphenDictionary objects are created, loading each hyph_es_*.dic file 82 times. It still doesn't spin the CPU usage of the WebProcess to 100% on my system, but it's obviously a problem. While the TinyLRUCache capacity could be bumped, it should be noted that at least in Debian packages a lot of these locale variations for one specific locale under /usr/share/hyphen are simply symbolic links to that one master dict file. For instance, there's 21 different Spanish locales under /usr/share/hyphen, from Bolivian to Venezuelan, but it's 20 files just linking to the master hyphen_es_ES.dic file. Same for the English locales -- hyph_en_AU.dic and hyph_en_ZA.dic link to hyph_en_GB.dic. So we should maybe also look into detecting symlinks when storing these filepaths in the availableLocales() HashMap. I have in /usr/share/hyphen/ 23 hyphen_en_??.dic files, all but one sym-linking hyph_en_US.dic. Checking for symlinks sounds like a good idea to me. (In reply to comment #5) > What locale are you using? Specifically, what locale is being passed (via > `localeIdentifier`) to lastHyphenLocation? My system locale is the German (de / de_DE) locale. I don't know about those variables. > In relation to that locale, how many dictionaries for that locale are stored > at /usr/share/hyphen or /usr/local/share/hyphen? /usr/share/hyphen: 29 locales, 6 are German (5 symlinks, one hyph_de_DE.dic file), 23 are English (22 symlinks, one hyph_en_US.dic file). /usr/locale/share/hyphen: doesn't exist > In my case, I'm using the 'en' locale, meaning that related locales will be > mapped via the availableLocales() HashMap to three dictionary files: > hyph_en_GB.dic, hyph_en_AU.dic and hyph_en_ZA.dic, as provided by the > hyphen-en-gb package. > > We're storing these in a TinyLRUCache with a default capacity of 4, meaning > that they will fit in just fine. No idea about that and the rest of your post. hyph_en_US.dic is 106.1 kiB, hyph_de_DE.dic is 50.1 kiB. Zan, can the problem you identified result in this assertion failure? I guess the assertion is a separate issue that just coincidentally is related to hyphenation and occurs on the same page? (In reply to comment #5) > With the 'en' locale, loading the Python PEP page, only three > HyphenDictionary objects are created, with the underlying hyph_en_*.dic file > opened and processed. My locale is en_US.UTF-8. But I have 23 hyph_en_*.dic dictionaries installed, of which all except hyph_en_US.dic itself are symlinks to hyph_en_US.dic. > With the 'es' locale, loading the same page, 1723 HyphenDictionary objects > are created, loading each hyph_es_*.dic file 82 times. It still doesn't spin > the CPU usage of the WebProcess to 100% on my system, but it's obviously a > problem. Thanks for debugging. :) > While the TinyLRUCache capacity could be bumped, it should be noted that at > least in Debian packages a lot of these locale variations for one specific > locale under /usr/share/hyphen are simply symbolic links to that one master > dict file. For instance, there's 21 different Spanish locales under > /usr/share/hyphen, from Bolivian to Venezuelan, but it's 20 files just > linking to the master hyphen_es_ES.dic file. > > Same for the English locales -- hyph_en_AU.dic and hyph_en_ZA.dic link to > hyph_en_GB.dic. > > So we should maybe also look into detecting symlinks when storing these > filepaths in the availableLocales() HashMap. I guess we really ought to do both. A class named TinyLRUCache seems appropriate here, but I would never have assumed the capacity was as low as 4.... (In reply to comment #8) > Zan, can the problem you identified result in this assertion failure? I > guess the assertion is a separate issue that just coincidentally is related > to hyphenation and occurs on the same page? > I don't know for sure, but I don't think it is. Created attachment 298478 [details]
WIP patch
Comment on attachment 298478 [details] WIP patch View in context: https://bugs.webkit.org/attachment.cgi?id=298478&action=review It looks sane, thanks. > Source/WebCore/platform/glib/FileSystemGlib.cpp:411 > +#if PLATFORM(GTK) Since there's nothing GTK-specific here, I would omit the guards. This function works for any GLib port. > Source/WebCore/platform/glib/FileSystemGlib.cpp:423 > + if (!g_strcmp0(dirname.get(), ".")) { What about ".."? It would probably be easier to just use realpath(). Is there some disadvantage to doing that? > Source/WebCore/platform/text/hyphen/HyphenationLibHyphen.cpp:181 > + static TinyLRUCache<AtomicString, RefPtr<WebCore::HyphenationDictionary>, 8>& cache() Maybe we should just use HashTable for this. Attachment 298478 [details] did not pass style-queue:
ERROR: Source/WebCore/ChangeLog:8: You should remove the 'No new tests' and either add and list tests, or explain why no new tests were possible. [changelog/nonewtests] [5]
Total errors found: 1 in 4 files
If any of these errors are false positives, please file a bug against check-webkit-style.
(In reply to comment #11) > Comment on attachment 298478 [details] > WIP patch > > View in context: > https://bugs.webkit.org/attachment.cgi?id=298478&action=review > > It looks sane, thanks. > > > Source/WebCore/platform/glib/FileSystemGlib.cpp:411 > > +#if PLATFORM(GTK) > > Since there's nothing GTK-specific here, I would omit the guards. This > function works for any GLib port. Yes, we don't need any GTK guards in this file, only in the header. This is a bit confusing, in the case of FileSystem, this used to be FileSystemGtk, because it was only used by the GTK port, but we renamed it to glib because there's nothing specific to GTK+, but it's still only used by GTK+ port, other glib based ports use their own FileSystemFoo. > > Source/WebCore/platform/glib/FileSystemGlib.cpp:423 > > + if (!g_strcmp0(dirname.get(), ".")) { > > What about ".."? > > It would probably be easier to just use realpath(). Is there some > disadvantage to doing that? The only advantage I see of using g_read_link is that it can read file paths of any length, because glib uses a loop to try with bigger buffers if previous call to readlink failed because the buffer size was not enough. When using realpath the buffer size should be PATH_MAX. That's usually enough, and realpath not only follows symlinks but also resolves '. and '..'. And that's what I ended up doing for the plugins, without adding any new function to FileSystem: Vector<String> result; char normalizedPath[PATH_MAX]; for (const auto& path : listDirectory(directory, String("*.so"))) { CString filename = fileSystemRepresentation(path); if (realpath(filename.data(), normalizedPath)) result.append(stringFromFileSystemRepresentation(normalizedPath)); } > > Source/WebCore/platform/text/hyphen/HyphenationLibHyphen.cpp:181 > > + static TinyLRUCache<AtomicString, RefPtr<WebCore::HyphenationDictionary>, 8>& cache() > > Maybe we should just use HashTable for this. That's out of the scope of this patch, I would say. Comment on attachment 298478 [details] WIP patch View in context: https://bugs.webkit.org/attachment.cgi?id=298478&action=review >>> Source/WebCore/platform/glib/FileSystemGlib.cpp:411 >>> +#if PLATFORM(GTK) >> >> Since there's nothing GTK-specific here, I would omit the guards. This function works for any GLib port. > > Yes, we don't need any GTK guards in this file, only in the header. This is a bit confusing, in the case of FileSystem, this used to be FileSystemGtk, because it was only used by the GTK port, but we renamed it to glib because there's nothing specific to GTK+, but it's still only used by GTK+ port, other glib based ports use their own FileSystemFoo. The flag is used because I think the implementation wouldn't work on OS(WINDOWS). >>> Source/WebCore/platform/glib/FileSystemGlib.cpp:423 >>> + if (!g_strcmp0(dirname.get(), ".")) { >> >> What about ".."? >> >> It would probably be easier to just use realpath(). Is there some disadvantage to doing that? > > The only advantage I see of using g_read_link is that it can read file paths of any length, because glib uses a loop to try with bigger buffers if previous call to readlink failed because the buffer size was not enough. When using realpath the buffer size should be PATH_MAX. That's usually enough, and realpath not only follows symlinks but also resolves '. and '..'. And that's what I ended up doing for the plugins, without adding any new function to FileSystem: > > Vector<String> result; > char normalizedPath[PATH_MAX]; > for (const auto& path : listDirectory(directory, String("*.so"))) { > CString filename = fileSystemRepresentation(path); > if (realpath(filename.data(), normalizedPath)) > result.append(stringFromFileSystemRepresentation(normalizedPath)); > } I would actually prefer using realpath() directly in HyphenationLibHyphen, rather than adding a GTK+-specific function to FileSystem.h. >>> Source/WebCore/platform/text/hyphen/HyphenationLibHyphen.cpp:181 >>> + static TinyLRUCache<AtomicString, RefPtr<WebCore::HyphenationDictionary>, 8>& cache() >> >> Maybe we should just use HashTable for this. > > That's out of the scope of this patch, I would say. A LRU cache is better because it does limit the amount of entries, and hence the memory consumption of the contained objects. OTOH if somehow the system ends up with multiple locale files being used, then this again is eating up CPU while opening these dictionaries over and over. A larger capacity would maybe fit better, say 32 objects. It also helps that the dictionary files are not big (though I don't know how much memory the parsed dictionary data consumes), but I would prefer it to be capped off at some value. Created attachment 298604 [details]
Patch
Comment on attachment 298604 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=298604&action=review Nice! > Source/WebCore/ChangeLog:8 > + No new tests (OOPS!). Seems like it's ready for r+, except for this bit. > Source/WebCore/platform/text/hyphen/HyphenationLibHyphen.cpp:189 > - static TinyLRUCache<AtomicString, RefPtr<WebCore::HyphenationDictionary>>& cache() > + static TinyLRUCache<AtomicString, RefPtr<WebCore::HyphenationDictionary>, 8>& cache() > { > - static NeverDestroyed<TinyLRUCache<AtomicString, RefPtr<WebCore::HyphenationDictionary>>> cache; > + static NeverDestroyed<TinyLRUCache<AtomicString, RefPtr<WebCore::HyphenationDictionary>, 8>> cache; 32 or even 16 seems a lot safer than 8, indeed. It's easy to imagine a user has more than 8 hyphenation dictionaries installed. Anyway, I expect the primary improvement will come from resolving the symlinks. Comment on attachment 298604 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=298604&action=review > Source/WebCore/platform/text/hyphen/HyphenationLibHyphen.cpp:75 > availableLocales.add(locale, Vector<String>()).iterator->value.append(filePath); Do we really want to add the path even when realpath fails? Some of the errors could be serious, like ELOOP. Comment on attachment 298604 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=298604&action=review >> Source/WebCore/platform/text/hyphen/HyphenationLibHyphen.cpp:75 >> availableLocales.add(locale, Vector<String>()).iterator->value.append(filePath); > > Do we really want to add the path even when realpath fails? Some of the errors could be serious, like ELOOP. OK, makes sense. >> Source/WebCore/platform/text/hyphen/HyphenationLibHyphen.cpp:189 >> + static NeverDestroyed<TinyLRUCache<AtomicString, RefPtr<WebCore::HyphenationDictionary>, 8>> cache; > > 32 or even 16 seems a lot safer than 8, indeed. It's easy to imagine a user has more than 8 hyphenation dictionaries installed. Anyway, I expect the primary improvement will come from resolving the symlinks. Let's do 32. Created attachment 298669 [details]
Patch
Comment on attachment 298669 [details] Patch Attachment 298669 [details] did not pass mac-wk2-ews (mac-wk2): Output: http://webkit-queues.webkit.org/results/2877278 New failing tests: fast/mediastream/MediaStream-video-element-track-stop.html Created attachment 298676 [details]
Archive of layout-test-results from ews107 for mac-elcapitan-wk2
The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: ews107 Port: mac-elcapitan-wk2 Platform: Mac OS X 10.11.6
Comment on attachment 298669 [details] Patch Clearing flags on attachment: 298669 Committed r210670: <http://trac.webkit.org/changeset/210670> All reviewed patches have been landed. Closing bug. |