Summary: | DumpRenderTree should report unknown options | ||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Product: | WebKit | Reporter: | Frédéric Wang (:fredw) <fred.wang> | ||||||||||||||||||||||||||||||||||||||||||
Component: | Tools / Tests | Assignee: | Nobody <webkit-unassigned> | ||||||||||||||||||||||||||||||||||||||||||
Status: | REOPENED --- | ||||||||||||||||||||||||||||||||||||||||||||
Severity: | Normal | CC: | commit-queue, darin, dkadu, ews-watchlist, Hironori.Fujii, lforschler, obrufau, rego, rniwa, tsavell, webkit-bug-importer | ||||||||||||||||||||||||||||||||||||||||||
Priority: | P2 | Keywords: | InRadar | ||||||||||||||||||||||||||||||||||||||||||
Version: | WebKit Nightly Build | ||||||||||||||||||||||||||||||||||||||||||||
Hardware: | Unspecified | ||||||||||||||||||||||||||||||||||||||||||||
OS: | Unspecified | ||||||||||||||||||||||||||||||||||||||||||||
Attachments: |
|
Description
Frédéric Wang (:fredw)
2018-11-06 03:04:09 PST
Original webkit-dev discussion: https://lists.webkit.org/pipermail/webkit-dev/2018-November/030239.html https://lists.webkit.org/pipermail/webkit-dev/2018-November/030247.html Probably there are some options that are already used in tests but not handled by TestOptions yet. Created attachment 358263 [details]
Added fallback else for unknown option
Added else for unknown option logged the option and break the loop
Attachment 358263 [details] did not pass style-queue:
ERROR: Tools/DumpRenderTree/TestOptions.cpp:111: Missing space after , [whitespace/comma] [3]
Total errors found: 1 in 1 files
If any of these errors are false positives, please file a bug against check-webkit-style.
Comment on attachment 358263 [details] Added fallback else for unknown option View in context: https://bugs.webkit.org/attachment.cgi?id=358263&action=review Thanks for the patch. Can you please rebase it (it seems it does not apply on tip of tree) and add the associated ChangeLog? Also, have you actually tested your patch with a currently unknown option? If you find-grep tests, I'm sure some of them already use such unhandled options and you might need to add them into TestOptions.cpp and update the expectations. Also, I would be curious to see if the LOG_ERROR actually shows up in the test output. > Tools/DumpRenderTree/TestOptions.cpp:113 > + LOG_ERROR("Unknown option %s",key.c_str()); Missing space after comma (please run check-webkit-style before uploading the patch) > Tools/DumpRenderTree/TestOptions.cpp:114 > + break; I think this break is not needed. We can just ignore the option (as we do now) and continue to the next pair. Created attachment 358456 [details]
Patch
Following are the options which are not handled in TestOptions.cpp applicationManifest experimental:CSSCustomPropertiesAndValuesEnabled useFlexibleViewport useThreadedScrolling experimental:DarkModeCSSEnabled experimental:CSSTypedOMEnabled punchOutWhiteBackgrounds enableEditableImages useCharacterSelectionGranularity spellCheckingDots experimental:CSSPaintingAPIEnabled experimental:PointerEventsEnabled useMockScrollbars language ignoresViewportScaleLimits shouldIgnoreMetaViewport enableProcessSwapOnNavigation runSingly internal:WebAPIStatisticsEnabled internal:SourceBufferChangeTypeEnabled Comment on attachment 358456 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=358456&action=review > Following are the options which are not handled in TestOptions.cpp Can you please give details about: (1) How did you get that list? And what are the associated tests (or the amount of such tests) (2) The changes you experimented in TestOptions.cpp and how they affect the webkit-test-runner results for the tests in (1). None of the patches you uploaded seem to have any effect on the EWS bots. > Tools/ChangeLog:10 > + ChangeLog still does not contain any useful information. > Tools/DumpRenderTree/TestOptions.cpp:111 > + fprintf(stderr, "Unknown option %s\n", key.c_str()); I think ideally we should re-use some logging feature provided in WTF. This does not seem better than LOG_ERROR (which e.g. provides file and line number). Created attachment 360474 [details]
Patch
> Can you please give details about:
> (1) How did you get that list? And what are the associated tests (or the
> amount of such tests)
So I did it manually searched for" webkit-test-runner [" in all test files,
noted down all the options and checked whether they are in if else condition in Testopstions.cpp
Created attachment 360733 [details]
Patch
Comment on attachment 360733 [details] Patch Attachment 360733 [details] did not pass mac-ews (mac): Output: https://webkit-queues.webkit.org/results/10974231 Number of test failures exceeded the failure limit. Created attachment 360737 [details]
Archive of layout-test-results from ews100 for mac-highsierra
The attached test failures were seen while running run-webkit-tests on the mac-ews.
Bot: ews100 Port: mac-highsierra Platform: Mac OS X 10.13.6
Comment on attachment 360733 [details] Patch Attachment 360733 [details] did not pass mac-debug-ews (mac): Output: https://webkit-queues.webkit.org/results/10973955 Number of test failures exceeded the failure limit. Created attachment 360738 [details]
Archive of layout-test-results from ews112 for mac-highsierra
The attached test failures were seen while running run-webkit-tests on the mac-debug-ews.
Bot: ews112 Port: mac-highsierra Platform: Mac OS X 10.13.6
Comment on attachment 360733 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=360733&action=review So now some tests are crashing, which is good news: Your patch really has an effect. Let's add support for the condition you mentioned in comment 6 now. > Tools/DumpRenderTree/TestOptions.cpp:115 > + RELEASE_ASSERT(false); It does not seem nice to use a hardcoded boolean here. Can you please use RELEASE_ASSERT_NOT_REACHED instead? View in context: https://bugs.webkit.org/attachment.cgi?id=360733&action=review So now some tests are crashing, which is good news: Your patch really has an effect. Let's add support for the condition you mentioned in comment 6 now. > Tools/DumpRenderTree/TestOptions.cpp:115 > + RELEASE_ASSERT(false); It does not seem nice to use a hardcoded boolean here. Can you please use RELEASE_ASSERT_NOT_REACHED instead? (In reply to Frédéric Wang (:fredw) from comment #16) > View in context: > https://bugs.webkit.org/attachment.cgi?id=360733&action=review > > So now some tests are crashing, which is good news: Your patch really has an > effect. Let's add support for the condition you mentioned in comment 6 now. > > > Tools/DumpRenderTree/TestOptions.cpp:115 > > + RELEASE_ASSERT(false); > > It does not seem nice to use a hardcoded boolean here. Can you please use > RELEASE_ASSERT_NOT_REACHED instead? Okay I will add
> So now some tests are crashing, which is good news: Your patch really has an
> effect. Let's add support for the condition you mentioned in comment 6 now.
Do I need to implement all the conditions? What about all the different action that need to take on each condition?
(In reply to darshan from comment #18) > > So now some tests are crashing, which is good news: Your patch really has an > > effect. Let's add support for the condition you mentioned in comment 6 now. > > Do I need to implement all the conditions? What about all the different > action that need to take on each condition? I think most of them are boolean values so it should be easy to implement the parsing and properly set the corresponding preferences. Alternatively, maybe you can handle them in a else with a FIXME comment and handle that in a separate bug. (In reply to Frédéric Wang (:fredw) from comment #19) > (In reply to darshan from comment #18) > > > So now some tests are crashing, which is good news: Your patch really has an > > > effect. Let's add support for the condition you mentioned in comment 6 now. > > > > Do I need to implement all the conditions? What about all the different > > action that need to take on each condition? > > I think most of them are boolean values so it should be easy to implement > the parsing and properly set the corresponding preferences. Alternatively, > maybe you can handle them in a else with a FIXME comment and handle that in > a separate bug. Yeah its seems okay! Thanks Comment on attachment 360733 [details] Patch Attachment 360733 [details] did not pass win-ews (win): Output: https://webkit-queues.webkit.org/results/10974618 New failing tests: fast/multicol/pagination/RightToLeft-rl-hittest.html js/dom/regress-157246.html fast/scrolling/rtl-scrollbars-animation-property.html css-custom-properties-api/registerProperty.html fast/text/combining-character-sequence-vertical.html fast/text/descent-clip-in-scaled-page.html fast/block/positioning/vertical-rl/fixed-positioning.html fast/dom/horizontal-scrollbar-in-rtl.html fast/text/vertical-quotation-marks.html fast/dom/horizontal-scrollbar-when-dir-change.html fast/dom/vertical-scrollbar-when-dir-change.html fast/block/positioning/rtl-fixed-positioning.html fast/dom/scroll-reveal-top-overflow.html fast/dom/scroll-reveal-left-overflow.html Created attachment 360743 [details]
Archive of layout-test-results from ews202 for win-future
The attached test failures were seen while running run-webkit-tests on the win-ews.
Bot: ews202 Port: win-future Platform: CYGWIN_NT-6.1-2.10.0-0.325-5-3-x86_64-64bit
Created attachment 360746 [details]
Patch
Comment on attachment 360746 [details] Patch Attachment 360746 [details] did not pass win-ews (win): Output: https://webkit-queues.webkit.org/results/10976681 New failing tests: js/dom/regress-157246.html Created attachment 360760 [details]
Archive of layout-test-results from ews201 for win-future
The attached test failures were seen while running run-webkit-tests on the win-ews.
Bot: ews201 Port: win-future Platform: CYGWIN_NT-6.1-2.10.0-0.325-5-3-x86_64-64bit
Comment on attachment 360746 [details] Patch Attachment 360746 [details] did not pass mac-ews (mac): Output: https://webkit-queues.webkit.org/results/10977474 New failing tests: fast/canvas/webgl/hide-some-renderer-info.html media/modern-media-controls/media-documents/insert-style-should-not-crash.html Created attachment 360762 [details]
Archive of layout-test-results from ews103 for mac-highsierra
The attached test failures were seen while running run-webkit-tests on the mac-ews.
Bot: ews103 Port: mac-highsierra Platform: Mac OS X 10.13.6
Comment on attachment 360746 [details] Patch Attachment 360746 [details] did not pass mac-debug-ews (mac): Output: https://webkit-queues.webkit.org/results/10977364 New failing tests: fast/canvas/webgl/hide-some-renderer-info.html media/modern-media-controls/media-documents/insert-style-should-not-crash.html Created attachment 360768 [details]
Archive of layout-test-results from ews117 for mac-highsierra
The attached test failures were seen while running run-webkit-tests on the mac-debug-ews.
Bot: ews117 Port: mac-highsierra Platform: Mac OS X 10.13.6
Comment on attachment 360746 [details] Patch Attachment 360746 [details] did not pass mac-wk2-ews (mac-wk2): Output: https://webkit-queues.webkit.org/results/10977843 New failing tests: applicationmanifest/display-mode-subframe.html Created attachment 360769 [details]
Archive of layout-test-results from ews106 for mac-highsierra-wk2
The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: ews106 Port: mac-highsierra-wk2 Platform: Mac OS X 10.13.6
Comment on attachment 360746 [details] Patch Attachment 360746 [details] did not pass ios-sim-ews (ios-simulator-wk2): Output: https://webkit-queues.webkit.org/results/10977745 New failing tests: applicationmanifest/display-mode-subframe.html Created attachment 360779 [details]
Archive of layout-test-results from ews125 for ios-simulator-wk2
The attached test failures were seen while running run-webkit-tests on the ios-sim-ews.
Bot: ews125 Port: ios-simulator-wk2 Platform: Mac OS X 10.13.6
Created attachment 360865 [details]
Patch
Created attachment 360913 [details]
Patch
Comment on attachment 360913 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=360913&action=review > Tools/ChangeLog:10 > + Added and assigned testoptions which are currently used in tests. test options > Tools/DumpRenderTree/TestOptions.cpp:117 > + experimentalCSSCustomPropertiesAndValuesEnabled = parseBooleanTestHeaderValue(value); Looking at the rest of the code, it seems keys starting with "experimental:" and "internal:" don't include the prefix in the variable name. Also it seems that sometimes the variable name is of the form enableXXX rather than XXXEnabled (although that's not very consistent). > Tools/DumpRenderTree/TestOptions.h:49 > + bool experimentalCSSCustomPropertiesAndValuesEnabled { false }; Ideally, you should check that the default value is consistent with the rest of the code (especially what is used by the WebKiTestRunner). Have you tried to do a search to see how the values are initialized elsewhere? Currently the new values are not used at all, so you should add a FIXME and open a separate bug if you don't plan to fix that in this bug. For Web Preferences, you can probably get the default values from Source/WebKit/Shared/WebPreferences.yaml and only need to initialize them in setWebPreferencesForTestOptions from Tools/DumpRenderTree/mac/DumpRenderTree.mm Created attachment 361201 [details]
Patch
Comment on attachment 361201 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=361201&action=review > Tools/DumpRenderTree/TestOptions.h:49 > + bool enableCSSCustomPropertiesAndValues { false }; Please address add a FIXME: (In reply to Frédéric Wang (:fredw) from comment #36) > Currently the new values are not used at all, so you should add a FIXME and > open a separate bug if you don't plan to fix that in this bug. > > For Web Preferences, you can probably get the default values from > Source/WebKit/Shared/WebPreferences.yaml and only need to initialize them in > setWebPreferencesForTestOptions from > Tools/DumpRenderTree/mac/DumpRenderTree.mm Created attachment 361302 [details]
Patch
(In reply to darshan from comment #39) > Created attachment 361302 [details] > Patch I tried to add default values of variables by referring Source/WebKit/Shared/WebPreferences.yaml and Tools/DumpRenderTree/mac/DumpRenderTree.mm Comment on attachment 361302 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=361302&action=review > Tools/DumpRenderTree/TestOptions.h:50 > + // Fixme: support following test options is not implemented nit: it should be uppercase FIXME. And ideally refer to a but number https://webkit.org/b/???? Created attachment 361683 [details]
Patch
(In reply to darshan from comment #42) > Created attachment 361683 [details] > Patch Created https://bugs.webkit.org/show_bug.cgi?id=194396 Comment on attachment 361683 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=361683&action=review > Tools/DumpRenderTree/TestOptions.h:50 > + // FIXME: support following test options is not implemented, I guess you mean something like "FIXME: Implement the following test options". Created attachment 361687 [details]
Patch
Comment on attachment 361687 [details] Patch Clearing flags on attachment: 361687 Committed r241269: <https://trac.webkit.org/changeset/241269> All reviewed patches have been landed. Closing bug. It appears that the changes in https://trac.webkit.org/changeset/241269/webkit made 5 tests on Mac WK1 crash: fast/text/international/system-language/arabic-glyph-cache-fill-combine.html fast/text/international/system-language/declarative-language.html fast/text/international/system-language/han-quotes.html fast/text/international/system-language/hindi-system-font-punctuation.html fast/text/international/system-language/system-font-punctuation.html History: http://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=fast%2Ftext%2Finternational%2Fsystem-language%2Farabic-glyph-cache-fill-combine.html%20fast%2Ftext%2Finternational%2Fsystem-language%2Fdeclarative-language.html%20fast%2Ftext%2Finternational%2Fsystem-language%2Fhan-quotes.html%20fast%2Ftext%2Finternational%2Fsystem-language%2Fhindi-system-font-punctuation.html%20fast%2Ftext%2Finternational%2Fsystem-language%2Fsystem-font-punctuation.html Test Results: https://build.webkit.org/results/Apple%20High%20Sierra%20Release%20WK1%20(Tests)/r241273%20(11135)/results.html Confirmed this by running the 5 tests together on a checkout of 241269 which caused the tests to constantly crash, and no crashes occurred on the revision before. This will need to be rolled out. Reverted r241269 for reason: Caused 5 layout tests crashes on Mac WK1 Committed r241277: <https://trac.webkit.org/changeset/241277> (In reply to darshan from comment #9) > > Can you please give details about: > > (1) How did you get that list? And what are the associated tests (or the > > amount of such tests) > > So I did it manually searched for" webkit-test-runner [" in all test files, > noted down all the options and checked whether they are in if else condition > in Testopstions.cpp What do you mean by "manually searched"? I really hope you've used a command line or any automated process to extract all these options. For example, in LayoutTests/fast/text we don't handle the "language" key and that's why the tests crash: find LayoutTests/fast/text -type f | xargs grep 'webkit-test-runner \[' | sed 's/.*webkit-test-runner \[ \(.\+\)\].*/\1/' | sort | uniq language=ar language=hi language=ko,zh language=zh-Hans useThreadedScrolling=false (probably there is a better command line option but you got the idea) (In reply to Frédéric Wang (:fredw) from comment #51) > (In reply to darshan from comment #9) > > > Can you please give details about: > > > (1) How did you get that list? And what are the associated tests (or the > > > amount of such tests) > > > > So I did it manually searched for" webkit-test-runner [" in all test files, > > noted down all the options and checked whether they are in if else condition > > in Testopstions.cpp > > What do you mean by "manually searched"? I really hope you've used a command > line or any automated process to extract all these options. For example, in > LayoutTests/fast/text we don't handle the "language" key and that's why the > tests crash: > > I used VS code to search 'webkit-test-runner [' and listed down all the option, I am confused why it did not fail on ews. I will add the options and submit the patch Created attachment 362238 [details]
Patch
(In reply to darshan from comment #52) > I used VS code to search 'webkit-test-runner [' and listed down all the > option, > I am confused why it did not fail on ews. EWS compiles the patch but does not run the tests on all platform. Have you tried to search the 'webkit-test-runner' instances using a more reliable and less error-prone method than "VS code search" + manual copy / sorting? Comment on attachment 362238 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=362238&action=review > Tools/DumpRenderTree/TestOptions.cpp:161 > + RELEASE_ASSERT_NOT_REACHED(); This is not "reporting" unknown options, it's *aborting* if passed an unknown option. That doesn’t seem like a good interface for a command line tool. Comment on attachment 362238 [details]
Patch
Why is it useful to parse and ignore so many options?
(In reply to Darin Adler from comment #55) > This is not "reporting" unknown options, it's *aborting* if passed an > unknown option. That doesn’t seem like a good interface for a command line > tool. I agree. I'm not sure exactly what Darshan tried, but he was not able to log the text in a way that test authors are made aware of the missing parsing. (In reply to Darin Adler from comment #56) > Comment on attachment 362238 [details] > Patch > > Why is it useful to parse and ignore so many options? I believe they should not be ignored. Darshan proposed to do it in a separate patch as that would lead to more changes (in test expectations?)... But actually it's possible that we just want to ignore some WK2-specific options in DumpRenderTree? If so we don't need to parse them. (In reply to Frédéric Wang (:fredw) from comment #54) > (In reply to darshan from comment #52) > > I used VS code to search 'webkit-test-runner [' and listed down all the > > option, > > I am confused why it did not fail on ews. > > EWS compiles the patch but does not run the tests on all platform. Have you > tried to search the 'webkit-test-runner' instances using a more reliable and > less error-prone method than "VS code search" + manual copy / sorting? Well I did not run any other command to do it and I agree this wasn't the best way to do this. (In reply to Darin Adler from comment #55) > Comment on attachment 362238 [details] > Patch > > View in context: > https://bugs.webkit.org/attachment.cgi?id=362238&action=review > > > Tools/DumpRenderTree/TestOptions.cpp:161 > > + RELEASE_ASSERT_NOT_REACHED(); > > This is not "reporting" unknown options, it's *aborting* if passed an > unknown option. That doesn’t seem like a good interface for a command line > tool. Idea was to fail the tests with unknown options. Getting the proper message why its failing would be the ideal solution for this! (In reply to darshan from comment #58) > Well I did not run any other command to do it and I agree this wasn't the > best way to do this. FWIW, you can compare with the corresponding code of WebKitTestRunner. https://trac.webkit.org/browser/webkit/trunk/Tools/WebKitTestRunner/TestController.cpp?rev=241755#L1221 |