webKit-patch fails to upload patch with 20000 files changed due to shell limitation: Argument list too long

[Kimmo Kinnunen]

WebKit patch fails to upload patch with 20000 files

 git show --summary |wc -l  
   22512

While large amount, it's not inconceivable.

The patch in question tried to introduce new WebGL conformance test suites, e.g. replace LayoutTests/{1.0.3,2.0.0} -> {1.0.4,2.0.1}.
The WebGL conformance test suite has ~2500 testcases. WebKit generates one test driver .html and one -expected.txt for each test. 


kkinnunen@Kimmos-iMac-Pro OpenSource % Tools/Scripts/webkit-patch upload -g HEAD 
Traceback (most recent call last):
  File "Tools/Scripts/webkit-patch", line 80, in <module>
    main()
  File "Tools/Scripts/webkit-patch", line 75, in main
    WebKitPatch(os.path.abspath(__file__)).main()
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/tool/multicommandtool.py", line 305, in main
    result = command.check_arguments_and_execute(options, args, self)
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/tool/multicommandtool.py", line 123, in check_arguments_and_execute
    return self.execute(options, args, tool) or 0
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/tool/commands/abstractsequencedcommand.py", line 55, in execute
    self._sequence.run_and_handle_errors(tool, options, state)
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/tool/commands/stepsequence.py", line 73, in run_and_handle_errors
    self._run(tool, options, state)
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/tool/commands/stepsequence.py", line 67, in _run
    step(tool, options).run(state)
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/tool/steps/checkstyle.py", line 66, in run
    self._tool.executive.run_and_throw_if_fail(self._tool.deprecated_port().check_webkit_style_command() + args, cwd=self._tool.scm().checkout_root)
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/common/system/executive.py", line 148, in run_and_throw_if_fail
    exit_code = self._run_command_with_teed_output(args, child_stdout, **kwargs)
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/common/system/executive.py", line 123, in _run_command_with_teed_output
    **kwargs)
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/common/system/executive.py", line 532, in popen
    result = subprocess.Popen(string_args, env=env, **kwargs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 394, in __init__
    errread, errwrite)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1047, in _execute_child
    raise child_exception
OSError: [Errno 7] Argument list too long



kkinnunen@Kimmos-iMac-Pro OpenSource % tools/Scripts/webkit-patch upload -g HEAD --ignore-style             
Traceback (most recent call last):
  File "tools/Scripts/webkit-patch", line 80, in <module>
    main()
  File "tools/Scripts/webkit-patch", line 75, in main
    WebKitPatch(os.path.abspath(__file__)).main()
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/tool/multicommandtool.py", line 305, in main
    result = command.check_arguments_and_execute(options, args, self)
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/tool/multicommandtool.py", line 123, in check_arguments_and_execute
    return self.execute(options, args, tool) or 0
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/tool/commands/abstractsequencedcommand.py", line 55, in execute
    self._sequence.run_and_handle_errors(tool, options, state)
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/tool/commands/stepsequence.py", line 73, in run_and_handle_errors
    self._run(tool, options, state)
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/tool/commands/stepsequence.py", line 67, in _run
    step(tool, options).run(state)
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/tool/steps/confirmdiff.py", line 73, in run
    diff = self.cached_lookup(state, "diff")
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/tool/steps/abstractstep.py", line 62, in cached_lookup
    state[key] = promise(self, state)
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/tool/steps/abstractstep.py", line 52, in <lambda>
    "diff": lambda self, state: self._tool.scm().create_patch(self._options.git_commit, changed_files=self._changed_files(state)),
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/common/checkout/scm/git.py", line 375, in create_patch
    return self.prepend_svn_revision(self.run(command, decode_output=False, cwd=self.checkout_root))
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/common/checkout/scm/scm.py", line 80, in run
    decode_output=decode_output)
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/common/system/executive.py", line 406, in run_command
    close_fds=self._should_close_fds())
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/common/system/executive.py", line 532, in popen
    result = subprocess.Popen(string_args, env=env, **kwargs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 394, in __init__
    errread, errwrite)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1047, in _execute_child
    raise child_exception
OSError: [Errno 7] Argument list too long


kkinnunen@Kimmos-iMac-Pro OpenSource % tools/Scripts/check-webkit-style  -g HEAD                
Traceback (most recent call last):
  File "tools/Scripts/check-webkit-style", line 46, in <module>
    sys.exit(CheckWebKitStyle().main())
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/style/main.py", line 159, in main
    patch_checker.check(patch)
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/style/patchreader.py", line 60, in check
    patch_string = string_utils.decode(patch_string, target_type=str)
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/string_utils.py", line 44, in decode
    return data.decode(encoding, errors=errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x82 in position 3450534: invalid start byte

[David Kilzer (:ddkilzer)]

We should probably break the check-webkit-style issue out into its own bug, since it has a different cause (unless we use this as an umbrella bug).

CC Jonathan just FYI.

[Jonathan Bedard]

Not sure how worth fixing this issue is, we'll be moving to pull-requests in the near future, which is the architectural fix for this sort of problem.

[Kimmo Kinnunen]

Filed bug 223367 about the check-webkit-style, which probably is not going away.

[Radar WebKit Bug Importer]

<rdar://problem/75748788>