Bug 91181 – nrwt: don't choke when printing invalid utf-8 to stderr

• Component: Tools / Tests
• Status: RESOLVED FIXED

[David Grogan]

This seems to only reproduce on one of my branches, so I've probably introduced an error that interacts badly with the recent nrwt changes.  Though I have no changes in Tools, just Source. Dirk, if you could try to glean some hints about a possible nrwt bug from this stacktrace, I'd appreciate it.

$ new-run-webkit-tests -f --debug --no-new-test-results --no-retry-failures storage/indexeddb

    File "/sdb1/chrome/1/src/third_party/WebKit/Tools/Scripts/webkitpy/common/message_pool.py", line 242, in run
      worker.handle(message.name, message.src, *message.args)
    File "/sdb1/chrome/1/src/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/worker.py", line 81, in handle
      self._run_test(test_input)
    File "/sdb1/chrome/1/src/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/worker.py", line 104, in _run_test
      result = self._run_test_with_timeout(test_input, test_timeout_sec)
    File "/sdb1/chrome/1/src/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/worker.py", line 146, in _run_test_with_timeout
      return self._run_test_in_this_thread(test_input)
    File "/sdb1/chrome/1/src/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/worker.py", line 231, in _run_test_in_this_thread
      return self._run_single_test(self._driver, test_input)
    File "/sdb1/chrome/1/src/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/worker.py", line 235, in _run_single_test
      test_input, driver, self._name)
    File "/sdb1/chrome/1/src/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py", line 46, in run_single_test
      return runner.run()
    File "/sdb1/chrome/1/src/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py", line 105, in run
      return self._run_compare_test()
    File "/sdb1/chrome/1/src/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py", line 118, in _run_compare_test
      test_result_writer.write_test_result(self._filesystem, self._port, self._test_name, driver_output, expected_driver_output, test_result.failures)
    File "/sdb1/chrome/1/src/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/test_result_writer.py", line 45, in write_test_result
      writer.write_stderr(driver_output.error)
    File "/sdb1/chrome/1/src/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/test_result_writer.py", line 163, in write_stderr
      self._write_text_file(filename, error)
    File "/sdb1/chrome/1/src/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/test_result_writer.py", line 136, in _write_text_file
      self._filesystem.write_text_file(path, contents)
    File "/sdb1/chrome/1/src/third_party/WebKit/Tools/Scripts/webkitpy/common/system/filesystem.py", line 226, in write_text_file
      f.write(contents)
    File "/usr/lib/python2.6/codecs.py", line 686, in write
      return self.writer.write(data)
    File "/usr/lib/python2.6/codecs.py", line 351, in write
      data, consumed = self.encode(object, self.errors)
worker/12 raised UnicodeDecodeError(''ascii' codec can't decode byte 0xef in position 914: ordinal not in range(128)'):
Exception raised, exiting

[Dirk Pranke]

the stack trace is saying that you printed something to stderr from DRT/WebKit that isn't a valid utf-8 codepoint, specifically a byte with value 0xef. Does that help?

[David Grogan]

Ah to stderr. Yes, there is a printf that prints invalid characters when running storage/indexeddb/odd-strings.html. Indeed, running odd-strings.html alone causes an exception. Though this just started happening today, at 122505.  Before then, running new-run-webkit-tests --debug storage/indexeddb/odd-strings.html succeeds.  After 122505, I get the stack trace below (practically the same as the first).  Any chance of making nrwt tolerate the unprintable characters?

$ new-run-webkit-tests --debug storage/indexeddb/odd-strings.html
Exception raised, exiting       
    File "/sdb1/chrome/1/src/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 129, in run
      unexpected_result_count = manager.run()
    File "/sdb1/chrome/1/src/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 866, in run
      interrupted, keyboard_interrupted, thread_timings, test_timings, individual_test_timings = self._run_tests(self._test_files_list, result_summary, int(self._options.child_processes))
    File "/sdb1/chrome/1/src/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 777, in _run_tests
      pool.run(('test_list', shard.name, shard.test_inputs) for shard in all_shards)
    File "/sdb1/chrome/1/src/third_party/WebKit/Tools/Scripts/webkitpy/common/message_pool.py", line 97, in run
      self.wait()
    File "/sdb1/chrome/1/src/third_party/WebKit/Tools/Scripts/webkitpy/common/message_pool.py", line 117, in wait
      self._workers[0].run()
    File "/sdb1/chrome/1/src/third_party/WebKit/Tools/Scripts/webkitpy/common/message_pool.py", line 242, in run
      worker.handle(message.name, message.src, *message.args)
    File "/sdb1/chrome/1/src/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/worker.py", line 81, in handle
      self._run_test(test_input)
    File "/sdb1/chrome/1/src/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/worker.py", line 104, in _run_test
      result = self._run_test_with_timeout(test_input, test_timeout_sec)
    File "/sdb1/chrome/1/src/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/worker.py", line 146, in _run_test_with_timeout
      return self._run_test_in_this_thread(test_input)
    File "/sdb1/chrome/1/src/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/worker.py", line 231, in _run_test_in_this_thread
      return self._run_single_test(self._driver, test_input)
    File "/sdb1/chrome/1/src/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/worker.py", line 235, in _run_single_test
      test_input, driver, self._name)
    File "/sdb1/chrome/1/src/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py", line 46, in run_single_test
      return runner.run()
    File "/sdb1/chrome/1/src/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py", line 105, in run
      return self._run_compare_test()
    File "/sdb1/chrome/1/src/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py", line 118, in _run_compare_test
      test_result_writer.write_test_result(self._filesystem, self._port, self._test_name, driver_output, expected_driver_output, test_result.failures)
    File "/sdb1/chrome/1/src/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/test_result_writer.py", line 45, in write_test_result
      writer.write_stderr(driver_output.error)
    File "/sdb1/chrome/1/src/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/test_result_writer.py", line 163, in write_stderr
      self._write_text_file(filename, error)
    File "/sdb1/chrome/1/src/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/test_result_writer.py", line 136, in _write_text_file
      self._filesystem.write_text_file(path, contents)
    File "/sdb1/chrome/1/src/third_party/WebKit/Tools/Scripts/webkitpy/common/system/filesystem.py", line 226, in write_text_file
      f.write(contents)
    File "/usr/lib/python2.6/codecs.py", line 686, in write
      return self.writer.write(data)
    File "/usr/lib/python2.6/codecs.py", line 351, in write
      data, consumed = self.encode(object, self.errors)

[Dirk Pranke]

As a workaround, all you need to do is change:

File "/sdb1/chrome/1/src/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/test_result_writer.py", line 163, in write_stderr
      self._write_text_file(filename, error)
   
to self._write_binary_file(filename, error)

this would probably be a reasonable change to make, period. I don't think we have a strong requirement that stderr is utf-8 :).

[David Grogan]

Created attachment 152137 [details]
Patch

[David Grogan]

Dirk, could you review this?

[Dirk Pranke]

Comment on attachment 152137 [details]
Patch

yes, I can :).

[WebKit Review Bot]

Comment on attachment 152137 [details]
Patch

Clearing flags on attachment: 152137

Committed r122551: <http://trac.webkit.org/changeset/122551>

[WebKit Review Bot]

All reviewed patches have been landed.  Closing bug.