Bug 91212 - ruby1.9 fails in PrettyPatch.rb with invalid byte sequence in UTF-8
Summary: ruby1.9 fails in PrettyPatch.rb with invalid byte sequence in UTF-8
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: 528+ (Nightly build)
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Simon Pena
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2012-07-13 03:58 PDT by Simon Pena
Modified: 2012-10-03 04:53 PDT (History)
3 users (show)

See Also:

Attachments
Patch (1.97 KB, patch)
2012-07-13 04:16 PDT, Simon Pena 		no flags Details | Formatted Diff | Diff
Patch (3.89 KB, patch)
2012-10-02 00:20 PDT, Simon Pena 		no flags Details | Formatted Diff | Diff
Show Obsolete (1) View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Simon Pena 2012-07-13 03:58:17 PDT 
After upgrading the setup in one of the GTK 64bits bots, we found that if ruby1.9 is used instead of ruby1.8, the following error happens when running test-webkitpy:

/home/spenap/Projects/WebKit/Websites/bugs.webkit.org/PrettyPatch/PrettyPatch.rb:108:in `gsub': invalid byte sequence in UTF-8 (ArgumentError)
	from /home/spenap/Projects/WebKit/Websites/bugs.webkit.org/PrettyPatch/PrettyPatch.rb:108:in `normalize_line_ending'
	from /home/spenap/Projects/WebKit/Websites/bugs.webkit.org/PrettyPatch/PrettyPatch.rb:19:in `prettify'
	from /home/spenap/Projects/WebKit/Websites/bugs.webkit.org/PrettyPatch/prettify.rb:26:in `<main>'

This is caused by the new encoding engine that Ruby 1.9 uses (see http://blog.grayproductions.net/articles/ruby_19s_string)

I took a look at http://smyck.net/2011/05/13/files-with-mixed-and-invalid-encodings-in-ruby/ to work around it, but iconv will be deprecated in the future, and String#encode is recommended instead. However, by looking at http://blog.segment7.net/2010/12/17/from-iconv-iconv-to-string-encode, the new alternative is not as straight-forward as using iconv.

We're currently keeping ruby in version 1.8, which works as expected.
Comment 1 Simon Pena 2012-07-13 04:03:55 PDT 
CCing Caio, who added the normalize_line_ending method originally.
Comment 2 Simon Pena 2012-07-13 04:16:13 PDT 
Created attachment 152212 [details]
Patch
Comment 3 Simon Pena 2012-07-13 04:17:48 PDT 
(In reply to comment #2)
> Created an attachment (id=152212) [details]
> Patch

I've attached a patch using Iconv, although, as mentioned in comment #0, it should be better to use String#encode.
Comment 4 Hajime Morrita 2012-07-18 19:25:09 PDT 
(In reply to comment #3)
> (In reply to comment #2)
> > Created an attachment (id=152212) [details] [details]
> > Patch
> 
> I've attached a patch using Iconv, although, as mentioned in comment #0, it should be better to use String#encode.
Let's do what you think the best :-)
Comment 5 Simon Pena 2012-07-19 01:38:04 PDT 
Comment on attachment 152212 [details]
Patch

OK, I'll try to come up with a proper fix later.
Comment 6 Simon Pena 2012-10-02 00:20:17 PDT 
Created attachment 166628 [details]
Patch
Comment 7 Simon Pena 2012-10-02 00:23:30 PDT 
This new version of the patch uses the "proper" implementation when Ruby's version is >= 1.9 (using the string encode method) and keeps the current one for 1.8.
Comment 8 Hajime Morrita 2012-10-02 22:04:04 PDT 
Comment on attachment 166628 [details]
Patch

Wow, I didn't know that encode() support :fallback option.
Comment 9 WebKit Review Bot 2012-10-03 04:53:45 PDT 
Comment on attachment 166628 [details]
Patch

Clearing flags on attachment: 166628

Committed r130276: <http://trac.webkit.org/changeset/130276>
Comment 10 WebKit Review Bot 2012-10-03 04:53:48 PDT 
All reviewed patches have been landed.  Closing bug.