Bug 168182 – Update custom line breaking iterators to the latest version of Unicode

• Component: Text
• Status: RESOLVED FIXED

[Myles C. Maxfield]

The custom rules we use are from 2014 and appear to be from ICU 54. We should update them.

[Myles C. Maxfield]

I went through the breaking rules line by line, and compared it to ICU's 54.1 release[1] . I found a few things:

1. This version of ICU has no concept of strict vs loose line breaking. Therefore, my comparisons are done ignoring the loose/normal/strict pieces of our custom rules. These pieces of our rules work by adding / removing characters from the existing unicode sets. Newer unicode does have a concept of strict / loose rules.
1. Our emoji handling is custom, and not included in ICU.
2. We have a couple declarations hidden behind ADDITIONAL_EMOJI_SUPPORT which ICU includes.
3. There are three constructions we have which the open source rules don't have:
$EXcm $INcm;
$CM* $IN $CM* $EX;
$CM+ $RI;

We should probably just opt all ports into the ADDITIONAL_EMOJI_SUPPORT flag and delete the flag.

The first two of the different constructions have to do with characters in the inseparable class, and the second one has to do with regional indicators. I'm not sure, but my theory right now is that these are just an oversight and aren't necessary.

[1] http://source.icu-project.org/repos/icu/icu/tags/release-54-1/source/data/brkitr/line.txt

[Myles C. Maxfield]

ICU 55.1 is the first release to support the concept of loose vs strict line breaking. However, I'm not sure which ICU is used on the oldest OS we support.

[Myles C. Maxfield]

I believe the oldest OS release we support is El Capitan, which uses ICU 55, which has loose & strict line breaking.

Perhaps the best solution is to just remove the custom rules entirely.

[Myles C. Maxfield]

(In reply to comment #3)
> which uses ICU 55,

*** 55.1

[Radar WebKit Bug Importer]

<rdar://problem/30481669>

[Myles C. Maxfield]

Created attachment 301320 [details]
WIP

[Myles C. Maxfield]

See also: https://bugs.webkit.org/show_bug.cgi?id=89235

[Myles C. Maxfield]

Created attachment 301329 [details]
Patch

[Myles C. Maxfield]

Committed r212235: <http://trac.webkit.org/changeset/212235>

[Saam Barati]

Comment on attachment 301329 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=301329&action=review

> Source/WTF/wtf/text/LineBreakIteratorPoolICU.h:54
> +        Vector<char> scratchBuffer(utf8Locale.length() + 11, 0);

Was this supposed to be + 1?

[Myles C. Maxfield]

Comment on attachment 301329 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=301329&action=review

>> Source/WTF/wtf/text/LineBreakIteratorPoolICU.h:54
>> +        Vector<char> scratchBuffer(utf8Locale.length() + 11, 0);
> 
> Was this supposed to be + 1?

Nope. The below code will add things like "@lb=normal" to the string, which is 10 characters long. Plus an additional one for the '\0' makes 11.