Bug 172205 – The latest version of Webkit can not correctly rendering 202F in mongolian home page

• Component: Layout and Rendering
• Status: RESOLVED CONFIGURATION CHANGED

[SiqinBilige]

Created attachment 310341 [details]
correct rendering

Summary:
The latest version of Safari can not  correctly display 202F in mongolian home page.
The latest version of Webkit can not correctly rendering 202F in mongolian home page

Steps to Reproduce:
Access mongolian home page http://www.mongolfont.com/mn/document/dj/dj033d.html.


Expected Results:
see firefox.png.

Actual Results:
see safari.png

Version:
macOS 10.12.4 (16E195)
Safari バージョン10.1 (12603.1.30.0.34)


Notes:
Not only safari on mac os, but also safari and webview on iOS.


Configuration:
All applications based on Webkit.


Attachments:
'safari.png' and 'firefox.png' were successfully uploaded.

[SiqinBilige]

Created attachment 310342 [details]
wrong rendering

[Radar WebKit Bug Importer]

<rdar://problem/32238286>

[Simon Fraser (smfr)]

Did this work in older iOS and macOS Safari versions?

[SiqinBilige]

(In reply to Simon Fraser (smfr) from comment #3)
> Did this work in older iOS and macOS Safari versions?

Yes.

[SiqinBilige]

(In reply to Simon Fraser (smfr) from comment #3)
> Did this work in older iOS and macOS Safari versions?
For this reason all browsers on Mac OS X and iOS are broken now.

[SiqinBilige]

This issue maybe related to https://bugs.webkit.org/show_bug.cgi?id=170284.
https://github.com/WebKit/webkit/blob/master/Source/WebCore/platform/graphics/FontCascade.cpp
FontCascade::CodePath FontCascade::characterRangeCodePath(const UChar* characters, unsigned len)
See 202F_code.png

The 202F and 200C are used as mongolian charators. We must passed it to shaping engine like harfbuzz together with other mongolian charators.
See 202F.png

[SiqinBilige]

Created attachment 314677 [details]
202F.png

[SiqinBilige]

Created attachment 314678 [details]
forgive it.

[SiqinBilige]

Created attachment 314681 [details]
202F_code.png

[SiqinBilige]

(In reply to SiqinBilige from comment #5)
> (In reply to Simon Fraser (smfr) from comment #3)
> > Did this work in older iOS and macOS Safari versions?
> For this reason all browsers on Mac OS X and iOS are broken now.

For this reason Safari on Mac OS X and all browsers on iOS are broken now.

[SiqinBilige]

(In reply to SiqinBilige from comment #6)
> This issue maybe related to https://bugs.webkit.org/show_bug.cgi?id=170284.
> https://github.com/WebKit/webkit/blob/master/Source/WebCore/platform/
> graphics/FontCascade.cpp
> FontCascade::CodePath FontCascade::characterRangeCodePath(const UChar*
> characters, unsigned len)
> See 202F_code.png
> 
> The 202F and 200C are used as mongolian charators. We must passed it to
> shaping engine like harfbuzz together with other mongolian charators.
> See 202F.png

Sorry, the 202F and 200D are used as mongolian charators. Not 200C.

[Simon Fraser (smfr)]

Created attachment 314688 [details]
Small test case

[SiqinBilige]

Use this test case please.
http://www.mongolfont.com/test/webkit/202F.html
The AAT font can download from
http://www.mongolfont.com/jAlmas/cms/documents/mongolfont/font/mnglwhiteaat.ttf.
If necessary, you can include this font to Webkit test case too.

[SiqinBilige]

Created attachment 314692 [details]
202F correct rendering in mongolian

[Myles C. Maxfield]

Raw CoreText draws this text correctly.

[Myles C. Maxfield]

We're finding that fontForCombiningCharacterSequence() is saying that the font doesn't support U+202F

[Myles C. Maxfield]

This is because we explicitly replace narrowNoBreakSpace with zeroWidthSpace in createAndFillGlyphPage(), and the font doesn't claim to support zeroWidthSpace, so we fall back to Lucida to render the space.

[Myles C. Maxfield]

Commenting out "overwriteCodePoint(narrowNoBreakSpace, zeroWidthSpace);" in overwriteCodePoint() fixes the problem.

If you are the creator of this font, you could also work around this problem by modifying the font to support zeroWidthSpace.

[Myles C. Maxfield]

What is the license of mnglwhiteaat.ttf? Is it compatible with the WebKit open source license? https://webkit.org/licensing-webkit/

[Myles C. Maxfield]

(When I say zeroWidthSpace, I mean U+0x200B)

[Myles C. Maxfield]

U+200B

[SiqinBilige]

(In reply to Myles C. Maxfield from comment #19)
> What is the license of mnglwhiteaat.ttf? Is it compatible with the WebKit
> open source license? https://webkit.org/licensing-webkit/

1. Because 202F and 200D　has different purpose in Mongolian. So we can not simply replace each other.
2. Mongolian do not use 200B.
3. The font licensed under the LGPL. ( Actually, the font which I created it no license at all. you can freely use it. ) If no font, Mongolian language difficult to test. mnglwhiteaat.ttf is for only mongolian(1800-1842). You can download the Universal font ( 1800-10AF which include all mongolian, todo, manchu, xibe, and aligali ). 

http://www.mongolfont.com/test/webkit/MongolianUniversalTest.ttf.

This  font licensed under the LGPL too. This font include both 202F and 200D.

[SiqinBilige]

By the way
Because the 202F is so problematic in mongolian on all platfoms. So, There is a plan to replace it with U+180F.

http://www.unicode.org/L2/L2017/17036-mongolian-suffix.pdf

If 180F special in Webkit, I want you to support it early.
Thank you.

[SiqinBilige]

(In reply to SiqinBilige from comment #22)
>( 1800-10AF which include all mongolian,
Sorry, there is a misspelling.
1800-10AF -> 1800-18AF

[Myles C. Maxfield]

(In reply to SiqinBilige from comment #22)
> (In reply to Myles C. Maxfield from comment #19)
> > What is the license of mnglwhiteaat.ttf? Is it compatible with the WebKit
> > open source license? https://webkit.org/licensing-webkit/
> 
> 1. Because 202F and 200D　has different purpose in Mongolian. So we can not
> simply replace each other.
> 2. Mongolian do not use 200B.
> 3. The font licensed under the LGPL. ( Actually, the font which I created it
> no license at all. you can freely use it. ) If no font, Mongolian language
> difficult to test. mnglwhiteaat.ttf is for only mongolian(1800-1842). You
> can download the Universal font ( 1800-10AF which include all mongolian,
> todo, manchu, xibe, and aligali ). 
> 
> http://www.mongolfont.com/test/webkit/MongolianUniversalTest.ttf.
> 
> This  font licensed under the LGPL too. This font include both 202F and 200D.

> Actually, the font which I created it no license at all. you can freely use it.

Which font did you create?

[Myles C. Maxfield]

Created attachment 314867 [details]
WIP

[Build Bot]

Attachment 314867 [details] did not pass style-queue:


ERROR: Source/WebCore/ChangeLog:8:  You should remove the 'No new tests' and either add and list tests, or explain why no new tests were possible.  [changelog/nonewtests] [5]
Total errors found: 1 in 1 files


If any of these errors are false positives, please file a bug against check-webkit-style.

[Alexey Proskuryakov]

rdar://problem/32194724

[Build Bot]

Comment on attachment 314867 [details]
WIP

Attachment 314867 [details] did not pass ios-sim-ews (ios-simulator-wk2):
Output: http://webkit-queues.webkit.org/results/4072174

New failing tests:
fast/text/narrow-non-breaking-space.html

[Build Bot]

Created attachment 314881 [details]
Archive of layout-test-results from ews124 for ios-simulator-wk2

The attached test failures were seen while running run-webkit-tests on the ios-sim-ews.
Bot: ews124  Port: ios-simulator-wk2  Platform: Mac OS X 10.12.5

[SiqinBilige]

> Which font did you create?

I created Both mnglwhiteaat.ttf and MongolianUniversalTest.ttf fonts, you can freely use it.

[SiqinBilige]

(In reply to SiqinBilige from comment #31)
> > Which font did you create?
> 
> I created Both mnglwhiteaat.ttf and MongolianUniversalTest.ttf fonts, you
> can freely use it.

It is better to use MongolianUniversalTest.ttf which include full set of 1800-18AF. include 200D and 202F too.

[Myles C. Maxfield]

Created attachment 315020 [details]
Fonts

[SiqinBilige]

For your reference, there are two case about usage of 202F in mongolian.
1. the word start with 202F.
2. the word include 202F.
(there is no word end with 202F in mongolian)

There is a sample.
  http://www.mongolfont.com/test/firefox/202f.html
  see 202f.jpg

[SiqinBilige]

Created attachment 328956 [details]
202f.jpg

[SiqinBilige]

For this reason all browsers on Mac OS X and iOS are broken now.
Can anyone fix it ?

[SiqinBilige]

(In reply to SiqinBilige from comment #36)
> For this reason all browsers on Mac OS X and iOS are broken now.
> Can anyone fix it ?

For this reason all browsers on iOS and safari on Mac OS X are broken now.
Can anyone fix it ?

[shrpne]

Similar issues:
https://bugs.webkit.org/show_bug.cgi?id=180744
https://bugs.webkit.org/show_bug.cgi?id=194985

Also want to note, that this problem occurs for all fonts I have tried, including system-ui, serif and sans-serif generic fonts.

And here is another test case reproducing no width for 202F:
https://jsfiddle.net/3g7yc5nf/1/

French uses 202F as a grouping/thousand separator as well. So this bug essentially breaks Intl.NumberFormat on iOS/macOS Safari when useGrouping is on.

[Myles C. Maxfield]

*** This bug has been marked as a duplicate of bug 215643 ***

[Myles C. Maxfield]

As far as I can tell, the Mongolian samples included here all work now. Can you verify that this is true?

[Myles C. Maxfield]

*** Bug 194985 has been marked as a duplicate of this bug. ***

[Myles C. Maxfield]

(I'm testing on macOS Monterey.)