Bug 172205

Summary: The latest version of Webkit can not correctly rendering 202F in mongolian home page
Product: WebKit Reporter: SiqinBilige <siqinbilige>
Component: Layout and RenderingAssignee: Myles C. Maxfield <mmaxfield>
Status: RESOLVED CONFIGURATION CHANGED    
Severity: Normal CC: bfulgham, buildbot, corsar89, dpaddock, felix.hageloh, icewil, mail, mmaxfield, simon.fraser, webkit-bug-importer, zalan
Priority: P1 Keywords: InRadar
Version: WebKit Nightly Build   
Hardware: Unspecified   
OS: Unspecified   
Bug Depends on:    
Bug Blocks: 206208    
Attachments:
Description Flags
correct rendering
none
wrong rendering
none
202F.png
none
forgive it.
none
202F_code.png
none
Small test case
none
202F correct rendering in mongolian
none
WIP
buildbot: commit-queue-
Archive of layout-test-results from ews124 for ios-simulator-wk2
none
Fonts
none
202f.jpg none

Description SiqinBilige 2017-05-16 18:37:55 PDT
Created attachment 310341 [details]
correct rendering

Summary:
The latest version of Safari can not  correctly display 202F in mongolian home page.
The latest version of Webkit can not correctly rendering 202F in mongolian home page

Steps to Reproduce:
Access mongolian home page http://www.mongolfont.com/mn/document/dj/dj033d.html.


Expected Results:
see firefox.png.

Actual Results:
see safari.png

Version:
macOS 10.12.4 (16E195)
Safari バージョン10.1 (12603.1.30.0.34)


Notes:
Not only safari on mac os, but also safari and webview on iOS.


Configuration:
All applications based on Webkit.


Attachments:
'safari.png' and 'firefox.png' were successfully uploaded.
Comment 1 SiqinBilige 2017-05-16 18:38:25 PDT
Created attachment 310342 [details]
wrong rendering
Comment 2 Radar WebKit Bug Importer 2017-05-16 18:41:22 PDT
<rdar://problem/32238286>
Comment 3 Simon Fraser (smfr) 2017-05-30 11:34:45 PDT
Did this work in older iOS and macOS Safari versions?
Comment 4 SiqinBilige 2017-07-05 18:24:41 PDT
(In reply to Simon Fraser (smfr) from comment #3)
> Did this work in older iOS and macOS Safari versions?

Yes.
Comment 5 SiqinBilige 2017-07-05 18:27:40 PDT
(In reply to Simon Fraser (smfr) from comment #3)
> Did this work in older iOS and macOS Safari versions?
For this reason all browsers on Mac OS X and iOS are broken now.
Comment 6 SiqinBilige 2017-07-05 19:19:57 PDT
This issue maybe related to https://bugs.webkit.org/show_bug.cgi?id=170284.
https://github.com/WebKit/webkit/blob/master/Source/WebCore/platform/graphics/FontCascade.cpp
FontCascade::CodePath FontCascade::characterRangeCodePath(const UChar* characters, unsigned len)
See 202F_code.png

The 202F and 200C are used as mongolian charators. We must passed it to shaping engine like harfbuzz together with other mongolian charators.
See 202F.png
Comment 7 SiqinBilige 2017-07-05 19:21:05 PDT
Created attachment 314677 [details]
202F.png
Comment 8 SiqinBilige 2017-07-05 19:21:21 PDT
Created attachment 314678 [details]
forgive it.
Comment 9 SiqinBilige 2017-07-05 19:30:56 PDT
Created attachment 314681 [details]
202F_code.png
Comment 10 SiqinBilige 2017-07-05 20:13:49 PDT
(In reply to SiqinBilige from comment #5)
> (In reply to Simon Fraser (smfr) from comment #3)
> > Did this work in older iOS and macOS Safari versions?
> For this reason all browsers on Mac OS X and iOS are broken now.

For this reason Safari on Mac OS X and all browsers on iOS are broken now.
Comment 11 SiqinBilige 2017-07-05 20:20:38 PDT
(In reply to SiqinBilige from comment #6)
> This issue maybe related to https://bugs.webkit.org/show_bug.cgi?id=170284.
> https://github.com/WebKit/webkit/blob/master/Source/WebCore/platform/
> graphics/FontCascade.cpp
> FontCascade::CodePath FontCascade::characterRangeCodePath(const UChar*
> characters, unsigned len)
> See 202F_code.png
> 
> The 202F and 200C are used as mongolian charators. We must passed it to
> shaping engine like harfbuzz together with other mongolian charators.
> See 202F.png

Sorry, the 202F and 200D are used as mongolian charators. Not 200C.
Comment 12 Simon Fraser (smfr) 2017-07-05 20:31:26 PDT
Created attachment 314688 [details]
Small test case
Comment 13 SiqinBilige 2017-07-05 20:58:59 PDT
Use this test case please.
http://www.mongolfont.com/test/webkit/202F.html
The AAT font can download from
http://www.mongolfont.com/jAlmas/cms/documents/mongolfont/font/mnglwhiteaat.ttf.
If necessary, you can include this font to Webkit test case too.
Comment 14 SiqinBilige 2017-07-05 21:08:48 PDT
Created attachment 314692 [details]
202F correct rendering in mongolian
Comment 15 Myles C. Maxfield 2017-07-06 13:24:46 PDT
Raw CoreText draws this text correctly.
Comment 16 Myles C. Maxfield 2017-07-06 17:34:28 PDT
We're finding that fontForCombiningCharacterSequence() is saying that the font doesn't support U+202F
Comment 17 Myles C. Maxfield 2017-07-06 17:40:38 PDT
This is because we explicitly replace narrowNoBreakSpace with zeroWidthSpace in createAndFillGlyphPage(), and the font doesn't claim to support zeroWidthSpace, so we fall back to Lucida to render the space.
Comment 18 Myles C. Maxfield 2017-07-06 17:42:29 PDT
Commenting out "overwriteCodePoint(narrowNoBreakSpace, zeroWidthSpace);" in overwriteCodePoint() fixes the problem.

If you are the creator of this font, you could also work around this problem by modifying the font to support zeroWidthSpace.
Comment 19 Myles C. Maxfield 2017-07-06 17:44:38 PDT
What is the license of mnglwhiteaat.ttf? Is it compatible with the WebKit open source license? https://webkit.org/licensing-webkit/
Comment 20 Myles C. Maxfield 2017-07-06 17:45:08 PDT
(When I say zeroWidthSpace, I mean U+0x200B)
Comment 21 Myles C. Maxfield 2017-07-06 17:45:49 PDT
U+200B
Comment 22 SiqinBilige 2017-07-06 18:39:18 PDT
(In reply to Myles C. Maxfield from comment #19)
> What is the license of mnglwhiteaat.ttf? Is it compatible with the WebKit
> open source license? https://webkit.org/licensing-webkit/

1. Because 202F and 200D has different purpose in Mongolian. So we can not simply replace each other.
2. Mongolian do not use 200B.
3. The font licensed under the LGPL. ( Actually, the font which I created it no license at all. you can freely use it. ) If no font, Mongolian language difficult to test. mnglwhiteaat.ttf is for only mongolian(1800-1842). You can download the Universal font ( 1800-10AF which include all mongolian, todo, manchu, xibe, and aligali ). 

http://www.mongolfont.com/test/webkit/MongolianUniversalTest.ttf.

This  font licensed under the LGPL too. This font include both 202F and 200D.
Comment 23 SiqinBilige 2017-07-06 18:58:20 PDT
By the way
Because the 202F is so problematic in mongolian on all platfoms. So, There is a plan to replace it with U+180F.

http://www.unicode.org/L2/L2017/17036-mongolian-suffix.pdf

If 180F special in Webkit, I want you to support it early.
Thank you.
Comment 24 SiqinBilige 2017-07-06 19:24:43 PDT
(In reply to SiqinBilige from comment #22)
>( 1800-10AF which include all mongolian,
Sorry, there is a misspelling.
1800-10AF -> 1800-18AF
Comment 25 Myles C. Maxfield 2017-07-07 11:32:34 PDT
(In reply to SiqinBilige from comment #22)
> (In reply to Myles C. Maxfield from comment #19)
> > What is the license of mnglwhiteaat.ttf? Is it compatible with the WebKit
> > open source license? https://webkit.org/licensing-webkit/
> 
> 1. Because 202F and 200D has different purpose in Mongolian. So we can not
> simply replace each other.
> 2. Mongolian do not use 200B.
> 3. The font licensed under the LGPL. ( Actually, the font which I created it
> no license at all. you can freely use it. ) If no font, Mongolian language
> difficult to test. mnglwhiteaat.ttf is for only mongolian(1800-1842). You
> can download the Universal font ( 1800-10AF which include all mongolian,
> todo, manchu, xibe, and aligali ). 
> 
> http://www.mongolfont.com/test/webkit/MongolianUniversalTest.ttf.
> 
> This  font licensed under the LGPL too. This font include both 202F and 200D.

> Actually, the font which I created it no license at all. you can freely use it.

Which font did you create?
Comment 26 Myles C. Maxfield 2017-07-07 13:16:12 PDT
Created attachment 314867 [details]
WIP
Comment 27 Build Bot 2017-07-07 13:19:10 PDT
Attachment 314867 [details] did not pass style-queue:


ERROR: Source/WebCore/ChangeLog:8:  You should remove the 'No new tests' and either add and list tests, or explain why no new tests were possible.  [changelog/nonewtests] [5]
Total errors found: 1 in 1 files


If any of these errors are false positives, please file a bug against check-webkit-style.
Comment 28 Alexey Proskuryakov 2017-07-07 13:49:29 PDT
rdar://problem/32194724
Comment 29 Build Bot 2017-07-07 14:43:49 PDT
Comment on attachment 314867 [details]
WIP

Attachment 314867 [details] did not pass ios-sim-ews (ios-simulator-wk2):
Output: http://webkit-queues.webkit.org/results/4072174

New failing tests:
fast/text/narrow-non-breaking-space.html
Comment 30 Build Bot 2017-07-07 14:43:51 PDT
Created attachment 314881 [details]
Archive of layout-test-results from ews124 for ios-simulator-wk2

The attached test failures were seen while running run-webkit-tests on the ios-sim-ews.
Bot: ews124  Port: ios-simulator-wk2  Platform: Mac OS X 10.12.5
Comment 31 SiqinBilige 2017-07-07 20:08:48 PDT
> Which font did you create?

I created Both mnglwhiteaat.ttf and MongolianUniversalTest.ttf fonts, you can freely use it.
Comment 32 SiqinBilige 2017-07-07 20:18:55 PDT
(In reply to SiqinBilige from comment #31)
> > Which font did you create?
> 
> I created Both mnglwhiteaat.ttf and MongolianUniversalTest.ttf fonts, you
> can freely use it.

It is better to use MongolianUniversalTest.ttf which include full set of 1800-18AF. include 200D and 202F too.
Comment 33 Myles C. Maxfield 2017-07-10 14:03:19 PDT
Created attachment 315020 [details]
Fonts
Comment 34 SiqinBilige 2017-12-11 01:55:01 PST
For your reference, there are two case about usage of 202F in mongolian.
1. the word start with 202F.
2. the word include 202F.
(there is no word end with 202F in mongolian)

There is a sample.
  http://www.mongolfont.com/test/firefox/202f.html
  see 202f.jpg
Comment 35 SiqinBilige 2017-12-11 01:55:36 PST
Created attachment 328956 [details]
202f.jpg
Comment 36 SiqinBilige 2018-01-30 17:23:25 PST
For this reason all browsers on Mac OS X and iOS are broken now.
Can anyone fix it ?
Comment 37 SiqinBilige 2018-01-30 17:26:34 PST
(In reply to SiqinBilige from comment #36)
> For this reason all browsers on Mac OS X and iOS are broken now.
> Can anyone fix it ?

For this reason all browsers on iOS and safari on Mac OS X are broken now.
Can anyone fix it ?
Comment 38 shrpne 2019-06-07 04:25:04 PDT
Similar issues:
https://bugs.webkit.org/show_bug.cgi?id=180744
https://bugs.webkit.org/show_bug.cgi?id=194985

Also want to note, that this problem occurs for all fonts I have tried, including system-ui, serif and sans-serif generic fonts.

And here is another test case reproducing no width for 202F:
https://jsfiddle.net/3g7yc5nf/1/
Comment 39 felix.hageloh 2020-02-28 07:42:49 PST
French uses 202F as a grouping/thousand separator as well. So this bug essentially breaks Intl.NumberFormat on iOS/macOS Safari when useGrouping is on.
Comment 40 Myles C. Maxfield 2021-07-23 11:54:56 PDT

*** This bug has been marked as a duplicate of bug 215643 ***
Comment 41 Myles C. Maxfield 2021-08-11 00:23:34 PDT
As far as I can tell, the Mongolian samples included here all work now. Can you verify that this is true?
Comment 42 Myles C. Maxfield 2021-08-11 11:58:55 PDT
*** Bug 194985 has been marked as a duplicate of this bug. ***
Comment 43 Myles C. Maxfield 2021-08-11 11:59:38 PDT
(I'm testing on macOS Monterey.)