Bug 20542 – Adding EOT Font Rendering capability

• Component: Layout and Rendering
• Status: NEW

[Prunthaban]

This patch is to provide 'EOT' (Embedded open Type) font rendering capability to Webkit along the "GDI only" path in Windows. There are a significant number of regional language sites (in India and some other countries) which uses EOT fonts for native characters (They don't use Unicode, they use non-unicode plain ASCII fonts with glyphs rendering native characters instead of standard ASCII characters). Without the support for EOT fonts,these sites will not be 'readable'.

[Prunthaban]

Created attachment 23022 [details]
A patch for supporting EOT rendering along GDI path

[Eric Seidel (no email)]

Comment on attachment 23022 [details]
A patch for supporting EOT rendering along GDI path

It seems there are missing files?  Can this little amount of code really support EOT fonts?

This will not compile in release mode:
+    if(!m_isEOT || renderingMode != AlternateRenderingMode)
+        ASSERT(m_cgFont);

ASSERT disappears in release mode.

Also, be sure to read:
http://webkit.org/coding/coding-style.html

There are lots of little style violations in this code.  A few which come to mind:

// comment
( generally we have a space after // before the comment)

if ( instead of if(

We don't use else after a return statement:
+    if(!m_isEOT)
+        return FontPlatformData(hfont, m_cgFont, size, bold, italic, renderingMode == AlternateRenderingMode);
+    else
+        return FontPlatformData(hfont, size, bold, italic, renderingMode == AlternateRenderingMode);	

{ goes on the next line after a function definition:

+void EOTStream::setInHeader(bool inHeader) {
+    m_inHeader = inHeader;
+}

Again, won't compile in Release mode (afaik):

+            if(!m_useGDI)
+                ASSERT(m_cgFont);


+bool isEOTFont(SharedBuffer* fontData) 

should have some minimum length check.  Otherwise it will crash if passed data shorter than sizeof(EOTPrefix)

According to our style guidelines, we don't use c-style casts in C++, this should be a reinterpret_cast<EOTPrefix*>:

+    EOTPrefix* prefix = (EOTPrefix*)(data);


No inner { } here:
+            if (localGlyphBuffer[i] == invalid_glyph) {
+                localGlyphBuffer[i] = 0;
+            }

r- for all the style violations.  I've CC'd hyatt and hixie, who are the two people who you would need to convince that EOT support is a good thing for WebKit before we could accept such a patch.

[Eric Seidel (no email)]

Comment on attachment 23022 [details]
A patch for supporting EOT rendering along GDI path

Oh, and *thanks* for the patch!  Congrats on your first WebKit patch.

One more thing is that we would require at least a few EOT test cases before we could land this.  I assume you could provide some test EOT fonts (from the web somewhere) and a couple HTML files which use them.  You can read more about creating layout tests here:
http://webkit.org/quality/testwriting.html

[Dave Hyatt]

We have a cross-platform solution (TTF) that works on other platforms besides just Windows.  

Implementing EOT undermines this solution, and will encourage vendors to write code that would only work on Windows.

If you're going to add EOT support, then really add EOT support.   A patch that only works on one platform (and only with one kind of text rendering mode) does not cut it, and it creates a gigantic compatibility difference in the downloadable fonts feature between the Windows platform and the other platforms.

[Ian 'Hixie' Hickson]

Hyatt is right, if we want to add EOT support to WebKit (and I understand that it might make sense to do that given Indic sites) then we absolutely must add it cross-platform. Just supporting one platform is bad.

[Prunthaban]

I have started working on providing corss-platform support for EOT. I will be updating this bug as and when I create new patches.

[Dave Hyatt]

Make sure you ifdef EOT support so that vendors are able to turn it off easily.

[Jungshik Shin]

*** Bug 18668 has been marked as a duplicate of this bug. ***

[Prunthaban]

Created attachment 24356 [details]
A patch to provide cross-platform EOT support

This patch will convert an EOT file into TTF file and pass it to the TTF handling routine. Since TTF handling already has a cross platform solution in Webkit, this patch should also work in all platforms.
This patch expects EOT_SUPPORT to be #defined. Otherwise, eot files will be ignored.
I have not tested this patch in Mac. This was tested on Windows and it works fine. 
The test-file attached expects broken.eot to be placed in resources directory (I will attach that too). If you need some additional test cases, I can create them. I need to know what type of test cases are expected.

[Prunthaban]

Created attachment 24357 [details]
EOT Font file to be used for testing

[Dave Hyatt]

I think I'd like to see the EOT stuff under its own subdirectory.  platform/graphics/eot/ would be fine.

You need to use ENABLE(EOT_SUPPORT) rather than EOT_SUPPORT.

I believe we use #ifndef NDEBUG for debugging code.

Not sure about the WINDOWS ifdef, but I think we use WIN_OS for that?

Are you respecting the restrictions that might be present in an EOT file (e.g., if the font says it can only be used with a certain URL etc.)?

[Prunthaban]

I will make those modifications.

The Root String (URL validation) handling is not done yet. I will be adding them before this patch is committed (I need to figure out a way to create a test-font without that restriction so that we can test it locally).

(In reply to comment #11)
> Are you respecting the restrictions that might be present in an EOT file (e.g.,
> if the font says it can only be used with a certain URL etc.)?

I think it is important to ensure that this conformance requirement does not get in users' way. For one, the Web Inspector should be able to show fonts regardless of the restriction. Non-browser WebKit clients may also want to do the same, so the restriction should probably be opt-in.

[Ian 'Hixie' Hickson]

What's the benefit to the users of honouring those restrictions? It seems like unless we have a good user-driven reason to honour the restrictions, we should just not bother.

[Srinivas Annam]

It is the UA that has the responsibility to enforce those restrictions. Otherwise, a web site operator can indiscriminately copy fonts from other web sites and make them available as their own.

[Dave Hyatt]

Comment on attachment 24356 [details]
A patch to provide cross-platform EOT support

Minusing based off my comments.

One thing I should say is that this patch has absolutely no vendor support.  In fact neither Safari nor Chrome are willing to add EOT support.

If there is no major vendor that is actually interested in shipping this code, then I don't think it should land in the WebKit tree.

[Prunthaban]

Hyatt,
Chrome is interested in adding EOT support (due to the fact that many major Indian regional sites use EOT) and that is the major reason behind this patch. (At present I am adding URL validation part to this patch).
Can you please clarify on this? If Chrome needs this patch, then we should not have a problem in landing this patch I believe.

[Dave Hyatt]

You say that this is wanted for Chrome, but when I brought this bug up today with Chrome team members they didn't seem to agree.

Regardless, I would strongly urge you not to add support for EOT to Chrome.

Adding EOT support has implications that go far beyond Web site compatibility.  Support of EOT undermines the current TTF support.

Mozilla, Opera and Apple are not going to add EOT support to their browsers.  If Chrome supports EOT this will create a significant fork for Chrome not only compared to shipping WebKit products but also with the other standards-compliant browsers.

I urge you to read up on the positions of Apple, Opera and Mozilla regarding why EOT support is not desired.

[Dave Hyatt]

https://lists.webkit.org/pipermail/webkit-dev/2008-October/005442.html

[Nicholas Shanks]

Indic sites need to move to standard encodings. Lots of fonts with unique encodings have had ISCII or Unicode converters created for them by users. (e.g. http://www.baraha.com/help/baraha/font_convert.htm or http://crl.nmsu.edu/~mleisher/nai.html )

Please persuade any sites you go to not to use font-dependant encodings. That not only means the use can't choose a font they are comfortable with, but it breaks indexing and searching.

If you choose ISCII rather than Unicode, serve the page with a Content-Type header containing one of the following charsets:

Codepage	Charset	Script
57002	x-iscii-de	Devanagri
57003	x-iscii-be	Bengali
57004	x-iscii-ta	Tamil
57005	x-iscii-te	Telugu
57006	x-iscii-as	Assamese
57007	x-iscii-or	Oriya
57008	x-iscii-ka	Kannada

When the user receives them, the browser will do the conversion from ISCII to Unicode, and render it with one of the user's installed fonts.

Sites that use fonts with custom encodings could also, at present, just serve the .ttf form of the 8-bit font to FF/Safari/Opera:

src: url(font.eot); /* ignored or overridden by everyone except IE */
src: url(font.ttf) format(truetype); /* ignored by IE due to unrecognised format() specifier */


Also, be aware that serving a full Unicode-encoded OpenType font to a Mac user via @font-face won't work. Apple's OpenType support is just getting started, and will not handle Indic for the foreseeable future, so an AAT font is required. The user should already have one for his language (either shipped with the OS, downloaded, or bought).

[Jungshik Shin]

(In reply to comment #20)
> Indic sites need to move to standard encodings. Lots of fonts with unique
> encodings have had ISCII or Unicode converters created for them by users. (e.g.
> http://www.baraha.com/help/baraha/font_convert.htm or
> http://crl.nmsu.edu/~mleisher/nai.html )

Ick. No ISCII, please. Switching to Unicode is a long over-due. No doubt about it. 

The reality is that they're slo........w in switching. In some Indian languages (like Malayalam), the switch has been accelerated recently, but in other languages, it appears that it's moving really slowly.  Sigh....

There are a lot of reasons to switch to Unicode. Even with EOT support in a browser, copy'n'paste, find-in-a-page, and history would not work.  And, obviously, the practice is not search-engine-friently, either. 

There is a Firefox extension that makes all the above possible by translating each textnode in a font-sepcific encoding (depending on the font used) to Unicode on-the-fly. ( http://padma.mozdev.org/ ) There was  a proposal to add that feature to Chrome (actually,  Webkit) natively, but I haven't  mentioned it here because it'd require rather invasive (and potentially perf-impacting) Webkit changes (as well as a few dozens of font-specific conversion tables) and I was not sure whether it'd fly here.  That's a certainly alternative to EOT support with a better user-experience, but with a lot to think about as mentioned earlier. I wonder what others think of this alternative. 

Just thinking aloud further:
How about enabling EOT supports for pre-configured list of (font, website) pairs? The vast majority of them will be Indian web sites with a small number of sites in Burmese,Khmer, etc (however, the speakers of the latter languages  have been rapidly embracing Unicode so it's perhaps 100% Indian web sites). 

[Jungshik Shin]

(In reply to comment #20)

> Sites that use fonts with custom encodings could also, at present, just serve
> the .ttf form of the 8-bit font to FF/Safari/Opera:
> 
> src: url(font.eot); /* ignored or overridden by everyone except IE */
> src: url(font.ttf) format(truetype); /* ignored by IE due to unrecognised
> format() specifier */

Yeah, that is the path of the least resistance to Indian web sites. If we cannot persuade them to switch to Unicode soon, at least we should ask them to do the above. Usually, they offer TTF available for download (for other browser users) so that there's no reason NOT to do the above. Needless to say, this does not solve find/search, and copy'n'paste. 

 
> Also, be aware that serving a full Unicode-encoded OpenType font to a Mac user
> via @font-face won't work. Apple's OpenType support is just getting started,
> and will not handle Indic for the foreseeable future, so an AAT font is
> required. The user should already have one for his language (either shipped
> with the OS, downloaded, or bought).


BTW, the current release version of Safari on Windows does not use OTF specified in font-face for complex scripts (Uniscribe codepath).  I haven't tried the trunk build, though. 

(In reply to comment #22)

> BTW, the current release version of Safari on Windows does not use OTF
> specified in font-face for complex scripts (Uniscribe codepath).  I haven't
> tried the trunk build, though.

It should work in TOT.

[Nicholas Shanks]

(In reply to comment #21)

> Ick. No ISCII, please.

There is nothing wrong in storing and serving the files in the various ISCII representations, just as there is nothing wrong in me serving GB 18030, ISO-8859-2, Windows 1256 or MacHebrew—because the browsers know how to convert them to their internal representation (which may not be Unicode anyway, especially low memory mobile devices in places like China and Japan).

> copy'n'paste, find-in-a-page, and history would not work

These features do not require Unicode, only that the encoding used is both specified by the server and understood by the client. In the case of WebKit, ICU is used to transcode from the served encoding to an internal representation (which happens to be Unicode).

If it is easier to talk sites into using ISCII than it is to get them to use Unicode, then that's great, lets do that. Anything to get them weened off font-specific encodings, and on to standardised ones.

[Maciej Stachowiak]

(In reply to comment #21)

> There is a Firefox extension that makes all the above possible by translating
> each textnode in a font-sepcific encoding (depending on the font used) to
> Unicode on-the-fly. ( http://padma.mozdev.org/ ) There was  a proposal to add
> that feature to Chrome (actually,  Webkit) natively, but I haven't  mentioned
> it here because it'd require rather invasive (and potentially perf-impacting)
> Webkit changes (as well as a few dozens of font-specific conversion tables) and
> I was not sure whether it'd fly here.  That's a certainly alternative to EOT
> support with a better user-experience, but with a lot to think about as
> mentioned earlier. I wonder what others think of this alternative. 

I'd love to hear more details about this proposal. Is it always perf-impacting, or only when a known oddly-encoded EOT font is used?

> Just thinking aloud further:
> How about enabling EOT supports for pre-configured list of (font, website)
> pairs? The vast majority of them will be Indian web sites with a small number
> of sites in Burmese,Khmer, etc (however, the speakers of the latter languages 
> have been rapidly embracing Unicode so it's perhaps 100% Indian web sites). 

That's also worth considering, though it sounds like translating on the fly to unicode could lead to a better user experience, in all the ways you mention.

[Jungshik Shin]

(In reply to comment #24)
> (In reply to comment #21)
> 
> > Ick. No ISCII, please.
> 
> There is nothing wrong in storing and serving the files in the various ISCII
> representations, 

You may not, but many people want to see *all* the web pages encoded in UTF-8 or UTF-16.  I know legacy-encoded pages will keep being put up for a long time to come.  However, do we really want to introduce any 'new legacy encoding' ('new' in the sense that they've not been used in any meaningful number of sites) into 'the web sphere' ?

> especially low memory mobile devices in places like China and Japan).
> 
> > copy'n'paste, find-in-a-page, and history would not work
> 
> These features do not require Unicode, 

Did I say that it requres Unicode?  Of course not ! Otherwise, all those non-Unicode encoded pages would be in big trouble. 

> If it is easier to talk sites into using ISCII than it is to get them to use
> Unicode, then that's great, lets do that.

I don't think it's any easier to persuade them to switch to ISCII than Unicode.  
Besides, note that Firefox does not support ISCII at all although it's relatively easy to add support for them.  Chrome does not either (it's even easier to support ISCII in Chrome than in Firefox), but again I'd not do that unless it's really necessary. 

[Nicholas Shanks]

(in reply to comment #26)

> You may not, but many people want to see *all* the web pages encoded in UTF-8
> or UTF-16.

FWIW I encode all my pages in UTF-8 by default, though also offer older encodings for non-Latin languages, with content negotiation turned on so older UAs can request something they understand.
I *am* a great proponent of Unicode, and have been actively involved in the Indian Unicode promotional community for several years. It is the best answer. I'm just saying it's not the only answer.

> I know legacy-encoded pages will keep being put up for a long time
> to come.

Do you ever envisage a time when they will not. Even 200 years from now, I think people will still be using pre-Unicode encodings.

> However, do we really want to introduce any 'new legacy encoding'
> ('new' in the sense that they've not been used in any meaningful number of
> sites) into 'the web sphere' ?

There are massive numbers of government documents in India that are ISCII encoded. Allowing these to be put online without intermediary steps will do wonders for e-governance in the country. Yes, we browser makers could dictate terms to the Indian government and force them to re-encode everything, but it's much easier for us to just support the encodings they are already using. I don't think your "not on the web at the moment" argument really passes muster. Sorry.

> > If it is easier to talk sites into using ISCII than it is to get them to use
> > Unicode, then that's great, lets do that.
> 
> I don't think it's any easier to persuade them to switch to ISCII than Unicode.

Unicode is unwieldy, foreign and unfamiliar, ISCII is something they've already heard of, home-grown, cosy and warm. I do think it will be easier to get people to use this than Unicode, though it hasn't been something I have pushed for previously.
There is also lots of ISCII-supporting software available in India, and it is older (therefore more likely to be installed on infrequently refreshed computers) than Unicode-compatible software. Without doubt there are large swathes of regional government that cannot use Unicode because of lack of modern software.

> Firefox does not support ISCII
> Chrome does not either
> I'd not do that unless it's really necessary.

I think ease-of-transition. Until it was suggested earlier in this thread, I hadn't though of doing the font-specific encoding conversion inside the browser itself. This may be a good short term solution, but still hurts other UAs like search engines etc, and may add another obstacle in the way of getting these sites to switch over long term. I think it's okay though for minority browsers that do not implement EOT to do this.

[Nicholas Shanks]

There seems to be a person/company/dunno offering a library for font-specific encoding conversion available at http://www.mukri.com/products/

[Nicholas Shanks]

There's a big list of converters (both Windows apps and online, but biased towards Devanagari) available at:
http://hi.wikipedia.org/wiki/इंटरनेट_पर_हिन्दी_के_साधन#.E0.A4.AB.E0.A4.BC.E0.A5.89.E0.A4.A3.E0.A5.8D.E0.A4.9F_.E0.A4.AA.E0.A4.B0.E0.A4.BF.E0.A4.B5.E0.A4.B0.E0.A5.8D.E0.A4.A4.E0.A4.95_.28Font_Converters.29

There is also a Windows app called Pavaritan supplied by the Indian government available for conversions:
http://ildc.gov.in/gistnew/Parivartan.zip

Perhaps someone who can read Hindi can go through these and check their licenses?

[John Daggett]

One word of caution here, the MTX compression scheme used in conjunction with the EOT format is a patented scheme, using it requires a license from the patent owner Monotype.  The EOT format has been proposed as a W3C format but until that proposal advances to the Recommendation phase, the submitted documentation does not imply a grant of license for its use.

Naturally, that doesn't apply to a Windows-only implementation that uses the t2embed library provided by Microsoft.

[Ian 'Hixie' Hickson]

My understanding is that nobody actually wants to implement EOT itself anymore; if anything happens, it'll just be a kind of text transformation that converts the input text to the right Unicode characters, without actually downloading the font at all.