Bug 207731 – import-w3c-tests breaks some tests that contain non UTF8 characters

• Status: NEW
• Component: Tools / Tests

[Carlos Alberto Lopez Perez]

The WPT test importer tries to rewrite some content inside imported tests, like for example:


1. We are adding "<!-- webkit-test-runner -->" comments to enable switches for experimental features
2. We are rewriting preffixes of some CSS properties to add a "-webkit" prefix. For example, we change "backface-visibility:" to "-webkit-backface-visibility:" or "user-select:" to "-webkit-user-select:"
3. We are rewriting the name of the test domains from "{{domains[www]}}" to "{{hosts[alt][]}}"
4. We are removing some JS scripts. See '<script src="testlib/testlib.js">' removed for example from test html/semantics/scripting-1/the-script-element/execution-timing/106-import.html
And more things...

To do that, we are parsing the files as UTF8 text, modifying the text of the test and writing it back.

But in the process of doing that we are making unwanted changes, sometimes breaking tests.

For example: the tests imported/w3c/web-platform-tests/encoding/eof-utf-8-one.html imported/w3c/web-platform-tests/encoding/eof-utf-8-three.html imported/w3c/web-platform-tests/encoding/eof-utf-8-two.html contain non-UTF8 byte truncated secuences at the end that are essential to those tests and the test importer removes that because of the fact that it reads the original file as UTF8 and then writes the parsed buffer to the destination.

[Radar WebKit Bug Importer]

<rdar://problem/69331940>

[Sam Sneddon [:gsnedders]]

I'd assume this is caused by the webkitpy.w3c.test_converter parse/serialize cycle. 

At the very least, waiting on bug 218987 and hoping we get rid of the test converter at that point will solve this (or waiting for part 2, bug 218992, which will be the latest point at which it is gone).