Bug 207731 - import-w3c-tests breaks some tests that contain non UTF8 characters
Summary: import-w3c-tests breaks some tests that contain non UTF8 characters
Status: NEW
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Nobody
URL:
Keywords: InRadar
Depends on:
Blocks: 207734 246222
  Show dependency treegraph
 
Reported: 2020-02-13 16:56 PST by Carlos Alberto Lopez Perez
Modified: 2024-01-29 02:48 PST (History)
2 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Carlos Alberto Lopez Perez 2020-02-13 16:56:08 PST
The WPT test importer tries to rewrite some content inside imported tests, like for example:


1. We are adding "<!-- webkit-test-runner -->" comments to enable switches for experimental features
2. We are rewriting preffixes of some CSS properties to add a "-webkit" prefix. For example, we change "backface-visibility:" to "-webkit-backface-visibility:" or "user-select:" to "-webkit-user-select:"
3. We are rewriting the name of the test domains from "{{domains[www]}}" to "{{hosts[alt][]}}"
4. We are removing some JS scripts. See '<script src="testlib/testlib.js">' removed for example from test html/semantics/scripting-1/the-script-element/execution-timing/106-import.html
And more things...

To do that, we are parsing the files as UTF8 text, modifying the text of the test and writing it back.

But in the process of doing that we are making unwanted changes, sometimes breaking tests.

For example: the tests imported/w3c/web-platform-tests/encoding/eof-utf-8-one.html imported/w3c/web-platform-tests/encoding/eof-utf-8-three.html imported/w3c/web-platform-tests/encoding/eof-utf-8-two.html contain non-UTF8 byte truncated secuences at the end that are essential to those tests and the test importer removes that because of the fact that it reads the original file as UTF8 and then writes the parsed buffer to the destination.
Comment 1 Radar WebKit Bug Importer 2020-09-21 15:35:48 PDT
<rdar://problem/69331940>
Comment 2 Sam Sneddon [:gsnedders] 2020-11-16 11:18:23 PST
I'd assume this is caused by the webkitpy.w3c.test_converter parse/serialize cycle. 

At the very least, waiting on bug 218987 and hoping we get rid of the test converter at that point will solve this (or waiting for part 2, bug 218992, which will be the latest point at which it is gone).