Bug 233921 – TextDecoder doesn't detect invalid UTF-8 sequences early enough

• Component: DOM
• Status: RESOLVED FIXED

[Andreu Botella]

WPT tests: https://wpt.fyi/results/encoding/textdecoder-eof.any.html?label=experimental&label=master&aligned (stream: true case), https://wpt.fyi/results/encoding/textdecoder-streaming.any.html?label=experimental&label=master&aligned, https://wpt.fyi/results/encoding/streams/decode-utf8.any.html?label=experimental&label=master&aligned (non-SharedArrayBuffer cases)

Related Chromium bug: https://bugs.chromium.org/p/chromium/issues/detail?id=796697

When the TextCodecUTF8 decoder finds a non-ASCII lead byte, it waits until enough bytes are consumed to make a valid sequence starting at that position, before starting to process the bytes. This goes against the encoding spec, which requires the replacement character to be emitted as soon as enough bytes are consumed to tell that the sequence is in fact invalid.

While this does not make a difference for non-streaming input, or for streaming data coming from the network, it does make a difference in that TextDecoder returns the wrong result as per the spec when in streaming mode:

const decoder = new TextDecoder();
console.log(decoder.decode(new Uint8Array([0xF0, 0x9F]), { stream: true }));
console.log(decoder.decode(new Uint8Array([0x41]), { stream: true }));
console.log(decoder.decode(new Uint8Array([0x42]), { stream: true }));

As per the spec, and in Firefox and Chromium 98, this prints "", "�A", "B". In WebKit and previous versions of Chromium, it prints "", "", "�AB".

[Alex Christensen]

I think we will need to do basically the same thing as Chromium, but it won't be a simple cherry pick because their DecodeNonASCIISequence returns a different value than our decodeNonASCIISequence and doesn't change the length, and their handlePartialSequence for LChar and UChar are more similar than ours are.

[Alex Christensen]

*** Bug 234030 has been marked as a duplicate of this bug. ***

[Andreu Botella]

Created attachment 446994 [details]
Patch

[EWS Watchlist]

This patch modifies the imported WPT tests. Please ensure that any changes on the tests (not coming from a WPT import) are exported to WPT. Please see https://trac.webkit.org/wiki/WPTExportProcess

[Andreu Botella]

Created attachment 447070 [details]
Patch

[Radar WebKit Bug Importer]

<rdar://problem/86461983>

[EWS]

Committed r287024 (245229@main): <https://commits.webkit.org/245229@main>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 447070 [details].