Bug 304238 – Big5 decoder doesn't recover to emit ASCII after an invalid leading byte

• Component: Platform
• Status: RESOLVED FIXED

[Nikita Skovoroda]

```
new TextDecoder('big5').decode(Uint8Array.of(0x81, 0x40))
```

Should be `'\ufffd@'` and instead returns just `'\ufffd'`

Chrome and Firefox behave correctly on this input.

See spec (https://encoding.spec.whatwg.org/#big5-decoder)

11.1.1. Big5 decoder, step 3.9

> If byte is an ASCII byte, restore byte to ioQueue.

Which means that `0x40` byte has to be attempted to be decoded again, which is what gives `@`.

[Nikita Skovoroda]

Related: https://github.com/nodejs/node/issues/40091 (Node.js also fails, but differently)
Chrome and Firefox are correct on that testcase too (it's similar)

[Darin Adler]

I’d be happy to fix the decoder, I’d also like to see tests covering this in WPT.

[Radar WebKit Bug Importer]

<rdar://problem/166672674>

[Darin Adler]

Pull request: https://github.com/WebKit/WebKit/pull/55538

[EWS]

Committed 304591@main (5f7d3882def0): <https://commits.webkit.org/304591@main>

Reviewed commits have been landed. Closing PR #55538 and removing active labels.

[Darin Adler]

https://github.com/web-platform-tests/wpt/pull/56844

[Nikita Skovoroda]

There are more tests that WPT misses, see https://github.com/ExodusOSS/bytes/blob/master/tests/encoding/mistakes.test.js

[Nikita Skovoroda]

WebKit fails on some of them

That (and the table at https://docs.google.com/spreadsheets/d/1pdEefRG6r9fZy61WHGz0TKSt8cO4ISWqlpBN5KntIvQ/edit) the primary place where I collect this
Didn't have time to file those to WPT yet, have been filing issues to platforms

Thanks for adding that test to WPT!

[Darin Adler]

I really appreciate the testing and reporting you have done for decoding bugs we can resolve in WebKit. It’s much more helpful to file new bug reports about each set of problems you find in WebKit rather than mentioning them in an already resolved bug report like this one. I’m not sure whether you intend to do that for the issues you mention above or if you are looking for someone else to do that.

[Nikita Skovoroda]

Filing over 50 reports is time-consuming
I will get to that someday

For now, see https://github.com/web-platform-tests/wpt/pull/56892 as the umbrella issue + all the tests combined

[Nikita Skovoroda]

Looking at this closer, this was a security issue
It violates critical requirement in https://encoding.spec.whatwg.org/#security-background