Detecting a UTF-8 string with control characters leads to unexpected behavior

[ruicc]

import Encoding from 'encoding-japanese';

const bs = Buffer.from(new Uint8Array([0x08]));
console.log('detect("<BS>"):', Encoding.detect(bs));

const utf8str = Buffer.from("UTF8の文字列です", 'utf8');
console.log('detect("UTF8の文字列です"):', Encoding.detect(utf8str));

const utf8BS = Buffer.concat([utf8str, bs]);
console.log('detect("UTF8の文字列です<BS>"):', Encoding.detect(utf8BS));

output:

detect("<BS>"): ASCII
detect("UTF8の文字列です"): UTF8
detect("UTF8の文字列です<BS>"): UNICODE

I think the final result can be "UTF8" because UTF-8 includes ASCII.

[hiros0921]

I reproduced this on 2.2.0 and traced it to isUTF8().

Appending a single control character to the same UTF-8 string:

appended byte	detect()
0x08 BS	UNICODE
0x0B VT	UNICODE
0x0C FF	UNICODE
0x1B ESC	UNICODE
0x7F DEL	UNICODE
0x09 TAB, 0x0A LF, 0x0D CR	UTF8
0x00 NUL, 0x07 BEL	BINARY

isUTF8() accepts only TAB, LF, CR and printable ASCII:

https://github.com/polygonplanet/encoding.js/blob/master/src/encoding-detect.js#L204-L207

isASCII(), on the other hand, accepts every byte in the ASCII range except ESC. So the same byte is reported as ASCII but not as UTF-8, even though every ASCII byte is valid UTF-8 by definition — which is why detect(<BS>) is ASCII while detect(<UTF-8 text> + <BS>) falls through to UNICODE.

Opened #64 with a fix and a test. It accepts the whole ASCII range except ESC, keeping ESC out to match isASCII(). detect() does not get looser, because isBINARY and isJIS are both tried before UTF8 in Encoding.orders.