Read the source and its declared rule
This source splits valid UTF-8 for 你 between chunks [E4, BD] and [A0]. A fresh decoder per chunk mistakes the fragments for separate inputs. The supplied import must carry decoder state and reject any unfinished final character.
| Record | Input JSON |
|---|---|
| Base input | {"chunks":[[228,189],[160]]} |
Compare the actual paths
The streaming path yields one 你. Decode with stream=true for each part, then flush; that final check exposes an incomplete last part.
| Path | Output JSON |
|---|---|
| Misleading path | {"accepted":true,"text":"\uFFFD\uFFFD"} |
| Corrected path | {"accepted":true,"text":"你"} |
import {DatabaseSync} from 'node:sqlite';
import {TextDecoder} from 'node:util';
import {inflateSync} from 'node:zlib';
import {posix} from 'node:path';
import {createHash} from 'node:crypto';
const rows=(db,sql)=>db.prepare(sql).all().map(r=>({...r}));
const failed=(action)=>{try{action();return null;}catch(e){return {code:e.errcode??e.code??e.name};}};
import assert from 'node:assert/strict';
const run=function run(i,corrected=true){
try{
if(!corrected)return {accepted:true,text:i.chunks.map(b=>new TextDecoder().decode(Uint8Array.from(b))).join('')};
const decoder=new TextDecoder('utf-8',{fatal:true});let text='';
for(const chunk of i.chunks)text+=decoder.decode(Uint8Array.from(chunk),{stream:true});
return {accepted:true,text:text+decoder.decode()};
}catch{return {accepted:false,error:'incomplete-or-invalid-utf8'};}
};
const cases=[{"label":"misleading base","input":{"chunks":[[228,189],[160]]},"expected":{"accepted":true,"text":"\uFFFD\uFFFD"},"corrected":false},{"label":"corrected base","input":{"chunks":[[228,189],[160]]},"expected":{"accepted":true,"text":"你"},"corrected":true},{"label":"one complete chunk","input":{"chunks":[[228,189,160]]},"expected":{"accepted":true,"text":"你"},"corrected":true},{"label":"incomplete final chunk","input":{"chunks":[[228,189]]},"expected":{"accepted":false,"error":"incomplete-or-invalid-utf8"},"corrected":true}];
for(const test of cases){
const actual=run(test.input,test.corrected);
assert.deepEqual(actual,test.expected,test.label);
console.log(test.label,actual);
}
Check boundaries before the next load
One complete chunk agrees with the streamed result. A final incomplete chunk is rejected rather than displayed as a repaired character.
| Case | Input JSON | Expected output JSON |
|---|---|---|
| One complete UTF-8 chunk | {"chunks":[[228,189,160]]} | {"accepted":true,"text":"你"} |
| Unfinished final character | {"chunks":[[228,189]]} | {"accepted":false,"error":"incomplete-or-invalid-utf8"} |
Is the split itself proof that the source encoding is invalid?
Show the worked answer
No. The combined bytes are valid and decode to 你. Loss occurs when decoder state is discarded between valid fragments.