We currently only escape C0 control characters, but we also need to escape C1 ones (0x80-0x9f; see Wikipedia).
The relevant code is here:
|
while chunk_off < chunk.len() |
|
&& (chunk[chunk_off] < 0x20 || chunk[chunk_off] == 0x7f) |
|
{ |
|
let ch = chunk[chunk_off]; |
|
chunk_off += 1; |
|
|
|
if ch == b'\t' { |
|
cursor_tab = self.cursor_move_to_offset_internal( |
|
cursor_tab, |
|
global_off + chunk_off - 1, |
|
); |
|
let tab_size = self.tab_size - (cursor_tab.column % self.tab_size); |
|
line.push_str(&TAB_WHITESPACE[..tab_size as usize]); |
|
|
|
// Since we know that we just aligned ourselves to the next tab stop, |
|
// we can trivially process any successive tabs. |
|
while chunk_off < chunk.len() && chunk[chunk_off] == b'\t' { |
|
line.push_str(&TAB_WHITESPACE[..self.tab_size as usize]); |
|
chunk_off += 1; |
|
} |
|
continue; |
|
} |
|
|
|
visualizer_buf[2] = if ch == 0x7F { |
|
0xA1 // U+2421 |
|
} else { |
|
0x80 | ch // 0x00..=0x1F => U+2400..=U+241F |
|
}; |
|
// Our manually constructed UTF8 is never going to be invalid. Trust. |
|
line.push_str(unsafe { str::from_utf8_unchecked(&visualizer_buf) }); |
|
} |
The primary issue with fixing this is that currently the code abuses the fact that C0 characters are ASCII and so it can trivially scan through the byte slices. C1 control characters however use a double-byte encoding under UTF8. One option to fix this holistically would be to move this logic into the Framebuffer code. In either case however, this requires us to use Utf8Chars to iterate through it, find and replace them.