…scii_line

Implement logic to increment column count in WidthMode::Characters, emitting output when width is reached. This ensures accurate line folding for multi-byte characters, enhancing Unicode support.

- Added conditional check in fold_file function to call emit_output when col_count >= width
- Ensures lines are properly wrapped based on byte or character width before final output flush
- Improves handling of incomplete lines that need early breaking to respect the specified width

In character width mode, emit output immediately after segments are added if column count exceeds width, preventing redundant flushes. Simplify the file folding logic by removing unnecessary conditional checks at the end, ensuring clean output writing. This fixes potential issues with extra line breaks or incorrect folding behavior.

…ory usage

Introduce a STREAMING_FLUSH_THRESHOLD constant and helper functions (maybe_flush_unbroken_output, push_byte, push_bytes) to periodically flush the output buffer when it exceeds 8KB and no spaces are being tracked, preventing excessive memory consumption when processing large files. This refactor replaces direct buffer pushes with checks for threshold-based flushing.

…d tests

Remove conditional checks that incorrectly emitted output when column count reached width in character mode, ensuring proper folding of wide characters and handling of edge cases. Add comprehensive tests for wide characters, invalid UTF-8, zero-width spaces, and buffer boundaries to verify correct behavior. This prevents issues with multi-byte character folding where output was prematurely flushed, improving accuracy for Unicode input.

Modify the fold implementation to process input in buffered chunks rather than line-by-line reading, ensuring correct handling of multi-byte characters split across buffer boundaries. Add process_pending_chunk function and new streaming logic to fold_file for better performance on large files. Update tests accordingly.

… mode

Only coalesce zero-width combining characters into base characters when folding by display columns (WidthMode::Columns). In character-counting mode, treat every scalar value as advancing the counter to match chars().count() semantics, preventing incorrect line breaking for characters with zero-width marks. This ensures consistent behavior across modes as verified by existing tests.

…vent unbounded buffering

Add explanatory comments to constants and functions in fold.rs, detailing the 8 KiB threshold for flushing in streaming mode to avoid unbounded buffer growth, and clarifying line folding behavior with the -s option. Improves code readability without altering functionality.

…nk handling

- Extract UTF-8 character processing into a new `process_utf8_chars` function for better code organization
- Add documentation to `process_pending_chunk` explaining its behavior with buffered bytes and invalid UTF-8
- Modify `process_pending_chunk` to properly handle the result of `process_non_utf8_line`, ensuring errors are propagated after clearing the buffer

Updated push_byte to only append bytes to the buffer without triggering flush checks, removing the call to maybe_flush_unbroken_output and adjusting the comment to reflect the new behavior. This change simplifies the function's logic, potentially improving performance by deferring flushes to other parts of the code.