Add proper grapheme cluster support for modern terminal UTF-8 handling #1583
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit addresses critical UTF-8 display and input issues, particularly with emoji variations and complex Unicode sequences that caused rendering problems in modern terminals like Ghostty. Key improvements: - Enhanced utf8.c with grapheme cluster detection using utf8proc - Fixed input field handling to preserve emoji variation selectors - Improved text measurement and cursor positioning for complex Unicode - Added smart paste processing for multi-codepoint characters - Bypassed problematic TRANSLIT when not needed for UTF-8 content - Enhanced GUI entry and readline to handle grapheme clusters properly The implementation maintains full backward compatibility while providing proper support for modern Unicode standards, fixing display corruption that occurred with emoji sequences containing variation selectors. Tested with emoji sequences like: 💕💋😘🥰💞❣️💓♥️ ♥️ 💓❣️🥰🥰😘💋💋
Addresses issue where emoji with variation selectors (like ❣️,♥️ ) were incorrectly calculated as width 1 instead of width 2, causing display overflow in modern terminals. - Add variation selector detection (0xFE0F) in string_advance_with_grapheme_support() - Apply special width handling for base emoji + variation selector combinations - Ensures consistent width calculation with input field processing - Fixes chat window overflow issues in terminals like Ghostty This brings string_advance logic in line with unichar_array_advance_cluster which already had proper variation selector handling.
dwfreed
requested changes
Sep 30, 2025
Relocate is_combining_char() from gui-entry.c to src/core/utf8.c as suggested in code review. This better organizes the codebase by placing UTF-8 character classification logic with other UTF-8 utilities in the core module. The function now uses is_utf8() instead of checking term_type directly, which is the appropriate method for core layer code. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR addresses critical UTF-8 display and input issues that occur in modern terminals like Ghostty when handling emoji with variation selectors and other complex Unicode sequences.
The problem manifests as display corruption where emoji sequences like
💕💋😘🥰💞❣️💓♥️♥️💓❣️🥰🥰😘💋💋cause text positioning issues and garbled output in sidepanels and input fields.Key Changes
Technical Details
src/core/utf8.c: New grapheme cluster advancement functions with utf8proc integrationsrc/fe-text/gui-entry.c: Fixed cursor positioning and text measurement for complex Unicodesrc/fe-text/gui-readline.c: Enhanced paste processing to handle multi-codepoint sequencessrc/core/recode.c: Bypass TRANSLIT when both input and output are valid UTF-8Test Case
The implementation was tested with the emoji sequence that originally caused issues:
💕💋😘🥰💞❣️💓♥️♥️💓❣️🥰🥰😘💋💋Compatibility