Thanks to visit codestin.com
Credit goes to github.com

Skip to content

[Web][Engine] Fix composingBaseOffset and composingExtentOffset value when input japanease text #161593

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 34 commits into
base: master
Choose a base branch
from

Conversation

koji-1009
Copy link
Contributor

@koji-1009 koji-1009 commented Jan 14, 2025

fix #159671

When entering Japanese text and operating shift + ← || → || ↑ || ↓ while composing a character, setSelectionRange set (0,0) and the composing text is disappeared. For this reason, disable shit + arrow text shortcuts on web platform.

Movie

fixed

fixed.mov

master branch

master.mov

Pre-launch Checklist

If you need help, consider asking for advice on the #hackers-new channel on Discord.

@github-actions github-actions bot added a: text input Entering text in a text field or keyboard related problems engine flutter/engine repository. See also e: labels. platform-web Web applications specifically labels Jan 14, 2025
@koji-1009 koji-1009 marked this pull request as ready for review January 14, 2025 23:10
@koji-1009 koji-1009 changed the title fix: Skip updating selection range during composition [Web][Engine] Skip updating selection range during composition Jan 14, 2025
@flutter-dashboard
Copy link

It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption, contact "@test-exemption-reviewer" in the #hackers channel in Discord (don't just cc them here, they won't see it!).

If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?

Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing. The test exemption team is a small volunteer group, so all reviewers should feel empowered to ask for tests, without delegating that responsibility entirely to the test exemption group.

@koji-1009 koji-1009 changed the title [Web][Engine] Skip updating selection range during composition [Engine][Web] Skip updating selection range during composition Jan 16, 2025
@justinmc justinmc requested a review from mdebbar February 13, 2025 21:50
Copy link
Contributor

@mdebbar mdebbar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for contributing a fix to this issue!

The test should go into this file: engine/src/flutter/lib/web_ui/test/engine/text_editing_test.dart. This file has many tests that do various things that you can copy from.

Here's how the test could look like:

  1. Create a text field using the showKeyboard helper.
  2. Set composingText to a non-null value.
  3. Send a TextInput.setEditingState.
  4. Expect the selection change hasn't been applied using checkInputEditingState.

These are some tests that may be relevant / similar to what you'll be writing:

I hope this helps. Let me know if you have any more questions.

@koji-1009 koji-1009 force-pushed the fix/web_composing_range branch from 5e8a061 to 04f2777 Compare March 21, 2025 12:32
}),
),
);
checkInputEditingState(textEditing!.strategy.domElement, 'へんかん', 4, 4);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In all these checks, the numbers match what was sent in selectionBase and selectionExtent. So it's not achieving the goal of the test which is to verify that selection wasn't updated on the DOM element.

To test this, you want to send some random numbers in selectionBase and selectionExtent, then check that the DOM element's selection hasn't changed to those random numbers.

Comment on lines 1960 to 1961
'composingBase': 0,
'composingExtent': 4,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does Flutter always send composingBase and composingExtent when the user is composing text? If yes, then we don't really need the new isComposing boolean, right? We can simply check for the presence of composingBase and/or composingExtent inside EditingState.applyToDomElement.

@koji-1009 koji-1009 force-pushed the fix/web_composing_range branch from 4aa715d to f5a2456 Compare March 22, 2025 00:01
@koji-1009
Copy link
Contributor Author

Sorry. I misunderstood that I needed to simulate the movement of Japanese IMEs. A test has been added to confirm this added process. Thank you for your support.

@koji-1009 koji-1009 requested a review from mdebbar March 22, 2025 05:20
Copy link
Contributor

@mdebbar mdebbar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks for contributing the fix!

Copy link
Contributor

@Renzo-Olivares Renzo-Olivares left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your contribution and patience @koji-1009. I'm not sure if the selection being updated while composing is the root cause of the issue here.

When logging myself I see a few things,

  1. Sometimes the composing range cannot be determined in cases where the selection extent becomes smaller than the composing text extent.
  2. The framework is sending an update to the engine when we press shift + arrow key right/left.

I think 2 is important here because we want all text editing shortcuts to be handled natively on the web, so the framework shouldn't be sending this update when shift + arrow key right/left is pressed. I think 1 is less important since it doesn't interfere with the native webs composition, since composing range does not exist in native web and is not applied to the dom element, right now it is only used by the framework to draw the composing underline (should be a separate issue).

I think the fix for this should be in the framework, we can add the shift + arrow left/right to the web disabling shortcuts here

static final Map<ShortcutActivator, Intent> _webDisablingTextShortcuts =
.

  const SingleActivator(LogicalKeyboardKey.arrowLeft, shift: true):
      const DoNothingAndStopPropagationTextIntent(),
  const SingleActivator(LogicalKeyboardKey.arrowRight, shift: true):
      const DoNothingAndStopPropagationTextIntent(),

I tried this out and it seems to match the behavior I am observing in html's native TextArea.

Copy link
Contributor Author

@koji-1009 koji-1009 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Renzo-Olivares
Thanks also for clarifying the root cause!

Because I hadn't noticed _webDisablingTextShortcuts, I thought it was correct not to call setSelectionRange while converting with the (Japanese) IME. It certainly seems to be a more suitable modification for flutter web to deal with on the framework side.

Adding 4 patterns, left and right, up and down, seems to solve the problem. Would the fix be to add a commit to this PR, or would it be better to open another PR?


Thank you so much for your support. I really appreciate it.

@Renzo-Olivares
Copy link
Contributor

@koji-1009, Doing it in this PR sounds good to me.

@koji-1009 koji-1009 force-pushed the fix/web_composing_range branch from 9e5212e to 532a258 Compare April 17, 2025 00:58
@github-actions github-actions bot added engine flutter/engine repository. See also e: labels. platform-web Web applications specifically labels Apr 17, 2025
@koji-1009 koji-1009 force-pushed the fix/web_composing_range branch from 89add93 to 24e8493 Compare April 18, 2025 11:36
@@ -81,7 +81,12 @@ mixin CompositionAwareMixin {
final int composingBase = editingState.extentOffset! - composingText!.length;

if (composingBase < 0) {
Copy link
Contributor Author

@koji-1009 koji-1009 Apr 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When we type Japanese, we may enter this if statement. For example, if you type “こんにちは” and then use shift + ← or → to change the range, we get the following log output.

code

  EditingState determineCompositionState(EditingState editingState) {
    print(
      'editingState = $editingState, composingText = $composingText, composingText length = ${composingText?.length}',
    );

log

editingState = EditingState("", base:0, extent:0, composingBase:-1, composingExtent:-1), composingText = null, composingText length = null
editingState = EditingState("k", base:1, extent:1, composingBase:-1, composingExtent:-1), composingText = k, composingText length = 1
editingState = EditingState("k", base:1, extent:1, composingBase:-1, composingExtent:-1), composingText = k, composingText length = 1
editingState = EditingState("こ", base:1, extent:1, composingBase:-1, composingExtent:-1), composingText = こ, composingText length = 1
editingState = EditingState("こ", base:1, extent:1, composingBase:-1, composingExtent:-1), composingText = こ, composingText length = 1
editingState = EditingState("こn", base:2, extent:2, composingBase:-1, composingExtent:-1), composingText = こn, composingText length = 2
editingState = EditingState("こn", base:2, extent:2, composingBase:-1, composingExtent:-1), composingText = こn, composingText length = 2
editingState = EditingState("こん", base:2, extent:2, composingBase:-1, composingExtent:-1), composingText = こん, composingText length = 2
editingState = EditingState("こん", base:2, extent:2, composingBase:-1, composingExtent:-1), composingText = こん, composingText length = 2
editingState = EditingState("こんn", base:3, extent:3, composingBase:-1, composingExtent:-1), composingText = こんn, composingText length = 3
editingState = EditingState("こんn", base:3, extent:3, composingBase:-1, composingExtent:-1), composingText = こんn, composingText length = 3
editingState = EditingState("こんに", base:3, extent:3, composingBase:-1, composingExtent:-1), composingText = こんに, composingText length = 3
editingState = EditingState("こんに", base:3, extent:3, composingBase:-1, composingExtent:-1), composingText = こんに, composingText length = 3
editingState = EditingState("こんにt", base:4, extent:4, composingBase:-1, composingExtent:-1), composingText = こんにt, composingText length = 4
editingState = EditingState("こんにt", base:4, extent:4, composingBase:-1, composingExtent:-1), composingText = こんにt, composingText length = 4
editingState = EditingState("こんにち", base:4, extent:4, composingBase:-1, composingExtent:-1), composingText = こんにち, composingText length = 4
editingState = EditingState("こんにち", base:4, extent:4, composingBase:-1, composingExtent:-1), composingText = こんにち, composingText length = 4
editingState = EditingState("こんにちh", base:5, extent:5, composingBase:-1, composingExtent:-1), composingText = こんにちh, composingText length = 5
editingState = EditingState("こんにちh", base:5, extent:5, composingBase:-1, composingExtent:-1), composingText = こんにちh, composingText length = 5
editingState = EditingState("こんにちは", base:5, extent:5, composingBase:-1, composingExtent:-1), composingText = こんにちは, composingText length = 5
editingState = EditingState("こんにちは", base:5, extent:5, composingBase:-1, composingExtent:-1), composingText = こんにちは, composingText length = 5
editingState = EditingState("こんにちは", base:0, extent:5, composingBase:-1, composingExtent:-1), composingText = こんにちは, composingText length = 5
editingState = EditingState("こんにちは", base:0, extent:5, composingBase:-1, composingExtent:-1), composingText = こんにちは, composingText length = 5
editingState = EditingState("今日は", base:0, extent:2, composingBase:-1, composingExtent:-1), composingText = 今日は, composingText length = 3
editingState = EditingState("今日は", base:0, extent:2, composingBase:-1, composingExtent:-1), composingText = 今日は, composingText length = 3
editingState = EditingState("こんに血は", base:0, extent:3, composingBase:-1, composingExtent:-1), composingText = こんに血は, composingText length = 5
editingState = EditingState("こんに血は", base:0, extent:3, composingBase:-1, composingExtent:-1), composingText = こんに血は, composingText length = 5
editingState = EditingState("今日は", base:0, extent:2, composingBase:-1, composingExtent:-1), composingText = 今日は, composingText length = 3
editingState = EditingState("今日は", base:0, extent:2, composingBase:-1, composingExtent:-1), composingText = 今日は, composingText length = 3
editingState = EditingState("こんにちは", base:0, extent:5, composingBase:-1, composingExtent:-1), composingText = こんにちは, composingText length = 5
editingState = EditingState("こんにちは", base:0, extent:5, composingBase:-1, composingExtent:-1), composingText = こんにちは, composingText length = 5
editingState = EditingState("こんにちは", base:5, extent:5, composingBase:-1, composingExtent:-1), composingText = こんにちは, composingText length = 5
editingState = EditingState("こんにちは", base:5, extent:5, composingBase:-1, composingExtent:-1), composingText = こんにちは, composingText length = 5
editingState = EditingState("こんにちは", base:5, extent:5, composingBase:-1, composingExtent:-1), composingText = こんにちは, composingText length = 5
editingState = EditingState("こんにちは", base:5, extent:5, composingBase:-1, composingExtent:-1), composingText = null, composingText length = null

If composingText is non-null, text is in composing. However, the previous process could not determine that composing is in progress because composingBase and composingExtent are -1.

This change will allow the framework to be notified that composing is in progress.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should try printing this log after this line

newEditingState = determineCompositionState(newEditingState);

printing at the top of determineCompositionState would mean the composition range has not been determined yet.

Copy link
Contributor Author

@koji-1009 koji-1009 Apr 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added print code, and test it.

    EditingState newEditingState = EditingState.fromDomElement(activeDomElement);
    newEditingState = determineCompositionState(newEditingState);
    print('newEditingState: $newEditingState');

Input "こんにちは" and compose it , then press shit + ←.

main branch

newEditingState: EditingState("", base:0, extent:0, composingBase:-1, composingExtent:-1)
newEditingState: EditingState("k", base:1, extent:1, composingBase:0, composingExtent:1)
newEditingState: EditingState("k", base:1, extent:1, composingBase:0, composingExtent:1)
newEditingState: EditingState("こ", base:1, extent:1, composingBase:0, composingExtent:1)
newEditingState: EditingState("こ", base:1, extent:1, composingBase:0, composingExtent:1)
newEditingState: EditingState("こn", base:2, extent:2, composingBase:0, composingExtent:2)
newEditingState: EditingState("こn", base:2, extent:2, composingBase:0, composingExtent:2)
newEditingState: EditingState("こん", base:2, extent:2, composingBase:0, composingExtent:2)
newEditingState: EditingState("こん", base:2, extent:2, composingBase:0, composingExtent:2)
newEditingState: EditingState("こんn", base:3, extent:3, composingBase:0, composingExtent:3)
newEditingState: EditingState("こんn", base:3, extent:3, composingBase:0, composingExtent:3)
newEditingState: EditingState("こんに", base:3, extent:3, composingBase:0, composingExtent:3)
newEditingState: EditingState("こんに", base:3, extent:3, composingBase:0, composingExtent:3)
newEditingState: EditingState("こんにt", base:4, extent:4, composingBase:0, composingExtent:4)
newEditingState: EditingState("こんにt", base:4, extent:4, composingBase:0, composingExtent:4)
newEditingState: EditingState("こんにち", base:4, extent:4, composingBase:0, composingExtent:4)
newEditingState: EditingState("こんにち", base:4, extent:4, composingBase:0, composingExtent:4)
newEditingState: EditingState("こんにちh", base:5, extent:5, composingBase:0, composingExtent:5)
newEditingState: EditingState("こんにちh", base:5, extent:5, composingBase:0, composingExtent:5)
newEditingState: EditingState("こんにちは", base:5, extent:5, composingBase:0, composingExtent:5)
newEditingState: EditingState("こんにちは", base:5, extent:5, composingBase:0, composingExtent:5)
newEditingState: EditingState("今日は", base:0, extent:3, composingBase:0, composingExtent:3)
newEditingState: EditingState("今日は", base:0, extent:3, composingBase:0, composingExtent:3)
newEditingState: EditingState("今日は", base:0, extent:2, composingBase:-1, composingExtent:-1)
newEditingState: EditingState("今日は", base:0, extent:2, composingBase:-1, composingExtent:-1)
newEditingState: EditingState("こんに血は", base:0, extent:3, composingBase:-1, composingExtent:-1)
newEditingState: EditingState("こんに血は", base:0, extent:3, composingBase:-1, composingExtent:-1)
newEditingState: EditingState("", base:0, extent:0, composingBase:0, composingExtent:0)
newEditingState: EditingState("", base:0, extent:0, composingBase:-1, composingExtent:-1)

We can infer from the logs and logic that the problem is caused by cases where the composing text is longer than the extent.

@@ -904,10 +902,10 @@ class EditingState {
final String? text;

Copy link
Contributor Author

@koji-1009 koji-1009 Apr 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

baseOffset and extentOffset are never set to null.

  EditingState({
    this.text,
    int? baseOffset,
    int? extentOffset,
    this.composingBaseOffset = -1,
    this.composingExtentOffset = -1,
  }) : // Don't allow negative numbers.
       baseOffset = math.max(0, baseOffset ?? 0),
       // Don't allow negative numbers.
       extentOffset = math.max(0, extentOffset ?? 0);

@koji-1009 koji-1009 changed the title [Web] Disable shift + arrow text shortcuts [Web][Engine] Fix composingBaseOffset and composingExtentOffset value when input japanease text Apr 18, 2025
@koji-1009 koji-1009 marked this pull request as ready for review April 24, 2025 01:06
// The length of the input string is set to the length of the composing string.
// This is a workaround for the case where the japanese IME is used.
return editingState.copyWith(
composingBaseOffset: editingState.baseOffset,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this accurate when composing? I would expect the editing.baseOffset and editing.extentOffset to be collapsed while composing text, and only becomes uncollapsed when pressing shift + <-/->.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#161593 (comment)
The log output of Japanese conversions, for example, will have the following.

editingState = EditingState("こんにちは", base:0, extent:5, composingBase:-1, composingExtent:-1), composingText = こんにちは, composingText length = 5
editingState = EditingState("今日は", base:0, extent:2, composingBase:-1, composingExtent:-1), composingText = 今日は, composingText length = 3
editingState = EditingState("今日は", base:0, extent:2, composingBase:-1, composingExtent:-1), composingText = 今日は, composingText length = 3
editingState = EditingState("こんに血は", base:0, extent:3, composingBase:-1, composingExtent:-1), composingText = こんに血は, composingText length = 5
editingState = EditingState("こんに血は", base:0, extent:3, composingBase:-1, composingExtent:-1), composingText = こんに血は, composingText length = 5

It seems that the “number of conversion blocks” rather than the “string length” is reflected in the extent during Japanese conversion, thinking from the log. From “こんにちは”, shift + ← becomes “今日は”. This is two blocks of “今日” and “は”. Then shift + ←, becoms "こんに血は". This is three blocks of "こんに" and "血" and "は".

In the current implementation, if we “update the conversion range” during composing, it can no longer be determined that composing is in progress. As a result, setSelectRange is updated with (0,0) and the character being input cleared.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replied in #161593 (comment), this will get us more accurate logs.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry. There is an incorrect comment above.

If we press shift + arrow key, the area of the character to be converted is changed. For example, if we composing “今日は”, base=0 extent=2 when ‘今日’ is selected, then pressing arrow right selects “は” and base=2 extent=3.
The current implementation does not work as expected when there is a 3-character composing text “今日は” and the “今日” part is selected.

baseOffset: 0,
extentOffset: 4,
extentOffset: 2,
composingBaseOffset: 0,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the composing range here should actually be (0,3) since it should cover the full length of the composingText.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
a: text input Entering text in a text field or keyboard related problems engine flutter/engine repository. See also e: labels. framework flutter/packages/flutter repository. See also f: labels. platform-web Web applications specifically
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Web] Composing Japanese letters disappear when shifting converting area
3 participants