[macOS] Remove isComposing workaround #33838

knopp · 2022-06-05T18:30:51Z

If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.

Pre-launch Checklist

I read the Contributor Guide and followed the process outlined there for submitting PRs.
I read the Tree Hygiene wiki page, which explains my responsibilities.
I read and followed the Flutter Style Guide and the C++, Objective-C, Java style guides.
I listed at least one issue that this PR fixes in the description above.
I added new tests to check the change I am making or feature I am adding, or Hixie said the PR is test-exempt. See testing the engine for instructions on
writing and running engine tests.
I updated/added relevant documentation (doc comments with ///).
I signed the CLA.
All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

justinmc

This looks straightforward to me. Here's my framework PR review: flutter/flutter#105407 (review)

If we're going to move forward with this we should test that kPerformIntent is invoked.

chinmaygarde · 2022-06-09T20:19:58Z

cc @justinmc Are you able to review this?

LongCatIsLooong · 2022-06-21T19:18:44Z

shell/platform/darwin/macos/framework/Source/FlutterTextInputPlugin.mm

@@ -620,6 +634,28 @@ - (void)doCommandBySelector:(SEL)selector {
    void (*func)(id, SEL, id) = reinterpret_cast<void (*)(id, SEL, id)>(imp);
    func(self, selector, nil);
  }
+


If the selector is implemented by FlutterTextInputPlugin I think we don't want to forward the selector to the framework? Otherwise the same selector could be handled twice?

All selectors are implemented by NSTextView (they just are mostly empty implementation). However this means that [self respondsToSelector:selector] always returns true.

We only seem to be handling insertNewline: in text input plugin (I'll double check) translating it into text, so perhaps we could just not forward this one.

We could also check with objc runtime to see whether particular selector is overridden, which seems a bit of overkill, but that will need special exception for insertTab:

There's a noop: selector (or maybe noop) when a shortcut isn't mapped to anything.

LongCatIsLooong · 2022-06-21T21:50:49Z

shell/platform/darwin/macos/framework/Source/FlutterTextInputPlugin.mm

+                                ? (__bridge CFStringRef)self.customRunLoopMode
+                                : kCFRunLoopCommonModes;
+
+  CFRunLoopPerformBlock(CFRunLoopGetMain(), runLoopMode, ^{


If I understand this correctly, this is trying to ensure when a shortcut/key equivalent is mapped to a sequence of selectors (e.g., Alt+↑ => moveBackward: then moveToBeginningOfParagraph:), these selectors are sent to the framework in the same channel message as a list?

Will this cause some events to reorder, for example, if a key is bound to 3 selectors [moveBackward, insert: a, moveForward], since insert:a will be handled by the input plugin, this essentially becomes [insert: a, moveBackward, moveForward], from the framework's perspective?

Maybe use dispatch_async instead of CFRunLoopPerformBlock? Then you don't have to set the mode?

nit: maybe don't schedule the block when it's already scheduled (e.g., selectors.length > 1)?

Maybe use dispatch_async instead of CFRunLoopPerformBlock? Then you don't have to set the mode?

We can't pump the dispatch queue manually in unit tests. We can however run the run loop in custom mode so that only these blocks are executed.

nit: maybe don't schedule the block when it's already scheduled (e.g., selectors.length > 1)?

Done.

Will this cause some events to reorder, for example, if a key is bound to 3 selectors [moveBackward, insert: a, moveForward], since insert:a will be handled by the input plugin, this essentially becomes [insert: a, moveBackward, moveForward], from the framework's perspective?

So it will indeed become that. However even if we flush buffered selectors and it will become [moveBackward, insert:a, moveForward] it still won't give expected result, because when TextInputPlugin processes insert:a the selection hasn't been updated by the framework yet so that first [moveBackward:] will only update selection after text is already inserted.

I don't think we can handle compound actions that involve inserting text correctly at this point. It may required some rather significant changes to get this working.

Something still needs to pump the main dispatch queue. I don't know any other way to do this except running the CFRunLoop in common modes. However running the CFRunLoop in default mode in tests crashes because of various scheduled NSView related tasks from other tests.

Dispatch barrier is unlikely to help here. Doing anything blocking on main thread (dispatch_sync, dispatch_barrier_sync, dispatch_barrier_async_and_wait) will crash with "LIBDISPATCH: dispatch_sync called on queue already owned by current thread". And async barrier will do not anything useful (it still needs the queue to be pumped).

This test passes (didn't need a barrier since the main queue is not concurrent):

func testExample() throws { var x = 0 let expectation = XCTestExpectation() DispatchQueue.main.async { print("execute") x += 1 } DispatchQueue.main.async { XCTAssertEqual(x, 1) expectation.fulfill() } print("wait") wait(for: [expectation], timeout: 10) }

xctwaiter.wait calls _dispatch_main_queue_drain(), so the main queue is pumped in a sense.

_dispatch_main_queue_drain() doesn't seem to be exported from libdispatch. Looking at XCTest it seems that waitFor actually runs the run loop, which would have same sideeffect as calling CFRunLoopRunInModewith default modes manually, as in it would crash flutter_desktop_darwin_unittests. Now why this crash happens is another issue (perhaps we could drain the run loop in each test while the appropriate cocoa objects still exist), but that should be tracked as another issue.

Created flutter/flutter#106509.

shell/platform/darwin/macos/framework/Source/FlutterTextInputPlugin.mm

justinmc

Just jumping in with a few nits here as I read through this PR.

shell/platform/darwin/macos/framework/Source/FlutterKeyboardManagerUnittests.mm

shell/platform/darwin/macos/framework/Source/FlutterTextInputPlugin.mm

cbracken · 2022-06-22T22:59:21Z

shell/platform/darwin/macos/framework/Source/FlutterTextInputPlugin.mm

@@ -568,6 +577,11 @@ - (NSTextInputContext*)inputContext {
 #pragma mark -
 #pragma mark NSTextInputClient

+- (void)insertTab:(id)sender {
+  // Implementing insertTab: makes AppKit send tab as command, instead of
+  // insertText with '\t'.


/cc @loic-sharma who's looking at tab handling on Windows.

Since flutter text doesn't define tab shortcut and thus it's always handled as focus, when I wanted to insert tab I simply used this Action

class InsertTabAction extends Action<NextFocusIntent> { final TextEditingController controller; InsertTabAction(this.controller); void _sendIntent(Intent intent) { final BuildContext? primaryContext = primaryFocus?.context; if (primaryContext != null) { Actions.invoke(primaryContext, intent); } } @override Object? invoke(NextFocusIntent intent) { _sendIntent(ReplaceTextIntent(controller.value, '', controller.selection, SelectionChangedCause.keyboard)); _sendIntent(ReplaceTextIntent(controller.value, '\t', controller.selection, SelectionChangedCause.keyboard)); return null; } }

and bound it to NextFocusIntent above text editor. It's not the most obvious solution as it requires binding to NextFocusIntent instead of using Shortcut and custom intent, but it works even with Mac IME input after this PR.

/cc @loic-sharma

Can you illustrate why using SingleActivator(LogicalKeyboardKey.tab) and some InsertTabIntent in a Shortcuts above the text editor wouldn't work?

Can you illustrate why using SingleActivator(LogicalKeyboardKey.tab) and some InsertTabIntent in a Shortcuts above the text editor wouldn't work?

It wouldn't. That's why I said it requires binding to NextFocusIntent. Tab has to be unconditionally passed to the IME (because we have no way of knowing whether the candidates popup is shown), and when we receive insertTab: from TextInputContext we send NextFocusIntent to current focus node. This is not ideal, but I don't think there's any way currently to synthesize key event and activate shortcuts, is there?

shell/platform/darwin/macos/framework/Source/FlutterTextInputPlugin.mm

LongCatIsLooong · 2022-06-23T18:20:52Z

LGTM modulo filtering out selectors already handled by the NSTextView implementation (e.g., insertNewline:)

cbracken

lgtm and +1 on a small comment to explain the setMarkedText behaviour; the Apple docs lack a lot of detail.

cbracken · 2022-06-30T07:15:35Z

@knopp is this good to land?

dkwingsmt · 2022-06-30T07:16:36Z

Sorry I'm still taking a look, trying to understand the data flow after the change.

knopp · 2022-06-30T21:29:55Z

@knopp is this good to land?

I'd prefer landing #34250 first and then rebasing this one.

Sorry I'm still taking a look, trying to understand the data flow after the change.

As far as engine is concerned the only change is that editing events are not forwarded to TextInputContext directly when compositing. The framework needs to make sure that events possibly related to compositing (mostly arrows keys/tab/enter/esc) are unhandled. Engine then translates this to selectors and in framework the selectors are translated to intents and sent to active focus node.

dkwingsmt · 2022-07-01T02:25:00Z

After talking with @LongCatIsLooong I think I understand the flow better. Let me phrase it and see if it's correct.

After this and flutter/flutter#105407, certain control keys (tab, arrows, etc.) will by rejected by the framework when a text input is being focused (formally, these keys are mapped to DoNothingAndStopPropagationTextIntent).

These events are sent back to the AppKit platform. Now, I assume, is when the platform gets to dispatch them to the IME (emoji picker.)

If the emoji picker doesn't handle them, the platform dispatches them to doCommandBySelector as actions represented as strings. (For this to work FlutterTextInputPlugin needs to implement insertTab with an empty body.)

These string-actions are sent to Framework's TextInput through a new channel, whose callback converts these messages into intents.

Since these string-actions now always dispatch fixed intents, for example, a Tab key always dispatches NextFocusIntent, the developer can change their behavior by overriding the intent-to-action map.

If so, I have a few questions:

First, the framework change only rejects control keys without any modifiers. For example, it rejects ArrowLeft, but not Alt-ArrowLeft, but Alt-ArrowLeft is still removed from _macShortcuts. It seems to me that this should not work. If they do work after the change, does mean these extra default text actions (such as for Alt-ArrowLeft) were not needed?

Second, while I'm still concerned with forcing the developer with override a NextFocusIntent with an InsertTabAction, I can't think of a better solution. However, this requires much more education, since overriding stuff is more confusing. We need a way to tell the developers to override the intent-to-action map, instead of the activator-to-intent map (which is understandably more intuitive), and the breakage otherwise will only be noticed in the corner case of emoji picker. Moreover, they will have to remember the enumerated intents to override, which basically means they're associating NextFocusIntent with pressing a Tab key.

I'm wondering if it will be a better choice to create a MacosSelectorIntent(String content) and map them to the actions for macOS. This also makes it look less weird to map to an InsertTabAction. What do you think?

And also, I definitely prefer documenting the data flow above somewhere in the engine.

knopp · 2022-07-05T14:11:18Z

If so, I have a few questions:

First, the framework change only rejects control keys without any modifiers. For example, it rejects ArrowLeft, but not Alt-ArrowLeft, but Alt-ArrowLeft is still removed from _macShortcuts. It seems to me that this should not work. If they do work after the change, does mean these extra default text actions (such as for Alt-ArrowLeft) were not needed?

It rejects ArrowLeft, because it is mapped to DirectionalFocusIntent(TraversalDirection.left) inside app.dart. So during text editing it needs to be mapped to DoNothingAndStopPropagationTextIntent in order for framework to not handle the event.

Alt-ArrowLeft on the other hand is not mapped to anything in framework. So framework does not handle the event, the engine will pass it to TextInputContext, which will produce the moveWordLeft: selector.

Second, while I'm still concerned with forcing the developer with override a NextFocusIntent with an InsertTabAction, I can't think of a better solution. However, this requires much more education, since overriding stuff is more confusing. We need a way to tell the developers to override the intent-to-action map, instead of the activator-to-intent map (which is understandably more intuitive), and the breakage otherwise will only be noticed in the corner case of emoji picker. Moreover, they will have to remember the enumerated intents to override, which basically means they're associating NextFocusIntent with pressing a Tab key.

Agreed. Ideally the framework should invoke whatever intent is mapped to Esc key above the editor. Users would expect that Shortcut with LogicalKeyboardKey.tab activator above text field will work, and it indeed will, except on macOS. Unfortunately I don't think this is possible to do currently with Shortcuts.

I'm wondering if it will be a better choice to create a MacosSelectorIntent(String content) and map them to the actions for macOS. This also makes it look less weird to map to an InsertTabAction. What do you think?

Which selectors would this be for? If I understand correctly you suggesting mapping

'insertTab:': MacosSelectorIntent('insertTab:), and then having NextFocusAction registered for MacosSelectorIntent('insertTab:)? What about cancelOperation: selector? That is currently mapped to DismissIntent, if that gets changed to MacosSelectorIntent('cancelOperation:') it might be confusing.

And also, I definitely prefer documenting the data flow above somewhere in the engine.

knopp · 2022-07-28T20:51:45Z

@chinmaygarde, this should be good to go, after the tests pass. I needs to be merged together with the framework PR, not sure how to do that.

dkwingsmt · 2022-07-28T20:55:41Z

I think merging the engine PR first shouldn't break anything. It adds the functionality that unhandled key presses will be converted to selectors, but since these text shortcuts are still handled, this functionality should not be triggered.

flutter-dashboard bot added the platform-macos label Jun 5, 2022

knopp force-pushed the remove_composing_hack branch from b8750e6 to f7641eb Compare June 5, 2022 18:33

This was referenced Jun 5, 2022

[macOS] Use editing intents from engine flutter/flutter#105407

Merged

Can't navigate composing candidates list with arrow keys on Mac flutter/flutter#85328

Closed

justinmc reviewed Jun 7, 2022

View reviewed changes

chinmaygarde added the Work in progress (WIP) Not ready (yet) for review! label Jun 9, 2022

knopp force-pushed the remove_composing_hack branch from 46cb18b to 4de9b63 Compare June 11, 2022 18:17

knopp removed the Work in progress (WIP) Not ready (yet) for review! label Jun 12, 2022

knopp changed the title ~~WIP: [macOS] Remove isComposing workaround~~ [macOS] Remove isComposing workaround Jun 12, 2022

knopp changed the title ~~[macOS] Remove isComposing workaround~~ WIP: [macOS] Remove isComposing workaround Jun 12, 2022

knopp added the Work in progress (WIP) Not ready (yet) for review! label Jun 12, 2022

This comment was marked as outdated.

Sign in to view

knopp removed the Work in progress (WIP) Not ready (yet) for review! label Jun 13, 2022

knopp changed the title ~~WIP: [macOS] Remove isComposing workaround~~ [macOS] Remove isComposing workaround Jun 13, 2022

knopp added affects: text input affects: desktop labels Jun 13, 2022

knopp force-pushed the remove_composing_hack branch from e210cc7 to 593ba19 Compare June 15, 2022 10:54

LongCatIsLooong reviewed Jun 21, 2022

View reviewed changes

knopp force-pushed the remove_composing_hack branch from 593ba19 to f53a793 Compare June 22, 2022 06:52

justinmc reviewed Jun 22, 2022

View reviewed changes

cbracken reviewed Jun 22, 2022

View reviewed changes

LongCatIsLooong mentioned this pull request Jun 23, 2022

Some tests in flutter_desktop_darwin_unittests leak into other tests flutter/flutter#106509

Closed

cbracken approved these changes Jun 23, 2022

View reviewed changes

knopp added 19 commits July 28, 2022 22:18

Forward editing intents to framework

d7d60a7

Respect suggested composing range

891adf6

Reformat

d2c4e44

Use performSelector call and forward all unmodified selectors

c6bf312

Remove forwardKeyEventsToSystemWhenComposing test

43bc30d

Add testSelectorsAreForwardedToFramework

1fe79ed

Add TestSetMarkedTextWithReplacementRange

737cc8c

Implement insertTab:

794d35d

Group selectors received in single runloop turn together

178229c

Fix test crash

a098e6d

Update comment

82c653f

Comment periods

8efd6fb

Typo

465a875

Plural

0b9a5c4

Remove SetSelection call

1e60b0a

Only schedule performBlock if needed

295e539

The framework

a0e98d2

Document replacementRange documentation being wrong

389dbb9

do not fowrard insertNewline: to framework

5f0b88a

knopp force-pushed the remove_composing_hack branch from 6f6ee70 to 5f0b88a Compare July 28, 2022 20:26

knopp changed the title ~~WIP: [macOS] Remove isComposing workaround~~ [macOS] Remove isComposing workaround Jul 28, 2022

knopp merged commit 4ad914e into flutter:main Jul 29, 2022

knopp deleted the remove_composing_hack branch July 29, 2022 07:21

engine-flutter-autoroll mentioned this pull request Jul 29, 2022

Roll Flutter Engine from 23a617aaed33 to 4ad914efddcb (1 revision) flutter/flutter#108602

Merged

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Jul 29, 2022

4ad914e [macOS] Remove isComposing workaround (flutter/engine#33838)

57931a3

emilyabest pushed a commit to emilyabest/engine that referenced this pull request Aug 12, 2022

[macOS] Remove isComposing workaround (flutter#33838)

9c39be2

Renzo-Olivares mentioned this pull request May 17, 2023

IME Deltas - Two-finger scrolling on Mac trackpad changes selection flutter/flutter#118642

Closed

cbracken mentioned this pull request May 17, 2023

Deflake flutter_desktop_darwin_unittests flutter/flutter#104789

Closed

[macOS] Remove isComposing workaround #33838

[macOS] Remove isComposing workaround #33838

Uh oh!

Conversation

knopp commented Jun 5, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pre-launch Checklist

Uh oh!

justinmc left a comment

Choose a reason for hiding this comment

Uh oh!

chinmaygarde commented Jun 9, 2022

Uh oh!

This comment was marked as outdated.

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LongCatIsLooong Jun 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

knopp Jun 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

knopp Jun 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

knopp Jun 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

justinmc left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

knopp Jun 23, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dkwingsmt Jun 30, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

LongCatIsLooong commented Jun 23, 2022

Uh oh!

cbracken left a comment

Choose a reason for hiding this comment

Uh oh!

cbracken commented Jun 30, 2022

Uh oh!

dkwingsmt commented Jun 30, 2022

Uh oh!

knopp commented Jun 30, 2022

Uh oh!

dkwingsmt commented Jul 1, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

knopp commented Jun 5, 2022 •

edited

Loading

LongCatIsLooong Jun 21, 2022 •

edited

Loading

knopp Jun 22, 2022 •

edited

Loading

knopp Jun 22, 2022 •

edited

Loading

knopp Jun 22, 2022 •

edited

Loading

knopp Jun 23, 2022 •

edited

Loading

dkwingsmt Jun 30, 2022 •

edited

Loading

dkwingsmt commented Jul 1, 2022 •

edited

Loading

knopp commented Jul 5, 2022 •

edited

Loading

dkwingsmt commented Jul 28, 2022 •

edited

Loading