Add support for iOS UndoManager #31415

fbcouch · 2022-02-11T23:00:09Z

This PR is modifies the engine to allow the framework to interact with the iOS UndoManager. That should enable 3-finger/shake gestures to undo/redo text editing and enable undo/redo from the iPad keyboard.

Related to flutter/flutter#34749 and flutter/flutter#77614

I haven't had a chance to test this out – still waiting for the engine to compile, but wanted to plant the flag on this and possibly feedback on the approach.

Framework PR: flutter/flutter#98294

Pre-launch Checklist

I read the Contributor Guide and followed the process outlined there for submitting PRs.
I read the Tree Hygiene wiki page, which explains my responsibilities.
I read and followed the Flutter Style Guide and the C++, Objective-C, Java style guides.
I listed at least one issue that this PR fixes in the description above.
I added new tests to check the change I am making or feature I am adding, or Hixie said the PR is test-exempt. See testing the engine for instructions on
writing and running engine tests.
I updated/added relevant documentation (doc comments with ///).
I signed the CLA.
All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

justinmc

Thanks for jumping on this, I was hoping we could support the iOS undo/redo gestures.

Is there any other benefit to using the native NSUndoManager versus detecting the gestures in the framework?

fbcouch · 2022-02-14T20:37:35Z

@justinmc I think using the NSUndoManager will actually allow us to support the 3-finger gestures, iPad keyboard, and voice control (flutter/flutter#77614) – I just tested that out and it seems to work properly. In the unlikely event that iOS adds another way to undo, I think having integrated with the NSUndoManager should support that as well.

fbcouch · 2022-02-14T20:53:32Z

justinmc · 2022-02-15T23:52:50Z

Ah nice I'm on board with that, it makes sense to do it this way instead of listening for the 3 finger swipe in the framework then. Let me know when you want me to review.

fbcouch · 2022-02-16T17:31:19Z

@justinmc Sounds good. I think this should be ready for review. Thanks!

chunhtai · 2022-02-17T17:57:29Z

shell/platform/darwin/ios/framework/Source/FlutterEngine.mm

@@ -950,6 +953,14 @@ - (void)flutterTextInputView:(FlutterTextInputView*)textInputView
                              arguments:@[ @(client) ]];
 }

+#pragma mark - Undo Manager Delegate
+
+- (void)flutterTextInputPlugin:(FlutterTextInputPlugin*)textInputPlugin


is textInputPlugin needed?

No, but my understanding from creating the Scribble delegate is that it's best practice to pass the plugin in as the first argument to delegate methods (see https://github.com/flutter/engine/blob/main/shell/platform/darwin/ios/framework/Source/FlutterIndirectScribbleDelegate.h and https://github.com/flutter/engine/blob/main/shell/platform/darwin/ios/framework/Source/FlutterEngine.mm#L919)

chunhtai · 2022-02-17T18:03:00Z

shell/platform/darwin/ios/framework/Source/FlutterTextInputPlugin.mm

@@ -1831,6 +1834,7 @@ - (void)insertText:(NSString*)text {
  [copiedRects release];
  _selectionAffinity = _kTextAffinityDownstream;
  [self replaceRange:_selectedTextRange withText:text];
+  [self ensureUndoEnabled];


Why do we only call this when insertText or delete text is called?

I believe insert/delete are the only actions that should enable the undo buttons.

There is a strange issue with the undoManager where the first call to - registerUndoWithTarget doesn't actually enable the undo button. To work around this, this method calls - registerUndoWithTarget, then - removeAllActionsWithTarget to remove that placeholder. After that, calling - registerUndoWithTarget again will correctly enable the undo button.

The framework could also perform actions on behalf of the user that are potentially undo/redo-able?

chunhtai · 2022-02-17T18:04:07Z

shell/platform/darwin/ios/framework/Source/FlutterTextInputPlugin.mm

+}
+
+- (void)setUndoManager:(NSUndoManager*)undoManager {
+  self.undoManagerOverride = undoManager;


why caches this?

So that I can set a mock object for tests – maybe there's a better way if I could mock the FlutterTextInput itself instead?

Anybody know if there's a better way to test this?

If not, maybe consider leaving a comment that this is done for testing.

Maybe just stub the method? https://ocmock.org/reference/#stubing-methods

Anyhow you don't need to expose this as a public method. A private category in the test should do it.

chinmaygarde · 2022-03-03T21:18:10Z

Any updates to @chunhtai queries?

fbcouch · 2022-03-03T22:30:58Z

Ah – sorry, I was moving last week and been catching up this week (and reworking the flutter side of this). I can go ahead and respond though, I think the responses will still be relevant when this gets reworked a bit.

fbcouch · 2022-03-09T19:14:56Z

All right, I split out the UndoManager interactions into their own channel and plugin, so I think this is ready for a review now

chinmaygarde · 2022-03-17T20:28:04Z

@LongCatIsLooong Ping on review?

justinmc

I left some comments on the framework PR but assuming everyone is ok with that overall approach, this engine PR LGTM 👍

justinmc · 2022-03-21T16:37:23Z

shell/platform/darwin/ios/framework/Source/FlutterTextInputPlugin.mm

+}
+
+- (void)setUndoManager:(NSUndoManager*)undoManager {
+  self.undoManagerOverride = undoManager;


Anybody know if there's a better way to test this?

If not, maybe consider leaving a comment that this is done for testing.

justinmc · 2022-03-21T16:40:50Z

shell/platform/darwin/ios/framework/Source/FlutterUndoManagerPlugin.mm

+  [self.undoManager beginUndoGrouping];
+  [self.undoManager registerUndoWithTarget:self
+                                   handler:^(id target) {
+                                     // register undo with opposite direction


Nit: Capital letter and period on these comments here and below.

LongCatIsLooong · 2022-03-22T17:19:39Z

shell/platform/darwin/ios/framework/Source/FlutterTextInputPlugin.mm

+}
+
+- (void)setUndoManager:(NSUndoManager*)undoManager {
+  self.undoManagerOverride = undoManager;


Maybe just stub the method? https://ocmock.org/reference/#stubing-methods

LongCatIsLooong · 2022-03-22T17:20:54Z

shell/platform/darwin/ios/framework/Source/FlutterTextInputPlugin.mm

+}
+
+- (void)setUndoManager:(NSUndoManager*)undoManager {
+  self.undoManagerOverride = undoManager;


Anyhow you don't need to expose this as a public method. A private category in the test should do it.

LongCatIsLooong · 2022-03-22T17:24:05Z

shell/platform/darwin/ios/framework/Source/FlutterUndoManagerPlugin.mm

+
+- (void)dealloc {
+  [self resetUndoManager];
+  if (_undoManager != nil) {


nit: just [_undoManager release] should be fine.

LongCatIsLooong · 2022-03-22T17:38:07Z

shell/platform/darwin/ios/framework/Source/FlutterTextInputPlugin.mm

@@ -1831,6 +1834,7 @@ - (void)insertText:(NSString*)text {
  [copiedRects release];
  _selectionAffinity = _kTextAffinityDownstream;
  [self replaceRange:_selectedTextRange withText:text];
+  [self ensureUndoEnabled];


The framework could also perform actions on behalf of the user that are potentially undo/redo-able?

LongCatIsLooong · 2022-03-22T17:44:58Z

shell/platform/darwin/ios/framework/Source/FlutterTextInputPlugin.mm

+
+- (void)ensureUndoEnabled API_AVAILABLE(ios(9.0)) {
+  if (![self.undoManager canUndo]) {
+    self.undoManager.groupsByEvent = NO;


Is setting this to NO and setting it back necessary? Shouldn't beginUndoGrouping be enough?

I can take another look at this – I tried a lot of different things to get the undo/redo buttons on the keyboard to show up the first time...this is where it ended up, but that doesn't mean it was all necessary.

Taken with your comment on line 1837, I wonder if this might make more sense in the UndoManagerPlugin – we might be able to make two calls across the channel, one to ensure undo is enabled, then set the state...that would keep this from being too tightly coupled to text input.

So I think here, it isn't necessary – in the FlutterUndoManagerPlugin, though, it does seem to be. I suspect it has something to do with the fact that we're potentially registering multiple undos in the same event loop, and without groupsByEvent off, they still get grouped together by iOS.

I fiddled with trying to move the ensureUndoEnabled stuff into FlutterUndoManagerPlugin, but couldn't get it to actually work correctly. I'm not sure why, but I think doing it within the same event loop as a key press might be the secret sauce there. The good news is, I think the only thing that's needed for is to get iOS to correctly update the UI of the iPad keyboard on the first keypress. After that, things work as expected. For non-text input use cases, the keyboard wouldn't be up anyway, so that should be less of a concern.

Ah interesting thanks for looking into this! Since we're using a dedicated NSUndoManager in the plugin, would it work if we just turn groupsByEvent off at the beginning once and for all since we'll be manually grouping the events then?

So we aren't actually using a dedicated NSUndoManager (except in tests), we're just grabbing the one from the FlutterViewController.

I do wonder, though, if some of the weirdness that requires the ensureUndoEnabled has to do with which NSUndoManager is active at the time – I did a bit of reading about it, and if I understand correctly, each NSResponder can have it's own UndoManager, and iOS walks up the responder chain to get the "active" undoManager.

I'm wondering if we should have our own FlutterUndoManager or override the FlutterTextInputView to return the one from the FlutterViewController or something like that. I ran out of time today before I could play around with that idea, but I'll try to get to it soon.

@LongCatIsLooong Just wanted to check in to see if you have any thoughts on this

Looked into this, it looks like -[TUISystemInputAssistantView setNeedsValidation] triggers the keyboard update. Calling [self.inputDelegate didChangeSelection: self] should trigger that.

(if you have access to the input view that is). Or [firstResponder reloadInputViews] if you have access to the current first responder. But I'm not sure if that's going to affect market text.

Is the beginUndoGrouping pair still needed? It sounded like it was a workaround to trigger a keyboard update?

I think the grouping is still needed – the workaround for keyboard state was in the TextInputPlugin where I was registering and immediately removing an undo group. That workaround is replaced by the UITextInputAssistantItem business in FlutterUndoManagerPlugin. Calling selectionDidChange on the inputDelegate seems to be an adequate replacement, though, and seems at least a little more correct than removing and re-adding the input bar items.

chinmaygarde · 2022-04-14T20:06:28Z

@LongCatIsLooong Another review please?

LongCatIsLooong

Great work! Just a few questions to make sure I'm understanding the code correctly.

LongCatIsLooong · 2022-04-22T00:17:42Z

shell/platform/darwin/ios/framework/Source/FlutterUndoManagerPlugin.mm

+  [self.undoManager undo];
+}
+
+- (void)setUndoState:(NSDictionary*)dictionary API_AVAILABLE(ios(9.0)) {


More of a framework question, but when should the framework update the undo state and how does it decide the value of canUndo/canRedo?

If we are in a text field that both operations are available then a new route pops up, I assume setUndoState needs to be called?

A different scenario, if we are in an editable text field that has an inline image widget, if the image is focused, has canUndo == false and canRedo == true, but for the text field both types of operations are available, is there a way to propagate the canUndo query up the focus chain to tell the system the undo operation is actually available?

I agree, that probably makes sense on the framework side.

I suppose the question is, what should happen in that case? I'll readily admit I'm not super familiar with the intricacies of how undo works...if the image is focused, is the expected behavior that undo would be available? My naive assumption was that it should only reflect the undo state of whatever has focus currently.

I'm sure we can find a way to propagate that up, perhaps either re-using the focus tree or building a separate undo one.

LongCatIsLooong · 2022-04-22T01:06:41Z

shell/platform/darwin/ios/framework/Source/FlutterUndoManagerPlugin.mm

+  BOOL canUndo = [dictionary[kCanUndo] boolValue];
+  BOOL canRedo = [dictionary[kCanRedo] boolValue];
+
+  [self resetUndoManager];


Does the plugin need to know how deep the undo stack is and how deep the redo stack is, so say after 3 consecutive left 3-finger swipes the undo button on the virtual keyboard becomes disabled? Or the framework is supposed to send another pair of booleans to update the canUndo and canRedo state?

Currently, the latter – the framework is responsible for telling the engine whether undo and redo are available.

zanderso · 2022-04-28T20:05:26Z

From PR triage: @LongCatIsLooong does this still look good to you? Also it looks like this PR needs to be rebased.

LongCatIsLooong · 2022-04-29T00:14:47Z

Sorry still doing some tests. Need a little bit more time.

cyanglaz · 2022-04-29T17:54:15Z

shell/platform/darwin/ios/framework/Source/FlutterUndoManagerPluginTest.mm

+  id engine;
+  FlutterUndoManagerPluginForTest* undoManagerPlugin;
+  FlutterViewController* viewController;
+  NSUndoManager* undoManager;


nits: Use properties instead

cyanglaz · 2022-04-29T18:02:59Z

shell/platform/darwin/ios/framework/Source/FlutterUndoManagerDelegate.h

+@class FlutterUndoManagerPlugin;
+
+@protocol FlutterUndoManagerDelegate <NSObject>
+- (void)flutterUndoManagerPlugin:(FlutterUndoManagerPlugin*)undoManagerPlugin


nits: a better Objective-C style naming would be:

- (void)flutterUndoManagerPlugin:(FlutterUndoManagerPlugin*)undoManagerPlugin handleUndoWithDirection:(FlutterUndoRedoDirection)direction;

cyanglaz · 2022-04-29T18:12:03Z

shell/platform/darwin/ios/framework/Source/FlutterUndoManagerPlugin.mm

+}
+
+- (void)resetUndoManager API_AVAILABLE(ios(9.0)) {
+  [self.undoManager removeAllActionsWithTarget:self];


nits:
I recommend against using dot notation for method invocations. Although It is legal in Objective-C, it decrease the readability of the code.
Dot is usually used to access properties in Objective-C, this confuses code readers as there's no property named undoManager.

Gotcha. I think I used to have a property for it (to allow overriding it in tests), so this was just left over from that. Thanks.

…d when changing text (used with the flutter_undo plugin)

…oManager

…into the framework to undo/redo appropriately

… think there might be some issue with which UndoManager is currently active in the responder chain

…h a workaround in FlutterUndoManagerPlugin to notify the UIAssistantBar that it needs to update it's button state

skia-gold · 2022-05-05T20:52:57Z

Gold has detected about 229 new digest(s) on patchset 23.
View them at https://flutter-engine-gold.skia.org/cl/github/31415

* Use properties in tests * Use objc-style method invocation * Rename delegate method to match objc convention

* Wrap setUndoState in groupsByEvent pair, rather than each method * Use slightly more straightforward workaround for iPadOS keyboard update issue

LongCatIsLooong · 2022-05-10T22:30:50Z

The approach LGTM. One last question: would it work better/worse, if we use our own NSUndoManager and override canUndo and canRedo to reflect the value sent by the framework?

fbcouch · 2022-05-11T01:16:15Z

That's a good question. I actually started down that road at one point while trying to get away from having that workaround in the TextInputPlugin. It's been a little while now, so I don't remember if there was an issue with it beyond just not replacing the workaround. Conceptually, I think it should work, and it would make the plugin code here a little less strange.

LongCatIsLooong · 2022-05-11T16:29:21Z

Would you like to try that or should we merge the pull request as it is? For UITextField there's a dedicated NSUndoManager subclass and you won't be able to undo/redo edits made to other UI components when a UITextField is the first responder. But I'm not sure what makes the most sense in an add-to-app scenario.

fbcouch · 2022-05-11T16:35:36Z

I'm good with merging it as is, just in case using our own subclass messes something up with the UITextField's NSUndoManager subclass.

LongCatIsLooong · 2022-05-11T19:07:47Z

We're not using UITextFields so that shouldn't be a problem.

flutter-dashboard bot added the platform-ios label Feb 11, 2022

fbcouch mentioned this pull request Feb 11, 2022

Add support for iOS UndoManager flutter/flutter#98294

Merged

8 tasks

fbcouch force-pushed the feature/undo-redo-gestures branch from dbfd62a to 8c82076 Compare February 14, 2022 03:53

justinmc reviewed Feb 14, 2022

View reviewed changes

chunhtai self-requested a review February 15, 2022 18:00

fbcouch marked this pull request as ready for review February 16, 2022 17:19

chunhtai reviewed Feb 17, 2022

View reviewed changes

fbcouch marked this pull request as draft March 3, 2022 22:41

fbcouch marked this pull request as ready for review March 9, 2022 19:14

justinmc requested a review from LongCatIsLooong March 9, 2022 22:49

justinmc approved these changes Mar 21, 2022

View reviewed changes

LongCatIsLooong reviewed Mar 22, 2022

View reviewed changes

fbcouch requested a review from LongCatIsLooong March 31, 2022 15:00

LongCatIsLooong reviewed Apr 22, 2022

View reviewed changes

cyanglaz reviewed Apr 29, 2022

View reviewed changes

fbcouch added 4 commits May 5, 2022 15:05

Add ensureUndoEnabled method to make sure the undo button gets enable…

70f72cf

…d when changing text (used with the flutter_undo plugin)

Add FlutterUndoManagerDelegate, initial thoughts at how to handle und…

f651ea8

…oManager

Framework can set up the undo manager as needed, delegate calls back …

c74bb85

…into the framework to undo/redo appropriately

Add setUndoState method – works pretty well

283a434

fbcouch added 10 commits May 5, 2022 15:06

Split out FlutterUndoManagerPlugin from FlutterTextInputPlugin

b09dbb3

Split out setUndoState test

1a07efe

Fix licenses

043399e

Fix up comments

c1f26df

Move test override for undoManager into tests

1cbe10c

WIP - trying to move ensureUndoEnabled to FlutterUndoManagerPlugin. I…

655261a

… think there might be some issue with which UndoManager is currently active in the responder chain

Remove ensureUndoEnabled stuff from FlutterTextInputView; replace wit…

76eee92

…h a workaround in FlutterUndoManagerPlugin to notify the UIAssistantBar that it needs to update it's button state

Do not commit development team

05c2af4

Remove unnecessary include

a532298

Clean up after rebase

df791ee

fbcouch force-pushed the feature/undo-redo-gestures branch from 854df54 to df791ee Compare May 5, 2022 20:16

fbcouch added 3 commits May 5, 2022 16:17

Address PR Feedback

6354526

* Use properties in tests * Use objc-style method invocation * Rename delegate method to match objc convention

More PR Feedback

8fd6dae

* Wrap setUndoState in groupsByEvent pair, rather than each method * Use slightly more straightforward workaround for iPadOS keyboard update issue

Merge branch 'main' into feature/undo-redo-gestures

ab85e69

LongCatIsLooong approved these changes May 11, 2022

View reviewed changes

LongCatIsLooong added the waiting for tree to go green This PR is approved and tested, but waiting for the tree to be green to land. label May 11, 2022

fluttergithubbot merged commit 7e56794 into flutter:main May 11, 2022

engine-flutter-autoroll mentioned this pull request May 11, 2022

Roll Flutter Engine from 94a5895388f7 to ccad2ef41eca (2 revisions) flutter/flutter#103533

Merged

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request May 11, 2022

7e56794 Add support for iOS UndoManager (flutter/engine#31415)

c66c849

engine-flutter-autoroll mentioned this pull request May 11, 2022

Roll Flutter Engine from 94a5895388f7 to ccad2ef41eca (2 revisions) flutter/flutter#103534

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request May 11, 2022

7e56794 Add support for iOS UndoManager (flutter/engine#31415)

ce66514

engine-flutter-autoroll mentioned this pull request May 11, 2022

Roll Flutter Engine from 94a5895388f7 to ccad2ef41eca (2 revisions) flutter/flutter#103535

Merged

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request May 11, 2022

7e56794 Add support for iOS UndoManager (flutter/engine#31415)

47ced80

fbcouch added a commit to twinsunllc/engine that referenced this pull request May 16, 2022

Add support for iOS UndoManager (flutter#31415)

0989929

Add support for iOS UndoManager #31415

Add support for iOS UndoManager #31415

Uh oh!

Conversation

fbcouch commented Feb 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pre-launch Checklist

Uh oh!

justinmc left a comment

Choose a reason for hiding this comment

Uh oh!

fbcouch commented Feb 14, 2022

Uh oh!

fbcouch commented Feb 14, 2022

Uh oh!

justinmc commented Feb 15, 2022

Uh oh!

fbcouch commented Feb 16, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LongCatIsLooong Mar 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chinmaygarde commented Mar 3, 2022

Uh oh!

fbcouch commented Mar 3, 2022

Uh oh!

fbcouch commented Mar 9, 2022

Uh oh!

chinmaygarde commented Mar 17, 2022

Uh oh!

justinmc left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LongCatIsLooong Mar 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fbcouch commented Feb 11, 2022 •

edited

Loading

LongCatIsLooong Mar 22, 2022 •

edited

Loading

LongCatIsLooong Mar 22, 2022 •

edited

Loading

LongCatIsLooong Apr 29, 2022 •

edited

Loading

LongCatIsLooong left a comment •

edited

Loading