Support Scribble Handwriting #75472

fbcouch · 2021-02-05T02:46:57Z

This PR is the framework side of adding scribble handwriting support. I still need to add some tests, but looking for feedback on the overall approach. I incorporated some feedback from @LongCatIsLooong on a previous draft, moving most of the logic of this into the framework and adding a widget (_ScribbleElement) to handle most of the scribble-specific logic.

The one remaining issue here (that I could use some ideas on how to solve) is that the UIIndirectScribbleInteraction can always be triggered, regardless of whether the text input in question is obscured by another element or otherwise non-interactive (e.g. behind something else in a stack).

#61278

Design doc: https://docs.google.com/document/d/1mjQbsSRQnHuAgMNdouaSgTS-Xv-w57fdKfOUUafWpRo

If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.

Pre-launch Checklist

I read the Contributor Guide and followed the process outlined there for submitting PRs.
I read the Tree Hygiene wiki page, which explains my responsibilities.
I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
I listed at least one issue that this PR fixes in the description above.
I added new tests to check the change I am making or feature I am adding, or Hixie said the PR is test exempt.
I updated/added relevant documentation (doc comments with ///).
I signed the CLA.
All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

LongCatIsLooong

Thank you for the PR and great work! Left some initial feedback and questions.

An additional question regarding the UIIndirectScribbleInteractionDelegate protocol:
when does UIKit call func indirectScribbleInteraction(UIInteraction, frameForElement: Self.ElementIdentifier) -> CGRect? Is it always immediately after one of the async calls (func indirectScribbleInteraction(UIInteraction, requestElementsIn: CGRect, completion: ([Self.ElementIdentifier]) -> Void) or func indirectScribbleInteraction(UIInteraction, focusElementIfNeeded: Self.ElementIdentifier, referencePoint: CGPoint, completion: ((UIResponder & UITextInput)?) -> Void))? I'd like to understand how much information we would have to send to the engine every frame.

LongCatIsLooong · 2021-02-09T05:21:33Z

packages/flutter/lib/src/rendering/editable.dart

@@ -1049,6 +1049,9 @@ class RenderEditable extends RenderBox with RelayoutWhenSystemFontsChangeMixin {
  /// The text to display.
  TextSpan? get text => _textPainter.text as TextSpan?;
  final TextPainter _textPainter;
+  /// The text painter currently used to paint text.
+  /// 
+  /// Currently used to find the selection rects for Scribble support


We probably don't want to expose the underlying TextPainter but feel free to expose the methods you need on RenderEditable (see RenderEditable.getBoxesForSelection).

LongCatIsLooong · 2021-02-09T05:27:36Z

packages/flutter/lib/src/services/text_input.dart

+
+  /// Called during a UIIndirectScribbleInteraction when the [ScribbleClient] should
+  /// receive focus
+  void onScribbleFocus(double x, double y);


should probably use Offset instead.

LongCatIsLooong · 2021-02-09T05:29:27Z

packages/flutter/lib/src/services/text_input.dart

+  void onScribbleFocus(double x, double y);
+
+  /// Tests whether the [ScribbleClient] overlaps the given rectangle bounds
+  bool inScribbleRect(double x, double y, double width, double height);


nit: maybe: isInScribbleRect(Rect scribbleRect).

Additionally you can have text fields that are offscreen (e.g. currently obstructed by other routes), so this is probably not enough to tell which client wants to be focused.

I agree! Do you have any thoughts on the best way to figure that out? Between that and other things that might obscure the fields from being interactive, I think that is the biggest remaining issue.

I think the rest of your feedback all makes sense at first glance, thank you for taking the time to review this!

LongCatIsLooong · 2021-02-09T05:35:45Z

packages/flutter/lib/src/services/text_input.dart

+  bool inScribbleRect(double x, double y, double width, double height);
+
+  /// The current bounds of the [ScribbleClient]
+  List<double> get bounds;


we should use more specific types when possible (i.e. Rect over List<double>).

LongCatIsLooong · 2021-02-09T05:40:24Z

packages/flutter/lib/src/services/text_input.dart

@@ -1236,10 +1281,24 @@ class TextInput {
  TextInputConnection? _currentConnection;
  late TextInputConfiguration _currentConfiguration;

+  final Map<String, ScribbleClient> _scribbleClients = <String, ScribbleClient>{};


It feels a bit strange to store the client list here. Would it be possible to move this to a widget?

Would a static member of EditableTextState make sense? Or maybe _ScribbleElement should become public and this and the register/unregister static methods would all live there?

LongCatIsLooong · 2021-02-09T05:47:09Z

packages/flutter/lib/src/widgets/editable_text.dart

@@ -2423,6 +2423,15 @@ class EditableTextState extends State<EditableText> with AutomaticKeepAliveClien
      final Size size = renderEditable.size;
      final Matrix4 transform = renderEditable.getTransformTo(null);
      _textInputConnection!.setEditableSizeAndTransform(size, transform);
+      final TextSpan textSpan = buildTextSpan();


Is this trying to get the Rect of the RenderEditable?

This is to get the length of the text for use on line 2430

Oh I see to implement closestPositionToPoint and friends you will need the bounding boxes for each glyph. This is going to be pretty heavy for long documents, and since a RenderEditable clips its content if it gets too long (i.e. RenderEditable is a scrollable container and only part of it will remain visible), you'll get false positives on glyphs that are not currently visible.

What scribble function(s) would it break if we leave the closestPositionToPoint family unimplemented?

Oh interesting. I will have to put some long text in and see how that affects things.

So, closestPositionToPoint itself allows UIKit to move the selection around based on where the user is writing. That one is not a huge deal to leave unimplemented, I suppose – the caret will just stay where it is using the previous implementation. The bigger ones are firstRectForRange and selectionRectsForRange which UIKit needs to do pretty much everything except inserting text at the caret: selecting, deleting, inserting/removing spaces

Okay, so you are right, text long enough to scroll does cause some issues. I pushed up some changes to deal with this:

Offset the selection rects by the current scroll offset

Cache the selection rects so that they are only recalculated if needed

Wait for the scroll direction to be .idle so that scrolling is smooth

LongCatIsLooong · 2021-02-09T05:52:57Z

packages/flutter/lib/src/services/text_input.dart

+  }
+
+  /// Deregisters a [ScribbleClient] with [elementIdentifier]
+  static void deregisterScribbleElement(String elementIdentifier) {


Did a search in a codebase and it seems we didn't name anything deregisterSomething. So maybe unregister?

LongCatIsLooong · 2021-02-09T06:08:26Z

packages/flutter/lib/src/services/text_input.dart

@@ -681,6 +683,7 @@ class TextEditingValue {
        start: encoded['composingBase'] as int? ?? -1,
        end: encoded['composingExtent'] as int? ?? -1,
      ),
+      scribbleInProgress: encoded['scribbleInProgress'] as bool? ?? false,


We probably should send scribbleInProgress to the framework in a separate text input channel call, since the framework doesn't change the value. And we can avoid having to send the value back, as it's not going to be changed by the framework

LongCatIsLooong · 2021-02-09T06:12:03Z

packages/flutter/lib/src/widgets/editable_text.dart

+  @override
+  String get elementIdentifier {
+  if (_elementIdentifier == null) {
+      final math.Random random = math.Random();


It seems we want the identifier to be unique but not necessarily random? Can we use increase-by-one counter instead?

fbcouch · 2021-02-11T20:19:34Z

Thank you for the PR and great work! Left some initial feedback and questions.

An additional question regarding the UIIndirectScribbleInteractionDelegate protocol:
when does UIKit call func indirectScribbleInteraction(UIInteraction, frameForElement: Self.ElementIdentifier) -> CGRect? Is it always immediately after one of the async calls (func indirectScribbleInteraction(UIInteraction, requestElementsIn: CGRect, completion: ([Self.ElementIdentifier]) -> Void) or func indirectScribbleInteraction(UIInteraction, focusElementIfNeeded: Self.ElementIdentifier, referencePoint: CGPoint, completion: ((UIResponder & UITextInput)?) -> Void))? I'd like to understand how much information we would have to send to the engine every frame.

I believe UIKit calls func indirectScribbleInteraction(UIInteraction, requestElementsIn: CGRect, completion: ([Self.ElementIdentifier]) -> Void) and waits for the completion handler. Then it calls func indirectScribbleInteraction(UIInteraction, frameForElement: Self.ElementIdentifier) -> CGRect. Since I have func indirectScribbleInteraction(UIInteraction, shouldDelayFocusForElement: Self.ElementIdentifier) -> Bool always returning true, it waits for a pause in the user input before calling func indirectScribbleInteraction(_ interaction: UIInteraction, focusElementIfNeeded elementIdentifier: Self.ElementIdentifier, referencePoint focusReferencePoint: CGPoint, completion: @escaping ((UIResponder & UITextInput)?) -> Void)

I don't think we're sending info to the engine on every frame during the indirect interaction now that I moved that to the framework side. Once the input is focused, we do calculate the selection boxes every frame, but we also cache those on the framework side so that they only actually get sent to the engine when changed.

LongCatIsLooong · 2021-02-23T00:16:39Z

Sorry for the delay I'll take a look tonight.

fbcouch · 2021-02-23T04:30:30Z

No problem! I actually have an idea for how to do the hit testing thing, just haven't had a chance to cherry pick that into the PR branch. That's here: twinsunllc@0dfa712#diff-f5f93c879cef9a102adbf148583de9f3d9a05f9678fdcad7bde106f62200bb34R2947

LongCatIsLooong

Sorry I just realized I can't test this on a simulator. I'll see if I can get a device.

LongCatIsLooong · 2021-02-23T07:19:57Z

packages/flutter/lib/src/widgets/editable_text.dart

@@ -2423,6 +2423,15 @@ class EditableTextState extends State<EditableText> with AutomaticKeepAliveClien
      final Size size = renderEditable.size;
      final Matrix4 transform = renderEditable.getTransformTo(null);
      _textInputConnection!.setEditableSizeAndTransform(size, transform);
+      final TextSpan textSpan = buildTextSpan();


Oh I see to implement closestPositionToPoint and friends you will need the bounding boxes for each glyph. This is going to be pretty heavy for long documents, and since a RenderEditable clips its content if it gets too long (i.e. RenderEditable is a scrollable container and only part of it will remain visible), you'll get false positives on glyphs that are not currently visible.

LongCatIsLooong · 2021-02-23T07:24:23Z

packages/flutter/lib/src/widgets/editable_text.dart

@@ -2423,6 +2423,15 @@ class EditableTextState extends State<EditableText> with AutomaticKeepAliveClien
      final Size size = renderEditable.size;
      final Matrix4 transform = renderEditable.getTransformTo(null);
      _textInputConnection!.setEditableSizeAndTransform(size, transform);
+      final TextSpan textSpan = buildTextSpan();


What scribble function(s) would it break if we leave the closestPositionToPoint family unimplemented?

LongCatIsLooong · 2021-02-23T07:36:40Z

packages/flutter/lib/src/widgets/editable_text.dart

+  RenderEditable? get renderEditable => widget.editableKey.currentContext?.findRenderObject() as RenderEditable?;
+
+  static int _nextElementIdentifier = 1;
+  String _elementIdentifier;


this can be final right?

LongCatIsLooong · 2021-02-28T04:55:48Z

packages/flutter/lib/src/services/text_input.dart

+      }
+      return;
+    } else if (method == 'TextInputClient.requestElementsInRect') {
+      final List<dynamic> args = methodCall.arguments as List<dynamic>;


Maybe we can have a widget the serves as the scope of the clients that will be considered visible, and put one such widget in each route.

The idea being to move the _scribbleClients and registerScribbleElement / unregisterScribbleElement members into a ScribbleContext or some such thing?

Yeah the same solution autofill uses to determine which clients are considered visible. But for autofill it's easier: the current scope is the one that containss the current TextInputClient. For scribble I'm not sure how this should work, since UIIndirectScribbleInteraction doesn't seem to require the subject text field to be focused. I don't think there is an app-wise "currentRoute", since you can have more than one navigators in an app and each of them will have a "currentRoute" (CupertinoTabScaffold for example). Even if there is, widgets in the "currentRounte" are not guaranteed to be visible and interactable.

It might be possible to solve this by adding something to the RenderObject class that resembles hit-testing (or generalize the existing hit-testing system) but with Rects instead of PointEvents.

LongCatIsLooong · 2021-02-28T05:02:31Z

packages/flutter/lib/src/widgets/editable_text.dart

+      final Offset rectOffset = Offset(_isMultiline ? 0.0 : -1 * scrollOffset, _isMultiline ? -1 * scrollOffset : 0.0);
+      final List<Rect> rects = List<Rect>.generate(
+              text.length, (int i) => renderEditable.getBoxesForSelection(TextSelection(baseOffset: i, extentOffset: i + 1)).first.toRect())
+          .map((Rect rect) => rect.translate(rectOffset.dx, rectOffset.dy))


this should probably happen in the RenderEditable public api you exposed in the PR.

So using localToGlobal on the render object there instead of offsetting by the scroll offset here?

The RenderEditable.getBoxesForSelection API you exposed in the PR should be responsible for doing the translate(rectOffset.dx, rectOffset.dy) thing. The caller would usually expect the TextBoxes to be described in the RenderEditable's coordiante space instead of _textPainter's. (Some other RenderEditable APIs are doing the same thing).

LongCatIsLooong · 2021-02-28T05:05:31Z

packages/flutter/lib/src/widgets/editable_text.dart

+      _cachedFirstRect = firstRect;
+      final Offset rectOffset = Offset(_isMultiline ? 0.0 : -1 * scrollOffset, _isMultiline ? -1 * scrollOffset : 0.0);
+      final List<Rect> rects = List<Rect>.generate(
+              text.length, (int i) => renderEditable.getBoxesForSelection(TextSelection(baseOffset: i, extentOffset: i + 1)).first.toRect())


I'm not sure if this is going to work when there's surrogate pairs & extended glypheme clusters. @justinmc

Will UIkit ever ask for the locations of the glyphs in a text field that is not focused? If that's the case the Rects can be outdated since we only update for the current text field.

I believe it will not. When not focused, we should only be dealing with UIIndirectScribbleInteraction which asks for the elements in a rect (requestElementsInRect) and then lets us know to focus on one of them at a given point (focusElementIfNeeded). Once an element has focus, we should be sending over the rects so that UIKit has what it needs to do the text operations that happen during a UIDirectScribbleInteraction

fbcouch · 2021-03-10T22:02:01Z

@LongCatIsLooong I think I have addressed your feedback except for a couple of items:

Support Scribble Handwriting #75472 (comment) - not sure what to do here
Support Scribble Handwriting #75472 (comment) - is there a way to test this? I believe Scribble only supports English and Chinese, so I'm not sure if we would run into those things

I also added an implementation for insertTextPlaceholder and removeTextPlaceholder coming from the engine side. PencilKit uses those to accomplish the Insert scribble method where you can long press with the pencil to get some extra space to write in.

Let me know your thoughts there. I have some merge conflicts now – I'm hoping this is close, but I'll wait to get your thumbs up before rebasing this onto the current master (or perhaps the last green commit on master?) and marking it as ready for review.

LongCatIsLooong

Great work!

The "finding the right scribble clients" problem seems to be way harder than I initially thought and unfortunately I don't really have a good solution.

/cc @justinmc for insights.

LongCatIsLooong · 2021-03-14T08:45:33Z

packages/flutter/lib/src/services/text_input.dart

+  void showToolbar();
+
+  /// UIKit has requested a text placeholder be added at the current selection
+  void insertTextPlaceholder(Size size);


Could you elaborate on what is a "text placeholder"? Is it possible for multiple placeholders to co-exist (for example, how does it work if I add a text placeholder in one field and switch to a different input field and add another)?

It does not appear that you can add multiple placeholders, that said, we do need to clean up the one in the current TextInputClient if the focus changes, since the removeTextPlaceholder call will get pushed to the new TextInputClient

Oh, I forgot to answer your first question: this is another scribble feature where you can long press to give yourself extra space to write in the middle of some text. I don't know if it's used for anything else...the docs about it are pretty terse, and don't event mention the scribble use.

Maybe add a little bit in these comments about what this is for, something like:

/// For example, this is called when responding to UIKit requesting - /// a text placeholder be added at the current selection. + /// a text placeholder be added at the current selection, such as when + /// requesting additional writing space with iPadOS14 Scribble.

LongCatIsLooong · 2021-03-14T09:48:56Z

packages/flutter/lib/src/services/text_input.dart

+        }
+        return <dynamic>[elementIdentifier, ...<dynamic>[bounds.left, bounds.top, bounds.width, bounds.height]];
+      }).where((List<dynamic> list) {
+        return list.length == 5;


This is a bit strange, as the meaning of the "where" is not immediately clear. Consider doing the filtering before the map or just do the whole thing in a for loop.

LongCatIsLooong · 2021-03-14T09:51:02Z

packages/flutter/lib/src/services/text_input.dart

+    if (method == 'TextInputClient.focusElement') {
+      final List<dynamic> args = methodCall.arguments as List<dynamic>;
+      if (_scribbleClients.containsKey(args[0])) {
+        _scribbleClients[args[0]]?.onScribbleFocus(Offset(args[1].toDouble() as double, args[2].toDouble() as double));


Do you still need the type case with toDouble()? Doesn't toDouble() return a double?

I think the issue is that args is List<dynamic>, so args[1] is dynamic which means args[1].toDouble() also returns dynamic. If i take out as double I get a the following error:

The argument type 'dynamic' can't be assigned to the parameter type 'double'.

LongCatIsLooong · 2021-03-14T10:00:23Z

packages/flutter/lib/src/services/text_input.dart

+      }
+      return;
+    } else if (method == 'TextInputClient.requestElementsInRect') {
+      final List<dynamic> args = methodCall.arguments as List<dynamic>;


Yeah the same solution autofill uses to determine which clients are considered visible. But for autofill it's easier: the current scope is the one that containss the current TextInputClient. For scribble I'm not sure how this should work, since UIIndirectScribbleInteraction doesn't seem to require the subject text field to be focused. I don't think there is an app-wise "currentRoute", since you can have more than one navigators in an app and each of them will have a "currentRoute" (CupertinoTabScaffold for example). Even if there is, widgets in the "currentRounte" are not guaranteed to be visible and interactable.

LongCatIsLooong · 2021-03-14T23:38:06Z

packages/flutter/lib/src/services/text_input.dart

+      }
+      return;
+    } else if (method == 'TextInputClient.requestElementsInRect') {
+      final List<dynamic> args = methodCall.arguments as List<dynamic>;


It might be possible to solve this by adding something to the RenderObject class that resembles hit-testing (or generalize the existing hit-testing system) but with Rects instead of PointEvents.

LongCatIsLooong · 2021-03-15T02:33:31Z

packages/flutter/lib/src/widgets/editable_text.dart

+  void insertTextPlaceholder(Size size) {
+    print('[scribble][flutter] insertTextPlaceholder $size');
+    setState(() {
+      _placeholderLocation = _value.text.length - widget.controller.selection.end;


unfortunately currently in the framework we use (-1, -1) to represent no selection instead of null. That means you'll get _value.text.length + 1 if there's no selection. Can that happen here?

Can the EditableText have focus but not a selection? The checks on where the placeholder gets inserted should handle that (they test that it falls within [0, _value.text.length]), so we wouldn't insert the placeholder anywhere...which...if there's no selection, we wouldn't know where to put it anyway.

Yeah the user can set the selection to (-1, -1) even if the EditableText is focused.

You might want to check if the selection is valid (https://master-api.flutter.dev/flutter/dart-ui/TextRange/isValid.html) first.

I'm curious how text selections interact with placeholders. When a placeholder is inserted and the current selection isn't empty, does the placeholder replace the currently selected text, or the placeholder is simply inserted at the end of the current selection? If it's the latter what happens if you press backspace after inserting a placeholder?

The placeholder should be inserted wherever the user is pressing – the selection actually gets moved there and collapsed anyway when the pencil is placed.

LongCatIsLooong · 2021-03-15T02:41:18Z

packages/flutter/lib/src/widgets/editable_text.dart

@@ -2519,6 +2545,24 @@ class EditableTextState extends State<EditableText> with AutomaticKeepAliveClien
    }
  }

+  int _placeholderLocation = -1;


nit: maybe use null to indicate there's no placeholder

I actually kinda like using -1 as a special value here, as I can easily test that the placeholder is within the bounds of the string (which I need to do anyway) without having to also make sure _placeholderLocation is not null

If we do go with -1, can we make sure it's fully explained in a comment somewhere? Like that -1 means there is no placeholder, otherwise it must be between 0 and the length of the text inclusive.

LongCatIsLooong · 2021-03-15T03:19:00Z

packages/flutter/lib/src/widgets/editable_text.dart

+
+  @override
+  bool isInScribbleRect(Rect rect) {
+    final Rect _bounds = bounds;


nit: why the local variable?

bounds is computed, so I figured it was worth getting once and setting it to a local variable to use in the later steps

LongCatIsLooong · 2021-03-15T06:25:39Z

packages/flutter/lib/src/widgets/editable_text.dart

+      intersection.bottomCenter,
+      intersection.bottomRight
+    ].any((Offset point) {
+      final HitTestResult result = HitTestResult();


Ah I see how difficult it would be to find hitable RenderObjects in a given Rect. Hitable RenderBoxes can partially overlap each other. So never mind the previous comment about adding things to RenderObject. But still we probably shouldn't sample points.

@goderbauer @justinmc any suggestions on how we can find currently "interactable" elements in a given rect?

Yeah, sampling "random" points doesn't seem right here.

Also it seems we are doing this hit testing multiple times if there are multiple Scribbles on screen?

Can you describe what the hit test is trying to accomplish?

@goderbauer there's an interesting scribble feature where you can scribble on screen without any text field focused. The text will be sent to the most relevant text field based on the scribble location. An example: in the "reminder" app you can launch the app and just scribble away, and a new "reminder" entry will be added.

To that end UIKit needs to ask us for "active scribble elements" in a certain Rect. But it's tricky to tell if a ScribbleClient has a "hotspot" in the given a Rect (as it can be behind an IgnorePointer or obscured by another ScribbleClient, we would have to hit test every Offset in the Rect).

Right, so the hit testing is last here, after making sure the _ScribbleFocusable intersects the rect that UIKit is interested in. I think in most cases that would just be a few elements. It also uses any so that in the nominal case where the whole text field is visible and interactive, only one point will need to be checked.

I am interested to hear some ideas on the best way to do this. FWIW I think this approach will work in most cases, since it is testing points on the intersection between the two rectangles. Unless someone has very large handwriting, that area should be relatively small.

We actually have a beta test going with a custom engine implementing this that seems to be going pretty well. Granted, that is on a separate branch as we haven't updated from flutter 1.22.2 – I will probably look into upgrading again once this makes it to the stable branch.

What if we just hit test the center of the intersection and that's it? Are there any real cases where that would fail and the user would reasonably expect it to succeed?

Unless I'm missing some case, it seems like hit testing the center of the intersection would only fail if the EditableText is more than half way obscured. In reality, if the EditableText is obscured, the user probably would forgive us for not entering their text. For example, if the field is partially (at least halfway) scrolled underneath the AppBar, it would be clear that the user needs to reveal it first to get it to work.

The main cases where this approach wouldn't work would be crazy polygon shaped widgets on top of text fields, which probably isn't vital to get correct in a feature like this.

LongCatIsLooong · 2021-03-15T06:29:23Z

packages/flutter/lib/src/widgets/editable_text.dart

+    final RenderBox? box = context.findRenderObject() as RenderBox?;
+    if (box == null || !mounted || !box.attached)
+      return Rect.zero;
+    final Offset topLeft = box.localToGlobal(Offset.zero);


This may give you the wrong size because only the origin is transformed. The transform can be retrieved using getTransformTo and it should be applied to the rect. There doesn't seem to a shorthand method for this unfortunately.

goderbauer

(not a full review)

goderbauer · 2021-03-15T17:45:07Z

packages/flutter/lib/src/rendering/editable.dart

@@ -62,6 +62,9 @@ enum SelectionChangedCause {
  /// The user used the mouse to change the selection by dragging over a piece
  /// of text.
  drag,
+
+  /// The user iPadOS 14 Scribble to change the selection.


nit: this is missing a verb

Also: the other SelectionChangedCause are expressed more abstract and not necessarily tied to a particular feature or OS. Can we do the same for this? Would it be fair to say that the selection change was triggered, e.g. by "stylus"?

Perhaps, but I don't know the behavior for those – for scribble specifically, this is needed so that we show the selection handles like native the iPadOS UITextField does when using scribble to change the selection...I don't know if that is the case for using a stylus to change the selection on android, for example

goderbauer · 2021-03-15T17:50:44Z

packages/flutter/lib/src/services/text_input.dart

+  void removeTextPlaceholder();
+}
+
+/// An interface for recieving focus during a UIIndirectScribbleInteraction


This shouldn't be tied to a particular platform implementation. E.g. when another platform adds a similar feature we want to reuse these APIs.

(here and elsewhere)

Instead of pointing to a particular implementation the docs should explain what this client is (and then could refer to a particular platform implementation as an example that makes use of this).

Do you have any thoughts on what parts of this would be abstracted out? This is pretty closely tied to the way iPadOS's scribble feature interacts with the UITextInput, so I'm not sure how generic it would be

I think I see what you mean – just keeping the docs at a more abstract level...I think I got all of those and made the docs more generic (e.g. handling x from engine) with the stuff about specific iOS features as a "for example"

goderbauer · 2021-03-15T17:56:33Z

packages/flutter/lib/src/services/text_input.dart

@@ -1425,4 +1524,15 @@ class TextInput {
      shouldSave ,
    );
  }
+
+  /// Registers a [ScribbleClient] with [elementIdentifier] that can be focused using an
+  /// UIIndirectScribbleInteraction on an iPad


nit (here and elsewhere): end this with a .

goderbauer · 2021-03-15T17:58:43Z

packages/flutter/lib/src/widgets/editable_text.dart

+
+  @override
+  void removeTextPlaceholder() {
+    print('[scribble][flutter] removeTextPlaceholder');


nit: please don't forget to remove prints before submitting this.

goderbauer · 2021-03-15T17:59:36Z

packages/flutter/lib/src/widgets/editable_text.dart

@@ -3053,3 +3119,135 @@ class _WhitespaceDirectionalityFormatter extends TextInputFormatter {
    return _ltrRegExp.hasMatch(String.fromCharCode(value)) ? TextDirection.ltr : TextDirection.rtl;
  }
 }
+
+class _ScribbleElement extends StatefulWidget {


nit: Element is a bad suffix for a widget, elements are something else in Flutter: https://master-api.flutter.dev/flutter/widgets/Element-class.html

Would _ScribbleFocusable be a better name? It is certainly more descriptive.

goderbauer · 2021-03-15T18:00:36Z

packages/flutter/lib/src/widgets/editable_text.dart

+  @override
+  void onScribbleFocus(Offset offset) {
+    widget.focusNode.requestFocus();
+    renderEditable?.selectPositionAt(from: offset, cause: SelectionChangedCause.keyboard);


cause: keyboard is odd. Is that correct here?

goderbauer · 2021-03-15T18:08:09Z

packages/flutter/lib/src/widgets/editable_text.dart

+      intersection.bottomCenter,
+      intersection.bottomRight
+    ].any((Offset point) {
+      final HitTestResult result = HitTestResult();


Yeah, sampling "random" points doesn't seem right here.

Also it seems we are doing this hit testing multiple times if there are multiple Scribbles on screen?

Can you describe what the hit test is trying to accomplish?

goderbauer · 2021-03-15T18:09:37Z

packages/flutter/lib/src/widgets/editable_text.dart

+  /// The size of the span, used in place of adding a placeholder size to the [TextPainter]
+  final Size size;
+
+  /// Adds a placeholder box to the paragraph builder if a size has been


no need to repeat the docs of an inherited method.

justinmc

Quick review. This is exciting! Mainly see the comments about extended grapheme clusters.

justinmc · 2021-03-25T21:41:19Z

packages/flutter/lib/src/widgets/editable_text.dart

+      _cachedFirstRect = firstRect;
+      _cachedSize = size;
+      final List<Rect> rects = List<Rect>.generate(
+              text.length, (int i) => renderEditable.getBoxesForSelection(TextSelection(baseOffset: i, extentOffset: i + 1)).first);


As was discussed in #75472 (comment), this will result in TextSelections of partial characters when there are multi-byte characters like 👨‍👩‍👦 or 😆 . I think you should use characters, but you'll probably need a full String and not just a StringBuffer.

Also nit: I think the usual Flutter style is more like this.

final List<Rect> rects = List<Rect>.generate( text.length, (int i) => renderEditable.getBoxesForSelection(TextSelection(baseOffset: i, extentOffset: i + 1)).first, );

@justinmc I did some refactoring to use characters, let me know how that looks. It seems to work well with at least those specific multi-byte characters that you mentioned now.

All right, so I did find one more issue that's pretty strange with characters like 👨‍👩‍👦 that contain zero width joiners. With the following text:

\n
👨‍👩‍👦 1 2 3 4 5 6 7 8

(Note the leading newline) – only "👨‍👩‍👦 1 2 3 4 5 " will be selectable with scribble. However, adding a trailing newline makes the whole line selectable.

I've been poking around various things and trying to compare them to the native UITextView versions of these, and really have no idea what's going on. It really looks like PencilKit basically searches (correctly) positions 1-24 at first, then comes back through and only looks at 1-20 before setting the selected text to 1-19. If you have any thoughts on why that might be, I'm all ears.

All that said, it works perfectly with 😆 in all of my testing, so I think this is limited to ZWJ-containing characters.

@justinmc All right, I think the ZWJ-containing issue is resolved with the latest updates to the engine part of this PR

justinmc · 2021-03-25T21:51:00Z

packages/flutter/lib/src/widgets/editable_text.dart

@@ -2519,6 +2545,24 @@ class EditableTextState extends State<EditableText> with AutomaticKeepAliveClien
    }
  }

+  int _placeholderLocation = -1;


If we do go with -1, can we make sure it's fully explained in a comment somewhere? Like that -1 means there is no placeholder, otherwise it must be between 0 and the length of the text inclusive.

justinmc · 2021-03-29T21:19:50Z

packages/flutter/lib/src/widgets/editable_text.dart

+      intersection.bottomCenter,
+      intersection.bottomRight
+    ].any((Offset point) {
+      final HitTestResult result = HitTestResult();


What if we just hit test the center of the intersection and that's it? Are there any real cases where that would fail and the user would reasonably expect it to succeed?

Unless I'm missing some case, it seems like hit testing the center of the intersection would only fail if the EditableText is more than half way obscured. In reality, if the EditableText is obscured, the user probably would forgive us for not entering their text. For example, if the field is partially (at least halfway) scrolled underneath the AppBar, it would be clear that the user needs to reveal it first to get it to work.

The main cases where this approach wouldn't work would be crazy polygon shaped widgets on top of text fields, which probably isn't vital to get correct in a feature like this.

fbcouch · 2021-06-04T18:52:43Z

@LongCatIsLooong @justinmc @goderbauer I believe I've addressed all of the comments here, let me know if you see anything else! Thanks!

fbcouch · 2021-06-28T15:01:19Z

@LongCatIsLooong @justinmc @goderbauer Just checking in here if there are any more changes needed. It looks like there are some merge conflicts now, so if everything looks good, I can clean those up and hopefully we can get this merged 😄

justinmc

Sorry for the slow response, I'm still really excited for Flutter to support this. No major blockers in my comments, but we should also get @LongCatIsLooong to sign off.

Am I able to try this out using the simulator on a Mac, or do I need an iPad?

justinmc · 2021-06-28T16:50:19Z

packages/flutter/lib/src/services/text_input.dart

+  void showToolbar();
+
+  /// UIKit has requested a text placeholder be added at the current selection
+  void insertTextPlaceholder(Size size);


Maybe add a little bit in these comments about what this is for, something like:

/// For example, this is called when responding to UIKit requesting - /// a text placeholder be added at the current selection. + /// a text placeholder be added at the current selection, such as when + /// requesting additional writing space with iPadOS14 Scribble.

justinmc · 2021-06-28T17:01:20Z

packages/flutter/lib/src/services/text_input.dart

+  /// [bounds].
+  const SelectionRect({required this.position, required this.bounds});
+
+  /// The position of this selection rect within the text.


Can you elaborate exactly what this position is in these docs? Is it the baseOffset of the selection, or is it whichever comes first depending on the TextDirection?

Should this go in the comment? I don't know how it would work with a rtl language (are Traditional Chinese or Cantonese right to left on iOS?), but this is essentially the base offset of a selection that would contain this SelectionRect, so for example, this will be 0 for the first character, 1 for the second, and so on, with some jumps for multi-byte characters, so an 8 byte character at 0 would jump to 8 for the next character.

Ok I see, I maybe just add String to clarify and then it's good: "The position of this selection rect within the text String."

justinmc · 2021-06-28T17:01:56Z

packages/flutter/lib/src/services/text_input.dart

+  /// The position of this selection rect within the text.
+  final int position;
+
+  /// The rectangle representing the bounds of this selection rect.


Is it possible for the selection to be on multiple lines? If so, is this like a bounding box?

No, this is not itself a selection, it just represents a bounding box around a given character within a string of text, so the string "abc" will have selection rects for "a", "b", and "c".

Oh got it, I didn't realize it was only for a single character.

From the code in editable_text.dart, this in the current focused RenderEditable's coordinate space?

justinmc · 2021-06-28T17:11:21Z

packages/flutter/lib/src/services/text_input.dart

+      }
+      return;
+    } else if (method == 'TextInputClient.requestElementsInRect') {
+      final List<dynamic> args = methodCall.arguments as List<dynamic>;


I think you need List.cast.

justinmc · 2021-06-28T17:21:29Z

packages/flutter/lib/src/services/text_input.dart

+  /// Requests that the client show the editing toolbar, for example when the
+  /// platform changes the selection through a non-flutter method such as
+  /// scribble.
+  void showToolbar();


I worry about adding one more way to show the toolbar, since it's already pretty complicated figuring out what's causing it to show or not. It does seem to be necessary in this case though...

justinmc · 2021-06-28T17:22:57Z

packages/flutter/lib/src/widgets/editable_text.dart

+  void _updateSelectionRects({bool force = false}) {
+    if (defaultTargetPlatform != TargetPlatform.iOS)
+      return;
+    // This is to avoid sending selection rects on non-iPad devices


Period at the end.

I guess we have no better way to determine if this is iPadOS... At least we should make 1536.0 a constant called _kIPadWidth or something like that.

justinmc · 2021-06-28T17:27:59Z

packages/flutter/lib/src/widgets/editable_text.dart

+    if (scrollDirection == ScrollDirection.idle && (force || text != _cachedText ||
+        _cachedFirstRect != firstRect || _cachedSize != size ||
+        _cachedPlaceholder != _placeholderLocation)) {


Would it make this any more readable if you split out some variable(s) out of this big conditional?

justinmc · 2021-06-28T17:47:34Z

packages/flutter/test/widgets/editable_text_test.dart

+    expect(textSpan.text, 'Lorem ipsum dolor sit amet');
+  }, skip: kIsWeb);
+
+


Remove this extra indent.

justinmc · 2021-06-28T17:47:55Z

packages/flutter/test/widgets/editable_text_test.dart

+    textSpan = findRenderEditable(tester).text!;
+    expect(textSpan.children, null);
+    expect(textSpan.text, 'Lorem ipsum dolor sit amet');
+  }, skip: kIsWeb);


Should some of these tests be TargetPlatform.iOS only?

fbcouch · 2021-07-01T02:14:18Z

@justinmc Thanks for your comments, I'll take a look through those individually – I have some memory management stuff to clean up on the engine side as well.

Yes, unfortunately there's no simulator support, so you will need a physical iPad with iPadOS 14 and a Pencil.

fbcouch · 2021-07-01T20:50:43Z

@justinmc I think I made all the changes you requested / responded to questions. Let me know if I missed anything

blasten · 2021-07-01T22:59:08Z

@fbcouch Awesome work! I was wondering if there's a design doc that provides an overview of this feature? framework <-> engine interactions, etc...

fbcouch · 2021-07-02T03:07:00Z

@blasten There isn't that I know of, but I can try to put one together if that would be helpful

…n text style changes

…noTextField to allow opting out of the scribble features

fbcouch · 2022-01-11T16:12:51Z

@LongCatIsLooong Thanks! I added a note to that effect, and rebasing onto master fixed the checks!

LongCatIsLooong · 2022-01-11T18:58:15Z

Let's try merging this!

This reverts commit 9490917.

zhoushuangjian001 · 2023-01-31T02:51:41Z

Flutter 3.7 version, ipad device, any page can be triggered, apple pencil's handwriting function

@fbcouch

Enables the Scribe feature, or Android stylus handwriting text input. ![scribe](https://github.com/flutter/flutter/assets/389558/25a54ae9-9399-4772-8482-913ec7a9b330) This PR only implements basic handwriting input. Other features will be done in subsequent PRs: * #155948 * #156018 I created and fixed issue about stylus hovering while working on this: #148810 Original PR for iOS Scribble, the iOS version of this feature: #75472 FYI @fbcouch ~~Depends on flutter/engine#52943 (merged). Fixes #115607 <details> <summary>Example code I'm using to test this feature (but any TextField works)</summary> ```dart import 'package:flutter/material.dart'; import 'package:flutter/scheduler.dart'; import 'package:flutter/services.dart'; void main() { runApp(const MyApp()); } class MyApp extends StatelessWidget { const MyApp({super.key}); @OverRide Widget build(BuildContext context) { return const MaterialApp( home: MyHomePage(), ); } } class MyHomePage extends StatefulWidget { const MyHomePage({super.key}); @OverRide State<MyHomePage> createState() => _MyHomePageState(); } class _MyHomePageState extends State<MyHomePage> { final FocusNode _focusNode1 = FocusNode(); final FocusNode _focusNode2 = FocusNode(); final FocusNode _focusNode3 = FocusNode(); final TextEditingController _controller3 = TextEditingController( text: 'Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.', ); @OverRide Widget build(BuildContext context) { return Scaffold( appBar: AppBar( title: const Text('Scribe demo'), ), body: Center( child: Padding( padding: const EdgeInsets.symmetric(horizontal: 74.0), child: Column( mainAxisAlignment: MainAxisAlignment.center, children: <Widget>[ TextField( focusNode: _focusNode1, autofocus: false, ), TextField( focusNode: _focusNode2, ), TextField( focusNode: _focusNode3, minLines: 4, maxLines: 4, controller: _controller3, ), TextButton( onPressed: () { _focusNode1.unfocus(); _focusNode2.unfocus(); _focusNode3.unfocus(); }, child: const Text('Unfocus'), ), TextButton( onPressed: () { _focusNode1.requestFocus(); SchedulerBinding.instance.addPostFrameCallback((Duration _) { SystemChannels.textInput.invokeMethod('TextInput.hide'); }); }, child: const Text('Focus 1'), ), ], ), ), ), ); } } ``` </details> --------- Co-authored-by: Nate Wilson <[email protected]>

flutter-dashboard bot added f: material design flutter/packages/flutter/material repository. framework flutter/packages/flutter repository. See also f: labels. labels Feb 5, 2021

google-cla bot added the cla: yes label Feb 5, 2021

LongCatIsLooong reviewed Feb 9, 2021

View reviewed changes

LongCatIsLooong reviewed Feb 23, 2021

View reviewed changes

flutter-dashboard bot added a: tests "flutter test", flutter_test, or one of our tests f: cupertino flutter/packages/flutter/cupertino repository labels Feb 26, 2021

LongCatIsLooong reviewed Feb 28, 2021

View reviewed changes

LongCatIsLooong self-requested a review March 12, 2021 07:56

LongCatIsLooong reviewed Mar 15, 2021

View reviewed changes

goderbauer reviewed Mar 15, 2021

View reviewed changes

justinmc reviewed Mar 29, 2021

View reviewed changes

fbcouch force-pushed the feature/scribble+master branch from a0ba019 to 3152493 Compare April 8, 2021 20:30

fbcouch force-pushed the feature/scribble+master branch from 2fab767 to 2f8c117 Compare May 5, 2021 21:35

LongCatIsLooong mentioned this pull request May 5, 2021

[flutter] hacked together global selection example #81849

Closed

fbcouch marked this pull request as ready for review May 6, 2021 01:26

fbcouch mentioned this pull request May 15, 2021

Support Scribble Handwriting flutter/engine#24224

Merged

9 tasks

Piinks added c: new feature Nothing broken; request for a new capability a: text input Entering text in a text field or keyboard related problems labels Jun 10, 2021

justinmc reviewed Jun 28, 2021

View reviewed changes

fbcouch added 5 commits January 11, 2022 09:25

Fix up comments

eef7913

Make new TextInputClient members; invalidate selection rect cache whe…

f494809

…n text style changes

Add scribbleEnabled parameter to EditableText, TextField, and Cuperti…

42d522f

…noTextField to allow opting out of the scribble features

Retry checks

7de1f73

Fix up after rebase

6ccdd12

fbcouch force-pushed the feature/scribble+master branch from 72b5534 to 6ccdd12 Compare January 11, 2022 15:33

Add note that scribbleEnabled only available on iPads

11c2618

LongCatIsLooong added the waiting for tree to go green label Jan 11, 2022

fluttergithubbot merged commit 9490917 into flutter:master Jan 11, 2022

LongCatIsLooong added a commit that referenced this pull request Jan 13, 2022

Revert "Support Scribble Handwriting (#75472)"

e3b68aa

This reverts commit 9490917.

LongCatIsLooong mentioned this pull request Jan 13, 2022

Revert "Support Scribble Handwriting" #96615

Merged

fbcouch mentioned this pull request Jan 13, 2022

Scribble text input client breaking change flutter/website#6679

Merged

3 tasks

engine-flutter-autoroll added a commit to engine-flutter-autoroll/plugins that referenced this pull request Jan 18, 2022

9490917 Support Scribble Handwriting (flutter/flutter#75472)

ff9a2d9

engine-flutter-autoroll added a commit to engine-flutter-autoroll/plugins that referenced this pull request Jan 18, 2022

9490917 Support Scribble Handwriting (flutter/flutter#75472)

4ff85f5

engine-flutter-autoroll added a commit to engine-flutter-autoroll/plugins that referenced this pull request Jan 18, 2022

9490917 Support Scribble Handwriting (flutter/flutter#75472)

e954927

fbcouch mentioned this pull request Jan 19, 2022

Re-land "Support Scribble Handwriting" (#96615) #96881

Merged

8 tasks

fbcouch mentioned this pull request Jan 28, 2022

Re-land "Support Scribble Handwriting" (#96615) #97437

Merged

8 tasks

engine-flutter-autoroll added a commit to engine-flutter-autoroll/plugins that referenced this pull request Feb 4, 2022

9490917 Support Scribble Handwriting (flutter/flutter#75472)

f50fa16

jmagman mentioned this pull request Apr 4, 2022

Provide default method call handler for SystemChannels.textInput #101087

Merged

8 tasks

justinmc mentioned this pull request Apr 15, 2022

Broken link on migration guide flutter/website#7033

Closed

This was referenced May 19, 2022

Scribble mixin #104128

Merged

Unable to focus TextField widget after scrolling #102084

Open

justinmc mentioned this pull request Nov 17, 2022

Support stylus handwriting input on Android #115607

Closed

Jethro87 mentioned this pull request Feb 20, 2023

[SuperEditor] Support Scribble on iOS superlistapp/super_editor#1040

Open

justinmc mentioned this pull request Oct 23, 2024

Scribe Android handwriting text input #148784

Merged

		expect(textSpan.text, 'Lorem ipsum dolor sit amet');
		}, skip: kIsWeb);

Support Scribble Handwriting #75472

Support Scribble Handwriting #75472

Uh oh!

Conversation

fbcouch commented Feb 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pre-launch Checklist

Uh oh!

LongCatIsLooong left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LongCatIsLooong Feb 9, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fbcouch commented Feb 11, 2021

Uh oh!

LongCatIsLooong commented Feb 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fbcouch commented Feb 23, 2021

Uh oh!

LongCatIsLooong left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LongCatIsLooong Mar 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

fbcouch commented Feb 5, 2021 •

edited

Loading

LongCatIsLooong Feb 9, 2021 •

edited

Loading

LongCatIsLooong commented Feb 23, 2021 •

edited

Loading

LongCatIsLooong Mar 1, 2021 •

edited

Loading