-
Notifications
You must be signed in to change notification settings - Fork 6k
Scribe (Android stylus handwriting text input) #52943
Conversation
(triage): I spoke to @justinmc last week and he says this is still on his radar. |
I have a few P1s keeping me from getting back to this, but it's still a priority 👍 |
sendToBinaryMessageHandler( | ||
binaryMessageHandler, ScribeChannel.METHOD_START_STYLUS_HANDWRITING); | ||
|
||
verify(mockReply).reply(any(ByteBuffer.class)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would have liked to test the exact argument that reply
was called with here, but I was struggling to do anything useful with the ByteBuffer. I was trying StandardMethodCodec.INSTANCE.decodeEnvelope
because it looks like that's what it's encoded with (e.g. here), but I always got an underflow error.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Instead of trying to reverse engineer the reply bytebuffer what if you used a spy on the result callback and verified that either callback.error or result.error was called.
I do not love that the error condition verifies the same thing as the success condition but at least we verify we do not call the methods we are trying to protect.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I changed this to JSONMethodCodec which I believe is what we're supposed to be using for these kinds of method channels now, and incidentally I was able to decode and test the value of result
then.
@@ -62,6 +63,11 @@ public void onMethodCall(@NonNull MethodCall call, @NonNull MethodChannel.Result | |||
|
|||
private void isStylusHandwritingAvailable( | |||
@NonNull MethodCall call, @NonNull MethodChannel.Result result) { | |||
if (Build.VERSION.SDK_INT < API_LEVELS.API_34) { | |||
result.error("error", "Requires API level 34 or higher.", null); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this mean that the dart side of the plugin needs to check if it is running on android and check the api level?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess in practice this probably means that you want to call this wrapped in a try/catch
in Dart. My thought process is that this method channel method is just a proxy for InputMethodManager.isStylusHandwritingAvailable. If I'm not even able to call that, I should error, rather than succeed with false
, which the app developer might interpret to mean that InputMethodManager.isStylusHandwritingAvailable returned false.
If that sounds reasonable then I'll at least make sure this is clearly documented in the framework.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Update here: #52943 (comment)
@NonNull private final View mView; | ||
@NonNull private final ScribeChannel mScribeChannel; | ||
@NonNull private final InputMethodManager mInputMethodManager; | ||
@NonNull public View mView; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you want this to be private or at least public and @VisibleForTesting not public.
Then add a setter for the view. My gut is that over time if the view changes then you want to be informed and be able to setup new callbacks or other events.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Non blocking but I think expecting the dart code to check api level feels like the check is at the wrong layer of abstraction. I think ScribePlugin should consider an additional method "isScribeFeatureAvailable" that checks for 34+ and returns true. I know this is another method and more tests but I think it will make the dart code make more sense and keeps the behavior you want in the boolean methods.
assertNotNull(scribePlugin); | ||
|
||
assertThrows( | ||
NoSuchMethodError.class, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nonblocking: This feels like an odd error to expect.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just confirming that it fails when the API level is too low. I could remove these tests if that's not necessary?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you should keep the test it just seems to me that the error would be something like "feature not available" or "api too low" or something. "No such method". That type of error feels like a programing mistake not an error that tells you what is wrong.
I think I've come around to Reid's approach. Here's what I've done:
I think it's important to match the underlying Android APIs, which I've done with isStylusHandwritingAvailable and startStylusHandwriting, but I'm not opposed to adding in a convenience method where it makes sense. In this case it makes the Dart side much easier. This is what the Dart code would have looked like without isFeatureAvailable: try {
if (!(await Scribe.isStylusHandwritingAvailable() ?? false)) {
// If isStylusHandwritingAvailable returns false then the device's API level
// supports Scribe, but for some other reason it's not able to accept stylus
// input right now.
return;
}
} on PlatformException catch (exception) {
if (exception.message == 'Requires API level 34 or higher.') {
// The device's API level is too low to support Scribe. It's unfortunate
// that we have to rely on matching this string from the engine.
return;
}
// Any other exception is unexpected and should not be caught here.
rethrow;
}
// Scribe is supported, so start it.
Scribe.startStylusHandwriting(); With isFeatureAvailable it looks like this: if (!(await Scribe.isFeatureAvailable() ?? false)) {
// The device doesn't support stylus input right now, or maybe at all.
return;
}
// Scribe is supported, so start it.
Scribe.startStylusHandwriting(); For users that need to use the Android API directly for more nuance, they can use isStylusHandwritingAvailable. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, on board with the addition of isFeatureAvailable
for convenience - it seems like it will make the end use in dart a good bit nicer
I really like the change to the dart code that comes with isFeatureAvailable. @justinmc you rock! |
There was a failure that seems irrelevant and I'm hoping will go away after I push a merge commit.
|
…157295) flutter/engine@de569ba...5eb21d2 2024-10-21 [email protected] [Impeller] Reland: one descriptor pool per frame. (flutter/engine#55960) 2024-10-21 [email protected] Scribe (Android stylus handwriting text input) (flutter/engine#52943) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-engine-flutter-autoroll Please CC [email protected],[email protected] on the revert to ensure that a human is aware of the problem. To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md
Enables the Scribe feature, or Android stylus handwriting text input.  This PR only implements basic handwriting input. Other features will be done in subsequent PRs: * #155948 * #156018 I created and fixed issue about stylus hovering while working on this: #148810 Original PR for iOS Scribble, the iOS version of this feature: #75472 FYI @fbcouch ~~Depends on flutter/engine#52943 (merged). Fixes #115607 <details> <summary>Example code I'm using to test this feature (but any TextField works)</summary> ```dart import 'package:flutter/material.dart'; import 'package:flutter/scheduler.dart'; import 'package:flutter/services.dart'; void main() { runApp(const MyApp()); } class MyApp extends StatelessWidget { const MyApp({super.key}); @OverRide Widget build(BuildContext context) { return const MaterialApp( home: MyHomePage(), ); } } class MyHomePage extends StatefulWidget { const MyHomePage({super.key}); @OverRide State<MyHomePage> createState() => _MyHomePageState(); } class _MyHomePageState extends State<MyHomePage> { final FocusNode _focusNode1 = FocusNode(); final FocusNode _focusNode2 = FocusNode(); final FocusNode _focusNode3 = FocusNode(); final TextEditingController _controller3 = TextEditingController( text: 'Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.', ); @OverRide Widget build(BuildContext context) { return Scaffold( appBar: AppBar( title: const Text('Scribe demo'), ), body: Center( child: Padding( padding: const EdgeInsets.symmetric(horizontal: 74.0), child: Column( mainAxisAlignment: MainAxisAlignment.center, children: <Widget>[ TextField( focusNode: _focusNode1, autofocus: false, ), TextField( focusNode: _focusNode2, ), TextField( focusNode: _focusNode3, minLines: 4, maxLines: 4, controller: _controller3, ), TextButton( onPressed: () { _focusNode1.unfocus(); _focusNode2.unfocus(); _focusNode3.unfocus(); }, child: const Text('Unfocus'), ), TextButton( onPressed: () { _focusNode1.requestFocus(); SchedulerBinding.instance.addPostFrameCallback((Duration _) { SystemChannels.textInput.invokeMethod('TextInput.hide'); }); }, child: const Text('Focus 1'), ), ], ), ), ), ); } } ``` </details> --------- Co-authored-by: Nate Wilson <[email protected]>
The engine API for Android's Scribe stylus handwriting input. Just bare bones handwriting itself, does not support special gestures, which will come in subsequent PR(s).
Enables the Scribe feature, or Android stylus handwriting text input. This is the bare-minimum feature set, while other features have been scoped out and have POCs (see flutter/flutter#155948, flutter/flutter#156018).
Depends on flutter/flutter#148784
Part of flutter/flutter#115607