Thanks to visit codestin.com
Credit goes to github.com

Skip to content
This repository was archived by the owner on Feb 25, 2025. It is now read-only.

Scribe (Android stylus handwriting text input) #52943

Merged
merged 50 commits into from
Oct 21, 2024
Merged

Conversation

justinmc
Copy link
Contributor

@justinmc justinmc commented May 20, 2024

Enables the Scribe feature, or Android stylus handwriting text input. This is the bare-minimum feature set, while other features have been scoped out and have POCs (see flutter/flutter#155948, flutter/flutter#156018).

Depends on flutter/flutter#148784

Part of flutter/flutter#115607

scribe

@goderbauer
Copy link
Member

(triage): I spoke to @justinmc last week and he says this is still on his radar.

@justinmc
Copy link
Contributor Author

I have a few P1s keeping me from getting back to this, but it's still a priority 👍

@justinmc justinmc mentioned this pull request Oct 1, 2024
sendToBinaryMessageHandler(
binaryMessageHandler, ScribeChannel.METHOD_START_STYLUS_HANDWRITING);

verify(mockReply).reply(any(ByteBuffer.class));
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would have liked to test the exact argument that reply was called with here, but I was struggling to do anything useful with the ByteBuffer. I was trying StandardMethodCodec.INSTANCE.decodeEnvelope because it looks like that's what it's encoded with (e.g. here), but I always got an underflow error.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of trying to reverse engineer the reply bytebuffer what if you used a spy on the result callback and verified that either callback.error or result.error was called.

I do not love that the error condition verifies the same thing as the success condition but at least we verify we do not call the methods we are trying to protect.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed this to JSONMethodCodec which I believe is what we're supposed to be using for these kinds of method channels now, and incidentally I was able to decode and test the value of result then.

@justinmc justinmc requested a review from reidbaker October 4, 2024 23:43
@@ -62,6 +63,11 @@ public void onMethodCall(@NonNull MethodCall call, @NonNull MethodChannel.Result

private void isStylusHandwritingAvailable(
@NonNull MethodCall call, @NonNull MethodChannel.Result result) {
if (Build.VERSION.SDK_INT < API_LEVELS.API_34) {
result.error("error", "Requires API level 34 or higher.", null);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this mean that the dart side of the plugin needs to check if it is running on android and check the api level?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess in practice this probably means that you want to call this wrapped in a try/catch in Dart. My thought process is that this method channel method is just a proxy for InputMethodManager.isStylusHandwritingAvailable. If I'm not even able to call that, I should error, rather than succeed with false, which the app developer might interpret to mean that InputMethodManager.isStylusHandwritingAvailable returned false.

If that sounds reasonable then I'll at least make sure this is clearly documented in the framework.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update here: #52943 (comment)

@NonNull private final View mView;
@NonNull private final ScribeChannel mScribeChannel;
@NonNull private final InputMethodManager mInputMethodManager;
@NonNull public View mView;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you want this to be private or at least public and @VisibleForTesting not public.
Then add a setter for the view. My gut is that over time if the view changes then you want to be informed and be able to setup new callbacks or other events.

Copy link
Contributor

@reidbaker reidbaker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non blocking but I think expecting the dart code to check api level feels like the check is at the wrong layer of abstraction. I think ScribePlugin should consider an additional method "isScribeFeatureAvailable" that checks for 34+ and returns true. I know this is another method and more tests but I think it will make the dart code make more sense and keeps the behavior you want in the boolean methods.

assertNotNull(scribePlugin);

assertThrows(
NoSuchMethodError.class,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nonblocking: This feels like an odd error to expect.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just confirming that it fails when the API level is too low. I could remove these tests if that's not necessary?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you should keep the test it just seems to me that the error would be something like "feature not available" or "api too low" or something. "No such method". That type of error feels like a programing mistake not an error that tells you what is wrong.

@justinmc
Copy link
Contributor Author

I think I've come around to Reid's approach. Here's what I've done:

  • Added isFeatureAvailable. I left off the "Scribe" because it felt redundant.
  • Made it call over to isStylusHandwritingAvailable, so from the Dart side you only need to call one method.

I think it's important to match the underlying Android APIs, which I've done with isStylusHandwritingAvailable and startStylusHandwriting, but I'm not opposed to adding in a convenience method where it makes sense. In this case it makes the Dart side much easier.

This is what the Dart code would have looked like without isFeatureAvailable:

try {
  if (!(await Scribe.isStylusHandwritingAvailable() ?? false)) {
    // If isStylusHandwritingAvailable returns false then the device's API level
    // supports Scribe, but for some other reason it's not able to accept stylus
    // input right now.
    return;
  }
} on PlatformException catch (exception) {
  if (exception.message == 'Requires API level 34 or higher.') {
    // The device's API level is too low to support Scribe. It's unfortunate
    // that we have to rely on matching this string from the engine.
    return;
  }
  // Any other exception is unexpected and should not be caught here.
  rethrow;
}

// Scribe is supported, so start it.
Scribe.startStylusHandwriting();

With isFeatureAvailable it looks like this:

if (!(await Scribe.isFeatureAvailable() ?? false)) {
  // The device doesn't support stylus input right now, or maybe at all.
  return;
}

// Scribe is supported, so start it.
Scribe.startStylusHandwriting();

For users that need to use the Android API directly for more nuance, they can use isStylusHandwritingAvailable.

Copy link
Member

@gmackall gmackall left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, on board with the addition of isFeatureAvailable for convenience - it seems like it will make the end use in dart a good bit nicer

@reidbaker
Copy link
Contributor

I really like the change to the dart code that comes with isFeatureAvailable. @justinmc you rock!
#52943 (comment)

@justinmc
Copy link
Contributor Author

There was a failure that seems irrelevant and I'm hoping will go away after I push a merge commit.

[ +189 ms] org-dartlang-app:///main.dart:1:1: Error: The specified language version is too high. The highest supported language version is 3.6.
[        ] // @dart = 3.7

@justinmc justinmc merged commit a489914 into flutter:main Oct 21, 2024
30 checks passed
@justinmc justinmc deleted the scribe branch October 21, 2024 18:55
engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Oct 21, 2024
auto-submit bot pushed a commit to flutter/flutter that referenced this pull request Oct 21, 2024
…157295)

flutter/engine@de569ba...5eb21d2

2024-10-21 [email protected] [Impeller] Reland: one descriptor pool per frame. (flutter/engine#55960)
2024-10-21 [email protected] Scribe (Android stylus handwriting text input) (flutter/engine#52943)

If this roll has caused a breakage, revert this CL and stop the roller
using the controls here:
https://autoroll.skia.org/r/flutter-engine-flutter-autoroll
Please CC [email protected],[email protected] on the revert to ensure that a human
is aware of the problem.

To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose

To report a problem with the AutoRoller itself, please file a bug:
https://issues.skia.org/issues/new?component=1389291&template=1850622

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md
github-merge-queue bot pushed a commit to flutter/flutter that referenced this pull request Nov 21, 2024
Enables the Scribe feature, or Android stylus handwriting text input.


![scribe](https://github.com/flutter/flutter/assets/389558/25a54ae9-9399-4772-8482-913ec7a9b330)

This PR only implements basic handwriting input. Other features will be
done in subsequent PRs:

 * #155948
 * #156018

I created and fixed issue about stylus hovering while working on this:
#148810

Original PR for iOS Scribble, the iOS version of this feature:
#75472
FYI @fbcouch 

~~Depends on flutter/engine#52943 (merged).

Fixes #115607

<details>

<summary>Example code I'm using to test this feature (but any TextField
works)</summary>

```dart
import 'package:flutter/material.dart';
import 'package:flutter/scheduler.dart';
import 'package:flutter/services.dart';

void main() {
  runApp(const MyApp());
}

class MyApp extends StatelessWidget {
  const MyApp({super.key});

  @OverRide
  Widget build(BuildContext context) {
    return const MaterialApp(
      home: MyHomePage(),
    );
  }
}

class MyHomePage extends StatefulWidget {
  const MyHomePage({super.key});

  @OverRide
  State<MyHomePage> createState() => _MyHomePageState();
}

class _MyHomePageState extends State<MyHomePage> {
  final FocusNode _focusNode1 = FocusNode();
  final FocusNode _focusNode2 = FocusNode();
  final FocusNode _focusNode3 = FocusNode();
  final TextEditingController _controller3 = TextEditingController(
    text: 'Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.',
  );

  @OverRide
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: const Text('Scribe demo'),
      ),
      body: Center(
        child: Padding(
          padding: const EdgeInsets.symmetric(horizontal: 74.0),
          child: Column(
            mainAxisAlignment: MainAxisAlignment.center,
            children: <Widget>[
              TextField(
                focusNode: _focusNode1,
                autofocus: false,
              ),
              TextField(
                focusNode: _focusNode2,
              ),
              TextField(
                focusNode: _focusNode3,
                minLines: 4,
                maxLines: 4,
                controller: _controller3,
              ),
              TextButton(
                onPressed: () {
                  _focusNode1.unfocus();
                  _focusNode2.unfocus();
                  _focusNode3.unfocus();
                },
                child: const Text('Unfocus'),
              ),
              TextButton(
                onPressed: () {
                  _focusNode1.requestFocus();
                  SchedulerBinding.instance.addPostFrameCallback((Duration _) {
                    SystemChannels.textInput.invokeMethod('TextInput.hide');
                  });
                },
                child: const Text('Focus 1'),
              ),
            ],
          ),
        ),
      ),
    );
  }
}
```

</details>

---------

Co-authored-by: Nate Wilson <[email protected]>
nick9822 pushed a commit to nick9822/flutter that referenced this pull request Dec 18, 2024
The engine API for Android's Scribe stylus handwriting input. Just bare bones handwriting itself, does not support special gestures, which will come in subsequent PR(s).
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants