Add enablers for custom/in-app virtual keyboards #69146
Conversation
|
/cc @justinmc P.S. The combined diff looks a bit intimidating. It might be better to look at the individual commits, which are in logical steps. |
There was a problem hiding this comment.
Just a quick initial review. Thanks for the clean commit history.
Could you walk me through how these changes are used by your virtual keyboard project? I see TextInput being used in keyboard.dart. Just want to make sure I understand how this will be used by an in-app keyboard. We might also want to include an example in the docs, or just some more information, for anyone else that might want to implement something like an in-app keyboard.
CC @gspencergoog for any desktop implications, and tagging @LongCatIsLooong as another reviewer.
There was a problem hiding this comment.
Just a quick initial review.
Thanks!
Could you walk me through how these changes are used by your virtual keyboard project?
Oh, it's not mine, actually. I just found someone's old (unmaintained?) package on pub.dev and used it as a testbed. It would be awesome to port LatinIME from AOSP to Flutter, but I have a feeling it's not as straight-forward as it sounds. :)
I see TextInput being used in keyboard.dart. Just want to make sure I understand how this will be used by an in-app keyboard.
Framework => VKB:
- EditableText calls TextInput.attach() to create a connection, and passes itself as a "client"
- EditableText calls TextInputConnection methods to communicate with the keyboard
- TextInputConnection handles text input related requests from the framework
- when to show or hide the keyboard
- what is the current input configuration/type (text, multiline, numeric, ...)
- the default implementation relays these over the method channel to the platform plugin
- a custom implementation can communicate with a local keyboard widget
VKB => Framework:
- Keyboard calls TextInputClient methods to communicate with the framework
- update the editing value
- perform an action
- ...
The connection factory is the glue between. It gives access to the current TextInputClient, and allows returning a custom TextInputConnection. This way a keyboard can have full control over both ways of the text input communication.
We might also want to include an example in the docs, or just some more information, for anyone else that might want to implement something like an in-app keyboard.
Yeah, I'd imagine covering this topic thoroughly with examples and all that could easily result in a separate dedicated doc page.
|
Thank you for the explanation! Are you actually communicating over the TextInputChannel in your forked virtual keyboard project, or are you using that project in another app? Could you point me to where the usage of the changes in this PR happens? I think you're right that this shouldn't be a breaking change. |
|
Yeah, the experiment is in the text-input-connection-factory branch. Registration:
Communication FW => VKB:
Communication VKB => FW:
The VKB is using a helper model to manage the text input state. It's a Dart port of the TextInputModel used by the Windows & Linux platform shells. |
4a73e0b to
c981686
Compare
| class TextInputConnection { | ||
| TextInputConnection._(this._client) | ||
| abstract class TextInputConnection { | ||
| /// Creates a connection for a [TextInputClient]. |
There was a problem hiding this comment.
Maybe document what the default implementation handles and what it does not, so developers can get an idea which to use, implements or extends.
|
The PR is slowly growing, but I like where we're heading. ;) With the latest version, it's now possible to switch the text input control/factory on the fly.
|
|
Would you prefer to offload some of the preparational steps to a separate PR? |
|
The branch was getting messy and falling behind so I've rebased and cleaned up the commits. Also, I felt like a factory method was no longer appropriate for what we were doing with it. I liked the term "input source" that was used in some comments, so I turned the factory method into a |
|
It's good to hear that things are starting to look good. I'm starting to feel that this is getting a bit out of hand, though. To make it easier to focus on the important parts, I would propose to start with the preparatory |
|
I have revised the docs and added a dummy VKB example snippet. I left out some irrelevant parts to keep it as short as possible. A full runnable version here: https://gist.github.com/jpnurmi/c70eb5226a26db9aa534f07ae58aa8c6 Some slightly more elaborate (but still far from feature complete) examples for inspiration: |
|
/cc @mdebbar |
There was a problem hiding this comment.
Thanks for the contribution, @jpnurmi!
The changes look good to me. One thing to keep in mind though, is that on web, the framework doesn't handle the keyboard events directly. Instead, we let the browser handle them, and we get change events from the browser's input element. See here to find how we ignore key events on the web.
I'm not sure how that would work with custom input sources. I'm assuming we have to allow the framework to handle key events in that case, right?
|
@jpnurmi Could you try out your virtual keyboard example in Flutter web and see what happens? |
|
Thanks for the comments. I'm glad you brought this up. A custom VKB works fine on any platform because it basically communicates directly with After digging a bit deeper, I can see now that the Linux platform plugin maintains its internal |
|
I have a new proposal that I'd like someone to double-check when you have a few spare minutes: jpnurmi@62aebc3 In a nutshell, we could categorize text input calls to state changes vs. visual changes. When a custom input control is registered, we clearly cannot cut off the text input method channel. Text input state updates must continue to be delivered to the platform backend to avoid that the platform plugin goes out of sync. This ensures that physical keyboards managed by the platform plugin and custom virtual keyboards work seamlessly together. Visual changes, however, would only be communicated to the current text input control to make sure only one input control shows up. What do you think? |
|
@jpnurmi I like that proposal. So your virtual keyboard would implement TextInputHandler and register itself via setInputControl? What about unregistering the current _customControl? Also, would there ever be a reason to register multiple _customControls? It seems like all of this could be more generic to support things like that, but there's probably no reason to do that if there is always one _platformControl and optionally one _customControl. |
|
@jpnurmi Are you still available to work on this PR? |
|
@justinmc Yes, I'm still very much interested in having something like this in Flutter! Should we continue exploring the "TextInputControl" proposal?
s/TextInputHandler/TextInputControl/ but basically yes :)
I think we could allow registering multiple input controls, but perhaps only activate one at a time? The currently active input control would be the one that shows up, but the others would also get to process text input events. |
|
Here's a documented version of Only one (visual) text input control can be set at a time, but it's now possible to install multiple (non-visual) text input handlers. The method channel based platform text input handler is always installed and cannot be removed, to ensure that platform text input works and the underlying input model stays in sync. Updated examples:
The old tests still need to be updated. I'll abandon this PR and leave it as a reference, and open a separate one for the new implementation when it's finished. |
|
A new attempt: #76072 |
Description
This PR introduces a new public class
TextInputSourceused to createTextInputConnectioninstances from withinTextInput.attach(). This makes it possible to replace the default method channel -based text input source with a custom in-app virtual keyboard on platforms that do not have a system-provided keyboard.Related Issues
Closes #68988
Tests
I added the following tests:
flutter/test/services/text_input_test.dart.Checklist
Before you create this PR, confirm that it meets all requirements listed below by checking the relevant checkboxes (
[x]). This will ensure a smooth and quick review process.///).flutter analyze --flutter-repo) does not report any problems on my PR.Breaking Change
Did any tests fail when you ran them? Please read Handling breaking changes.