Bytedance Terminal Technology — Hou Huayong & Lin Xuebin
The background,
In the process of using Flutter, we often encountered problems related to the keyboard. There were also many problems when we searched the official issue of Flutter using keyboard as the key word. We also encountered and solved some related problems in the process of business development. This article mainly describes the process of Flutter to invoke the soft keyboard to help you understand how the keyboard pops up and provide some solutions to the known keyboard problems.
Ii. Process and principle of Flutter keyboard
Next, this article will introduce the keyboard eject flow, the Flutter page redraw and the page shrink animation, as well as the problems we know.
Figure 2-1 Flutter TextField tunes up the keyboard
View the Flutter source to see the keyboard pop-up flow, using TextField as an example:
Figure 2-2 Flowchart for invoking the keyboard on the Flutter Android terminal
As shown in Figure 2-2, after clicking TextField on the Android terminal, showSoftInput method of InputMethonManager of the system is called through TextInputPlugin to realize the tuning logic of the keyboard. The process on the iOS end is basically similar, which is that FlutterTextInputView instance of UITextInput realizes the keyboard pop-up by calling becomeFirstResponder on the Native end. In Figure 2-1, we can see that the whole page of Flutter moves up after the keyboard is lifted, and the keyboard appears after a fade and pan animation. How is this implemented? The above process is divided into two points:
- The keyboard pop-up animation is triggered by the system and not controlled by the Flutter.
- The Flutter page moves up and the added keyboard starts to trigger a change in the Properties of the FlutterView’s WindowInsets, causing the page to redraw.
2.1. Page redrawing logic after keyboard tuning
Figure 2-1-1 shows the WindowInsets parameter changes and transfer paths after setting the keyboard
The above process looks like a long path, but the logic is not complicated. It can be summarized as the following steps:
- The keyboard pop-up occupies the space of FlutterView, resulting in the change of FlutterView’s WindowInsets property
- When the WindowInsets change, Metrics changes are passed from the Platform thread to the UI thread
- Finally, scheduleForceFrame is called to force the drawing process to be triggered
2.2. Page shrinkage animation
As you can see from Figure 2-1, the change in Metrics causes a refresh of the page and only one frame is drawn. The change is rather hard. Add AnimatedContainer to the outer box of the page Widget. And according to the window.viewinsets. Bottom/window.devicepixelRatio value changes, set different Padding, to achieve a smoother animation effect. The effect is as follows:
Figure 2-2-1 Keyboard animation
Three, keyboard related issues
3.1 Keyboard animation stuck
According to some of our business feedback, the page of some models of mobile phones is seriously stuck in the process of keyboard pop-up. In the following GIF, we can obviously feel that there is some lag in the animation of keyboard pop-up page.
Figure 3-1-1 keyboard animation stuck
We then used Systrace in the same scenario with different phone models for comparison:
FIG. 3-1-2 Systrace diagram of keyboard stuck on Samsung S10
By comparing FIG. 3-1-2 and FIG. 3-1-3, we can intuitively perceive the cause of this problem — normal mobile phones only trigger the page build once at the beginning of animation, while S10 re-triggers every frame. The key now is to find out why the page is triggering the Build operation. To do this, we need to grab the timeline of Flutter in profie mode by using the track-Widget-creation function of Flutter:
As many classes in Figure 3-1-3 involve business logic, the analysis results are directly described here: Except for the part in red box in Figure 3-1-3, it is the real content that needs to be animated, so it is normal for build behavior to occur. Take a close look at the number of child widgets, from top to bottom, all of which contain a content called MediaQuery. A brief introduction to MediaQuery:
Figure 3-1-4 MediaQuere UML Diagram MediaQuery inherits the InheritedWidget, and the InheritedWidget is a class in Flutter that is used to pass data into the widget. The core method is updateShouldNotify, which is used to determine whether related data has changed behavior. The MeidaQuery updateShouldNotify function is as follows:
@override // oldWidget.data is a MediaQueryData bool updateShouldNotify(MediaQuery oldWidget) => data ! = oldWidget.data;Copy the codeMediaQueryData ==
@override bool operator ==(Object other) { if (other.runtimeType ! = runtimeType) return false; return other is MediaQueryData && other.size == size && other.devicePixelRatio == devicePixelRatio && other.textScaleFactor == textScaleFactor && other.platformBrightness == platformBrightness && other.padding == padding && other.viewPadding == viewPadding && other.viewInsets == viewInsets && other.alwaysUse24HourFormat == alwaysUse24HourFormat && other.highContrast == highContrast && other.disableAnimations == disableAnimations && other.invertColors == invertColors && other.accessibleNavigation == accessibleNavigation && other.boldText == boldText && other.navigationMode == navigationMode;Copy the codeDoes the analysis reveal anything at this point? In Chapter 2, we mentioned that “when the keyboard is up, it causes the WindowInset of FlutterView to change”, which happened to change the ViewInsets. MediaQuery’s updateShouldNotify returns true, causing the build behavior of the subtree. Let’s look at what a normal Hello World code looks like:
void main() { runApp(MyApp()); } class MyApp extends StatelessWidget { // This widget is the root of your application. @override Widget build(BuildContext context) { return MaterialApp( title: 'Flutter Demo', theme: ThemeData( primarySwatch: Colors.blue, ), home: MyHomePage(title: 'Flutter Demo Home Page'), ); }}Copy the codeSo we can easily create a widget tree hierarchy:
MyApp
MaterialApp
WidgetsApp
Shortcuts
Actions
FocusTraversalGroup
_MediaQueryFromWindow
MediaQuery
Localizations
...
HomePage
Copy the codeLet’s look at the core function of _MediaQueryFromWindow:
class _MediaQueryFromWindow extends StatefulWidget { const _MediaQueryFromWindow({Key key, this.child}) : super(key: key); final Widget child; @override _MediaQueryFromWindowsState createState() => _MediaQueryFromWindowsState(); } class _MediaQueryFromWindowsState extends State<_MediaQueryFromWindow> with WidgetsBindingObserver { @override void initState() { super.initState(); / / register WidgetsBinding listening WidgetsBinding. Instance. AddObserver (this); } @override void didChangeMetrics() {// When size changes, setState(() {}); } @override Widget build(BuildContext Context) {// Update MediaQueryData MediaQueryData data = MediaQueryData.fromWindow(WidgetsBinding.instance.window); return MediaQuery( data: data, child: widget.child, ); }}Copy the codeFrom the code above, you can see that MediaQueryFromWindows listens for system actions such as Viewport Size, screen brightness, and font Size by listening for WidgetsBinding. This is done by changing MediaQueryData and passing information from the top down through MediaQuery. Have you had any ideas since then? On normal phones, the “WindowInsets” change occurs once while the keyboard is suspended. On the Samsung S10, the “WindowInsets” change occurs multiple times. Here you can see how both systems handle the keyboard bounce animation.
- Normal phone: apply for a space of 400 height once, then make a scene animation by changing the translateY of keyboard View
- Samsung S10: Different height for each application, 0, 10, 40,…. 300, 350, 400 so the animation process
This triggers a change in the Flutter Metirics each time, resulting in a large area of buid behavior. Solution: 1, we add in Flutter Perforamce setCurrentIsKeyboardScene function, when in need keyboard scene, put the switch marked as true, So we will block MediaQuery changes caused by WindowInsets within 300 ms of calling the Show and hide functions of keyboard. 2. For Katton’s Samsung S10 and model, we actively monitored the changes of Metrics. If we received two consecutive changes of Metrics within 32 ms, we changed the AnimatorContaner mentioned in Chapter 3 into Padding. The effect is shown below:
Figure 3-1-5 Systrace diagram of optimized keyboard on Samsung S10
3.2 The keyboard cannot be retracted after the screen is locked
Another problem we encountered was that the screen was locked when the keyboard was in the pop-up state, but the keyboard could not be put away after the screen was unlocked again. The specific problem is shown in the following GIF:
Figure 3-2-1 Input box loses focus after the screen is locked and the keyboard is not retracted. First of all, let’s pay attention to the logic diagram of retracted keyboard. Based on Figure 2-1, we can quickly get the corresponding flow chart:
Figure 3-2-2 Flow chart of hidden keyboard on Flutter Android terminal First we observe that EditText is out of focus after the screen is opened, so _handleFoucusChanged must be called. Anyway we can add logs on key nodes first. The EditText loses focus and triggers a textinput. hide MessageChannel call. Let’s look at TextInputChannel’s onMethodCall method:
public void onMethodCall(@NonNull MethodCall call, @NonNull MethodChannel.Result result) { if (textInputMethodHandler == null) { return; } switch (method) { case "TextInput.hide": textInputMethodHandler.hide(); isKeyBoardShow = false; result.success(null); break; . }}Copy the codeAs you can see from the code above, at some point the textInputMethodHandler is assigned null, causing the current problem.
SomeActivity.onPause()
FlutterView.detachFromFlutterEngine()
TexInputPlugin.destroy
TextInputChannel.setTextInputMethodHandler(null)
Copy the codeNote: detachFromFlutterEngine is used in onPause because of mixed routing and engine reuse.
How to fix it? We record the keyboard ever show, after the TextInputChannel. SetTextInputMethodHandler (null), call the hide to fix the problem.
3.3 Sogou Input method on iOS Long press to send without line breaking
At the same time, the business feedback to us is also some problems in the use of three-way input method, here is sogou input method, when long press enter can not be newline, but added a space behind.
As shown in Figure 3-3-1, the Flutter appears to add a carriage return after the keyboard operation, but after the Flutter is repaired, the behavior of line break is normal. This problem is stable, we can directly write a simple Example debugging, code as follows:
TextField(keyboardType: TextInputType. Multiline, // Must be multiline, otherwise enter also does not work maxLines: 5, minLines: 1, textInputAction: Textinputaction. send, // Display the return key as the send button onChanged: (value) {// Callback for text changes}, onSubmitted: (_) {// Click send button callback}, decoration: const InputDecoration(// HintText: 'input ', filled: true, fillColor: Colors. White, contentPadding: const EdgeInsets. Symmetric (horizontal: 10, vertical: 5), isDense: true, border: const OutlineInputBorder( gapPadding: 0, borderRadius: const BorderRadius.all(Radius.circular(4)), borderSide: BorderSide( width: 1, style: BorderStyle.none, ), ), ), ),Copy the codeWe can just print out each char of the String, so we can just override the onChanged callback:
for( int v in value.codeUnits) {
print('char code is ${v}');
}
Copy the codeWhen we hold down the send button, we get 13. We then comment out textInputAction: textinputAction. send to return to normal carriage return mode, which gives us 10. Then we query the ASCII table and get:
| coding | meaning | Representation in String |
|---|---|---|
| 10 | LF newline, a new line | ‘\n’ |
| 13 | CR returns, usually to the beginning of the current row | ‘\r’ |
This verifies the hypothesis that there is something wrong with the input character. Change is relatively easy, because EditableTextState updateEditgingValue relationship can be modified at the Framework layer, can also be modified in FlutterTextInputPlugin, the character to replace.
3.4 iOS cursor animations cause CPU spikes
In a simple test on iPhone 12 (don’t use Debug mode for Profile mode, performance, etc.), once EditText gets the cursor, the CPU goes up from 4% to 16%.
Figure 3-4-1 CPU usage of iOS cursor animationEditableTextStateIt takes 250ms to animate from alpha 1.0 to 0.0 or 0.0 value 1.0, then 150 ms, then another 250ms, and so on. The initial suspicion was that cursor related drawing was time-consuming. Currently cursor and Text related painting is done in a paint, so as long as the two are separated, the CPU usage is reduced. But after analysis, it was found that this was a logical problem with the refresh of the Flutter animation frame. It is currently feasible to align the cursor animation with the Android side to reduce CPU usage, as shown below:
Figure 3-4-3 CPU usage of iOS cursor animation set to Android mode
3.5 The Cursor Still Exists after the keyboard is folded up on iOS
The cursor appears and flashes when the native input field of iOS is in the input state, and disappears when the input method is withdrawn. The behavior after Flutter was slightly different, with the cursor still flashing after the keyboard was retracted.
As you can see from the image above, the cursor disappears after the user manually folds the virtual keyboard in the native app on iOS. But Flutter still keeps the cursor animated. In and of itself this is not a huge problem, but the existence of the 3.4 problem caused additional CPU consumption and consumed resources without doing anything. Fixed: we on the iOS side of the keyboard put up the action to do the corresponding listening, implementation and native always behavior logic, listening to the notification of the disappearance of the keyboard, cursor processing. Q: What about Android? A: The Android native is still flashing the cursor after the keyboard is folded.
3.6 iOS12+ Long Press The System input method Space cursor is slow and insensitive
After iOS 12, you can use the input method provided by the system to hold down the space to move the cursor quickly. Moving the cursor quickly can effectively help us improve typing efficiency. It is necessary to frequently modify and move the cursor position for editing when typing text on the mobile phone. Moving the cursor can quickly locate the text to be changed, as shown in the following figure
Figure 3-6-1 Long-press selection function of system input method This function has certain defects in Flutter. When non-English characters are entered, the cursor will become stuck and cannot move smoothly.
FIG. 3-6-2 Long-press selection function of Flutter as mentioned above, the overall framework of Flutter text input is implemented based on Native. Then, data synchronization between Flutter end and Native end was carried out through FlutterTextInputPlugin, while keyboard related operations were basically carried out on the Native side and then synchronized to Flutter. This system input method long press selection problem also occurs in many Native implementation of custom input controls, in the official Apple UITextInteraction documentation has such a passage:
PS: UITextInteraction | Apple Developer Documentation and then add a UITextInteraction FlutterTextInputView is normal.
If (@ the available (iOS 13.0, *)) { UITextInteraction* interaction = [UITextInteraction textInteractionForMode:UITextInteractionModeEditable]; interaction.textInput = self; [self addInteraction:interaction]; }Copy the codeGoogle official fix MR: github.com/flutter/eng…
Four,
When you encounter keyboard-related problems with Flutter, it is much easier to identify problems and resolve them by knowing the entire keyboard execution process. The keyboard in Flutter is closely connected with the input function. The essence of the input function of Flutter is to synchronize data between Flutter and Native side through channels with the input ability of Native. Data changes on either side are synchronized to the other side (such as text changes, selections, and cursor movements) and some problems arise during this synchronization process. When the keyboard pops up, the page changes are caused by Metrics changes after the WindowInset changes, and then scheduleForceFrame is called to force the drawing. What we need to do is to analyze the process and code for the different scenarios in which the problem arises. Tools such as Systrace and Instruments can also help us find clues when analyzing the problem. There are some performance related issues that we are constantly exploring when using the keyboard. If you have good ideas, please feel free to raise them.
About the Byte Terminal technology team
Bytedance Client Infrastructure is a global r&d team of big front-end Infrastructure technology (with r&d teams in Beijing, Shanghai, Hangzhou, Shenzhen, Guangzhou, Singapore and Mountain View), responsible for the construction of the whole big front-end Infrastructure of Bytedance. Improve the performance, stability and engineering efficiency of the company’s entire product line; The supported products include but are not limited to Douyin, Toutiao, Watermelon Video, Feishu, Kechedi, etc. We have in-depth research on mobile terminal, Web, Desktop and other terminals.
Now! Client/front-end/server/side intelligent algorithm/test development for global recruitment! Let’s change the world with technology. If you are interested, please contact [email protected], email subject resume – Name – Job intention – Desired city – Phone number.
Bytes to beat application suite MARS is byte to beat terminal technology team in trill, today’s headlines over the past nine years, watermelon video, books, understand car such as emperor App development practice, for mobile research and development, the front-end development, QA, operations, product managers, project managers and operating roles, one-stop research and development of the overall solution, We will help enterprises upgrade their R&D models and reduce their overall r&d costs. Click the link to enter the official website for more product information.