preface

Recently, I encountered a problem of event distribution under the mixed stack when RESEARCHING the rendering of flutter. For this, I took a look at the event distribution process of flutter after it was connected to the native view. In order to facilitate later access, do this record, also hope to help people in need.

The native view discussed in this article is AndroidViewSurface (hybird Composition) and the startup process of Flutter. If you are not familiar with this view, you are advised to browse the following article:

Flutter startup Process analysis on Android platform

What does the Native layer do when Flutter launches on Android?

Analysis of the principle of Flutter — Hybrid Composition Layers

Platform layer: Android

According to the startup process, the first FlutterView created is similar to the container that holds the surface required for flutter drawing, so we can see from its onTouchEvent method.

In fact, based on the positioning of the Flutter framework, we can also guess that the platform is responsible for event distribution.Copy the code

Flutter:onTouchEvent

@Override public boolean onTouchEvent(@NonNull MotionEvent event) { ... Irrelevant code... return androidTouchProcessor.onTouchEvent(event); }Copy the code

Here call androidTouchProcessor. OnTouchEvent (event).

AndroidTouchProcessor: onTouchEvent

public boolean onTouchEvent(@NonNull MotionEvent event) {
  return onTouchEvent(event, IDENTITY_TRANSFORM);
}
Copy the code

This method further calls the function of the same name:

public boolean onTouchEvent(@NonNull MotionEvent event, Matrix transformMatrix) { int pointerCount = event.getPointerCount(); // Prepare a data packet of the appropriate size and order. And passed to flutter side ByteBuffer packet = ByteBuffer. AllocateDirect (POINTER_DATA_FIELD_COUNT pointerCount * * BYTES_PER_FIELD); packet.order(ByteOrder.LITTLE_ENDIAN); / /... Leaving out a lot of code... // Mainly based on the event type, with addPointerForIndex(...) Methods for packet data fill / / to send packet to flutter the renderer. DispatchPointerDataPacket (packet, the packet. The position ()); return true; }Copy the code

Eventually call FlutterRenderer. DispatchPointerDataPacket event passed, we continue to see.

dispatchPointerDataPacket

Here is relatively simple, the renderer dispatchPointerDataPacket method to directly invoke the FlutterJni dispatchPointerDataPacket method:

//FlutterRenderer public void dispatchPointerDataPacket(@NonNull ByteBuffer buffer, int position) { flutterJNI.dispatchPointerDataPacket(buffer, position); } / / FlutterJNI / * * this pointer will be packet to the corresponding method of engine layer: nativeDispatchPointerDataPacket * / @ UiThread public void dispatchPointerDataPacket(@NonNull ByteBuffer buffer, int position) { ensureRunningOnMainThread(); ensureAttachedToNative(); / / call the method that engine layer nativeDispatchPointerDataPacket (nativeShellHolderId, buffer, position); }Copy the code

Event distribution at the platform level is fairly straightforward, and we follow events further down the line.

Middle layer: Engine

Platform Thread

What does the Native layer do when Flutter launches on the Android platform via its inception? Platform_view_android_jni_imp.cc: platform_view_android_jni_imp.cc:

bool RegisterApi(JNIEnv* env) { static const JNINativeMethod flutter_jni_methods[] = { //... {omit part of the registration method. The name = "nativeDispatchPointerDataPacket", / / the android side method name. The signature = "(JLjava/nio ByteBuffer; I) V ", / / android the signature of the method name (parameter types). FnPtr = reinterpret_cast < void * > (& DispatchPointerDataPacket), / / engine layer method pointer}, / /... Omitting partially registered methods}Copy the code

When we call nativeDispatchPointerDataPacket methods in android side, will call the engine DispatchPointerDataPacket method, its implementation is as follows:

static void DispatchPointerDataPacket(JNIEnv* env, jobject jcaller, jlong shell_holder, jobject buffer, // GetDirectBufferAddress = uint8_t* data = static_cast<uint8_t*>(env->GetDirectBufferAddress(buffer)); / / by the position that ultimately make a pointer to the packet auto packet = STD: : make_unique < flutter: : PointerDataPacket > (data, position); / / and then delivers the packet to the platform of view DispatchPointerDataPacket method ANDROID_SHELL_HOLDER->GetPlatformView()->DispatchPointerDataPacket( std::move(packet)); }Copy the code

Android the packet by the above method, passing to the platform view DispatchPointerDataPacket method.

// Holder returns a subclass of PlatformViewAndroid FML ::WeakPtr<PlatformViewAndroid> AndroidShellHolder::GetPlatformView() { FML_DCHECK(platform_view_); return platform_view_; }Copy the code

Although returns are PlatformViewAndroid, but the parent class PlatformViewAndroid DispatchPointerDataPacket method does not require a subclass of rewriting, PlatformViewAndroid does not override this method, so let’s look directly at the implementation of its parent class:

void PlatformView::DispatchPointerDataPacket(
    std::unique_ptr<PointerDataPacket> packet) {
  delegate_.OnPlatformViewDispatchPointerDataPacket(
      pointer_data_packet_converter_.Convert(std::move(packet)));
}
Copy the code

AttachJNI (AndroidShellHolder) {AttachJNI (AndroidShellHolder) {AttachJNI (AndroidShellHolder) {AttachJNI (AndroidShellHolder) {AttachJNI (AndroidShellHolder);

AttachJNI/AndroidShellHolder etc. can be seen aboveCopy the code

AndroidShellHolder::AndroidShellHolder( flutter::Settings settings, std::shared_ptr<PlatformViewAndroidJNI> jni_facade, bool is_background_view) : settings_(std::move(settings)), jni_facade_(jni_facade) { ... // When the shell is initialized, Pass this callback Shell::CreateCallback<PlatformView> on_create_platform_view = [is_background_view, &jni_facade, &weak_platform_view](Shell& shell) { std::unique_ptr<PlatformViewAndroid> platform_view_android; // Here, through the callback, We pass the shell as a delegate PlatformViewAndroid platform_view_android = STD ::make_unique<PlatformViewAndroid>(shell, // delegate shell.GetTaskRunners(), // task runners jni_facade, // JNI interop shell.GetSettings() .enable_software_rendering, // use software rendering ! is_background_view // create onscreen surface ); weak_platform_view = platform_view_android->GetWeakPtr(); auto display = Display(jni_facade->GetDisplayRefreshRate()); shell.OnDisplayUpdates(DisplayUpdateType::kStartup, {display}); return platform_view_android; }; . Omit some code}Copy the code

Ok, by the code above, we know how to delegate instance, then get back to business, to continue to see PlatformView: : DispatchPointerDataPacket:

void PlatformView::DispatchPointerDataPacket(
    std::unique_ptr<PointerDataPacket> packet) {
  delegate_.OnPlatformViewDispatchPointerDataPacket(
      pointer_data_packet_converter_.Convert(std::move(packet)));
}
Copy the code

Its again call Shell OnPlatformViewDispatchPointerDataPacket method:

void Shell::OnPlatformViewDispatchPointerDataPacket( std::unique_ptr<PointerDataPacket> packet) { ... Omit part of the code / / made a Thread here, add a task to the UI Thread, / / and eventually DispatchPointerDataPacket method of execution engine / / note: Task_runners_.GetUITaskRunner()->PostTask(//shell is a subclass of engine, Weak_engine_ FML ::MakeCopyable([engine = Weak_Engine_, packet = STD :: Move (packet), flow_id = next_pointer_flow_id_]() mutable { if (engine) { engine->DispatchPointerDataPacket(std::move(packet), flow_id); }})); next_pointer_flow_id_++; }Copy the code

UI Thread

Engine – > DispatchPointerDataPacket this method a little round, its implementation is as follows:

void Engine::DispatchPointerDataPacket(
    std::unique_ptr<PointerDataPacket> packet,
    uint64_t trace_flow_id) {
  ...
  pointer_data_dispatcher_->DispatchPacket(std::move(packet), trace_flow_id);
}
Copy the code

It looks like it’s calling dispatcher again, but it’s still calling the engine interface method DoDispatchPacket:

void Engine::DoDispatchPacket(std::unique_ptr<PointerDataPacket> packet, uint64_t trace_flow_id) { animator_->EnqueueTraceFlowId(trace_flow_id); if (runtime_controller_) { runtime_controller_->DispatchPointerDataPacket(*packet); }}Copy the code

Enter the RuntimeController: : DispatchPointerDataPacket

bool RuntimeController::DispatchPointerDataPacket( const PointerDataPacket& packet) { if (auto* platform_configuration =  GetPlatformConfigurationIfAvailable()) { ... Other code / / here for the window id 0 and invoke the DispatchPointerDataPacket platform_configuration->get_window(0)->DispatchPointerDataPacket(packet); return true; } return false; }Copy the code

This window, which you will be familiar with if you have read about the source of Flutter (e.g., the rendering process), corresponds to the Window on the flutter side.

void Window::DispatchPointerDataPacket(const PointerDataPacket& packet) { std::shared_ptr<tonic::DartState> dart_state =  library_.dart_state().lock(); if (! dart_state) { return; } tonic::DartState::Scope scope(dart_state); // Const STD ::vector<uint8_t> &buffer = packet.data(); Dart_Handle data_handle = tonic::DartByteData::Create(buffer.data(), buffer.size()); if (Dart_IsError(data_handle)) { return; } // A jNI-like call, Pull up the flutter the tonic "_dispatchPointerDataPacket" method: : LogIfError (tonic: : DartInvokeField (library_. Value (), "_dispatchPointerDataPacket", {data_handle})); }Copy the code

The next shift is to the flutter side.

Flutter layers

If you are not familiar with WidgetsFlutterBinding and its mixins, check out the article on Flutter. There are many.Copy the code

Let’s dive right into the GestureBinding class and see the initialization method:

@override
void initInstances() {
  super.initInstances();
  _instance = this;
  window.onPointerDataPacket = _handlePointerDataPacket;
}
Copy the code

window.onPointerDataPacket

We look at the window. OnPointerDataPacke

set onPointerDataPacket(PointerDataPacketCallback? callback) {
  platformDispatcher.onPointerDataPacket = callback;
}
Copy the code

PlatformDispatcher methods:

PointerDataPacketCallback? get onPointerDataPacket => _onPointerDataPacket;
PointerDataPacketCallback? _onPointerDataPacket;
Zone _onPointerDataPacketZone = Zone.root;
set onPointerDataPacket(PointerDataPacketCallback? callback) {
  _onPointerDataPacket = callback;
  _onPointerDataPacketZone = Zone.current;
}
Copy the code

The _handlePointerDataPacket method is eventually assigned to platformDispatcher’s _onPointerDataPacket, and _onPointerDataPacket is called in this method:

// Called from the engine, via hooks.dart
void _dispatchPointerDataPacket(ByteData packet) {
  if (onPointerDataPacket != null) {
    _invoke1<PointerDataPacket>(
      onPointerDataPacket,
      _onPointerDataPacketZone,
      _unpackPointerDataPacket(packet),
    );
  }
}
Copy the code

As you can see from the comments above, this method is ultimately associated with our call chain in the Engine layer, which is called by the following method:

  tonic::LogIfError(tonic::DartInvokeField(
      library_.value(), "_dispatchPointerDataPacket", {data_handle}));
Copy the code

Of course, the associated nodes are done in the extension.dart file.

@pragma('vm:entry-point')
// ignore: unused_element
void _dispatchPointerDataPacket(ByteData packet) {
  PlatformDispatcher.instance._dispatchPointerDataPacket(packet);
}
Copy the code

Next, _handlePointerDataPacket

_handlePointerDataPacket

This method is bound to the window and used in response to engine’s callback,

void _handlePointerDataPacket(ui.PointerDataPacket packet) { // We convert pointer data to logical pixels so that e.g. the touch slop can be // defined in a device-independent manner. _pendingPointerEvents.addAll(PointerEventConverter.expand(packet.data, window.devicePixelRatio)); if (! locked) _flushPointerEventQueue(); }Copy the code

This article aims to analyze the event distribution process of the native View. In order to avoid getting off topic and reduce the length, the event distribution on the flutter side will be briefly summarized.

A relatively simple chain of internal calls leads to the following method:

Void _handlePointerEventImmediately (PointerEvent event) {/ / first step HitTestResult? hitTestResult; if (event is PointerDownEvent || event is PointerSignalEvent || event is PointerHoverEvent) { assert(! _hitTests.containsKey(event.pointer)); hitTestResult = HitTestResult(); hitTest(hitTestResult, event.position); if (event is PointerDownEvent) { _hitTests[event.pointer] = hitTestResult; } assert(() { if (debugPrintHitTestResults) debugPrint('$event: $hitTestResult'); return true; } ()); } else if (event is PointerUpEvent || event is PointerCancelEvent) { hitTestResult = _hitTests.remove(event.pointer); } else if (event.down) { // Because events that occur with the pointer down (like // [PointerMoveEvent]s) should be dispatched to the same place that their // initial PointerDownEvent was, we want to re-use the path we found when // the pointer went down, rather than do hit detection each time we get // such an event. hitTestResult = _hitTests[event.pointer]; } assert(() { if (debugPrintMouseHoverEvents && event is PointerHoverEvent) debugPrint('$event'); return true; } ()); if (hitTestResult ! = null || event is PointerAddedEvent || event is PointerRemovedEvent) { assert(event.position ! = null); DispatchEvent (event, hitTestResult); }}Copy the code

This method can be roughly divided into two steps, here is a brief summary:

The first step:

Create a root hitTestResult with an internal _path of type List, then call the hitTest method of rootView and then the hitTest method of its child, traversing the entire Render tree.

The 'root hitTestResult' will always be passedCopy the code

Whenever a node is iterated over to render, it is added to _path based on whether its _size contains pointer Event positon.

The second step:

Call the dispatchEvent method, which iterates through the nodes in _path and calls the handleEvent method:

For (final HitTestEntry entry in hittestresult. path) {try {// Target is renderObject, It implements the HitTestTarget interface entry.target.handleEvent(Event.Transformed (entry.transform), entry); } catch (exception, stack) { ... other code } }Copy the code

After the above section, we have an overview of flutter event distribution. Now let’s get back to the basics and see how AndroidViewSurface handles events.

AndroidViewSurface

For more information about how to create and implement “Flutter” layers, please refer to this article: Analysis of the principles of Flutter — Hybrid Composition layers.

AndroidViewSurface inherits from PlatformViewSurface. An important method inside AndroidViewSurface is:

// Create a render object, @override RenderObject createRenderObject(BuildContext Context) {return PlatformViewRenderBox(controller: controller, gestureRecognizers: gestureRecognizers, hitTestBehavior: hitTestBehavior); }Copy the code

Let’s move on to PlatformViewRenderBox:

// Look at its parent class, combine the previous content, Class PlatformViewRenderBox extends RenderBox with _PlatformViewGestureMixin{PlatformViewRenderBox({required) PlatformViewController controller, required PlatformViewHitTestBehavior hitTestBehavior, required Set<Factory<OneSequenceGestureRecognizer>> gestureRecognizers, }) : assert(controller ! = null && controller.viewId ! = null && controller.viewId > -1), assert(hitTestBehavior ! = null), assert(gestureRecognizers ! _controller = controller {this.hitTestBehavior = hitTestBehavior; UpdateGestureRecognizers (gestureRecognizers) }... non-relative code }Copy the code

As you can see, the dispatchPointerEvent method is taken from _Controller (AndroidViewController).

void updateGestureRecognizers(Set<Factory<OneSequenceGestureRecognizer>> gestureRecognizers) {
  _updateGestureRecognizersWithCallBack(gestureRecognizers, _controller.dispatchPointerEvent);
}
Copy the code

Let’s take a look at _controller. DispatchPointerEvent implementation:

@override Future<void> dispatchPointerEvent(PointerEvent event) async { if (event is PointerHoverEvent) { return; } if (event is PointerDownEvent) { _motionEventConverter.handlePointerDownEvent(event); } _motionEventConverter.updatePointerPositions(event); // Convert a flutter event to an Android event final AndroidMotionEvent? androidEvent = _motionEventConverter.toAndroidMotionEvent(event); if (event is PointerUpEvent) { _motionEventConverter.handlePointerUpEvent(event); } else if (event is PointerCancelEvent) { _motionEventConverter.handlePointerCancelEvent(event); } if (androidEvent ! = null) {// send the converted event await sendMotionEvent(androidEvent); }}Copy the code

Then go to the sendMotionEvent(androidEvent) method:

Future<void> sendMotionEvent(AndroidMotionEvent event) async {
  await SystemChannels.platform_views.invokeMethod<dynamic>(
    'touch',
    event._asList(viewId),
  );
}
Copy the code

(○´ · д ·) Blue, here sends the click event back to the Android…… Without further ado, this event will eventually be consumed by the corresponding native view.

conclusion

When I first saw this, I was quite puzzled, thinking that the event had gone a long way around, but then I thought that Flutter is the main consumption area of the upper layer, and if it is skipped, it may cause the issue of event error.

However, I am still wondering if the event consumption problem in the blending layer can be optimized in the Engine layer. After all, blending layer rendering is optimized in Engine.

At this point, the process to comb has been completed, thank you for reading, if there is any mistake welcome to point out.

mo4tech.com (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.

Flutter – the Touch event distribution process of native View

preface

Platform layer: Android

Flutter:onTouchEvent

AndroidTouchProcessor: onTouchEvent

dispatchPointerDataPacket

Middle layer: Engine

Platform Thread

UI Thread

Flutter layers

window.onPointerDataPacket

_handlePointerDataPacket

AndroidViewSurface

conclusion

Other Flutter related articles

Flutter – the Touch event distribution process of native View

preface

Platform layer: Android

Flutter:onTouchEvent

AndroidTouchProcessor: onTouchEvent

dispatchPointerDataPacket

Middle layer: Engine

Platform Thread

UI Thread

Flutter layers

window.onPointerDataPacket

_handlePointerDataPacket

AndroidViewSurface

conclusion

Other Flutter related articles

Related Posts

Android screen adaptation has never been easier (deprecated)

Window Activity View ViewRootImpl

BottomSheetBehavior Bottom popover Bluetooth sharing