Introduction:
After release the app developers the most headache problem is how to solve the problem of the user side of the delivery after reduction and positioning, is the lack of a set of system solution of blank areas, idle fish technical team with his business pain points put forward a new set of technical ideas to solve this problem and in the online satisfactory practical results have been achieved.
We capture the flow of UI events and business data through the underlying system, and use the captured data to reproduce problems online through event playback mechanisms. This article first introduces the overall framework for recording and playback, and then introduces the three key technical points involved, which are the most complex technologies here (simulated touch events, unified interceptor implementation, unified Hook block).
background
Nowadays, apps generally provide an entrance for users to feedback problems. However, there are two ways to provide users with feedback problems:
-
Direct text input expression, or screenshots
-
Record video feedback directly
These two types of feedback often lead to the following complaints:
-
User: Input text is time-consuming and laborious
-
Dev 1: Can’t understand what user feedback is saying?
-
Development 2: Basically understand what the user said, but I can’t reproduce offline
-
Development 3: I watched the video recorded by the user, but I could not reproduce it offline, nor locate the problem
Therefore: In order to solve the above problems, we use a set of new ideas to design the online problem replay system
Significance of online problem replay system
-
Users do not need to input text feedback, just need to re-operate the app to reproduce the problem steps
-
The developer gets the script of the problem feedback from the user and plays it back offline to see the problem at a glance, just like recording a video. Yes, you read that right, just like watching a video.
-
Real-time access to app runtime data (local data, network data, stack, etc.) through script playback for troubleshooting
-
Open the door to automated testing — you get the idea
Effect of video
play
The technology principle
1. The relationship between APP and external environment
As can be seen from the above diagram, the operation of the whole app is nothing more than user UI operation, and then trigger the APP to obtain data from the outside world, including network data, GPS data, etc., as well as local data from the phone, such as album data, machine data, system data, etc. Therefore, to achieve problem playback, we only need to record user UI operations and external data, as well as app’s own data.
App recording = USER UI operations + external data (inside and outside the phone) + app data
2. The online problem replay architecture consists of two parts: recording and replay
Recording is for playback. The more detailed the recorded information is, the higher the playback success rate is and the easier it is to locate problems
Recording is actually to record the UI and data, playback is actually app automatic
Drive UI operations and stuff recorded data back into place.
Copy the code3. Record the architecture diagram
The recording process
4. Play back the architecture diagram
Playback process:
The playback flow chart is omitted here
1. Start app and click the playback button 2. Engine loads playback script 3. 5. Start the player and read one event after another from the consumption queue to play. If it is a UI event, play it directly; if it is a static data event, replace the data value directly according to the instruction requirements. If it is a non-UI runtime event, the event instruction rules will determine whether to play the event actively or wait for the corresponding event to be intercepted. If the corresponding event needs to be intercepted, the player will wait for this event until the event is consumed by the APP. Only if this event is consumed can the player play the next event. 6. After intercepting the registered event, the corresponding data will be inserted into the corresponding field according to the requirements of the event instruction 7. Jump back to 6 and continue running until the event in the consumption queue is consumed
Note: The corresponding stack information and event data will be automatically printed in real time when each event is played back, which is conducive to troubleshooting
Introduction to key technologies
1. Simulate touch events
Parse out the view being touched and the hierarchical relationship in the view tree where the view is located from the UI event data, find the corresponding view on the current playback interface, and then send UI operation events (click, double click, etc.) to the view, and bring the coordinate information of the touch event. In fact, this is the simulated touch event.
Let’s start with the process of handling touch events
Waiting for touch stage
-
The phone screen is in standby mode, waiting for the touch event to occur
-
Fingers start touching the screen
System response stage
-
The on-screen sensor picks up the touch and passes the touch data to IOKit.
-
IOKit encapsulates the touch event as an IOHIDEvent object
-
IOKit then forwards the IOHIDEvent object to the SpringBoard process
The SpringBoard process is the iOS system desktop. It exists in the iDevice process and cannot be cleared. Its operation principle is similar to that of explorer. It is responsible for interface management, so only it knows who is responding to the current touch.
SpringBoard receiving phase
-
SpringBoard after receiving IOHIDEvent news, triggering the runloop Source1 callback __IOHIDEventSystemClientQueueCallback () method.
-
SpringBoard starts to check if there is a running app in the foreground. If so, SpringBoard forwards the touch event to the current app in the foreground via process communication. If not, SpringBoard enters its own internal response process.
App processing phase
-
The foreground app main thread Runloop received a message from SpringBoard forward, and triggers the corresponding to the Runloop Source1 callback _UIApplicationHandleEventQueue ().
-
_UIApplicationHandleEventQueue () wrap IOHIDEvent processing as a UIEvent processing distribution
-
Soucre0 calls back to the sendEvent: method of the internal UIApplication, passing UIEvent to UIWindow
-
HitTest (_:with:) and point(inside:with:) are used to recursively find the view that responds to the touch event in the whole view tree with UIWindow as the root node.
-
Once the final leaf node view is found, events bound to that view, such as jump pages, are triggered.
From the touch event processing process above, we can see that to record UI events, we only need to capture the touch data at the UIApplication sendEvent method in the app processing stage, which is also where the touch is simulated back during playback.
The key point here is to record the timestamp of touch.timestamp and the time between the current touch event and the last touch event, Because this involves touch induced inertial acceleration. For example, when we normally swipe a list view, the list view will slide for a short period of time after our finger leaves the screen.
- (void)handleUIEvent:(UIEvent *)event
{
if (! self.isEnabled) return;
if (event.type ! = UIEventTypeTouches) return;
NSSet *allTouches = [event allTouches];
UITouch *touch = (UITouch *)[allTouches anyObject];
if (touch.view) {
if (self.filter && ! self.filter(touch.view)) {
return;
}
}
switch (touch.phase) {
case UITouchPhaseBegan:
{
self.machAbsoluteTime = mach_absolute_time();
self.systemStartUptime = touch.timestamp;
self.tuochArray = [NSMutableArray array];
[self recordTouch:touch click:self.machAbsoluteTime];
break;
}
case UITouchPhaseStationary:
{
[self recordTouch:touch click:mach_absolute_time()];
break;
}
case UITouchPhaseCancelled:
{
[self recordTouch:touch click:mach_absolute_time()];
[[NSNotificationCenter defaultCenter] postNotificationName:@"notice_ui_test" object:self.tuochArray];
break;
}
case UITouchPhaseEnded:
{
[self recordTouch:touch click:mach_absolute_time()];
[[NSNotificationCenter defaultCenter] postNotificationName:@"notice_ui_test" object:self.tuochArray];
break;
}
case UITouchPhaseMoved:
{
[self recordTouch:touch click:mach_absolute_time()];
}
default:
break;
}
}
Copy the codeLet’s take a look at how the code emulates the click-touch event (I’ve removed some of the non-critical and complex code for ease of understanding).
Next, let’s look at the simulated touch event code. A basic touch event generally consists of three parts:
-
1.UITouch object – will be used for touch
-
2. The first UIEvent Began touching
-
3. The second UIEvent Ended touch
Implementation steps:
-
1. In the front part of the code, there are some private interfaces of UITouch and UIEvent, as well as private variable fields. Since Apple does not disclose them, we need to include these fields in order to make it compile without error.
-
2. Construct touch objects: UITouch and UIEvent, and insert the corresponding field value of the record back into the corresponding field. You plug it back in with private interfaces and private fields
-
[[UIApplication sharedApplication] sendEvent:event]; [UIApplication sharedApplication] sendEvent: Event]
-
4. To play back these touch events, we need to drop them into CADisplayLink
//
// SimulationTouch.m
//
// Created by Jg on 2018/5/15
//
#import "SimulationTouch.h"
#import <objc/runtime.h>
#include <mach/mach_time.h>
@implementation UITouch (replay)
- (id)initPoint:(CGPoint)point window:(UIWindow *)window
{
NSParameterAssert(window);
self = [super init];
if (self) {
[self setTapCount:1];
[self setIsTap:YES];
[self setPhase:UITouchPhaseBegan];
[self setWindow:window];
[self _setLocationInWindow:point resetPrevious:YES];
[self setView:[window hitTest:point withEvent:nil]];
[self _setIsFirstTouchForView:YES];
[self setTimestamp:[[NSProcessInfo processInfo] systemUptime]];
}
return self;
}
@end
@interface UIInternalEvent : UIEvent
- (void)_setHIDEvent:(IOHIDEventRef)event;
@end
@interface UITouchesEvent : UIInternalEvent
- (void)_addTouch:(UITouch *)touch forDelayedDelivery:(BOOL)delayedDelivery;
- (void)_clearTouches;
@end
typedef enum {
kIOHIDDigitizerEventRange = 0x00000001,
kIOHIDDigitizerEventTouch = 0x00000002,
kIOHIDDigitizerEventPosition = 0x00000004,
} IOHIDDigitizerEventMask;
IOHIDEventRef IOHIDEventCreateDigitizerFingerEvent(CFAllocatorRef allocator,
AbsoluteTime timeStamp,
uint32_t index,
uint32_t identity,
IOHIDDigitizerEventMask eventMask,
IOHIDFloat x,
IOHIDFloat y,
IOHIDFloat z,
IOHIDFloat tipPressure,
IOHIDFloat twist,
Boolean range,
Boolean touch,
IOOptionBits options);
@implementation SimulationTouch
- (void)performTouchInView:(UIView *)view start:(bool)start
{
UIWindow *_window = view.window;
CGRect fInWindow;
if ([view isKindOfClass:[UIWindow class]])
{
fInWindow = view.frame;
}
else
{
fInWindow = [_window convertRect:view.frame fromView:view.superview];
}
CGPoint point =
CGPointMake(fInWindow.origin.x + fInWindow.size.width/2,
fInWindow.origin.y + fInWindow.size.height/2);
if(start)
{
self.touch = [[UITouch alloc] initPoint:point window:_window];
[self.touch setPhase:UITouchPhaseBegan];
}
else
{
[self.touch _setLocationInWindow:point resetPrevious:NO];
[self.touch setPhase:UITouchPhaseEnded];
}
CGPoint currentTouchLocation = point;
UITouchesEvent *event = [[UIApplication sharedApplication] _touchesEvent];
[event _clearTouches];
uint64_t machAbsoluteTime = mach_absolute_time();
AbsoluteTime timeStamp;
timeStamp.hi = (UInt32)(machAbsoluteTime >> 32);
timeStamp.lo = (UInt32)(machAbsoluteTime);
[self.touch setTimestamp:[[NSProcessInfo processInfo] systemUptime]];
IOHIDDigitizerEventMask eventMask = (self.touch.phase == UITouchPhaseMoved)
? kIOHIDDigitizerEventPosition
: (kIOHIDDigitizerEventRange | kIOHIDDigitizerEventTouch);
Boolean isRangeAndTouch = (self.touch.phase ! = UITouchPhaseEnded);
IOHIDEventRef hidEvent = IOHIDEventCreateDigitizerFingerEvent(kCFAllocatorDefault,
timeStamp,
0,
2,
eventMask,
currentTouchLocation.x,
currentTouchLocation.y,
0,
0,
0,
isRangeAndTouch,
isRangeAndTouch,
0);
if ([self.touch respondsToSelector:@selector(_setHidEvent:)]) {
[self.touch _setHidEvent:hidEvent];
}
[event _setHIDEvent:hidEvent];
[event _addTouch:self.touch forDelayedDelivery:NO];
[[UIApplication sharedApplication] sendEvent:event];
}
@end
Copy the codeHow to call the private interface, and use which private interfaces that don’t need to explain again, if you are interested, please pay attention to our public, the subsequent I write an article to expose this aspect of technology, in general is download apple with touch events source library, analyze the source code, and then set the broken debugging, even the disassembly to understand the principle of touch events.
2. Unified interceptors
Recording and playback are handled by the event stream, and the data event stream is actually a hook to some key methods. Because we want to ensure that the business code is not intrusive and extensible (registration of events), we need to unify the hook for all methods, all methods are handled by the same hook. As shown in the figure below
This hook is written to use assembler, because the assembly code is more, and difficult to read, so here temporarily not include source code, compiled layer data read out the inside of the hardware of number, such as general purpose register data and floating point register data, stack, and so on, even in front of the method parameters can be read out in front of, Finally, it is forwarded to the C language layer for processing.
After assembling hardware related information, the assembly layer calls the UNIFIED interception interface of the C layer. The assembly layer serves the C layer. The C layer cannot read hardware information, so assembly is used here. The interface of layer C locates the event to which the current method belongs through hardware related information. Knowing the event also means knowing the event instruction, the event instruction, the fields to be plugged back, and the original method to be hooked.
C layer code is described as follows: since the interceptor is called uniformly, the interceptor does not know which business code is executing the current business method, nor how many arguments the current business method has, what is the type of each argument, etc., the interface code processing is roughly as follows
-
Get the object self through the register
-
Get method SEL through register
-
Get the corresponding event instruction through self and sel
-
An event command calls back to the upper layer to decide whether to proceed
-
Gets the data to play back the event
-
To stuff data back, such as into a register, or into a field of an object to which a register points, etc
-
If immediate playback is required, the original hook method is called. If immediate playback is not required, the live information needs to be saved and wait for the appropriate time to play by the playback queue (call).
//xRegs indicates that the unified assembler passes in all the current generic register data, and their addresses are stored in an array pointer
//dRegs means that the unified assembler passes in all the current floating-point register data, and their addresses are stored in an array pointer
//dRegs indicates that the unified assembler passes the current stack pointer
//fp represents the call stack frame pointer
void replay_entry_start(void* xRegs, void* dRegs, void* spReg, CallBackRetIns *retIns,StackFrame *fp, void *con_stub_lp)
{
void *objAdr = (((void **)xRegs)[0]); // Get the object itself self or block object itself
EngineManager *manager = [EngineManager sharedInstance];
ReplayEventIns *node = [manager getEventInsWithBlock:objAdr];
id obj = (__bridge id)objAdr;
void *xrArg = ((void **)xRegs)+2;
if(nil == node)
{
SEL selecter = (SEL)(((void **)xRegs)[1]); // The method called by the corresponding object
Class tclass = [obj class]; //object_getClass(obj); The object_getClass method can only get its class from an object. It cannot pass in the class and return the class itself.
do
{
node = [manager getEventIns:tclass sel:selecter]; // Get the corresponding event instruction node by object and method
}while(nil == node && (tclass = class_getSuperclass(tclass)));
}
else
{
xrArg = ((void **)xRegs)+1;
}
assert(node && "node is nil in replay_call_start");
// The callback notifies the upper layer whether the current playback is interrupted
if(node.BreakCurReplayExe && node.BreakCurReplayExe(obj,node,xrArg,dRegs))
{
retIns->nodeAddr = NULL;
retIns->recordOrReplayData = NULL;
retIns->return_address = NULL;
return;
}
bool needReplay = true;
// Callback notifies the upper layer that the event is currently about to be played back
if(node.willReplay)
{
needReplay = (*(node.willReplay))(obj,node,xrArg,dRegs);
}
if(needReplay)
{
ReplayEventData *replayData = nil;
if(node.getReplayData)
{
// Get the data corresponding to the playback event
replayData = (*(node.getReplayData))(obj,node,xrArg,dRegs);
}
Else // Default method
{
replayData = [manager getNextReplayEventData:node];
}
// This is the real playback, that is, the data is inserted back, and the original hook method is called
if(replayData)
{
if(replay_type_intercept_call == node.replayType)
{
sstuffArg(xRegs,dRegs,spReg,node,replayData.orgDic);
NSArray *arglist = fetchAllArgInReplay(xRegs, dRegs, spReg, node);
ReplayInvocation *funobj = [[ReplayInvocation alloc] initWithFunPtr:node.callBack ? node.callBack : [node getOrgFun]
args:arglist
argType:[node getFunTypeStr]
retType:rf_return_type_v];
if([[EngineManager sharedInstance] setRepalyEventReady:replayData funObj:funobj])
{
// Put it in the queue, return the uncalled address, so that it does not go down
retIns->return_address = NULL;
return ;
}
}
else
{
/ / data
sstuffArg(xRegs,dRegs,spReg,node,replayData.orgDic);
}
}
retIns->nodeAddr = (__bridge void *)node;
retIns->recordOrReplayData = (__bridge void *)replayData;
retIns->return_address = node.callBack ? node.callBack : [node getOrgFun];
replayData.runStatus = relay_event_run_status_runFinish;
}
else
{
retIns->nodeAddr = NULL;
retIns->recordOrReplayData = NULL;
retIns->return_address = [node getOrgFun];
}
}
Copy the code3. How to unify hook block
If you just want to understand the underlying technology of Block, you just need to Google it. If you want a thorough understanding of the underlying technology of Block, the resources available online are not enough. I could only read the source code of Apple compiler Clang and list the source code of representative block examples. Then I turned it into C language and assembly, and studied the underlying details through C language and assembly.
What is oc block
-
A block is a closure, much like a callback, which is also an object
-
Blcok has the following features: 1. Can have a parameter list 2. Can have a return value 3. Memory management strategy with object reference counting (Block life cycle)
-
There are three common memory formats for storing blocks _NSConcretStackBlock _NSConcretGlobalBlock _NSConcretMallocBlock
How do you express blocks underneath the system
Let’s take a look at the block example:
void test()
{
__block int var1 = 8; // Context variables
NSString *var2 = @" I'm the second variable "; // Context variables
Void (^block)(int) = ^(int arg)// Argument list
{
var1 = 6;
NSLog(@"arg = %d,var1 = %d, var2 = %@", arg, var1, var2);
};
block(1); // Invoke block syntax
dispatch_async(dispatch_get_global_queue(0, 0), ^
{
block(2); // Call block asynchronously
});
}
Copy the codeThis code first defines two variables, then defines a block, and finally calls the block.
-
Two variables: both variables are referenced by blocks. The first variable has the keyword _block, indicating that the variable can be assigned a value in the block. The second variable has no _block keyword and can only be read, not written, in the block.
-
Two statements that call a block: the first is called directly from the current method test(), while the block’s memory is on the stack, and the second is asynchronous, meaning that by the time block(2) is executed, test() may have run out and the test() stack may have been destroyed. So in this case the block must not be on the stack, it must be on the heap or in the global partition.
Several important data structures of block expressed at the bottom of the system are as follows:
-
Note: Although these constructs are used to express blocks underneath, they are not source code, they are binary code
enum
{
BLOCK_REFCOUNT_MASK = (0xffff),
BLOCK_NEEDS_FREE = (1 << 24),
BLOCK_HAS_COPY_DISPOSE = (1 << 25),
BLOCK_HAS_CTOR = (1 << 26),//todo = = BLOCK_HAS_CXX_OBJ?
BLOCK_IS_GC = (1 << 27),
BLOCK_IS_GLOBAL = (1 << 28),
BLOCK_HAS_DESCRIPTOR = (1 << 29),//todo = = BLOCK_USE_STRET?
BLOCK_HAS_SIGNATURE = (1 << 30),
OBLOCK_HAS_EXTENDED_LAYOUT = (1 << 31)
};
enum
{
BLOCK_FIELD_IS_OBJECT = 3,
BLOCK_FIELD_IS_BLOCK = 7,
BLOCK_FIELD_IS_BYREF = 8,
OBLOCK_FIELD_IS_WEAK = 16,
OBLOCK_BYREF_CALLER = 128
};
typedef struct block_descriptor_head
{
unsigned long int reserved;
unsigned long int size; // Represents the memory size of the main block structure
}block_descriptor_head;
typedef struct block_descriptor_has_help
{
unsigned long int reserved;
unsigned long int size; // Represents the memory size of the main block structure
void (*copy)(void *dst, void *src); // This function pointer is executed when the block is retained
void (*dispose)(void *); // called when a block is destroyed
struct block_arg_var_descriptor *argVar;
}block_descriptor_has_help;
typedef struct block_descriptor_has_sig
{
unsigned long int reserved;
unsigned long int size;
const char *signature; // Block signature information
struct block_arg_var_descriptor *argVar;
}block_descriptor_has_sig;
typedef struct block_descriptor_has_all
{
unsigned long int reserved;
unsigned long int size;
void (*copy)(void *dst, void *src);
void (*dispose)(void *);
const char *signature;
struct block_arg_var_descriptor *argVar;
}block_descriptor_has_all;
typedef struct block_info_1
{
void *isa; // Indicates whether the current BLCOK is on the heap or stack, or in the global section _NSConcreteGlobalBlock
int flags; // Corresponds to the above enum values, which I copied from the compiler source
int reserved;
void (*invoke)(void *, ...) ; // Block corresponding method body (execution body, that is, code block)
void *descriptor; // This points to one of the above structures, depending on the flags value, which is used to further describe block information
// Starting with this field, the following fields represent variables referenced by the block.
NSString *var2;
byref_var1_1 var1;
} block_info_1;
Copy the codeThe underlying representation of the block in this example is something like this:
First, block_info_1 is used to express the block itself, and then block_DESc_1 is used to describe specific information about the block (such as the size of the block_info_1 structure, on the heap or on the stack? Which method to call when dispose or copy, etc.), however, the specific structure of block_DESc_1 is determined by flags field in block_info_1. The invoke field in block_Info_1 points to the block method body, that is, the code segment. The call to a block is to execute the function pointer. Since var1 is writable, we need to design a structure (byref_var1_1) to express var1. Why does var2 use its original type and VAR1 use a structure? I don’t have enough space. Why don’t you think about that?
Block summary
-
In order to express block, three structures are designed at the bottom layer: block_info_1, block_DESc_1, byref_VAR1_1, and three function Pointers: block invoke method body, copy method, and Dispose method
-
In fact, expressing blocks is very complicated and involves the block life cycle, memory management issues and so on. I have simply introduced it through the main flow here, leaving out many details.
How to unify hook block
From the above analysis, that oc block is a structure pointer, so I can directly convert it into a structure pointer in the source code to handle. Unified Hook Block source code is as follows
VoidfunBlock createNewBlock(VoidfunBlock orgblock, ReplayEventIns *blockEvent,bool isRecord)
{
if(orgblock && blockEvent)
{
VoidfunBlock newBlock = ^(void)
{
orgblock();
if(nil == blockEvent)
{
assert(0);
}
};
trace_block_layout *blockLayout = (__bridge trace_block_layout *)newBlock;
blockLayout->invoke = (void (*)(void *, ...) )(isRecord? hook_var_block_callBack_record:hook_var_block_callBack_replay);
return newBlock;
}
return nil;
}
Copy the codeWe first create a newBlock newBlock, and then package the original block orgblock and the event instruction blockEvent into the new blcok to achieve the effect of reference. The new block is then turned into a structure pointer and the invoke (method body) field in the structure pointer points to the unified callback method. You may wonder if the new block has no parameter type. The original block has parameter type. The answer is no, because we only use the block data structure to construct the new block. The block callback method field has been neutered, and the callback method points to the unified method, which can accept any type of argument, including no argument type. This unified method is also assembly implementation, the code implementation is similar to the assembly layer code above, there is no attached source code.
So how do I read the old block and event instruction objects in the new BLCOK? The code is as follows:
void var_block_callback_start_record(trace_block_layout * blockLayout)
{
VoidfunBlock orgBlock = (__bridge VoidfunBlock)(*((void **)((char *)blockLayout + sizeof(trace_block_layout))));
ReplayEventIns *node = (__bridge ReplayEventIns *)(*((void **)((char *)blockLayout + 40)));
}
Copy the codeconclusion
Idle fish technology team is a dapper engineering technology team. We not only pay attention to the effective solution of business problems, but also promote the cutting edge practice of computer vision technology in mobile terminals by breaking the division of division of technology stack (the unification of android/iOS/Html5/Server programming model and language). As a software engineer in the Idle Fish technology team, you have the opportunity to demonstrate all your talents and courage in the evolution of the entire product and user problem solving to prove that technology development is a life-changing force.
Resume: [email protected]
Identify two-dimensional code, forward-looking technology is in your grasp