Introduction:
According to the requirements of online teaching, in addition to synchronizing the data of line drawing, it is also necessary to ensure the smoothness and immediacy of line drawing. Because the underlying background is pictures and video stream, higher requirements are put forward for the performance of the white board. Eraser is not supported in many open source projects (those that do support it simply change the color of the “eraser” to the background color) because eraser completely negates the smoothness optimization strategy. (Main reference: netease Whiteboard Demo, BHBDrawBoarderDemo, several Github open source projects)
HYWhiteboard Demo address: https://github.com/HaydenYe/HYWhiteboard.git
I. Whiteboard optimization
1. Memory optimization
BHBDrawBoarderDemo solves the problem of memory inflation caused by using CALayer as the drawing layer.
When CALayer calls drawRect:, the CPU assigns it a context CTX, which is the length * width *4 of rect. If our layer size is screen size (in fact, the image will be larger because it needs to be scaled), it will be tens or even hundreds of megabytes of memory.
CAShapeLayer solves this problem because CAShapeLayer uses hardware acceleration and does not generate intermediate bitmaps (and certainly does not call drawRect), so there is no memory inflation. While reducing memory overhead, it speeds up drawing and reduces CPU usage (Core Graphics draws use CPU).
Original link: mp.weixin.qq.com/s?__biz=MjM…
2. Add the eraser function
According to the requirements of the project, the background color is not a single and unchanging color, so we cannot use the same color as the background color to cover other lines.
General drawing line:
// Initialize the Bezier curve
UIBezierPath *path = [UIBezierPath new];
path.lineJoinStyle = kCGLineJoinRound;
path.lineWidth = 1.f;
path.lineCapStyle = kCGLineCapRound;
UIColor *lineColor = [UIColor redColor];
[lineColor setStroke];
// Normal overwrite mode
[path strokeWithBlendMode:kCGBlendModeNormal alpha:1.0];
Copy the codeEraser: pay attention to the color calculation of the two modes. Find the mode that suits your background color
// Initialize the Bezier curve
UIBezierPath *path = [UIBezierPath new];
path.lineJoinStyle = kCGLineJoinRound;
path.lineWidth = 1.f;
path.lineCapStyle = kCGLineCapSquare;
// Clear mode for layer with transparent background color
[path strokeWithBlendMode:kCGBlendModeClear alpha:1.0];
Copy the codeOr:
// Copy mode
UIColor *lineColor = [UIColor clearColor];
[lineColor setStroke];
[path strokeWithBlendMode:kCGBlendModeCopy alpha:1.0];
Copy the code3. Caton optimization
The reason for the lag is that every time we add a point, we have to redraw all the previous points on the layer, and we lose frames when we draw faster than the screen refresh rate. If the data is synchronized over the network, it can also cause long delays.
Steps of optimization plan:
1. Control the layer refresh frequency. CADisplayLink is used to control the refresh rate of view drawing. When frameInterval is 1, the refresh rate is the fastest, 60 times per second. The larger the frameInterval is, the slower the refresh rate is. When the frameInterval is 2, the refresh rate is 30 times per second, and so on.
NSInteger frameInterval = 1;
CADisplayLink *displayLink = [CADisplayLink displayLinkWithTarget:self selector:@selector(onDisplayLink:)];
[displayLink setFrameInterval:frameInterval];
[displayLink addToRunLoop:[NSRunLoop mainRunLoop] forMode:NSRunLoopCommonModes];
Copy the code2. Each line generates a CAShapeLayer. Each point is drawn and only the path of the layer is updated.
UIBezierPath *path = [self _singleLine:currentLine needStroke:NO];
CAShapeLayer *realTimeLy = [CAShapeLayer layer];
realTimeLy.backgroundColor = [UIColor clearColor].CGColor;
realTimeLy.path = path.CGPath;
realTimeLy.strokeColor = [[UIColor redColoe] CGColor];
realTimeLy.fillColor = [UIColor clearColor].CGColor;
realTimeLy.lineWidth = path.lineWidth;
realTimeLy.lineCap = kCALineCapRound;
Copy the code3. There are two CAShapeLayer, each point is drawn, and only the upper layer is updated. When one line is drawn (finger leaves the screen), all lines are redrawn on the lower layer. (Code for v1.0)
@interface HYWhiteboardView(a)
@property (nonatomic.strong)CAShapeLayer *realTimeLy; // Real-time display layer
@end
@implementation HYWhiteboardView
// Set the layer of the view to CAShapeLayer, which causes CAShapeLayer to call drawRect:
+ (Class)layerClass {
return [CAShapeLayer class];
}
// Render non-eraser lines (render only the live display layer)
- (void)onDisplayLinkFire:(HYCADisplayLinkHolder *)holder duration:(NSTimeInterval)duration displayLink:(CADisplayLink *)displayLink {
if (_dataSource && [_dataSource needUpdate]) {
// Need real-time display layer for line drawing
NSArray *lines = [[_dataSource allLines] objectForKey:UserOfLinesMine];
// Clear the underlined render
if (lines.count <= 0) {[self.layer setNeedsDisplay];
self.realTimeLy.hidden = YES;
return;
}
// Eraser lines need to be rendered directly to the view layer, so no longer
NSArray *currentLine = lines.lastObject;
HYWbPoint *firstPoint = [currentLine objectAtIndex:0];
if (_isEraserLine) {
return;
}
// Render the line to the live display layer
UIBezierPath *path = [self _singleLine:currentLine needStroke:NO];
self.realTimeLy.path = path.CGPath;
_realTimeLy.strokeColor = [[_dataSource colorArr][firstPoint.colorIndex] CGColor];
_realTimeLy.fillColor = [UIColor clearColor].CGColor;
_realTimeLy.lineWidth = path.lineWidth;
_realTimeLy.lineCap = firstPoint.isEraser ? kCALineCapSquare : kCALineCapRound;
_realTimeLy.hidden = NO;
// If it is the last point, update the view layer and draw the line to the view layer
HYWbPoint *theLastPoint = [currentLine lastObject];
if (theLastPoint.type == HYWbPointTypeEnd) {
// The marker layer needs to be redrawn
[self.layer setNeedsDisplay];
_realTimeLy.hidden = YES; }}}// Redraw all lines in the view layer
- (void)drawRect:(CGRect)rect {
[self _drawLines];
}
@endCopy the codeAt this time, the drawing line has been fixed, but it is found that the lines in kCGBlendModeClear or kCGBlendModeCopy mode must be at the same layer as other lines to have effect, so we need to further optimize the rubber drawing line.
4. Eraser points are drawn directly in the lower layer, but setNeedsDisplayInRect: method is used to partially draw to solve the problem of too many redrawn lines. When one line is drawn, all lines are redrawn.
// Eraser renders directly to the view
- (void)drawEraserLineByPoint:(HYWbPoint *)wbPoint {
// A line has been drawn and rendered to the view layer
if (wbPoint.type == HYWbPointTypeEnd) {
_isEraserLine = NO;
[self.layer setNeedsDisplay];
[self.layer display];
return ;
}
_isEraserLine = YES;
CGPoint point = CGPointMake(wbPoint.xScale * self.frame.size.width, wbPoint.yScale * self.frame.size.height);
if (wbPoint.type == HYWbPointTypeStart) {
_lastEraserPoint = point;
}
// Render eraser to draw lines
[self _drawEraserPoint:point lineWidth:wbPoint.lineWidth];
_lastEraserPoint = point;
}
// Render eraser to draw lines
- (void)_drawEraserPoint:(CGPoint)point lineWidth:(NSInteger)width {
// Only redraw parts to improve efficiency
CGRect brushRect = CGRectMake(point.x - lineWidth /2.f, point.y - lineWidth/2.f, lineWidth, lineWidth);
[self.layer setNeedsDisplayInRect:brushRect];
// It is critical to render immediately
[self.layer display];
}
Copy the codeLocally drawn points (which should be called regions) are incoherent because the UIPanGestureRecognizer gets finger coordinates at a fixed time interval (screen refresh rate) rather than listening for changes in coordinates. Since you can’t use Bezier to draw the curve, you have to calculate the missing points (regions). The downside is that there is a 1pt offset between each point, which means the lines are jagged, but repainting (fingers off the screen) doesn’t have this problem.
Calculation method:
1. Calculate the offsets offsetX and offsetY of x and y between the two points
2. Calculate the maximum number of points (interval: 1pt) according to Max(fabS (offsetX), FABs (offsetY)); Calculate the interval between each point of Min(FABS (offsetX), FABS (offsetY)) based on the number of points.
3. Determine the recT coordinates drawn locally according to offsetX and offsetY
static float const kMaxDif = 1.f; // The maximum offset of two eraser positions when calculating the eraser track
// Calculate the points between two points of the eraser
- (void)_addEraserPointFromPoint:(CGPoint)point lineWidth:(NSInteger)lineWidth {
// 1. The offset of x and y between two points
CGFloat offsetX = point.x - self.lastEraserPoint.x;
CGFloat offsetY = point.y - self.lastEraserPoint.y;
// The interval between x and y between each point
CGFloat difX = kMaxDif;
CGFloat difY = kMaxDif;
// Start point x, y cheap zero, draw directly, prevent Nan crash (also can not draw)
if (offsetX == 0 && offsetY == 0) {[self _drawEraserpoint:point line lineWidth:lineWidth];
return ;
}
// 2. Calculate the number of points to be added and the interval
NSInteger temPCount = 0;
if (fabs(offsetX) > fabs(offsetY)) {
difY = fabs(offsetY) / fabs(offsetX);
temPCount = fabs(offsetX);
} else {
difX = fabs(offsetX) / fabs(offsetY);
temPCount = fabs(offsetY);
}
// Render additional paint points
// 3. Confirm the direction of the points above the x and y components
if (offsetX > kMaxDif) {
for (int i = 0; i < temPCount ; i ++) {
CGPoint addP = CGPointMake(_lastEraserPoint.x + difX * i, _lastEraserPoint.y);
if (offsetY > kMaxDif) {
addP.y = addP.y + difY * i;
}
else if (offsetY < - kMaxDif) {
addP.y = addP.y - difY * i;
}
[self_drawEraserPoint:addP lineWidth:lineWidth]; }}else if (offsetX < - kMaxDif) {
for (int i = 0; i < temPCount ; i ++) {
CGPoint addP = CGPointMake(_lastEraserPoint.x - difX * i, _lastEraserPoint.y);
if (offsetY > kMaxDif) {
addP.y = addP.y + difY * i;
}
else if (offsetY < - kMaxDif) {
addP.y = addP.y - difY * i;
}
[self_drawEraserPoint:addP lineWidth:lineWidth]; }}else if (offsetY > kMaxDif) {
for (int i = 0; i < temPCount ; i ++) {
CGPoint addP = CGPointMake(_lastEraserPoint.x, _lastEraserPoint.y + difY * i);
if (offsetX > kMaxDif) {
addP.x = addP.x + difX * i;
}
else if (offsetX < - kMaxDif) {
addP.x = addP.x - difX * i;
}
[self_drawEraserPoint:addP lineWidth:lineWidth]; }}else if (offsetY < - kMaxDif) {
for (int i = 0; i < temPCount ; i ++) {
CGPoint addP = CGPointMake(_lastEraserPoint.x, _lastEraserPoint.y - difY * i);
if (offsetX > kMaxDif) {
addP.x = addP.x + difX * i;
}
else if (offsetX < - kMaxDif) {
addP.x = addP.x - difX * i;
}
[self_drawEraserPoint:addP lineWidth:lineWidth]; }}// No need to add points
else{[self_drawEraserPoint:point lineWidth:lineWidth]; }}Copy the code4. Optimization of smoothness of lines
1. Since the coordinates of the acquisition point are obtained according to the screen refresh frequency, that is, the faster the finger moves and the time is constant, the farther the distance between the two coordinates may be, leading to edges and corners of the first-order Bezier, so we decide to use the second-order Bezier. Second order Bessel principle :(web image)
Since the distance between the two points is variable, the selection of control points is important in order to form a smooth curve. We take the midpoint of the last control point and the point to be added as the new control point.
- (instancetype)init {
if (self = [super init]) {
// Initialize the control point
_controlPoint = CGPointZero;
}
return self;
}
// Get a Bezier curve
- (UIBezierPath *)_singleLine:(NSArray<HYWbPoint *> *)line needStroke:(BOOL)needStroke {
// Get the starting point of the line
HYWbPoint *firstPoint = line.firstObject;
// Initialize the Bezier curve
UIBezierPath *path = [UIBezierPath new];
path.lineJoinStyle = kCGLineJoinRound;
path.lineWidth = firstPoint.isEraser ? firstPoint.lineWidth * 2.f : firstPoint.lineWidth;
path.lineCapStyle = firstPoint.isEraser ? kCGLineCapSquare : kCGLineCapRound;
// Draw the line color
UIColor *lineColor = [_dataSource colorArr][firstPoint.colorIndex];
// Generate bezier curve
for (HYWbPoint *point in line) {
CGPoint p = CGPointMake(point.xScale * self.frame.size.width, point.yScale * self.frame.size.height);
if (point.type == HYWbPointTypeStart) {
[path moveToPoint:p];
}
// Optimize the curve smoothness, second order Bessel
else {
if(_controlPoint.x ! = p.x || _controlPoint.y ! = p.y) { [path addQuadCurveToPoint:CGPointMake((_controlPoint.x + p.x) / 2, (_controlPoint.y + p.y) / 2) controlPoint:_controlPoint];
}
}
_controlPoint = p;
}
// Need to render
if (needStroke) {
if (firstPoint.isEraser) {
[path strokeWithBlendMode:kCGBlendModeClear alpha:1.0];
}
else {
[lineColor setStroke];
[path strokeWithBlendMode:kCGBlendModeNormal alpha:1.0]; }}return path;
}
Copy the code2. After using the second-order Bezier, the lines are drawn with curves, so you can see that the lines are zigzagged. This is because apple’s Retina screen uses higher pixels, so you just need to set the Layer contentsScale property.
layer.contentsScale = [UIScreen mainScreen].scale;
Copy the code5. Optimized space
When eraser is used, the CPU usage is high and there is room for optimization.
2. Data synchronization
1. Accuracy of line drawing position
The terminals in the project include Android TV, Android tablet, Android phone, Apple phone and Apple tablet. Among them, TV is displayed in horizontal screen while other terminals are displayed in vertical screen, so it will lead to inaccurate line drawing position.
Solution:
Send the proportion of coordinate points in the drawing area, rather than absolute coordinates, and ensure that the drawing area is the same size as the picture, then the aspect ratio is the same, then the coordinate conversion is accurate.
// The point to send
HYWbPoint *point = [HYWbPoint new];
point.xScale = (p.x) / _wbView.frame.size.width;
point.yScale = (p.y) / _wbView.frame.size.height;
// The conversion after receiving the point
CGPoint p = CGPointMake(point.xScale * self.frame.size.width , point.yScale * self.frame.size.height);
Copy the code2. Accuracy of line thickness
1. Line drawing: There is a mistake in the design. It is not that both ends need to be calculated according to the proportion, but according to the proportion of the area of the picture, so as to calculate the thickness of the line drawing. For example, if the lineWidth is set to 1% of the picture width picWidth, the line thickness is:
CGFloat result = lineWidth * picWidth / 100.f;
Copy the code2. Eraser: the optimized processing of eraser, in the local drawing, if the width of recT is calculated according to the thickness of the drawing line, the error will be very large when drawing the diagonal line, so we think the thickness of the drawing line is actually the length of the diagonal line of recT.
// Render eraser to draw lines
- (void)_drawEraserPoint:(CGPoint)point lineWidth:(NSInteger)width {
// Calculate the true width from the diagonal width
CGFloat lineWidth = width * 2.f / 1.414f;
// Only redraw parts to improve efficiency
CGRect brushRect = CGRectMake(point.x - lineWidth /2.f, point.y - lineWidth/2.f, lineWidth, lineWidth);
[self.layer setNeedsDisplayInRect:brushRect];
// It is critical to render immediately
[self.layer display];
}
Copy the code3. Line data synchronization
Line drawing data is realized by sending the collection of points to each other through socket (Intranet) or IM. Ideally, one point is drawn and one point is sent, but SOcekt has a buffer pool, so frequent sending can cause sticky packets, which is less than ideal (data parsing is cpu-intensive). Generally, when the network condition is good, you can set 60ms to send a packet.
// Start the whiteboard command timer
- (void)_startSendingCmd {
if (_cmdTimer) {
[_cmdTimer invalidate];
_cmdTimer = nil;
}
_cmdTimer = [NSTimer timerWithTimeInterval:kTimeIntervalSendCmd target:self selector:@selector(_sendWhiteboardCommand) userInfo:nil repeats:YES];
[[NSRunLoop currentRunLoop] addTimer:self.cmdTimer forMode:NSRunLoopCommonModes];
}
// Send whiteboard command
- (void)_sendWhiteboardCommand {
if (_cmdBuff.count > 0) {
// A collection of sending points
NSArray<NSString *> *cmds = [NSArray arrayWithArray:_cmdBuff];
[self _sendWhiteboardMessage:cmds successed:nil failed:nil]; [_cmdBuff removeAllObjects]; }}Copy the codeIf IM sending is used according to the network fluctuation is the best adjust the contract time interval and CPU usage, because IM is the way forward through the server, so the time is too short messaging can lead to send the order and the order to reach the server is not consistent, do not match the order will receive a message, so you also need to reorder.
Third, other
There are some other considerations when designing a whiteboard application for a distance learning scenario:
1. The design of communication protocol should take into account the use of traffic, compatibility of protocol versions and compatibility of terminals.
2. The differences between mobile iOS and Android devices, such as picture zooming and line drawing optimization, need to be timely communicated.
3. Synchronize all line drawing data of the other party at one time, which requires subcontracting processing due to the large amount of data.
4. If the sending frequency is too high or the amount of data to be sent is too large or the socket buffer pool overflows, you need to implement an ARQ policy to ensure the sending of critical messages.
Iv. Supplement & Update
2018.9.19 update
In iOS 12, due to an Apple bug, the setNeedsDisplayInRect: method was no longer drawn locally, so the eraser optimization failed. You can verify that the recT returned is inconsistent with the recT passed in the drawRect: method. The bug has been submitted and we are waiting for the reply. The eraser optimization function will be closed in the iOS12 system of the project recently.
2018.10.15 Updated version v1.2
- Fixed a BUG where the CPU was too high when receiving line drawing
- Fixed BUG of unsynchronized eraser mode switch
- Optimized line drawing core logic, support file to look back
- Optimize the accuracy of line width
- Optimized the amount of line drawing data, slightly improved performance
When drawing lines for synchronous remote users, we consider that a packet may contain multiple points drawing lines, so it is inaccurate to render only the last drawing line. We introduced the dirtyCount attribute, which records the number of lines that have been rendered so that unrendered lines can be found exactly.
// Render non-eraser lines (render only the live display layer)
- (void)onDisplayLinkFire:(HYCADisplayLinkHolder *)holder duration:(NSTimeInterval)duration displayLink:(CADisplayLink *)displayLink {
if (_dataSource && [_dataSource needUpdate]) {
HYWbAllLines *allLines = [_dataSource allLines];
// Clear all lines
if (allLines.allLines.count == 0) {[self.layer setNeedsDisplay];
_realTimeLy.hidden = YES;
return ;
}
// Eraser lines need to be rendered directly to the view layer, so no longer
if (_isEraserLine) {
return;
}
// All of the user's points have been rendered, possibly an undo or restore operation
if (allLines.dirtyCount >= allLines.allLines.count) {
[self.layer setNeedsDisplay];
return;
}
// Do I need to redraw all the lines
BOOL needUpdateLayer = NO;
// Render the unrendered lines to the live display layer first (optimize the line drawing lag)
if (allLines.allLines.count - allLines.dirtyCount == 1) {
// There is an unrendered line
HYWbLine *line = allLines.allLines.lastObject;
// Render the line to the live display layer
[self _drawLineOnRealTimeLayer:line.points color:line.color.CGColor];
// Whether to draw a line
HYWbPoint *lastPoint = [line.points lastObject];
if (lastPoint.type == HYWbPointTypeEnd) {
allLines.dirtyCount += 1;
needUpdateLayer = YES; }}else {
// There are multiple unrendered lines
NSArray *points = [NSArray new];
CGColorRef color = [UIColor clearColor].CGColor;
for (int i = (int)allLines.dirtyCount; i<allLines.allLines.count; i++) { HYWbLine *line = allLines.allLines[i]; color = line.color.CGColor; points = [points arrayByAddingObjectsFromArray:line.points]; }// Render the line to the live display layer
if (points.count) {
[self _drawLineOnRealTimeLayer:points color:color];
// Whether the last line has been drawn
HYWbPoint *lastPoint = [points lastObject];
if (lastPoint.type == HYWbPointTypeEnd) {
allLines.dirtyCount = allLines.allLines.count;
}
else {
allLines.dirtyCount = allLines.allLines.count - 1;
}
needUpdateLayer = YES; }}// The marker layer needs to be redrawn
if (needUpdateLayer) {
[self.layer setNeedsDisplay];
_realTimeLy.hidden = YES; }}}// Render the line to the live display layer
- (void)_drawLineOnRealTimeLayer:(NSArray *)line color:(CGColorRef)color {
UIBezierPath *path = [self _singleLine:line needStroke:NO];
self.realTimeLy.path = path.CGPath;
_realTimeLy.strokeColor = color;
_realTimeLy.fillColor = [UIColor clearColor].CGColor;
_realTimeLy.lineWidth = path.lineWidth;
_realTimeLy.lineCap = kCALineCapRound;
_realTimeLy.hidden = NO;
}
Copy the code