background

Dynamic stickers are one of the effects of face effects (based on face recognition SDK). Such as short video apps such as Douyin and Kuaishou, or camera apps such as Beauty Camera and Meitu Xiuxiu. Dynamic stickers are most commonly used in 2D, 3D stickers are not introduced here. 2D stickers can be divided into static and dynamic ones. The material of 2D static stickers is only one picture, while dynamic stickers are rendered by multiple pictures in the form of sequence frames. Let’s take a look at the implementation.

implementation

In general, stickers are meant to be downloaded dynamically, so we need to build a Json. The following structure can be expanded according to service requirements. Resource structure (F_CatMustache and F_MouceHeart image folders in the same format as Filename_000.png)

Config. json configuration format

{"name" : "white cat ", // for UI display "icon" : "baixiaomaohuxu_icon.png", // UI icon" Nodes" : [// Each model can have multiple effect combinations {"type" : "2dAnim", // Model type: such as: '2dAnim', '3dModel' (3D stickers), '3dAnim' (3D animation), etc. "dirName" : "F_CatMustache", // the name of the folder where the mustache is stored, if the first mustache is stored all in the F_CatMustache folder "facePos" : 46, // the center point of the key point of the face "startIndex" : 1, // stickers relative to the starting point in the key point of the face, with the end point used to calculate the width of the sticker on the face "endIndex" : 31, // the end point in the key point of the face "offsetX" : 0, // stickers x axis offset "offsetY" : 0, // sticker y offset "ratio" : 1, // sticker zoom ratio (relative to human face) "number" : 72, // the number of material pictures, i.e. the total number of pictures under the dirname folder. "Height" : 100, "duration" : 100, // The duration of each image, in milliseconds, can be different for each image under different dirnames. "Maxcount" : 5 // Maximum number of faces supported}, {"type" : "2dAnim", "dirName" : {"type" : "2dAnim", "dirname" : F_MouceHeart", "facePos" : 45, "startIndex" : 52, "endIndex" : 43, "offsetX" : -1.2, "offsetY" : -0.3, "ratio" : 1, "number" : 72, "width" : 200, "height" : 150, "duration" : 100, "isloop" : 1, "maxcount" : 5 }, ] }Copy the code

Apply colours to a drawing

1. Construction of visual vertebral body:

- (void)generateTransitionMatrix {
    
    float mRatio = outputFramebuffer.size.width/outputFramebuffer.size.height;
    
    _projectionMatrix = GLKMatrix4MakeFrustum(-mRatio, mRatio, -1, 1, 3, 9);
    
    _viewMatrix = GLKMatrix4MakeLookAt(0, 0, 6, 0, 0, 0, 0, 1, 0);
}
Copy the code

The optic vertebra constructed here adds the aspect ratio, and the view points (0.0, 0.0, 6.0) are exactly twice that of the near plane 3 for subsequent NDC coordinates calculation.

2. Calculate vertices and transformation matrices

- (void)drawFaceNode:(MKNodeModel *)node withfaceInfo:(MKFaceInfo *)faceInfo { GLuint textureId = [self getNodeTexture:node]; If (textureId <= 0) return; glEnable(GL_BLEND); glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA); [outputFramebuffer activateFramebuffer]; [_program use]; GLfloat tempPoint[8]; CGFloat mImageWidth = MKLandmarkManager.shareManager.detectionWidth; CGFloat mImageHeight = MKLandmarkManager.shareManager.detectionHeight; Float stickerWidth = getDistance(([faceInfo.points[node.startIndex] CGPointValue].x * 0.5 + 0.5) * mImageWidth, ([faceinfo. points[node.startIndex] CGPointValue]. Y * 0.5 + 0.5) * mImageHeight, ([faceinfo.points [node.endIndex] CGPointValue].x * 0.5 + 0.5) * mImageWidth, ([faceinfo.points [node.endIndex] CGPointValue]. Y * 0.5 + 0.5) * mImageHeight); float stickerHeight = stickerWidth * node.height/node.width; Float centerX = 0.0 f; Float centerY = 0.0 f; CenterX = ([faceinfo.points [node.facepos] CGPointValue]. X * 0.5 + 0.5) * mImageWidth; CenterY = ([faceinfo.points [node.facepos] CGPointValue]. Y * 0.5 + 0.5) * mImageHeight; centerY = ([faceinfo.points [node.facepos] CGPointValue]. centerX = centerX / mImageHeight * ProjectionScale; centerY = centerY / mImageHeight * ProjectionScale; // Because frustumM sets the aspect ratio, the NDC coordinates need to be changed to mRatio:1. Here need to change my float ndcCenterX = (centerX - outputFramebuffer. Size. Width/outputFramebuffer. The size, height) * ProjectionScale; Float ndcCenterY = (centerY - 1.0f) * ProjectionScale; float ndcCenterY = (centerY - 1.0f) * ProjectionScale; Float ndcStickerWidth = stickerWidth/mImageHeight * ProjectionScale; float ndcStickerWidth = mImageHeight * ProjectionScale; float ndcStickerHeight = ndcStickerWidth * (float) node.height / (float) node.width; // NDC offset coordinates float offsetX = (stickerWidth * node.offsetx)/mImageHeight * ProjectionScale; float offsetY = (stickerHeight * node.offsetY) / mImageHeight * ProjectionScale; Float anchorX = ndcCenterX + offsetX; float anchorX = ndcCenterX + offsetX; float anchorY = ndcCenterY + offsetY; // The actual vertex coordinates tempPoint[0] = anchorx-ndcstickerWidth; tempPoint[1] = anchorY - ndcStickerHeight; tempPoint[2] = anchorX + ndcStickerWidth; tempPoint[3] = anchorY - ndcStickerHeight; tempPoint[4] = anchorX - ndcStickerWidth; tempPoint[5] = anchorY + ndcStickerHeight; tempPoint[6] = anchorX + ndcStickerWidth; tempPoint[7] = anchorY + ndcStickerHeight; Static const GLfloat textureCoordinates[] = {0.0f, 0.0f, 1.0f, 0.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f,}; // Float pitchAngle = faceinfo.pitch; float yawAngle = faceInfo.yaw; float rollAngle = -faceInfo.roll; _modelViewMatrix = GLKMatrix4Identity; _modelViewMatrix = GLKMatrix4Translate(_modelViewMatrix, ndcCenterX, ndcCenterY, 0); _modelViewMatrix = GLKMatrix4RotateZ(_modelViewMatrix, rollAngle); _modelViewMatrix = GLKMatrix4RotateY(_modelViewMatrix, yawAngle); _modelViewMatrix = GLKMatrix4RotateX(_modelViewMatrix, pitchAngle); _modelViewMatrix = GLKMatrix4Translate(_modelViewMatrix, -ndcCenterx, -ndccentery, 0); GLKMatrix4 mvpMatrix = GLKMatrix4Multiply(_projectionMatrix, _viewMatrix); mvpMatrix = GLKMatrix4Multiply(mvpMatrix, _modelViewMatrix); glActiveTexture(GL_TEXTURE3); glBindTexture(GL_TEXTURE_2D, textureId); glUniform1i(_inputTextureUniform, 3); glUniformMatrix4fv(_mvpMatrixSlot, 1, GL_FALSE, mvpMatrix.m); glVertexAttribPointer(_positionAttribute, 2, GL_FLOAT, 0, 0, tempPoint); glEnableVertexAttribArray(_positionAttribute); glVertexAttribPointer(_inTextureAttribute, 2, GL_FLOAT, 0, 0, textureCoordinates); glEnableVertexAttribArray(_inTextureAttribute); glDrawArrays(GL_TRIANGLE_STRIP, 0, 4); glDisable(GL_BLEND); }Copy the code

Note: Euler Angle needs to be adjusted according to different SDK

3. Get the texture

Int frameIndex = (int)(([MKTool getCurrentTimeMillis] -nodemillis)/node.duration);

-(GLuint )getNodeTexture:(MKNodeModel *)node { uint64_t nodeMillis = 0; // Get the start milliseconds of node cache. If null, get the current system milliseconds. NodeMillis = [MKTool getCurrentTimeMillis]; if (_nodeFrameTime[node.dirName] == nil) {nodeMillis = [MKTool getCurrentTimeMillis]; _nodeFrameTime[node.dirname] = [[NSNumber alloc] initWithUnsignedLongLong:nodeMillis]; } else { nodeMillis = [_nodeFrameTime[node.dirname] unsignedLongLongValue]; } int frameIndex = (int)(([MKTool getCurrentTimeMillis] -nodemillis)/node.duration); // Compare the total number of materials, If (frameIndex >= node.number) {if (node.isloop) {_nodeFrameTime[node.dirName] = [[NSNumber alloc] initWithUnsignedLongLong:[MKTool getCurrentTimeMillis]]; frameIndex = 0; } else { return 0; NSString *imageName = [NSString stringWithFormat:@"%@_% 03D.png ", Node.dirName,frameIndex]; NSString *path = [node.filePath stringByAppendingPathComponent:imageName]; UIImage *image = [UIImage imageWithContentsOfFile:path]; Gpuimageture *picture1 = [[gpuImageture alloc] initWithImage:iamge]; GPUImageFramebuffer *frameBuffer1 = [picture1 framebufferForOutput]; return [frameBuffer1 texture]; }Copy the code

4. Shader is relatively simple

NSString *const kMKGPUImageDynamicSticker2DVertexShaderString = SHADER_STRING ( attribute vec3 vPosition; attribute vec2 in_texture; varying vec2 textureCoordinate; uniform mat4 u_mvpMatrix; Void main() {gl_Position = u_mvpMatrix * vec4(vPosition, 1.0); textureCoordinate = in_texture; });Copy the code

rendering

The code has been uploaded to MagicCamera, your star and fork are the best support and motivation for me

Jane books address www.jianshu.com/u/4cc792175…

Refer to the link www.jianshu.com/p/122bedf3a…