We first try to identify 33 human body feature points from the static picture, and draw them, so that readers can intuitively observe the 33 feature points in the human body position. Recognition from static images is also relatively simple compared with real-time camera recognition, which is a good way to get started.

Project creation

  1. The introduction ofML Kit SDK

We create a new XCODE project (MLKit project requires XCODE version 12.5.1 or higher). This project is an App project based on iOS platform, named MLKit-ios. The lowest target version of this project is set to iOS10, as described at the end of the previous section. Use the terminal command pod init to create a Podfile and import the MLKit SDK into the Podfile directory.

# Uncomment the next line to define a global platform for your project
# platform :ios, '9.0'

target 'MLKit-iOS' do
  # Comment the next line if you don't want to use dynamic frameworks
  use_frameworks!

  # Pods for MLKit-iOS
  # If you want to use the base implementation:
    pod 'GoogleMLKit/PoseDetection', '2.5.0'

  # If you want to use the accurate implementation:
    pod 'GoogleMLKit/PoseDetectionAccurate', '2.5.0'

  target 'MLKit-iOSTests' do
    inherit! :search_paths
    # Pods for testing
  end

  target 'MLKit-iOSUITests' do
    # Pods for testing
  end
end
Copy the code
  1. Engineering to create

After the Podfile is created, in the project directory, use the terminal command Pod Install to create the project, generating the.xcworkspace file, which we then double-click to open the project instead of.xcodeProj.

在AssetsInto two human body images, according toML KitThe picture must contain the face of the portrait.

Core coding

  1. Display of the whole body

This step is relatively simple, just create an ImageView to display a whole body image of Assets in the upper part, and place a recognition button to trigger the code of recognition of human feature points. ViewController is the first page of the App, in which an ImageView is built and a full body photo of James is placed. The safeAreaFrame() function is used to get the size of the SafeArea in the screen security area.

override func viewDidLoad() { super.viewDidLoad() // Do any additional setup after loading the view. view.backgroundColor = .white //ImageView imageView = UIImageView(frame: safeAreaFrame(self)) imageView! .contentMode = .scaleAspectFit view.addSubview(imageView!) imageView! .image = uiimage.init (named: "James1") .custom) detectButton.frame = CGRect(x:0, y:80, width: 200, height: X = self.view.center.x detectButton.setTitle(" DetectButton.setTitle ", for: UIControl.State.normal) detectButton.setTitleColor(.white, for: UIControl.State.normal) detectButton.backgroundColor = UIColor.orange detectButton.addTarget(self, action:#selector(detectButton(_:)), for: .touchUpInside) self.view.addSubview(detectButton) }Copy the code
  1. PoseDetectorObject creation

PoseDetector is the core recognition class in the ML Kit SDK. Since this project is written by Swift and the ML Kit SDK is written by Objective-C, the real class name of PoseDetector in THE ML Kit SDK is MLKPoseDetector. The PoseDetector construction method is simple:

+ (instancetype)poseDetectorWithOptions:(MLKCommonPoseDetectorOptions *)options
    NS_SWIFT_NAME(poseDetector(options:));
Copy the code

We have said in the previous section, there are two kinds of ML Kit SDK, is a kind of basic SDK, another kind is accurate SDK, so here corresponds to the MLKCommonPoseDetectorOptions also has two kinds: PoseDetectorOptions and AccuratePoseDetectorOptions, they are all MLKCommonPoseDetectorOptions subclasses. MLKCommonPoseDetectorOptions a MLKPoseDetectorMode parameter, which has two values. SingleImage and. Stream:

  • singleImageSingle picture mode

In this mode, the PoseDetector first identifies the human body in the image and then performs pose detection. Each image will undergo a human body test, so the delay will be higher. And there’s no body tracking. This mode is usually used with still images or without the need to track the human body.

  • streamFlow pattern

In this mode, the PoseDetector first detects the most prominent human body in the image before performing pose detection. In subsequent frames, the human body is not detected unless it becomes blurred or less reliable. The posture detector will attempt to track the most prominent human body and return their posture on each inference. This reduces latency and smoothes detection. Use this mode when you want to detect posture in a video stream. So in this section, we talk about identifying human body feature points from static pictures, obviously using singleImage mode. PoseDetector initialization, here we use the exact SDK

private var poseDetector: PoseDetector? Let the options = AccuratePoseDetectorOptions () / / using accurate ` SDK ` options. DetectorMode =. SingleImage / / ` singleImage ` mode Self. PoseDetector = poseDetector. PoseDetector (options: options) / / initializes the poseDetectorCopy the code
  1. Feature point recognition

Before feature point recognition, IT is necessary to transform UIImage into the class object required by ML Kit SDK, that is, the MLImage class object.

guard let inputImage = MLImage(image: Print ("Failed to create MLImage from UIImage.") return} inputImage. Orientation = ImageOrientation // Here we need to assign the orientation value of 'UIImage' to 'MLImage'Copy the code

Use – (void)processImage:(id

)image completion:(MLKPoseDetectionCallback)completion method to identify feature points:

poseDetector.process(inputImage) { poses, error in guard error == nil, let poses = poses, ! Poses. IsEmpty else {// Identifies error... return } ... }Copy the code

You can see the feature point identification by entering the inputImage parameter into the Process method. The recognized callback is an MLKPoseDetectionCallback closure that contains the error value for the recognized error, or the Boolean value for the recognized success. Error readers can try to handle it themselves, or view it in the source code. The object we need to focus on is the pose value, which is an NSArray array object:

NSArray<MLKPose *> *_Nullable poses
Copy the code

The array contains the MLKPose object. For a single static image, we only need to fetch the first element of the pose:

let pose = poses[0]
Copy the code

Let’s look at the MLKPose class, which has one core attribute Landmarks, which are all 33 feature point objects MLKPoseLandmark identified in this static image

@property(nonatomic, readonly) NSArray<MLKPoseLandmark *> *landmarks;
Copy the code

The MLKPoseLandmark class obviously contains the x, y, and Z coordinates of the feature point, the type of the feature point (eyes, nose, hands, feet, etc.), and the reliability of the result of the feature point recognition. The reliability range is [0, 1], which we discussed in the previous section.

@interface MLKPoseLandmark : NSObject @property(nonatomic, readonly) MLKPoseLandmarkType type; @property(nonatomic, readonly) MLKVision3DPoint *position; @property(nonatomic, readonly) float inFrameLikelihood; / / credibilityCopy the code

In order to see the pose value more intuitively, we run the program to identify an image and set a breakpoint to look at the specific attribute value in XCODE. Here, due to the length limitation of the screenshot window, I only intercepted the attribute value of 20 feature points, which should be 33 in total:

After this step of feature point recognition, we are happy to have learned how to recognize human feature points in still images using ML Kit SDK. The next step is to draw the identified feature points on the static image to achieve the following effect, the feature points are represented by small blue circles, and the feature points are connected by color line segments:

  1. Coordinate matrix transformation

Let’s first give the UIImageView an orange color and then explain the problem:



The orange area is oursUIImageView“, while Full-length shots of James, etc. were scaled upUIImageViewThis is done by settingcontentModefor.scaleAspectFitThe implementation.

imageView! .contentMode = .scaleAspectFitCopy the code

We’re going to add a mask transparent annotationOverlayView to UIImageView, and we’re going to do all of our drawing on this transparent annotationOverlayView, This annotationOverlayView must overwrite the UIImageView and have a consistent frame.

private lazy var annotationOverlayView: UIView = {precondition(isViewLoaded) // the annotationOverlayView must overlay the UIImageView and be frame consistent. let annotationOverlayView = UIView(frame: self.imageView.frame) annotationOverlayView.translatesAutoresizingMaskIntoConstraints = false annotationOverlayView.clipsToBounds = true return annotationOverlayView }()Copy the code

And we’ll see that the place that we need to draw is annotationOverlayView, and the feature points that we recognize are identified from UIImage (James), which is obviously scaled at equal proportions and put into UIImageView, The two frames are inconsistent, which involves a coordinate matrix problem, converting the coordinate system of UIImage image itself to the coordinate system of UIImageView, core code:

private func transformMatrix() -> CGAffineTransform { guard let image = imageView.image else { return CGAffineTransform ()} / / picture box wide high let imageViewWidth = imageView. Frame. The size, width let imageViewHeight = ImageView. Frame. The size, wide high height / / picture (this is a picture in the wide after the imageView is high, scaleAspectFit) let imageWidth = image.size.width let imageHeight = image.size.height let imageViewAspectRatio = ImageViewWidth/imageViewHeight let imageAspectRatio = imageWidth/imageHeight //ImageView aspect ratio and Image aspect ratio comparison let scale = (imageViewAspectRatio > imageAspectRatio) ? imageViewHeight / imageHeight : ImageViewWidth/imageWidth // Image original size let scaledImageWidth = imageWidth * scale let scaledImageHeight = imageHeight * Scale let xValue = (imageViewWidth-ScalediMageWidth)/CGFloat(2.0) let yValue = (imageViewHeight - scaledImageHeight) / / / CGFloat (2.0) coordinate matrix transformation var transform = CGAffineTransform. Identity. TranslatedBy (x: xValue, y: yValue) transform = transform.scaledBy(x: scale, y: scale) return transform }Copy the code

Thus, we get such a transformed coordinate system matrix:

Let transform = self.transformmatrix () //Copy the code

Then, the coordinate values of 33 feature points are transformed using the new coordinate matrix to obtain the new coordinate object of feature points, which is a two-dimensional CGPoint object:

let newPoint: CGPoint = oldPoint.applying(transform)
Copy the code
  1. Feature point drawing

The drawing of feature points is relatively simple. The converted 33 coordinate point objects CGpoints are cyclically traversed and drawn on annotationOverlayView one by one. Draw a single CGPoint object code:

public static func addCircle( atPoint point: CGPoint, to view: UIView, color: UIColor, radius: CGFloat ) { let divisor: CGFloat = 2.0 let xCoord = point. x-radius/divisor let yCoord = Point. y-radius/divisor let circleRect = CGRect(x:  xCoord, y: yCoord, width: radius, height: radius) guard circleRect.isValid() else { return } let circleView = UIView(frame: circleRect) circleView.layer.cornerRadius = radius / divisor circleView.alpha = Constants.circleViewAlpha circleView.backgroundColor = color view.addSubview(circleView) }Copy the code

Call this method to draw:

Uiutilities. addCircle(atPoint: newPoint, to: annotationOverlayView, color: uicolor. blue, // Feature point color radius: DotRadius // feature dotRadius)Copy the code
  1. Line segment drawing between feature points

Finally, to draw the associated feature points with line segments of a certain color, the dictionary set of associated feature points needs to be defined first: part of the associated feature points are intercepted here. The collection of associated Connections dictionaries is defined in the PoseConnectionsHolder structure, which we can use to understand the code shown in Figure 1 in the previous section. The dictionary key is the start point of the feature, and the dictionary value is the end point of the feature.

private static func poseConnections() -> [PoseLandmarkType: [PoseLandmarkType]] { struct PoseConnectionsHolder { static var connections: [PoseLandmarkType: [PoseLandmarkType]] = [ PoseLandmarkType.leftEar: [PoseLandmarkType.leftEyeOuter], PoseLandmarkType.leftEyeOuter: [PoseLandmarkType.leftEye], PoseLandmarkType.leftEye: [PoseLandmarkType.leftEyeInner], PoseLandmarkType.leftEyeInner: [PoseLandmarkType.nose], PoseLandmarkType.nose: [PoseLandmarkType.rightEyeInner], PoseLandmarkType.rightEyeInner: [PoseLandmarkType.rightEye], PoseLandmarkType.rightEye: [PoseLandmarkType.rightEyeOuter], PoseLandmarkType.rightEyeOuter: [PoseLandmarkType.rightEar], PoseLandmarkType.mouthLeft: [PoseLandmarkType.mouthRight], PoseLandmarkType.leftShoulder: [ PoseLandmarkType.rightShoulder, PoseLandmarkType.leftHip, ], ... ]  } return PoseConnectionsHolder.connections }Copy the code

Similarly, the starting point and end point of features are iterated and line segments are drawn. The code for drawing a line segment is as follows, the core of which is to use UIBezierPath Bezier curve to draw line segments.

private static func addLineSegment( fromPoint: CGPoint, toPoint: CGPoint, inView: UIView, colors: [UIColor], width: CGFloat) {let viewWidth = inView. Bounds. Width let viewHeight = inView. Bounds. The height if viewWidth = = 0.0 | | viewHeight == 0.0 {return} let path = UIBezierPath() path.move(to: fromPoint) path.addline (to: fromPoint) toPoint) let lineMaskLayer = CAShapeLayer() lineMaskLayer.path = path.cgPath lineMaskLayer.strokeColor = Opacity = 1.0 LinemAsklayer. lineWidth = width let gradientLayer = CAGradientLayer() gradientLayer.startPoint = CGPoint(x: fromPoint.x / viewWidth, y: fromPoint.y / viewHeight) gradientLayer.endPoint = CGPoint(x: toPoint.x / viewWidth, y: toPoint.y / viewHeight) gradientLayer.frame = inView.bounds var CGColors = [CGColor]() for color in colors { CGColors.append(color.cgColor) } if CGColors.count == 1 { CGColors.append(colors[0].cgColor) } gradientLayer.colors = CGColors gradientLayer.mask = lineMaskLayer let lineView = UIView(frame: inView.bounds) lineView.layer.addSublayer(gradientLayer) inView.addSubview(lineView) }Copy the code