instructions

ARKit series of articles directory

The Parallax View

The Parallax View is a software program created by a foreign developer that uses The iPhoneX to create glasses-free 3D.

The principle characteristics

Depth illusion obtained by 3D head tracking using ARKit and iPhone X.

For best results, only one eye should be open (the app allows you to select which eyes to track, or you can try to select the eyes automatically)

By tracking the orientation and position of the user’s head, the position of the eyes in 3D can be found.

The application can render the correct view from that location on the display.

To render this view, an off-axis projection (an asymmetric camera truncation body) is used.

This gives the illusion that objects appear in front and behind the screen.

imitation

Inspired by this, I tried to achieve a low-precision glasses-free 3D effect on a non-iphone X device using a human face recognition framework. The principle is:

  • Using face recognition to detect face position;
  • The face of the two-dimensional position into the three-dimensional space (z axis fixed, XY proportional conversion);
  • Assign the converted 3D coordinates to the camera;

The original plan was to identify the size of the face as a basis for determining the position of the camera on the Z axis, but the test found that the size of the face recognition is very variable, irregular and changeable.

Build a model

Face recognition

I thought Apple’s new framework, Vision, would be better, but it didn’t work on my iPhone SE, and the Vision framework requires orientation when recognizing faces. All this ends up using metadataObjectTypes in AVFoundation.

if session.canAddOutput(metaDataOutput) {
       session.addOutput(metaDataOutput)
}
 //7.AVFoundation framework identification type
metaDataOutput.metadataObjectTypes = [.face]
Copy the code

Processing the recognized face in the proxy method:

//AV frame face recognition
func metadataOutput(_ output: AVCaptureMetadataOutput, didOutput metadataObjects: [AVMetadataObject], from connection: AVCaptureConnection) {
    
    for obj in metadataObjects {
        if obj.type == .face {
            print("face---\(obj.bounds)")
            // Convert coordinates
            let oldRect = obj.bounds;
            let w = oldRect.size.height * self.previewView.bounds.size.width;
            let h = oldRect.size.width * self.previewView.bounds.size.height;
            let x = oldRect.origin.y * self.previewView.bounds.size.width;
            let y = oldRect.origin.x * self.previewView.bounds.size.height;
            
            // Add the rectangle
            rectLayer.frame = CGRect(x: x, y: y, width: w, height: h)
            rectLayer.isHidden = false
            
            // Put together reasonable data
            let cameraX = (oldRect.origin.y - 0.3) * 2
            let cameraY = (0.4 - oldRect.origin.x) * 2
            // Move the camera
            self.cameraNode.position = SCNVector3(cameraX, cameraY, 20)}else {
            rectLayer.isHidden = true}}}Copy the code

Preliminary results:

But we should also see that our 3D effect is achieved by directly moving The position of The camera, so The range of View of The camera doesn’t change, so it looks like you’re looking through a window, rather than like The Parallax View, where The 3D object appears to be attached to The screen of your phone.

The main reason why it is Parallax is that it uses an off-axis projection to measure The extent Of The object it sees. As shown below, The Field Of View (FOV) always aligns The four angles Of The base no matter how it is moved:

Projection transformation

In SceneKit, self. CameraNode. Camera? .projectionTransform

Apple also states that once this is set, zFar,zNear, and fieldOfView for the camera will not work because these values cannot be obtained from the matrix (mathematically irreversible). Also note that the ARSCNView class in ARKit overrides the projection transform, so don’t change it in an AR application.

If you have a good knowledge of 3D mathematics, you can just rewrite this projection transformation matrix and get the off-axis projection effect. Unfortunately, I don’t have this right now. I need to study the configuration and update it later.

code

Code address: NakedEye3D