“This is the 31st day of my participation in the First Challenge 2022. For details: First Challenge 2022”

preface

In our blog post, Tagless AUGMENTED Reality In Detail, we have studied the basic concepts of augmented reality and how to incorporate computer-generated virtual elements into a view of the real world for augmented reality. In this article, we’ll learn how to create fun Snapchat-based augmented reality, and we’ll introduce a hands-on project to add beard pendant between nose and mouth on a detected face.

Snapchat-based augmented reality

In the project, we covered the detected face with a mustache. We can use continuous video frames captured from the camera, or we can use a single test image. Before actually walking through the key steps of the program, look at the expected output image of the application:

The first step in the project was to detect faces in the images. Use cyan rectangles to plot the positions and sizes of faces detected in the image, as shown above; Next, all the faces detected in the image are iterated, and noses are searched in their regions. The pink rectangle represents noses detected in the image. Once the nose is detected, we need to adjust the area we want to cover the “beard” pendant based on the previously calculated position and size of the nose. The blue rectangle represents the area where the beard will be covered. If the target of processing is a continuous video frame, after processing all the detected faces, the next frame will be analyzed.

As described above, the program should first detect the face and nose in the image. To detect these objects, two classifiers are created, one for faces and one for noses:

# Cascade classifier for face and nose detection
face_cascade = cv2.CascadeClassifier("haarcascade_frontalface_default.xml")
nose_cascade = cv2.CascadeClassifier("haarcascade_mcs_nose.xml")
Copy the code

Once the classifier is created, the next step is to detect these objects in the image using the cv2.DetectMultiScale () function. The cv2.DetectMultiScale () function detects objects of different sizes in the input grayscale image and returns the detected objects as a rectangular list. For example, the following code is used to detect faces:

faces = face_cascade.detectMultiScale(gray, 1.3.5)
Copy the code

Next, walk through the detected faces and try to detect noses:

# Walk through the faces detected
for (x, y, w, h) in faces:
    # Create area of interest (ROI) based on the size of faces detected
    roi_gray = gray[y:y + h, x:x + w]
    roi_color = frame[y:y + h, x:x + w]

    # Detect the nose within the detected face
    noses = nose_cascade.detectMultiScale(roi_gray)
Copy the code

Once the nose is detected, traverse all detected noses and calculate the area that will be covered by the “beard” pendant. After filtering out the wrong nose position, overlay the “beard” pendant on the image according to the previously calculated area:

for (nx, ny, nw, nh) in noses:
    # Calculate the coordinates of the area to be covered by the "beard" pendant
    x1 = int(nx - nw / 2)
    x2 = int(nx + nw / 2 + nw)
    y1 = int(ny + nh / 2 + nh / 8)
    y2 = int(ny + nh + nh / 4 + nh / 6)

    if x1 < 0 or x2 < 0 or x2 > w or y2 > h:
        continue

    # Calculate the size of the area to be covered by the "beard" pendant
    img_moustache_res_width = int(x2 - x1)
    img_moustache_res_height = int(y2 - y1)

    # Adjust the mask size so that it is equal to the area where the "beard" pendant is placed
    mask = cv2.resize(img_moustache_mask, (img_moustache_res_width, img_moustache_res_height))

    # Flip mask
    mask_inv = cv2.bitwise_not(mask)

    # Adjust the "Beard" pendant to the desired area
    img = cv2.resize(img_moustache, (img_moustache_res_width, img_moustache_res_height))
    # Get the ROI of the original image
    roi = roi_color[y1:y2, x1:x2]
    # Create the ROI context and ROI vision
    roi_bakground = cv2.bitwise_and(roi, roi, mask=mask_inv)
    roi_foreground = cv2.bitwise_and(img, img, mask=mask)
    # get results
    res = cv2.add(roi_bakground, roi_foreground)
    Place the RES in the original image
    roi_color[y1:y2, x1:x2] = res
    break
Copy the code

The key to the above program is the img_mustache_mask image. This image was created using the Alpha channel of the “Beard” image to be fused. Using this image, it will be possible to draw only the foreground of the overlaid image. The “beard” mask created based on the Alpha channel of the fused image is as follows:

img_moustache = cv2.imread('moustache.png', -1)
img_moustache_mask = img_moustache[:, :, 3]
cv2.imshow("img moustache mask", img_moustache_mask)
Copy the code

The obtained beard mask is as follows:

Here’s what happens when augmented reality is combined with the Beard widget:

The processing of video frames is not explained here, as it is the same as a single image. The code for augmented reality of continuous video frames is given in the following complete code.

The complete code

import cv2

# Cascade classifier for face and nose detection
face_cascade = cv2.CascadeClassifier("haarcascade_frontalface_default.xml")
nose_cascade = cv2.CascadeClassifier("haarcascade_mcs_nose.xml")

# Load the beard image
img_moustache = cv2.imread('moustache.png', -1)
# Create a beard mask
img_moustache_mask = img_moustache[:, :, 3]
# Convert the beard image to a BGR image
img_moustache = img_moustache[:, :, 0:3]

Create VideoCapture object
video_capture = cv2.VideoCapture(0)

while True:
    Capture frames from the VideoCapture object
    ret, frame = video_capture.read()
    # Convert frame to grayscale image
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
    # Face detection
    faces = face_cascade.detectMultiScale(gray, 1.3.5)

    # Iterate over the faces detected
    for (x, y, w, h) in faces:
        # Create ROI based on detected face size
        roi_gray = gray[y:y + h, x:x + w]
        roi_color = frame[y:y + h, x:x + w]
        # Detect noses among detected faces
        noses = nose_cascade.detectMultiScale(roi_gray)

        for (nx, ny, nw, nh) in noses:
            # Calculate the coordinates where the "beard" pendant will be placed
            x1 = int(nx - nw / 2)
            x2 = int(nx + nw / 2 + nw)
            y1 = int(ny + nh / 2 + nh / 8)
            y2 = int(ny + nh + nh / 4 + nh / 6)

            if x1 < 0 or x2 < 0 or x2 > w or y2 > h:
                continue

            # Calculate the size of the "beard" pendant area
            img_moustache_res_width = int(x2 - x1)
            img_moustache_res_height = int(y2 - y1)

            # Scale the "Beard" mask according to the pendant area
            mask = cv2.resize(img_moustache_mask, (img_moustache_res_width, img_moustache_res_height))
            # Flip mask
            mask_inv = cv2.bitwise_not(mask)
            # Zoom the "Beard" widget
            img = cv2.resize(img_moustache, (img_moustache_res_width, img_moustache_res_height))

            # Get ROI from raw images
            roi = roi_color[y1:y2, x1:x2]
            # Create ROI vision and context
            roi_bakground = cv2.bitwise_and(roi, roi, mask=mask_inv)
            roi_foreground = cv2.bitwise_and(img, img, mask=mask)

            # roi_bakground > roi_foreground
            res = cv2.add(roi_bakground, roi_foreground)
            roi_color[y1:y2, x1:x2] = res

            break
    # display result
    cv2.imshow('Snapchat-based OpenCV moustache overlay', frame)

    Press the "Q" key to exit
    if cv2.waitKey(1) & 0xFF= =ord('q') :break

# Release resources
video_capture.release()
cv2.destroyAllWindows()
Copy the code

A link to the

OpenCV is based on unmarked augmented reality details