Optical flow and video feature point tracking in OpenCV

This blog will introduce the concept of optical flow and how to estimate optical flow using the Lucas-Kanade method, and demonstrate how to use Cv2.calCopticalFlowPyRlk () to track feature points in a video.

1. The rendering

The effect picture of optical flow tracking is as follows:

It shows a ball moving in 5 consecutive frames. The arrow represents its displacement vector.

Not very rigorous — The effect picture of sparse optical flow feature point tracking is as follows:

It tracks the edges of the drivers, co-drivers, and pedestrians in the video:

This code does not check the correctness of the next key point. So even if any feature points in the image disappear, it is possible for the optical flow to find the next point that seems likely to be close to it. For robust tracing, corner points should detect points at specific time intervals.

The first process diagram is as follows: The tracking effect of sparse optical flow feature points in the optimized version is as follows:

Find the feature points and check the optical flow points backwards every 30 frames, keeping only the feature points that still exist on the screen. There is no such thing as a car that has passed by and has a long and incorrect track track. The first process diagram is as follows:

The original image VS dense optical flow tracking GIF is as follows:

The original VS Hsv effect is as follows:

Principle 2.

2.1 What is optical flow? The premise and principle of optical flow tracking

Optical flow is the pattern of apparent motion of an image object between two consecutive frames caused by the motion of the object or camera. It is a 2D vector field where each vector is a displacement vector showing the movement of a point from frame 1 to frame 2.

The premise of optical flow tracking is: 1. The pixel intensity of the object does not change between consecutive frames; 2. Adjacent pixels have similar motion.

Principle of optical flow tracking:

Cv2. goodFeaturesToTrack() : shi-tomasi corner detector determines the feature points to track

Cv2.calcopticalflowpyrlk () : Trace the sparse feature points in the video

Cv2. CalcOpticalFlowFarneback () : track intensive feature point in the video

Take the first frame, detect some Shi-Tomasi corner points, and track these points iteratively using Lucas-Kanade optical flow. For the function cv2.calCopticalFlowPyRlk (), pass the previous frame, the previous point, and the next frame. It returns the next point as well as some state number, 1 if the next point is found and zero otherwise. These next points are then iteratively passed as the previous point in the next step.

Harris corner detector was used to check the similarity of the inverse matrix. It means that corner points are better tracking points. Shi-tomasi corner detector is better than Harris corner detector.

2.2 Application of optical flow

Optical flow has many applications in the following areas:

Structure of motion
Video compression
Video stabilization

2.3 Two methods of optical flow

OpenCV provides two kinds of calculation optical flow algorithm, respectively by: cv2. CalcOpticalFlowPyrLK (), cv2. CalcOpticalFlowFarneback implementation;

Sparse optical flow: The optical flow of the sparse feature set is calculated by the Lucas-Kanade method (corners detected by the Shi-Tomasi algorithm).
Dense Light Flow: Find dense light flow with Gunner Farneback. It calculates the optical flow at all points in the frame.

Sparse optical flow calculation:

This method passes the previous frame, the previous point, and the next frame; It returns the next point as well as some state number, 1 if the next point is found and zero otherwise.

p1, st, err = cv2.calcOpticalFlowPyrLK(old_gray, frame_gray, p0, None, winSize=(15, 15), maxLevel=2, The criteria = (cv2 TERM_CRITERIA_EPS | cv2. TERM_CRITERIA_COUNT, 10, 0.03))

-old_gray: single-channel gray map of the previous frame -frame_gray: single-channel gray map of the next frame -prepts: P0 Coordinates of the previous frame Pts-nextpts: None-winsize: Size of the search window at each pyramid level - maxLevel: maximum number of pyramid layers - Criteria: Specifies the termination condition of the iterative search algorithm after the specified maximum number of iterations is 10 or the search window moves less than 0.03Copy the code

Dense optical flow calculation:

This method results in a 2-channel array with an optical streamer (u, V). You can find their size and orientation, and then color code the results for better visualization. In HSV images, the direction corresponds to the tone of the image and the amplitude corresponds to the value plane.

Flow = cv2. CalcOpticalFlowFarneback (PRVS, next, None, 0.5, 3, 15, 3, 5, 1.2, 0)

-PRVS: last frame single channel grayscale -next: next frame single channel grayscale -flow: flow none-pyr_scale: 0.5 Classic pyramid, build pyramid scale - Level: 3 Number of pyramid layers in the initial image - Winsize: 3 Average window size, the larger the value is, the more robust the algorithm is to the image - Iterations: 15 iterations - poly_n: The parametric polygon size of the 5-pixel neighborhood, used to find polynomial expansions in each pixel; Larger values mean that the image will be approximated using smoother surfaces, resulting in higher resolution, robust algorithms, and fuzzier playing fields; Usually polygons n=5 or 7. -flags: can be a combination of the following flags: OPTFLOW_USE_INITIAL_FLOW: Uses the input flow as the initial flow approximation. OPTFLOW_FARNEBACK_GAUSSIAN: GAUSSIAN filters are used instead of box filters of the same size.Copy the code

3. The source code

3.2 Sparse optical flow tracking

# Optical flow tracking
The premise of optical flow tracking is: 1. The pixel intensity of the object does not change between consecutive frames; 2. Adjacent pixels have similar motion.
# -cv2.goodFeaturestoTrack () identifies feature points to track
# -cv2.calCopticalFlowPyRlk () tracks feature points in the video

# Take the first frame, detect some shi-Tomasi corner points, and track these points iteratively using Lucas-Kanade optical flow.
# For the function cv2.calCopticalFlowPyRlk (), pass the previous frame, the previous point, and the next frame. It returns the next point as well as some state number, 1 if the next point is found and zero otherwise.
These next points are then iteratively passed as the previous point in the next step.

# USAGE
# python video_optical_flow.py

import imutils
import numpy as np
import cv2

cap = cv2.VideoCapture('images/slow_traffic_small.mp4')

# ShiTomasi corner detection parameters
feature_params = dict(maxCorners=100,
                      qualityLevel=0.3,
                      minDistance=7,
                      blockSize=7)

# Lucas Kanada optical flow detection parameters
lk_params = dict(winSize=(15.15),
                 maxLevel=2,
                 criteria=(cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT, 10.0.03))

# Build random colors
color = np.random.randint(0.255, (100.3))

Get the first frame and find the corner
ret, old_frame = cap.read()
old_gray = cv2.cvtColor(old_frame, cv2.COLOR_BGR2GRAY)
p0 = cv2.goodFeaturesToTrack(old_gray, mask=None, **feature_params)

Construct a Mask to map the optical flow tracing
mask = np.zeros_like(old_frame)

num = 0
while (1):
    ret, frame = cap.read()

    if not ret:
        break

    frame_gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

    # Using the iterative Lucas Kanade method to calculate the optical flow of sparse feature sets
    # -old_gray: single channel grayscale image of the previous frame
    # -frame_gray: Next single channel grayscale image
    # -prepts: p0 last frame coordinates PTS
    # - nextPts: None
    # -winsize: Size of search window at each pyramid level
    # -maxlevel: maximum number of pyramid layers
    # -criteria: Specifies the end condition of the iterative search algorithm, after the specified maximum number of iterations. MaxCount or the search window moves less than criteria. Epsilon
    p1, st, err = cv2.calcOpticalFlowPyrLK(old_gray, frame_gray, p0, None, **lk_params)

    # Select the trajectory point
    good_new = p1[st == 1]
    good_old = p0[st == 1]

    # Draw trajectory
    for i, (new, old) in enumerate(zip(good_new, good_old)):
        a, b = new.ravel()
        c, d = old.ravel()
        mask = cv2.line(mask, (a, b), (c, d), color[i].tolist(), 2)
        frame = cv2.circle(frame, (a, b), 5, color[i].tolist(), -1)
    img = cv2.add(frame, mask)

    cv2.imshow('frame', img)
    cv2.imwrite('videoof-imgs/' + str(num) + '.jpg', imutils.resize(img, 500))
    print(str(num))
    num = num + 1
    k = cv2.waitKey(30) & 0xff
    if k == 27:
        break

    # Update the previous frame and point
    old_gray = frame_gray.copy()
    p0 = good_new.reshape(-1.1.2)

cv2.destroyAllWindows()
cap.release()
Copy the code

3.2 Optimized sparse optical flow tracking

# Optimized optical flow tracking -- Lucas-Kanade Tracker
Even if any feature points in the image disappear, it is possible for optical flow to find the next point that appears to be close to it when the next key point is not checked for correctness. In fact, for robust tracing, corner points should detect points at specific time intervals.
# After finding the feature points, backward check the optical flow points every 30 frames, select only the good ones.
# Lucas Kanade Sparse optical flow demonstration. Use GoodFeatures trace between backtracking frames used to track initialization and match validation.
# Lucas-Kanade sparse optical flow demo. Uses goodFeaturesToTrack for track initialization and back-tracking for match verification between frames.

# Usage
# pyhton lk_track.py images/slow_traffic_small.mp4
# Press ESC to exit

from __future__ import print_function

import imutils
import numpy as np
import cv2


def draw_str(dst, target, s) :
    x, y = target
    cv2.putText(dst, s, (x + 1, y + 1), cv2.FONT_HERSHEY_PLAIN, 1.0, (0.0.0), thickness=2, lineType=cv2.LINE_AA)
    cv2.putText(dst, s, (x, y), cv2.FONT_HERSHEY_PLAIN, 1.0, (255.255.255), lineType=cv2.LINE_AA)


lk_params = dict(winSize=(15.15),
                 maxLevel=2,
                 criteria=(cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT, 10.0.03))

feature_params = dict(maxCorners=500,
                      qualityLevel=0.3,
                      minDistance=7,
                      blockSize=7)


class App:
    def __init__(self, video_src) :
        self.track_len = 10
        self.detect_interval = 30
        self.tracks = []
        self.cam = cv2.VideoCapture(video_src)
        self.frame_idx = 0

    def run(self) :
        while True:
            _ret, frame = self.cam.read()
            if not _ret:
                break

            frame_gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
            vis = frame.copy()

            if len(self.tracks) > 0:
                img0, img1 = self.prev_gray, frame_gray
                p0 = np.float32([tr[-1] for tr in self.tracks]).reshape(-1.1.2)
                p1, _st, _err = cv2.calcOpticalFlowPyrLK(img0, img1, p0, None, **lk_params)
                p0r, _st, _err = cv2.calcOpticalFlowPyrLK(img1, img0, p1, None, **lk_params)
                d = abs(p0 - p0r).reshape(-1.2).max(-1)
                good = d < 1
                new_tracks = []
                for tr, (x, y), good_flag in zip(self.tracks, p1.reshape(-1.2), good):
                    if not good_flag:
                        continue
                    tr.append((x, y))
                    if len(tr) > self.track_len:
                        del tr[0]
                    new_tracks.append(tr)
                    cv2.circle(vis, (x, y), 2, (0.255.0), -1)
                self.tracks = new_tracks
                cv2.polylines(vis, [np.int32(tr) for tr in self.tracks], False, (0.255.0))
                draw_str(vis, (20.20), 'track count: %d' % len(self.tracks))

            if self.frame_idx % self.detect_interval == 0:
                mask = np.zeros_like(frame_gray)
                mask[:] = 255
                for x, y in [np.int32(tr[-1]) for tr in self.tracks]:
                    cv2.circle(mask, (x, y), 5.0, -1)
                p = cv2.goodFeaturesToTrack(frame_gray, mask=mask, **feature_params)
                if p is not None:
                    for x, y in np.float32(p).reshape(-1.2):
                        self.tracks.append([(x, y)])

            self.prev_gray = frame_gray
            cv2.imshow('lk_track', vis)
            print(self.frame_idx)
            cv2.imwrite('videoOof-imgs/' + str(self.frame_idx) + '.jpg', imutils.resize(vis, 500))
            self.frame_idx += 1

            ch = cv2.waitKey(1)
            if ch == 27:
                break

def main() :
    import sys
    try:
        video_src = sys.argv[1]
    except:
        video_src = 0

    App(video_src).run()
    print('Done')


if __name__ == '__main__':
    print(__doc__)
    main()
    cv2.destroyAllWindows()
Copy the code

2.3 Intensive optical flow tracking

# Dense optical flow in OpenCV
# Lucas-Kanade method computes the optical flow of the sparse feature set (corners detected using the Shi-Tomasi algorithm).
# OpenCV offers another algorithm: Gunner Farneback to find dense optical streams. It calculates the optical flow at all points in the frame.
# by cv2. CalcOpticalFlowFarneback () will get a with optical flow vector array (u, v) of the two channels. You can find their size and orientation, and then color code the results for better visualization.
# In HSV images, the direction corresponds to the tone of the image and the amplitude corresponds to the value plane.

import cv2
import imutils
import numpy as np

cap = cv2.VideoCapture('images/slow_traffic_small.mp4')

ret, frame1 = cap.read()
prvs = cv2.cvtColor(frame1, cv2.COLOR_BGR2GRAY)
hsv = np.zeros_like(frame1)
hsv[..., 1] = 255

num = 0
while (1):
    ret, frame2 = cap.read()

    if not ret:
        break
    next = cv2.cvtColor(frame2, cv2.COLOR_BGR2GRAY)

    # The iterative Gunner Farneback method is used to compute optical flow with dense features
    # -PRVS: Single channel grayscale image of last frame
    # -next: Next single channel grayscale image
    # -flow: Flow None
    # -pyr_scale: 0.5 classic pyramid, build pyramid scale
    # -level: 3 Number of pyramid layers for the initial image
    # -winsize: 3 Average window size, the larger the value, the stronger the algorithm's robustness to the image
    # -Iterations: 15 iterations
    # -poly_n: the parametric polygon size of the 5-pixel neighborhood, used to find polynomial expansions in each pixel; Larger values mean that the image will be approximated using smoother surfaces, resulting in higher resolution, robust algorithms, and fuzzier playing fields; Usually polygons n=5 or 7.
    # -poly_sigma: 1.2 Gaussian standard deviation, used to smooth derivatives
    # -flags: can be a combination of the following: OPTFLOW_USE_INITIAL_FLOW: Use the input flow as the initial flow approximation. OPTFLOW_FARNEBACK_GAUSSIAN: GAUSSIAN filters are used instead of box filters of the same size.
    flow = cv2.calcOpticalFlowFarneback(prvs, next.None.0.5.3.15.3.5.1.2.0)

    mag, ang = cv2.cartToPolar(flow[..., 0], flow[..., 1])
    hsv[..., 0] = ang * 180 / np.pi / 2
    hsv[..., 2] = cv2.normalize(mag, None.0.255, cv2.NORM_MINMAX)
    rgb = cv2.cvtColor(hsv, cv2.COLOR_HSV2BGR)

    cv2.imshow('Origin VS frame2', np.hstack([frame2, rgb]))
    cv2.imwrite('dof-imgs/' + str(num) + '.jpg', imutils.resize(np.hstack([frame2, rgb]), 600))
    k = cv2.waitKey(30) & 0xff
    num = num + 1
    if k == 27:
        break
    elif k == ord('s'):
        cv2.imwrite('dof-imgs/origin VS dense optical flow HSVres' + str(num) + ".jpg",
                    imutils.resize(np.hstack([frame2, rgb]), width=800))
    prvs = next

cap.release()
cv2.destroyAllWindows()
Copy the code

reference

Docs.opencv.org/3.0-beta/do…

Optical flow and video feature point tracking in OpenCV

Optical flow and video feature point tracking in OpenCV

1. The rendering

Principle 2.

2.1 What is optical flow? The premise and principle of optical flow tracking

2.2 Application of optical flow

2.3 Two methods of optical flow

3. The source code

3.2 Sparse optical flow tracking

3.2 Optimized sparse optical flow tracking

2.3 Intensive optical flow tracking

reference

Related Posts

CB013: TensorFlow, TensorBoard, seq2seq

Most complete machine learning field | SKLearn application guide

7 Regression Models you Should Know!