The target

In this section

  • We will learn the basics of multi-view geometry
  • We will learn about poles, pole lines, pole line constraints, etc.

Basic concept

When we take an image with a pinhole camera, we lose important information, which is the depth of the image. Or how far each point in the image is from the camera, because it’s a 3D to 2D conversion. Therefore, it is an important question whether depth information can be found using these cameras. The answer is to use more than one camera. With two cameras (two eyes), our eyes work in a similar way, which is called stereoscopic vision. So let’s take a look at what OpenCV provides in this field.

(Learning OpenCV through Gary Bradsky has a lot of information in this area.)

Before delving into images, let’s look at some basic concepts in multi-view geometry. In this section, we will discuss antipolar geometry. See the figure below, which shows the basic setup for taking an image of the same scene using two cameras.

If only the left camera is used, the 3D points corresponding to the points in the image cannot be found because each point on the line is projected onto the same point on the image plane. But also think about the right image. Now, points on the line $OX$project to points on the right plane ($x’$). So, using these two images, we can triangulate the correct 3D points. That’s the whole idea.

The projection of the different points forms a line on the right plane $OX$. This is called an Epiline corresponding to this point. This means that to find the point on the correct image, search along the contour. It should be somewhere along the line (thought of this way, matches can be found in other images, rather than searching the entire image, just along the Epiline. Therefore, it can provide better performance and accuracy). This is called the polar constraint. Similarly, all points will have their corresponding epilines in the other image. This plane is called the antipolar plane.

$O$and $O’$are camera centers. From the Settings given above, you can see a projection of the right camera $O’$on the left image at the point. It’s called the pole. Poles are the intersection of lines passing through the center of the camera and the image plane. The same goes for the pole of the left camera. In some cases, you won’t be able to find poles in the image; they may be outside the image (meaning one camera can’t see the other).

All polar lines pass through their poles. Therefore, to find the location of the center line, we can find many center lines and find their intersection.

Therefore, in section, we focus on finding antipolar lines and polar lines. But to find them, we need two other components, the base matrix (F) and the base matrix (E), which contains information about translation and rotation that describes the position of the second camera relative to the first in global coordinates. See the image below (image courtesy of Gary Bradsky: Learning OpenCV):

But we would prefer to measure it in pixel coordinates, right? In addition to the intrinsic information about the two cameras, the basic matrix also contains the same information as the basic matrix, so we can associate the pixel coordinates of the two cameras. If we use the corrected image and divide the focal length by the normalized point, $F=E$. In short, the fundamental matrix F maps points in one image to lines (up) in the other. This is calculated from the matching points of the two images. At least 8 such points are required to find the fundamental matrix (when using the 8-point algorithm). Selecting more points and using RANSAC will yield more reliable results.

code

Therefore, first we need to find as many matches as possible between the two images to find the fundamental matrix. To do this, we use the SIFT descriptor in conjunction with flann-based matchers and ratio tests.

Import numpy as NP import cv2 as CV from matplotlib import pyplot as PLT img1 = CV. Imread ('myleft.jpg',0) # index image # left Image img2 = cv.imread('myright.jpg',0) # right image sift = Cv. sift () # des1 = sift.detectAndCompute(img1,None) kp2, Des2 = sift. DetectAndCompute (IMg2,None) # FLANN_INDEX_KDTREE = 1 index_params = dict(algorithm = FLANN_INDEX_KDTREE, trees = 5) search_params = dict(checks=50) flann = cv.FlannBasedMatcher(index_params,search_params) matches = Flann. KnnMatch (DES1, DES2,k=2) good = [] pts1 = [] pts2 = [] # Pts2.append (kp1[m.kueryidx].pt) pts2.append(kp1[m.kueryidx].pt) pts2.append(kp1[m.kueryidx].pt)Copy the code

We now have a list of the best matches for the two images. Let’s find the fundamental matrix.

pts1 = np.int32(pts1) pts2 = np.int32(pts2) F, Pts1 = pts1[mask.ravel()==1] pts2 = PTS2 [mask.ravel()==1]Copy the code

Next, we find the Epilines. Epilines corresponding to the points in the first image are drawn on the second image. Therefore, it is important to mention the right image here. We’ve got a line. Therefore, we defined a new function to draw these lines on the image.

def drawlines(img1,img2,lines,pts1,pts2): Shape IMg1 = CV. CvtColor (IMG1, CV.COLOR_GRAY2BGR) IMG2 = r,c = IMG1.shape IMG1 = CV cv.cvtColor(img2,cv.COLOR_GRAY2BGR) for r,pt1,pt2 in zip(lines,pts1,pts2): Color = tuple(np.random.randint(0,255,3).tolist()) x0,y0 = map(int, [0, -r[2]/r[1]]) x1,y1 = map(int, [c, -(r[2]+r[0]*c)/r[1] ]) img1 = cv.line(img1, (x0,y0), (x1,y1), color,1) img1 = cv.circle(img1,tuple(pt1),5,color,-1) img2 = cv.circle(img2,tuple(pt2),5,color,-1) return img1,img2Copy the code

The Epiline is now found and plotted in both images.

# Find the pole corresponding to the point in the right image (second image), And then the left map polar lines1 = cv.com puteCorrespondEpilines (pts2. Reshape,1,2 (- 1), 2, F) lines1 = lines1. Reshape img5 (1, 3), img6 = drawlines (img1, img2, lines1, pts1 pts2) # found in the left (the first photo) and corresponding point Epilines, Then in the right image rendering polar lines2 = cv.com puteCorrespondEpilines (pts1. Reshape,1,2 (- 1), 1, F) lines2 = lines2. Reshape img3 (1, 3), img4 = drawlines (img2 img1, lines2, pts2, pts1) PLT, subplot (121), PLT. Imshow (img5) plt.subplot(122),plt.imshow(img3) plt.show()Copy the code

Here’s what we got:

You can see in the image on the left that all poles converge to the outside of the image on the right. That confluence is the pole. For better results, images with good resolution and many non-flat points should be used.

Additional resources

practice

  1. An important theme is camera moving forward. Poles are then seen at the same location in both locations and appear from the fixed point. See this discussion.
  2. Basic matrix estimation is sensitive to the quality of matches, outliers, etc. The situation gets worse if all the selected matches are on the same plane. Check out this discussion.

Panchuang.net/ OpenCV中文 版 : woshicver.com/