Using Python and OpenCV to achieve lane line detection, detailed steps.

Compiled by Dt Pham McGL

In this project, I built a pipeline using Python and OpenCV to detect lane lines. The pipeline contains the following steps:

  • Camera Calibration
  • Perspective Transformation
  • Color thresholding and Region masking
  • Finding Lane Pixels
  • Measuring Lane curves and curvatures
  • Display the result back to original image

1. Calibrate the camera

Distortion occurs when a camera looks at a 3D object in the real world and converts it into a 2D image; The transition is not perfect. The distortion actually changes the shape and size of these 3D objects. So the first step in analyzing a camera image is to eliminate this distortion so you can get useful information from it.

Two types of distortion are of interest to us: radial distortion and tangential distortion.

To calibrate the camera, we can take pictures of known shapes and correct for any distortions. The most common target for this task is the checkerboard because of its high contrast mode.

First, we need to take a lot of checkerboard images with our camera and detect all the corners of these images. This can be done using the function cv2.findChesspardCorners () in OpenCV.

After that, we will use Cv2. CalibrateCamera () to solve for the distortion factor. Three coefficients are needed to correct radial distortion: K1, K2 and K3; The correction of tangential distortion requires two coefficients: P1 and P2. The formula for calculating the coordinates of distortion points is as follows, where R is the known distance between a point in the undistorted image and the distortion center of the image, which is usually the center of the image (X_c, y_c).

▲ Formula of radial distortion

▲ Tangential distortion formula

▲ Results after distortion correction

2. Perspective transformation

In this step, we will convert the image to a bird ‘s-eye view. This will make subsequent steps such as measuring lane curvature easier. To image transform, first you need to take four points from the source image, and from the target image from four points, and then use the function cv2. GetPerspectiveTransform transform (). This function evaluates a 3×3 transformation matrix, which is required when we want to transform the image through the cv2.warpPerspective() function.

▲ Images before and after perspective transformation

3. Color threshold and area mask

▲ HSV and HLS color space

HLS (Hue, lightness, Saturation) is an alternative representation of the RGB model. HLS is cylindrical geometry, and hue (the Angle of dimension) is the actual color. Lightness refers to how much white (or black) is mixed into the color, while saturation refers to how much gray is in the color. A saturation value of 0 means mostly gray and 100% light (L=255) is white.

Lane lines are white and yellow, and both have saturation values within a specific range. Therefore, I select only pixels with saturation values in this range. By filtering out pixels with smaller brightness values, white can be detected in the brightness channel.

▲ Hue, brightness and saturation values

Area mask is the process of eliminating parts of an image that are unlikely to contain lane lines. We can see that lane lines appear mostly in the lower half of the image, so masking pixels in the upper half of the image will improve our accuracy in detecting lane lines, since the image now only contains pixels that are more likely to be part of the lane lines.

4. Look for lane pixels

Next, we need to classify which pixels are in the left lane, right lane, or neither. After that, we will find the polynomial equation that best fits all left-lane pixels and another equation that best fits all right-lane pixels.

First, I made a histogram of all the columns in the bottom half of the image. In our thresholding binary image, the pixels are either 0 or 1, so the two most prominent peaks in this histogram will be a good indicator of the x position of the lane line baseline.

We can use that as a starting point for our search line. Then, on the image we can use a sliding window to move up (further along the road) to determine where the lane line is going.

▲ Use sliding window to detect lane line pixels

5. Find the curve of the lane and measure the curvature

We have estimated which pixels belong to the left and right lines (shown in blue and red, respectively), and we have fitted the polynomial for the positions of these pixels. We can use this polynomial to calculate the radius of lane curvature and the distance of the vehicle from the center of the lane.

Recall our equation for finding lane lines:

The reason this is a function of y and not x is that, since the lane lines are nearly vertical in the transformed image, it is possible for multiple Y values to have the same X value.

Here x and y are in pixels, but we want to convert this to meters to calculate lane curvature and vehicle position. Assume that the distance between the two lane lines (pixel coordinates) is 700 pixels. In real life, that distance is about 12 feet, or 3.7 meters. So each pixel in the horizontal direction is equivalent to 3.7/700 meters in real life. We call this value mx = 3.7/700. Doing the same in the vertical direction, we get my = 30/720, 30 meters for every 720 vertical pixels. The conversion process looks like this:

▲ Conversion from pixel coordinates to real world coordinates

The radius of curvature at any point of function x=f(y) can be obtained as follows:

This is easy to calculate, because we only need to compute the first and second derivatives of our function.

Next, we calculate how far our car is from the center of the lane. This can be done by calculating the horizontal distance, in pixels, between the center of the lane and the center of the image. After that, we can multiply by the constant mx to convert this to a real distance.

6. Display the result back to the original image

The final step is to combine the results of the previous step with the original image.

To do this, we need to restore the perspective transform we made earlier and place the output image on top of the original.

▲ Original image and output image

conclusion

This particular pipeline worked well under normal and cloudy conditions (where lane lines are hard to see). However, because this pipeline has a limited number of parameters, it is not rubon enough to perform well in extreme conditions such as rain and snow.

All in all, by putting a few algorithms together, we can create a pipeline that can detect lane lines. First, we corrected the camera distortion. We then converted it to a bird’s eye view, filtered out the irrelevant parts of the image, and used a “sliding window” to find lane pixels. Finally, the lane line equation is calculated and the lane curvature is measured. For detailed implementation, please check out my Github project: github.com/Dt-Pham/Adv…

Source: medium.com/@dt99.pham/…