Writing in the front
QR Code is a super popular coding method on mobile devices in recent years (also known as TWO-DIMENSIONAL barcode, QR Code, QR full name QuickResponse), it can store more information than the traditional Bar Code (Bar Code), but also can represent more data types. With the popularity of QR code, almost most handheld devices and mobile apps support qr code scanning recognition. In iQiyi mobile terminal, the main function of two-dimensional code is to connect PC client, web terminal, TV terminal tools, can quickly guide users to login account, membership and other payment business payment, operation activities through, small program entrance, etc. In order to provide users with a faster and more convenient experience, iQiyi technical product team has specially optimized the qr code scanning performance. The recognition rate has increased from 30% to about 75%, and the average time has decreased from 2.1s to 722ms. Iqiyi’s experience in scanning code optimization practice will be shared in detail below.
background
Two-dimensional code has become the main online and offline connection tool due to its advantages of low cost, good compatibility and more stored information, and provides a faster entrance for mobile device interaction. For most of the standard clear TWO-DIMENSIONAL code scene, the use of open source ZXing or ZBar library can be a good completion of two-dimensional code recognition. However, because the TWO-DIMENSIONAL code is a dot-matrix information coding method, any visual loss, bending deformation, lighting, screen noise and so on will greatly interfere with the success rate and speed of recognition, affecting the user experience and conversion rate. In some special scenarios, ZXing/ZBar is difficult to meet business requirements due to low recognition rate.
These special scenarios include the following aspects:
1. The QR code has a dark background with a black box around it and no clear dividing line
2. The QR code is gray with low contrast
3, two-dimensional code tensile deformation, tilt distortion
4. Due to illumination interference, two-dimensional code image illumination is uneven and local area is too dark
Originally, the open source ZXing library directly integrated with iQiyi APP realized the scanning function without too much customization and optimization, resulting in poor recognition experience in these scenarios. After 2-3 iterations, iQiyi APP’s scanning performance index has been improved qualitatively. The internal sample set performance test results show that the recognition rate has increased from the initial 30% to about 75%, and the average time spent has decreased from 2.1s to 722ms. Targeted optimization was mainly carried out from three aspects: (1) scanning speed optimization, (2) interactive experience optimization, (3) recognition rate optimization.
Due to the difference of qr code sample set, the test performance data is only for reference:
| version | The success rate | Average Time consuming (ms) |
|---|---|---|
| V10.11.0 | 75% | 722.18 |
| V10.10.0 | 48% | 1471.9 |
| V10.9.0 | 30% | 2096.7 |
Scan speed optimization
1. Remove unnecessary format conversion and rotation operations ****
When analyzing the code, I found that the original logic in the camera frame ‘onPreviewFrame()’ callback processing frame pixel, first YUV data format into RGB format, and at the same time due to the relationship of vertical and horizontal screen, the pixel array rotated 90°. The processed pixel array is then passed to ‘RGBLuminanceSource’, which is handed to the ZXing decoder. After breakpoint analysis, it was found that YUV to RGB and rotation took a lot of time, and the larger the preview array was, the longer the time was.
ZXing itself provides’ YUVLuminanceSource ‘which can directly decode YUV data. In YUV format, Y component is gray component, with higher contrast than RGB. Directly hand over YUV format data to ZXing decoding, saving conversion time.
Camera preview is horizontal coordinate system, and the mobile APP is usually portrait sweep pattern, and we found ZXing decoding qr code and its analysis is horizontal or vertical relationship of the image is not large, can identify, so can not rotate pixel array, a rotating coordinate system, cut out of the area only need the rectangular box four do the coordinates transformation, Setting it to ‘YUVLuminanceSource’ saves time by eliminating the need to rotate all the pixels.
2. Reduce the decoding format ****
With ZXing decoding, the MultiFormatReader is generally used, which maintains an array of readers. By default, it supports one-dimensional bar codes (UPC-A, UPC-E, EAN-8, Code 39, Code 93, etc.) and two-dimensional bar codes (QR Code, QR Code, etc.). Data Matrix, PDF 417, MaxiCode) and other more than 20 formats, the cycle traversal decoding failure time will be prolonged, affecting the decoding of the next frame. In combination with the features of iQiyi APP scan code service and practical application, we cut the ZXing decoding format to support only two-dimensional code, which greatly reduces the decoding failure time (unrecognized frames) and indirectly optimizes the overall decoding time.
3. Set the preview size and clip the scan box ****
By default, ZXing provides the preview frame size selection algorithm to obtain the screen size on most devices. The higher the screen resolution, the clearer the image, the more pixels, and the more time it takes to decode a frame. For two-dimensional code scanning, locating the zigzag detection image and sampling decoding of two-dimensional code does not require a very high resolution. Therefore, modify the preview frame size Settings, adjust the matching algorithm, select the most close to the conventional preview size, and then cut the size of the TWO-DIMENSIONAL code scanning box to ZXing decoding. In order to increase the robustness of scan recognition, a square area of screen width is selected as the decoding region at intervals, which not only improves the robustness of recognition, but also speeds up the decoding recognition speed.
4. Serial change to parallel processing ****
ZXing decoding operation is to use a HandlerThread message queue processing, is the single thread model, when a frame processing is completed through ` setOneShotPreviewCallback () ` requests the camera ready for the next frame, downside is that if processing a frame data for a long time, It will block the processing of the next frame. For example, if a frame is blurred or there is no TWO-DIMENSIONAL code data, the user still needs to wait for a period of time to recognize the two-dimensional code after focusing on it.
When we use ‘setPreviewCallback()’ to generate consecutive preview frames, modify the serial decoding to parallel decoding, and submit an AsynTask to the thread pool each time the ‘onPreviewFrame()’ data arrives. The thread pool size can be dynamically configured. Properly setting the size of thread pool can greatly speed up parallelization and identification.
Optimization of interactive experience
1. Optimize the focusing mode and set fixed-point focusing ****
ZXing default is the ‘AUTO_FOCUS’ focusing mode, interval 1.5s trigger an autoFocus, the time is very long, poor experience on some devices, can shorten the focusing time, but in the ‘autoFocus()’ instant image will become blurred. FOCUS_MODE_CONTINUOUS_PICTURE defaults to FOCUS_MODE_CONTINUOUS_PICTURE, so that after the first focus is clear, subsequent frames are clear.
When the focus area is not set, the camera defaults to full-screen focus, and when the QR code picture is far away and the surrounding environment is dark, auto focus is very slow. By using ‘setFocusAreas()’ and ‘setMeteringAreas()’ to set the focusing area and metering area as the scanning frame area, the focusing speed and recognition success rate of the two-dimensional code in the remote image can be greatly improved.
2. Support automatic scaling ****
When the QR code area is very small and far away, automatic enlargement can greatly improve the user experience and recognition rate. The qr code decoding by ‘QRCodeReader’ is mainly divided into two steps :(1) identify the coordinates of the detection position of the qr code glyphs and obtain the matrix by sampling; (2) restore the data by RS decoding according to the decoding rules. The ‘DetectorResult’ decoded in the first step contains the coordinates of the anchor point and the alignment point. If the QR code is very small, incorrect pixels may be collected in the sampling process, and the second decoding process may still fail. In this scenario, we can roughly estimate the size of the two-dimensional code according to the information of the registration point, and then compare it with the size of the scanning frame, and make automatic amplification according to the focal length of the camera. After several amplification operations, if the decoding still fails, and reset to the default multiple, to prevent some noise location error estimate two-dimensional code size deviation, resulting in has been in the amplification state, poor experience.
3. Click Focus, double-click zoom ****
When the focus of some cameras is slow or the illumination environment is not uniform, users can click a point on the screen to focus, improving the clarity of the camera preview image. When the user double-clicks the screen, the user can choose to zoom in or out according to the current camera magnification.
4 support double finger zoom to adjust the focal length ****
When the automatic zoom to the unreasonable area or the TWO-DIMENSIONAL code has been too far to recognize, the user can manually use the two-finger zoom camera, to achieve the purpose of auxiliary recognition.
Scan code recognition rate optimization
In the process of two-dimensional code scanning recognition, it is mainly divided into the following steps: 1) binarization 2) positioning position detection graph 3) looking for alignment points 4) perspective transformation correction 5) pixel sampling decoding. In our two-dimensional code test set, the naked eye can see very clear images, but ZXing can not recognize how, in order to further optimize the recognition rate, can only study and improve the Algorithm of ZXing. There are probably the following scenarios and strategy optimization:
Policy 1: Add the N: 1:3:1:1 scan mode
When ZXing locates the zigzag position pattern, it is designed for the TWO-DIMENSIONAL code with a white background, and the state machine mode adopted is only 1:1:3:1:1 when the icon is located. In fact, for some black box or dark background two-dimensional code, because the zigzag pattern and background are fused together, there is no obvious boundary, ZXing state machine can not be located.
Figure 1 Example of dark QR code
To support this scenario, the logic of the ZXing anchor point needs to be modified, with the core code in the ‘FinderPatternFinder’ class adding consideration for boundary overlap scenarios. There are two cases of boundary coincidence :N :1:3:1:1 or 1:1:3:1:N. When the state machine array is 5, check whether the central part meets 1:3:1, and then determine whether the left boundary or the right boundary overlaps, and make certain corrections. The corresponding diagonal check logic in the source code also needs to be modified synchronously, while increasing the reverse diagonal check strategy.
Strategy 2: Optimize the estimation algorithm of the point in the lower right corner
For the TWO-DIMENSIONAL code with version 1, there is no correction point. In the scene without parallel front 90°, or the scene with pollution or damage of correction point, perspective transformation degrades to affine transformation, and ZXing recognition effect is poor.
Figure 2 Example of correcting qr code
For this scenario, we use the lower-left and upper-right glyph, locate the lower-right point by locating the lower-right boundary and the lower-right boundary, and then use direct intersection to find the correct lower-right point. In the case of finding the correction point, the correction point is used to do perspective transformation. In the case of the scene without the correction point, the perspective transformation is made by using the point at the lower right corner of the re-estimation, which greatly enhances the two-dimensional code recognition of the tilted version 1.
Strategy 3: Adjust the registration point filtering threshold and sorting rules
After finding potential anchor point coordinates, ZXing will carry out a round of screening and filtering, find three points that meet the conditions, and then sort to determine the order of lower left, upper left and upper right points. In the screening process, the moduleSize with the maximum registration point should not exceed 40% of the moduleSize with the minimum. However, for some two-dimensional codes (FIG. B) with serious skew, the difference will be greater than 40%. Therefore, the tolerance threshold can be adjusted appropriately to more than 60% to increase fault tolerance. For other two-dimensional codes with very serious tilt (FIG. A), when the length of adjacent sides is larger than the diagonal, the longest side is regarded as the diagonal by default by ZXing, so the sequence relation of the three points will be located incorrectly, resulting in the wrong estimation of the position of the fourth vertex, resulting in recognition failure.
For this scenario, we made the following policy adjustments. In ‘ResultPoint#orderBestPatterns()’, increase the maximum width and height of the image, estimate the position of the fourth point in the first round using the longest edge as a diagonal, and sort again using the second edge as a diagonal if the fourth point is clearly outside the image. This is a very serious tilt of the TWO-DIMENSIONAL code can also be normal correction identification.
FIG. 3 Example of tilted TWO-DIMENSIONAL code
Strategy 4: Use different binarization algorithms
The binarization algorithm determines the accuracy of locating points. The so-called binarization processing is to set a threshold value, pixels larger than the threshold value are set as white, and pixels smaller than the threshold value are set as black. For some gray pictures with low contrast or uneven illumination, the built-in binarization algorithm of ZXing cannot be used to restore the image well and separate the zigzag pattern.
We analyze and study ZXing with binarization ‘HibridBinarizer’ and several commonly used local binarization algorithm effect. After comprehensive evaluation, the auxiliary strategy of local mean jump frame execution is finally selected. Some frames use ZXing with binarization recognition, and some frames use local mean binarization recognition, which greatly improves the recognition rate of anti-illumination uneven and low contrast TWO-DIMENSIONAL code scenes.
FIG. 4 Binary image
Strategy 5: Integrate OpencV preprocessing ****
For some of the two-dimensional code in the center of the zigzag blur or with salt and pepper noise, it is easy to interfere with the location and identification of the zigzag after binarization. It is necessary to do some filtering and noise reduction pretreatment on the image, such as median filtering/Gaussian filtering, which can significantly reduce the interference of salt and pepper noise.
For the two-dimensional code of partial rotation tilt Angle, because ZXing uses horizontal and vertical line scanning algorithm to locate the zigzag pattern, after the Angle tilt is actually scanned to the diagonal of the TWO-DIMENSIONAL code, although the diagonal will follow the principle of 1:1:3:1:1, but only in a small area near the center point to meet the rule. And ZXing under normal circumstances is skipping scan, hard mode is progressive scan, it is easy to skip key points, resulting in locating pattern search failure. In fact, as long as the TWO-DIMENSIONAL code is rotated to the horizontal direction, it can be immediately recognized.
According to this part of the scene, we integrated the OpenCV library, computer vision for camera to qr code gray image preprocessing, filtering noise reduction, interception center of qr code, affine matching (change) to rotation changes, stretching into a fixed pixel size of binary image, and then to ZXing do decoding, raised some scenarios.
Other strategies
In addition to the general scan code optimization mentioned above, we also carried out some strategic scheduling optimization for some special scenarios.
1. For the two-dimensional code with partial bending distortion, when there is curved surface bending, it is difficult to sample correctly using normal screen perspective transformation. It is possible to fit this relationship by assuming that the sampling coordinate system follows a more complex mapping, such as a quadratic function. When the version number of two-dimensional code is greater than 7, there are more than 4 auxiliary correction points. By sampling more points, this matrix relationship can be constructed to achieve the effect of recognizing the surface scene.
2, increase the recognition ability of invert color TWO-DIMENSIONAL code. By default, ZXing only supports two-dimensional code with white background and black background. In the process of ZXing positioning zigzag position detection pattern, we increase the mechanism of collecting potential anti-color pile points. When forward identification cannot be done, fallback to reverse identification strategy.
future
\
We in the process of scanning code optimization, also refer to some public information, from which some experience summary, we have made a lot of targeted modifications in ZXing source code, enhance the robustness, the strategy has also been tuned, the follow-up plan to optimize the version of ZXing open source contribution to github community.
The size, clarity and complexity of qr codes are also factors affecting the recognition rate. Scanning code optimization is a long-term work, we will further combine iQiyi’s own business characteristics to continue to optimize, from the generation of TWO-DIMENSIONAL code to client identification and other links to explore optimization. For example, standard standardization of TWO-DIMENSIONAL code, exploration of using TensorFlow image recognition and detection of two-dimensional code area, attempt to use RenderScript GPU to do binarization calculation, etc.
Resources ****
1. The most comprehensive technical scheme of Alipay qr code scanning optimization in history
www.infoq.cn/article/hZg…
2. Lark QR code scanning optimization
zhuanlan.zhihu.com/p/44845942
3. Optimization of QR code decoding on smart devices
Cardinfolink. Making. IO / 2017/06/28 /…
end
Maybe you’d like to see more
PWA technology analysis and iQiyi PC terminal practice
Android article | iQIYI App start sharing of best practice
Scan the qr code below, more exciting content to accompany you!
