CameraX + Huawei ScanKit: The ultimate solution for QR code scanning

This article has participated in the good article call order activity, click to see: back end, big front end double track submission, 20,000 yuan prize pool for you to challenge!

At the beginning of the year, I wrote an article about the use of CameraX, which helped some friends and received some suggestions. I recently learned about the excellent performance of Huawei ScanKit in code scanning scenarios, so I decided to integrate the solution and improve some functions.

The previous Demo was slightly rough, and this time the UI has been adjusted. I added a translucent background to the top action bar and a translucent border to the toggle button to improve contrast. Some color schemes for shooting and recording scenes were also changed.

1. What is Huawei ScanKit

ScanKit can scan, parse, and generate qr codes and bar codes in a convenient way, helping you quickly build the code scanning function in your application.

It has many advantages, including the support of up to 13 code formats, in reflective, defaced, distorted, fuzzy and other complex scenes can be well recognized, in the case of scanning code from a distance can be adaptive amplification code, but also support multi-code recognition function and so on.

ScanKit provides developers with four integration modes, including Default View Mode in the fixed scan window, Customized View Mode in the Customized scan window, and Customized View Mode in the Customized scan window. Bitmap Mode and MultiProcessor Mode are fully customized by the developer.

The scanning process of the first two modes is controlled by ScanKit, and Camera1 is used for its internal implementation. You can only choose the latter two modes if you want to integrate into the CameraX. MultiProcessor Mode Applies to multi-code identification scenarios. The single-code identification Bitmap Mode is integrated first.

For more information about Huawei ScanKit, visit the official website:

Developer.huawei.com/consumer/cn…

And a full demonstration of Yi Dong:

Juejin. Cn/post / 696789…

2. Select the scanning scheme

Zxing was used in the previous scanning code scheme. After integrating ScanKit this time, Zxing is also retained for comparison and learning. After clicking the scan code button, a Fragment will pop up at the bottom to select the scan code scheme, and then inform CameraX’s ImageAnalysis of the corresponding scheme through ViewModel.

※ Google ML Kit is a more powerful OCR solution and will be integrated later

ScanKit may be a concern compared to the advantages of Zxing can refer to the following this review article: developer.huawei.com/consumer/cn…

It is mentioned in this article that ScanKit’s recognition speed and success rate are better than Zxing in such scenarios as long distance scanning code, oblique code body and fuzzy scanning code. You can also use the Demo in this article to choose Zxing and ScanKit schemes respectively and actually compare the scanning code experience.

3. The integrated ScanKit

Add ScanKit’s repository address to the Gradle file of project and add dependencies to the Gradle file of app for rapid integration. ※Demo relies on the scanPlus dependency package, which has better recognition capabilities

// build.gradle
buildscript {
    repositories {
        ...
        mavenCentral()
        maven {url 'https://developer.huawei.com/repo/'}
    }
}

allprojects {
    repositories {
        ...
        mavenCentral()
        maven {url 'https://developer.huawei.com/repo/'}
    }
}
Copy the code

// app/build.gradle dependencies { ... / / Huawei scan kit implementation 'com. Huawei. HMS: scanplus: 1.3.2.300'}Copy the code

3.1 ImageProxy Converts bitmaps

The image instance ImageProxy sent back by CameraX ImageAnalysis is in YUV format. You need to convert it into Bitmap through YuvImage first, and then call the Bitmap scanning code mode of ScanKit.

private fun proxyToBitmap(image: ImageProxy): Bitmap {
    val planes: Array<ImageProxy.PlaneProxy> = image.planes
    val yBuffer: ByteBuffer = planes[0].buffer
    val uBuffer: ByteBuffer = planes[1].buffer
    val vBuffer: ByteBuffer = planes[2].buffer

    val ySize: Int = yBuffer.remaining()
    val uSize: Int = uBuffer.remaining()
    val vSize: Int = vBuffer.remaining()

    val nv21 = ByteArray(ySize + uSize + vSize)
    yBuffer.get(nv21, 0, ySize)
    vBuffer.get(nv21, ySize, vSize)
    uBuffer.get(nv21, ySize + vSize, uSize)

    val yuvImage = YuvImage(nv21, ImageFormat.NV21, image.width, image.height, null)
    val out = ByteArrayOutputStream()
    yuvImage.compressToJpeg(Rect(0.0, yuvImage.width, yuvImage.height), 75.out)

    val imageBytes = out.toByteArray()
    val opt = BitmapFactory.Options()
    opt.inPreferredConfig = Bitmap.Config.ARGB_8888

    var bitmap = BitmapFactory.decodeByteArray(imageBytes, 0, imageBytes.size, opt)
    return bitmap
}
Copy the code

3.2 Calling the Bitmap Scan Mode

Create scanKit-specific scan parameters and pass the converted Bitmap instance to ScanUtil to start identification. The returned recognition result includes the content, coordinates, four corner position and other information, which is encapsulated in the HmsScan object. What ScanUtil actually returns after recognition is an array of HmsScan, the first element of which is the result of single-code recognition. The originalValue property of the HmsScan object is the resolved content.

class HuaweiScanAnalysis: RealTimeAnalysis {
    override fun analyzeContent(imageProxy: ImageProxy, context: Context): AnalysisResult {
        val bitmap = proxyToBitmap(imageProxy)
        imageProxy.close()

        // Create ScanKit scan code parameters
        val options = HmsScanAnalyzerOptions.Creator()
            .setHmsScanTypes(HmsScan.ALL_SCAN_TYPE)
            .setPhotoMode(false)
            .create()

        // Get the scan result
        val result = ScanUtil.decodeWithBitmap(
            context,
            bitmap,
            options
        )

        val content = if(result ! =null && result.isNotEmpty() && result[0].originalValue ! =null)
            result[0].originalValue else "".// Encapsulate the scan result into our custom instance
        return AnalysisResult(content, scale, rect)
    }
}
Copy the code

3.3 Automatic amplification of remote scanning code

When a code is scanned remotely or the code body is too small, ScanKit will calculate the appropriate magnification and assign the value to the zoomValue property of the HmsScan object. The value can be used to inform CameraX to adjust the multiplier of image acquisition in time, so as to improve the subsequent recognition rate. The implementation idea is very simple, using the setZoomRatio provided by CameraControl to enlarge the image preview and analysis of the multiplier. In order not to affect the next scanning experience, the multiplier must be set after the completion of scanning code.

class MyAnalyzer(...). : Analyzer {override fun analyze(image: ImageProxy) {
        viewModel.analysePicture(image).also {
            if(Constants.DEFAULT_ZOOM_SCALE ! = it.zoomScale && Constants.MIN_ZOOM_SCALE ! = it.zoomScale ) { callback.onZoomPreview(it.zoomScale) }else {
                callback.onAnalyzeResult(it)
            }
        }
    }
}

fun onAnalyzeGo(view: View?). {
    if (mAnalyzer == null) {
        mAnalyzer = MyAnalyzer(viewModel, object : AnalyzeCallback {
            override fun onZoomPreview(scale: Double){ mCamera.cameraControl.setZoomRatio(scale.toFloat()) } ... }}})Copy the code

3.4 Success tone and vibration

In order to improve the user experience, you can play the preset prompt tone or vibration feedback while scanning the code successfully, which can be realized by using the open source BeepManager tool class.

fun onAnalyzeGo(view: View?). {
    if (mAnalyzer == null) {
        mAnalyzer = MyAnalyzer(viewModel, object : AnalyzeCallback {
            override fun onAnalyzeResult(result: AnalysisResult){ synchronized(isAnalyzing) { showQRCodeResult(result.content) ... }}}private fun showQRCodeResult(result: String) {
    stopAnalysis()
    beepManager.playBeepSoundAndVibrate()
    ...
}
Copy the code

3.5 Position indicated by drawing code body

At the moment of success, wechat and Alipay App will display a dot on the QR code, which is a better reminder design. The borderRect property of the HmsScan class represents the rectangle position of the code body. The calculated centerX and centerY help to retrieve the center of the code body where an indicator View can be displayed.

Keep in mind that the Analyse pictures in portrait mode are off by 90 degrees, so you need to convert the position coordinates. Of course, if the Bitmap instance has already been rotated 90 degrees, the borderRect value needs no additional transformation. Unfortunately, there are some errors in the calculation of coordinates, and it is difficult to ensure that the indicated position is drawn in the center every time.

override fun onAnalyzeResult(result: AnalysisResult) {
    synchronized(isAnalyzing) {
        showQRCodeResult(result.content)
        val centerPoint = Utils.convertRectToPoint(result.rect, binding.previewView)
        showPointView(centerPoint)
    }
}

private fun showPointView(point: Point) {
    val popupWindow = PopupWindow(
        ViewGroup.LayoutParams.WRAP_CONTENT,
        ViewGroup.LayoutParams.WRAP_CONTENT
    )
    val imageView = ImageView(this)

    runOnUiThread {
        popupWindow.contentView = imageView
        imageView.setImageResource(R.drawable.ic_point_view)
        popupWindow.showAsDropDown(binding.previewView, point.x, point.y)
        binding.previewView.postDelayed({ popupWindow.dismiss() }, 1000)}}Copy the code

3.6 Drawing the border of code body

The border of the QR code can also be displayed when the identification process or success. Although such a design is not necessary, we can try to implement it.

The raw value of the borderRect property is the width and height of the box, and a rectangular box can be drawn on top of it based on the center position above. In fact, in addition to the borderRect, the cornerPoints property gets the exact position of the four corners of the code body and is used as a source of data for drawing the box.

override fun onAnalyzeResult(result: AnalysisResult) {
    synchronized(isAnalyzing) {
        showQRCodeResult(result.content)
        val centerPoint = Utils.convertRectToPoint(result.rect, binding.previewView)
        showRectView(centerPoint, result.rect)
    }
}

private fun showRectView(point: Point, rect: Rect) {
    val popupWindow = PopupWindow(
        rect.height(),
        rect.width()
    )
    val imageView = ImageView(this)

    runOnUiThread {
        popupWindow.contentView = imageView
        imageView.setImageResource(R.drawable.ic_rect_view)
        imageView.scaleType = ImageView.ScaleType.FIT_XY
        try {
            popupWindow.showAsDropDown(binding.previewView,
            point.x - (rect.width() / 2), point.y)
        } catch (e: Exception) {}
        binding.previewView.postDelayed({ popupWindow.dismiss() }, 1000)}}Copy the code

※ I don’t know whether there is a problem with the shooting Angle or the identification error of ScanKit. The drawing position of the frame is always a little wrong, and the position of the frame drawn by the official Demo is also not accurate

4. Gesture support when necessary

The previous Demo focused on using the CameraX API and neglected to support the necessary gestures. This time, I will add support for common gestures.

4.1 Double click to zoom

The setLinearZoom() API provided by CameraControl allows you to zoom in and out of the shot view linearly, making it useful for double-clicking or sliding a view to zoom in and out of the scene. It accepts a parameter value between 0 and 1, as follows:

• 0 is the minimum scaling, that is, the original size
• 1 is the maximum scale

Switch between the original scale of 0f and the middle scale of 0.5f by listening to the double click gesture.

private fun listenGesture(a) {
    binding.previewView.setOnTouchListener { view, event ->
        ...
        // Zoom when double click.
        doubleClickZoom(event)
        true}}private fun doubleClickZoom(event: MotionEvent) {
    if (doubleClickDetector == null) {
        doubleClickDetector = GestureDetector(this@NewCameraXActivity.object : GestureDetector.SimpleOnGestureListener() {
                override fun onDoubleTap(e: MotionEvent?).: Boolean{ cameraZoomState.value? .let {val zoomRatio = it.zoomRatio
                        val minRatio = it.minZoomRatio

                        // Ratio parameter from 0f to 1f.
                        if (zoomRatio > minRatio) {
                            mCamera.cameraControl.setLinearZoom(Constants.MIN_ZOOM_SCALE.toFloat())
                        } else {
                            mCamera.cameraControl.setLinearZoom(Constants.MIDDLE_ZOOM_SCALE.toFloat())
                        }
                    }
                    return true} }) } doubleClickDetector? .onTouchEvent(event) }Copy the code

4.2 Pinch gesture zoom

CameraControl’s setZoomRatio API provides a more accurate scaling ratio on the basis of linear scaling, which can realize pinch gesture zoom scene.

private fun listenGesture(a) {
    binding.previewView.setOnTouchListener { view, event ->
        ...
        // Listen to zoom gesture.
        scalePreview(event)
        true}}private fun scalePreview(event: MotionEvent) {
    if (scaleDetector == null) {
        scaleDetector = ScaleGestureDetector(this@NewCameraXActivity.object : SimpleOnScaleGestureListener() {
                override fun onScale(detector: ScaleGestureDetector): Boolean{ cameraZoomState.value? .let {val zoomRatio = it.zoomRatio
                        mCamera.cameraControl.setZoomRatio(zoomRatio * detector.scaleFactor)
                    }
                    return true} }) } scaleDetector? .onTouchEvent(event) }Copy the code

4.3 Optimization of manual focus

In the past, the focus was performed manually according to the coordinates when the Touch (ACITON_DOWN) was introduced. After the introduction of zoom gesture support, the focus operation will be touched by mistake during zooming. The solution is to limit the focus to the SingleTap gesture, where only a click triggers the focus.

private fun listenGesture(a) {
    binding.previewView.setOnTouchListener { view, event ->
        ...
        // Singe tap for focus.
        singleTapForFocus(event)
        true}}private fun singleTapForFocus(event: MotionEvent) {
    if (singleTapDetector == null) {
        singleTapDetector = GestureDetector(this@NewCameraXActivity.object : GestureDetector.SimpleOnGestureListener() {
                override fun onSingleTapConfirmed(e: MotionEvent?).: Boolean {
                    focusOnPosition(event.x, event.y, true)
                    return super.onSingleTapConfirmed(e) } }) } singleTapDetector? .onTouchEvent(event) }Copy the code

5. Continuous code improvement

Before the improvement, a lot of logic was stacked in the Activity. Now, the implementation of each UseCase is broken down to reduce the burden on the Activity. At the same time, some problems of CameraX are improved.

5.1 Preventing Repeated Crash

The control PreviewView, which shows a preview of the camera, is not yet added to the view Tree, and there is a problem if the binding operation of the CameraX is performed. The phenomenon is that Crash will occur when you open it again after shooting. The solution is simple: listen for the PreviewView control to attach, and only perform the CameraX binding operation when the attach succeeds.

override fun onCreate(savedInstanceState: Bundle?).{... setContentView(binding.root) startCameraWhenAttached() }private fun startCameraWhenAttached(a) {
    binding.previewView.addOnAttachStateChangeListener(object : View.OnAttachStateChangeListener{
        override fun onViewAttachedToWindow(v: View?). {
            ensureCameraPermission()
        }
    })
}

private fun ensureCameraPermission(a){... setupCamera(binding.previewView) }Copy the code

Remember to end the image analysis call when the picture is not visible to save memory.

override fun onStop(a) {
    super.onStop() mImageAnalysis? .clearAnalyzer() }Copy the code

5.2 Crash suppression of Consecutive Screen Taps

The following crash will occasionally occur in the case of a quick click on the video record and stop.

java.lang.IllegalStateException: Failed to stop the muxer

Looking at the source code for CameraX, the request and release of the Audio instance occurred at the beginning and end of the recording. In order to alleviate this phenomenon, the state control of recorded video is added in this improvement, and the termination of recording is prohibited within 500ms from the beginning of recording.

However, in the extremely fast recording and stop the repeated operation, the recorded part of the file may be damaged. Since CameraX’s video recording API is still experimental, wait for CameraX to work out.

private fun videoRecordingPrepared(a) {
    isCameraXHandling = false
    // Keep disabled status for a while to avoid fast click error with "Muxer stop failed!" .
    binding.capture.postDelayed({ binding.capture.isEnabled = true }, 500)}Copy the code

5.3 Mirror inversion

Photographs taken by CameraX are mirrored by default. Tell CameraX to reverse the image before shooting so that what you see is what you get.

private fun takenPictureInternal(isExternal: Boolean){...// Mirror image
    ImageCapture.Metadata().apply {
        isReversedHorizontal = true} mImageCapture? .takePicture(outputFileOptions, lightExecutor, MyCaptureCallback(picCount,this))}Copy the code

5.4 Selecting a Camera

Many of the devices, such as the security code and temperature integration devices that have been popular during the pandemic, don’t have more than one camera. So sometimes you can’t just switch back and forth, you need to switch by the ID of the lens.

private fun bindPreview(...). {
    // Select specified camera.
    val cameraSelector = CameraSelector.Builder().addCameraFilter(AllCameraFilter()).build()
    ...
}

class AllCameraFilter: CameraFilter {
    override fun filter(cameraInfos: MutableList<CameraInfo>): MutableList<CameraInfo> {
        val result: MutableList<CameraInfo> = mutableListOf()
        for (cameraInfo in cameraInfos) {
            val id = (cameraInfo as CameraInfoInternal).cameraId
            // Specify the camera id that U need, such as front camera which id is 0.
            if (CameraSelector.LENS_FACING_FRONT.equals(id)) {
                result.add(cameraInfo)
            }
        }
        return result
    }
}
Copy the code

In fact, the latest version of CameraX provides a new API (CameraInfo#getCameraSelector()) that returns a selector instance for a particular lens.

6. Summary of relevant APIS

A quick review of the main apis used by CameraX.

An interface or implementation for managing camera instances	role
CameraController	Interface for obtaining and managing camera instances
LifecycleCameraController	LifecycleOwner provides an interface for life cycle management of Camera instances
ProcessCameraProvider	LifecycleOwner implementation class for managing Camera instances in a singleton pattern

API to access camera functionality and properties	role
Camera	Provides the main interface for lens operation
CameraControl	This interface is used to zoom and focus the Camera. An instance is obtained through the Camera interface
CameraInfo	The ifs used to get lens parameters, such as zoom ratio, whether there is a flash, etc., are also provided by the Camera interface
CameraConfig	This interface is used to obtain the configuration information of the Camera and obtain the instance through the Camera interface
CameraSelector	Filters and matches the class for the corresponding shot, passing in the instance when the CameraController executes to initialize the corresponding shot

Scenario UseCase class implementation class	role
Preview	Preview the scene
ImageAnalysis	Image analysis
ImageCapture	Images taken
VideoCapture	Video recording

Camera effects extension class	role
PreviewExtender	Show preview extension effect, the implementation class has BeautyPreviewExtender, NightPreviewExtender and so on
ImageCaptureExtender	Show shooting extended effects, the same effects such as beauty class

And part of the ScanKit API:

API	role
ScanUtil	Bitmap A tool supported by functions such as scan code mode and Bitmap compression
HmsScanAnalyzerOptions	Specifies parameter classes such as scan code format
HmsScan	Scan code result encapsulation class, including content, code body coordinates, four corner position and other information

conclusion

The integration of Huawei ScanKit is very simple and smooth, so you can try it out when selecting scanning technology. Those who are worried about recognition rate or speed can download ScanKit and Zxing’s official Apk to experience and compare.

Scankit official Sample download address:

Developer.huawei.com/consumer/en…

Zxing official Sample download address:

Play.google.com/store/apps/…

Hope for CameraX scan code integration and practical improvements, to help you.

In this paper, the DEMO

Github.com/ellisonchan…

The resources

Huawei Official Documents

Perfect substitute for ZXing, unified scan code service

Getting started with Android CameraX

Recommended reading

Why is Jetpack CameraX recommended?

A look at Jetpack in terms of Preference component changes

The new widgets on Android 12: beautiful, easy and useful

The new app splash screen on Android 12.