Use OpenCV WeChat QR code recognition under CameraX

Preface

Earlier, we have introduced two ways to integrate wechat_qrcode WeChat QR code recognition capabilities:

  • Fully compile OpenCV and OpenCV Contrib
  • Native C++ integrates the wechat_qrcode module separately

The preview and recognition of these two methods are based on the JavaCamera2View provided by OpenCV.

Today I will introduce how to call the capabilities of the wechat_qrcode module through the CameraX framework.

CameraX basic use

CameraX has the following minimum version requirements:

  • Android API level 21
  • Android Architecture Components 1.1.1

Add dependency

allprojects {
    repositories {
        google()
        jcenter()
    }
}
compileOptions {
    sourceCompatibility = JavaVersion.VERSION_1_8
    targetCompatibility = JavaVersion.VERSION_1_8
}
kotlinOptions {
    jvmTarget = "1.8"
}
val cameraXVersion = "1.0.0-rc05"
implementation ("androidx.camera:camera-core:${cameraXVersion}")
implementation("androidx.camera:camera-camera2:$cameraXVersion")
implementation("androidx.camera:camera-lifecycle:$cameraXVersion")
implementation("androidx.camera:camera-view:1.0.0-alpha20")
implementation("androidx.camera:camera-extensions:1.0.0-alpha24")

Preview

Add PreviewView to the layout

<androidx.camera.view.PreviewView
    android:id="@+id/viewFinder"
    android:layout_width="match_parent"
    android:layout_height="match_parent" />

Request CameraProvider and check availability

/** Initialize CameraX, and prepare to bind the camera use cases  */
private fun setUpCamera() {
    val cameraProviderFuture = ProcessCameraProvider.getInstance(this)
    cameraProviderFuture.addListener({
        // CameraProvider
        cameraProvider = cameraProviderFuture.get()
        // Build and bind the camera use cases
        bindCameraUseCases()
    }, ContextCompat.getMainExecutor(this))
}

Select the camera and bind the life cycle and use cases

val cameraSelector = CameraSelector.Builder().requireLensFacing(lensFacing).build()

// Preview
preview = Preview.Builder()
    .setTargetAspectRatio(screenAspectRatio)
    .setTargetRotation(rotation)
    .build()
……
// Must unbind the use-cases before rebinding them
cameraProvider.unbindAll()
try {
    camera = cameraProvider.bindToLifecycle(
        this, cameraSelector, preview, imageCapture, imageAnalyzer
    )
    preview?.setSurfaceProvider(mBinding.viewFinder.surfaceProvider)
} catch (exc: Exception) {
    Log.e(App.TAG, "Use case binding failed", exc)
}

taking photos

Create use cases and bind life cycles

// ImageCapture
imageCapture = ImageCapture.Builder()
    .setTargetAspectRatio(screenAspectRatio)
    .setTargetRotation(rotation)
    .build()
……
camera = cameraProvider.bindToLifecycle(
                this, cameraSelector, preview, imageCapture, imageAnalyzer
            )
……

Perform a photo

private fun takePhoto() {
    // Get a stable reference of the modifiable image capture use case
    val imageCapture = imageCapture ?: return

    // Create time-stamped output file to hold the image
    val photoFile = File(
        outputDirectory,
        SimpleDateFormat(
            FILENAME_FORMAT, Locale.US
        ).format(System.currentTimeMillis()) + ".jpg"
    )

    val outputOptions = ImageCapture.OutputFileOptions.Builder(photoFile).build()

    imageCapture.takePicture(
        outputOptions,
        ContextCompat.getMainExecutor(this),
        object : ImageCapture.OnImageSavedCallback {
            override fun onError(exc: ImageCaptureException) {
                Log.e(App.TAG, "Photo capture failed: ${exc.message}", exc)
            }

            override fun onImageSaved(output: ImageCapture.OutputFileResults) {
                val savedUri = Uri.fromFile(photoFile)
                val msg = "Photo capture succeeded: $savedUri"
                Toast.makeText(baseContext, msg, Toast.LENGTH_SHORT).show()
                Log.d(App.TAG, msg)
            }
        })
}

Picture analysis

Image analysis use cases provide your application with images that can be accessed by the CPU to perform image processing, computer vision, or machine learning inference. The application implements the analyze method that runs on each frame.

Implement the Analyzer interface

public interface Analyzer {
    void analyze(@NonNull ImageProxy image);
}
private class WeChatAnalyzer(
    private val weChatQRCode: WeChatQRCode,
    private val listener: (results: List<String>) -> Unit
) : ImageAnalysis.Analyzer {
    override fun analyze(image: ImageProxy) {
        Log.d(App.TAG, "size = ${image.width} * ${image.height}")
        Log.d(App.TAG, "rotationDegrees = ${image.imageInfo.rotationDegrees}")

        val rectangles = ArrayList<Mat>()
        val results = weChatQRCode.detectAndDecode(gray(image), rectangles)
        listener(results)
        image.close()
    }
    ……
}

Create use cases and bind life cycles

// ImageAnalysis
imageAnalyzer = ImageAnalysis.Builder()
    .setTargetAspectRatio(screenAspectRatio)
    .setTargetRotation(rotation)
    .build()
    // The analyzer can then be assigned to the instance
    .also {
        it.setAnalyzer(
            cameraExecutor,
            WeChatAnalyzer(
                mWeChatQRCode,
            ) { results ->
                runOnUiThread {
                    if (results.isNotEmpty()) {
                        Toast.makeText(this, results.toString(), Toast.LENGTH_SHORT).show()
                        finish()
                    }
                }
            })
    }

……
camera = cameraProvider.bindToLifecycle(
                this, cameraSelector, preview, imageCapture, imageAnalyzer
            )
……

Identify the QR code

The identification interface adopts the WeChat QR code identification interface that was separately integrated through Native C++ in the previous article. The identification process is mainly divided into two steps:

  • Data format conversion
  • Call recognition API

Data format conversion

Interface transfer CameraX analyze our data in a given category ImageProxy encapsulated in the data format of YUV_420_888format. YUV_420_888The format is a generalized format of YCbCr, which can represent any 4:2:0 plane and half-plane format, and each component is represented by 8 bits. The image with this format is represented by 3 independent Buffers, and each Buffer represents a color plane (Plane). And we need to pass parameters of the interface is Mat type, so it is necessary to YUV_420_888convert the data format for Mat.

fun gray(image: ImageProxy): Mat {
    val planeProxy = image.planes // 获取分量数组
    val width = image.width // 获取宽高
    val height = image.height
    val yPlane = planeProxy[0].buffer // 获取亮度分量方便我们转为灰度图
    val yPlaneStep = planeProxy[0].rowStride // 获取步长
    return Mat(height, width, CvType.CV_8UC1, yPlane, yPlaneStep.toLong())
}

Call API

val results = weChatQRCode.detectAndDecode(gray(image), rectangles)

effect

Scanning result

Source code

https://github.com/onlyloveyd/LearningAndroidOpenCV