This is the first day that I participate in August more text challenge, activity details check: August more text challenge has been a little strong interest in Android audio and video development, but there is no system to learn the corresponding knowledge, can only study hard, borrow this more text activity to record their own a little exploration.
Audio and video basic knowledge need to master
- Ffmpeg powerful audio and video processing library, (CPU software etc.)
- MediaCodec Is a codec that comes with the Android SDK.
- Opengles uses gpus for image processing
- H264, H265 image coding compression algorithm
- Yuv420p,nv21, YUv_420_888, I420 Video encoding formats to understand
- Yuv and RGB conversion
- PCM, ACC needs to understand the audio coding format
- Camerax, mediaRecorder, audioRecorder acquisition related API
Audio and video in Android generally has two forms of expression, one is to play online or local video (recording and playing), the other is live (push and pull stream), below we first from the video collection to play the whole process to do a detailed exploration.
Video collection
- The use of Camerax
Camerax is a new camera library added to Jetpack. It is very well designed, not as cumbersome as camerA1 and CamerA2, and is tied to the life cycle, making it easy for developers to manage the life cycle.
- Add the dependent
implementation "androidx.camera:camera-camera2:$camerax_version" implementation "Androidx. Camera: the camera - lifecycle: $camerax_version" implementation "androidx. Camera: the camera - view: 1.0.0 - alpha10"Copy the code- Creating a preview layout
<androidx.camera.view.PreviewView
android:id="@+id/viewFinder"
app:layout_constraintTop_toTopOf="parent"
app:layout_constraintBottom_toTopOf="@+id/camera_capture_button"
android:layout_width="match_parent"
android:layout_height="0dp"/>
Copy the code3. Obtain camera permissions (omitted) 4. Open the camera and preview it
@suppresslint ("RestrictedApi") private fun startCamera() {initMediaCodec(480,640,20) val cameraProviderFuture = ProcessCameraProvider.getInstance(this) cameraProviderFuture.addListener(Runnable { bindImage(cameraProviderFuture) }, ContextCompat.getMainExecutor(this)) } private fun bindImage(cameraProviderFuture:ListenableFuture<ProcessCameraProvider>) { val cameraProvider: ProcessCameraProvider = cameraProviderFuture.get() preview = Preview.Builder().build() val cameraSelector =CameraSelector.Builder().requireLensFacing(CameraSelector.LENS_FACING_BACK).build() val imageAnalysis = ImageAnalysis.Builder() .setTargetResolution(Size(1280, 720)) .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST) .build() imageAnalysis.setAnalyzer(cameraExecutor, ImageAnalysis.Analyzer { image -> Thread { val data =ImageUtils.getDataFromImage2(image,ImageUtils.COLOR_FormatI420) val out = FileOutputStream(File(Environment.getExternalStorageDirectory(),"hhh.yuv")) out.write(data) image.close(); }.start() }) try { cameraProvider.unbindAll() camera = cameraProvider.bindToLifecycle(this, cameraSelector, imageAnalysis, preview) preview? .setSurfaceProvider(viewFinder.createSurfaceProvider(camera? .cameraInfo)) } catch (exc: Exception) { Log.e(TAG, "Use case binding failed", exc) } }Copy the code5. Obtain recorded video data. Use the setAnalyzer method of ImageAnalysis to obtain the original recorded video data. The default video data format is YUV_420_888, which cannot be directly encoded with FFMEPEG or Mediacodec. I needed to convert to the i420 format, which I did a lot of digging into.
Yuv_420_888 is a generalization of YCbCr, capable of representing any 4:2:0 plane and half-plane formats with 8 bits for each component. Images with this format are represented by three separate buffers, each representing a color Plane. In addition to the Buffer, it also provides rowStride, pixelStride to describe the corresponding Plane.
Officially, the first plane contains all the y data, the second plane contains all the U data, and the third plane contains all the V data. We just need to parse out the yuV components of the original data and arrange and scale them according to i420 to get the data we can finally use. However, in the actual process of parsing, I found that there were more padding data in the original data, which was used for alignment processing according to the Internet, and I didn’t have an in-depth understanding of it here. If you know the old iron, please let me know in the comment section. I’ve roughly written a utility class for the transformation along these lines.
public class ImageUtils { private static final boolean VERBOSE = true; private static final String TAG = "ImageUtils"; public static final int COLOR_FormatI420 = 1; public static final int COLOR_FormatNV21 = 2; public static boolean isImageFormatSupported(ImageProxy image) { int format = image.getFormat(); switch (format) { case ImageFormat.YUV_420_888: case ImageFormat.NV21: case ImageFormat.YV12: return true; } return false; } public static byte[] getDataFromImage2(ImageProxy image, int colorFormat) { if (colorFormat ! = COLOR_FormatI420 && colorFormat ! = COLOR_FormatNV21) { throw new IllegalArgumentException("only support COLOR_FormatI420 " + "and COLOR_FormatNV21"); } if (! isImageFormatSupported(image)) { throw new RuntimeException("can't convert Image to byte array, format " + image.getFormat()); } Rect crop = image.getCropRect(); int format = image.getFormat(); // 888 int width = crop.width(); int height = crop.height(); ImageProxy.PlaneProxy[] planes = image.getPlanes(); int yRe = planes[0].getBuffer().remaining(); int uvRe = planes[1].getBuffer().remaining(); int vuRe = planes[2].getBuffer().remaining(); int yStride = planes[0].getRowStride(); int uvStride = planes[1].getRowStride(); int vuStride = planes[2].getRowStride(); int yLength = yRe - (yStride-width)*(height-1); int uvLength = uvRe - (uvStride-width)*(height/2-1); int vuLength = vuRe - (vuStride-width)*(height/2-1); byte[] data = new byte[width * height * 12 /8]; byte[] yData = new byte[yLength]; byte[] uData = new byte[uvLength/2+1]; byte[] vData = new byte[vuLength/2 + 1]; [] uvData = new byte[uvLength]; [] vuData = new byte[vuLength]; byte[] rowData = new byte[planes[0].getRowStride()]; for (int i = 0; i < planes.length; i++) { ImageProxy.PlaneProxy plane = planes[i]; ByteBuffer buffer = plane.getBuffer(); int offset = 0; Log.e("getDataFromImage",plane.getPixelStride() + ""); buffer.position(0); int col = height / plane.getPixelStride(); If (plane.getpixelstride () == 1){// the matrix is i420 for (int j = 0; j < col; j++) { if(i == 0){ //y if(j < height/plane.getPixelStride() -1){ buffer.get(yData,offset,width); }else { buffer.get(yData,offset,yRe - (plane.getRowStride() * (height-1))); } offset +=width; if(j < height/plane.getPixelStride()-1){ buffer.position(buffer.position()+ plane.getRowStride() - width); } }else if(i == 1){ //uv buffer.get(uData,offset,width); offset +=width; if(j < height/plane.getPixelStride() -1){ buffer.position(buffer.position()+ plane.getRowStride() - width); } }else { // vu buffer.get(vData,offset,width); offset +=width; if(j < height/plane.getPixelStride()-1){ buffer.position(buffer.position()+ plane.getRowStride() - width); }}}}else {// nv12 or nv21 for (int j = 0; j < col; j++) { if(i == 0){ //y if(j < height/plane.getPixelStride() -1){ buffer.get(yData,offset,width); }else { buffer.get(yData,offset,yRe - (plane.getRowStride() * ( height/plane.getPixelStride() -1))); } offset +=width; if(j < height/plane.getPixelStride() -1){ buffer.position(buffer.position()+ plane.getRowStride() - width); } }else if(i == 1){ //uv if(j < height/plane.getPixelStride() -1){ buffer.get(uvData,offset,width); }else { buffer.get(uvData,offset,uvRe - (plane.getRowStride() * (height/plane.getPixelStride()-1))); } offset +=width; if(j < height/plane.getPixelStride() -1){ buffer.position(buffer.position()+ plane.getRowStride() - width); } }else { // vu if(j < height/plane.getPixelStride() -1){ buffer.get(vuData,offset,width); }else { buffer.get(vuData,offset,vuRe - (plane.getRowStride() * (height/plane.getPixelStride()-1))); } offset +=width; if(j < height/plane.getPixelStride()-1){ buffer.position(buffer.position()+ plane.getRowStride() - width); } } } } } boolean isI420 = image.getPlanes()[1].getPixelStride() == 1; //ydata uvdata vudata // extract u,v if(! isI420){ // for (int i = 0; i < uvData.length; i+=2) { uData[i/2] = uvData[i]; } for (int i = 0; i < vuData.length; i+=2) { vData[i/2] = uvData[i]; } Log.e("getDataFromImage",uData.length + ""); Log.e("getDataFromImage",vData.length + ""); } / / assembly i420 System. Arraycopy (yData, 0, data, 0, yData length); System.arraycopy(uData,0,data,yData.length,uData.length); System.arraycopy(vData,0,data,yData.length + uData.length ,vData.length); return data; }}Copy the codeThe final result, after parsing one frame, the direction and Angle can be adjusted by yourself, the color value seems to be a little bit wrong, I will study it tomorrow
If there is a problem with the following playback, it may be that YUV has a problem reading and writing data and needs to check the length of the byte array
The next section will record the process of learning, and again pay tribute to Thor
`