This is the second day of my participation in the August More text Challenge. For details, see:August is more challenging

The previous article described the process of using camerax and PreviewView to preview and get a frame of YUV image. This is a brief introduction to making Opengles preview camera data.

Opengles on Android

Android supports high performance 2D and 3D graphics through open graphics libraries (OpenGL®), especially the OpenGL ES API. OpenGL is a cross-platform graphics API that specifies standard software interfaces for 3D graphics processing hardware. OpenGL ES is a form of the OpenGL specification for embedded devices. Android supports multiple versions of OpenGL ES API:

  • OpenGL ES 1.0 and 1.1 – This API specification is supported by Android 1.0 and later.
  • OpenGL ES 2.0 – This API specification is supported by Android 2.2 (API level 8) and later.
  • OpenGL ES 3.0 – This API specification is supported by Android 4.3 (API level 18) and later.
  • OpenGL ES 3.1 – This API specification is supported by Android 5.0 (API level 21) and later.

Camerex implements previews with GlSurfaceView

// Turn on the camera
val cameraProviderFuture = ProcessCameraProvider.getInstance(this)
cameraProviderFuture.addListener(Runnable {
    bindImage(cameraProviderFuture)
}, ContextCompat.getMainExecutor(this))

Copy the code
/ / camerax GlSurfaceView
private fun bindImage(cameraProviderFuture: ListenableFuture<ProcessCameraProvider>) {
    cameraProvider = cameraProviderFuture.get()
    preview = Preview.Builder().build()
    try{ val imageCapture = ImageCapture.Builder().build() gl_view.attachPreview(preview) cameraProvider? .unbindAll() val camera:Camera? = cameraProvider? .bindToLifecycle(this, cameraSelector,imageCapture ,preview)
    } catch (exc: Exception) {
        Log.e(TAG, "Use case binding failed", exc)
    }
}
Copy the code
/ / camera and share a texture opegles public void attachPreview (Preview, Preview) {Preview. SetSurfaceProvider (new Preview. SurfaceProvider ()  { @Override public void onSurfaceRequested(@NonNull SurfaceRequest request) { mResolution = request.getResolution(); / / need to give texture set width is high, otherwise the picture might be very fuzzy surfaceTexture. SetDefaultBufferSize (mResolution. GetWidth (), mResolution. GetHeight ()); Surface surface = new Surface(surfaceTexture); request.provideSurface(surface, executor, new Consumer<SurfaceRequest.Result>() { @Override public void accept(SurfaceRequest.Result result) { surface.release(); surfaceTexture.release(); }}); }}); }Copy the code

Using Opengles requires setting up the version and setting up Render

setEGLContextClientVersion(3); setRenderer(this); SetRenderMode (glSurfaceView.rendermode_when_dirty);Copy the code

Process the data in Render

    @Override
    public void onSurfaceCreated(GL10 gl, EGLConfig config) {
        int[] textureIds = new int[1];
        GLES30.glGenTextures(1, ids, 0);
        textureId = textureIds[0];
        surfaceTexture = new SurfaceTexture(textureId);
        surfaceTexture.setOnFrameAvailableListener(this);
        directDrawer = new DirectDrawer(textureId);
    }

    @Override
    public void onSurfaceChanged(GL10 gl, int width, int height) {
        GLES30.glViewport(0, 0, width, height);
    }
    
    @Override
    public void onFrameAvailable(SurfaceTexture surfaceTexture) {
        requestRender();
    }

Copy the code
Public class DirectDrawer {// vertex shaders attribute: Input matrix variable, varying: Private final String vertexShaderCode = "Attribute vec4 vPosition;" "Attribute vec2 inputTextureCoordinate;" + // Varying VEC2 textureCoordinate; + "void main()" + "{"+ "gl_Position = vPosition;" + "textureCoordinate = inputTextureCoordinate;" + "} "; // Element shader, Private final String fragmentShaderCode = "#extension GL_OES_EGL_image_external: require\n"+ "precision mediump float;" + // Specifies the precision of float "VARYING vec2 textureCoordinate; \n" + "uniform samplerExternalOES s_texture; \n" + "void main() {" + "gl_FragColor = texture2D( s_texture, textureCoordinate ); \n" + "}"; private FloatBuffer vertexBuffer, textureVerticesBuffer; private ShortBuffer drawListBuffer; private final int mProgram; private int mPositionHandle; private int mTextureCoordHandle; Private static final int COORDS_PER_VERTEX = 2; Private final int vertexStride = COORDS_PER_VERTEX * 4; private final int vertexStride = COORDS_PER_VERTEX * 4; Static float squareCoords[] = {-1.0f, 1.0f, -1.0f, -1.0f, 1.0f, 1.0f, 1.0f, 1.0f, 1.0f, -1.0f,}; Static float textureVertices[] = {0f,1f, 0f,0f, 1f,1f, 1f,1f, 0f}; private int texture; public DirectDrawer(int texture) { this.texture = texture; ByteBuffer bb = ByteBuffer.allocateDirect(squareCoords.length * 4); bb.order(ByteOrder.nativeOrder()); vertexBuffer = bb.asFloatBuffer(); vertexBuffer.put(squareCoords); vertexBuffer.position(0); / / the data into FlostBuffer ByteBuffer DLB. = ByteBuffer allocateDirect (drawOrder. Length * 2); dlb.order(ByteOrder.nativeOrder()); drawListBuffer = dlb.asShortBuffer(); drawListBuffer.put(drawOrder); drawListBuffer.position(0); ByteBuffer bb2 = ByteBuffer.allocateDirect(textureVertices.length * 4); bb2.order(ByteOrder.nativeOrder()); textureVerticesBuffer = bb2.asFloatBuffer(); textureVerticesBuffer.put(textureVertices); textureVerticesBuffer.position(0); Int vertexShader = loadShader(gles30.gl_vertex_shader, vertexShaderCode); int fragmentShader = loadShader(GLES30.GL_FRAGMENT_SHADER, fragmentShaderCode); // link shader mProgram = gles30.glCreateProgram (); // create empty OpenGL ES Program GLES30.glAttachShader(mProgram, vertexShader); // add the vertex shader to program GLES30.glAttachShader(mProgram, fragmentShader); // add the fragment shader to program GLES30.glLinkProgram(mProgram); // creates OpenGL ES Program executables} // Creates OpenGL ES Program executables public void draw(float[] MTX){ GLES30.glUseProgram(mProgram); GLES30.glActiveTexture(GLES30.GL_TEXTURE0); GLES30.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, texture); mPositionHandle = GLES30.glGetAttribLocation(mProgram, "vPosition"); GLES30.glEnableVertexAttribArray(mPositionHandle); GLES30.glVertexAttribPointer(mPositionHandle, COORDS_PER_VERTEX, GLES30.GL_FLOAT, false, vertexStride, vertexBuffer); mTextureCoordHandle = GLES30.glGetAttribLocation(mProgram, "inputTextureCoordinate"); GLES30.glEnableVertexAttribArray(mTextureCoordHandle); GLES30.glVertexAttribPointer(mTextureCoordHandle, COORDS_PER_VERTEX, GLES30.GL_FLOAT, false, vertexStride, textureVerticesBuffer); //GL_TRIANGLE_STRIP Gles30.gldrawarrays (gles30.gl_triangle_strip,0,4); gles30.gl_triangle_strip,0,4); GLES30.glDisableVertexAttribArray(mPositionHandle); GLES30.glDisableVertexAttribArray(mTextureCoordHandle); } private int loadShader(int type, String shaderCode){ int shader = GLES30.glCreateShader(type); GLES30.glShaderSource(shader, shaderCode); GLES30.glCompileShader(shader); return shader; }}Copy the code

Here I used only camera preview processing, and the effect is as follows:

Now we can do a little matrix transformation and add a filter effect

private final String fragmentShaderCode = "#extension GL_OES_EGL_image_external : require\n"+ "precision mediump float;" + // Specifies the type float "VARYING vec2 textureCoordinate; \n" + "uniform samplerExternalOES s_texture; \n" + "void main() {" + "mediump vec4 textureColor = texture2D(s_texture, textureCoordinate); \n"+ "float gray = texturecolor. r * 0.2125 + texturecolor. g * 0.7154 + texturecolor. b * 0.0721; \n"+ "gl_FragColor = vec4(gray, gray, gray, textureColor.w);" + "} ";Copy the code

The effect is as follows:

Ps: I have tried many combinations of texture coordinates transformation to make the graphics normal, if there is a detailed transformation principle, please kindly point out in the comments section, I would be very grateful