The original article was first published on the wechat official account Byteflow

Finally, many friends asked, “How do you learn to write filters in OpenGL ES?” “, “How do YOU learn shader?” .

Recently, I consulted some big shots, and they all agreed that the right thing to do is to “imitate.” First try to imitate others’ filters, such as observing some simple filters of Douyin, and then try to implement one yourself.

Of course, the most efficient way is to study some relevant open source projects, such as the well-known Android-GpuImage project, which basically implements a variety of common filters and is easy to use. Students who learn Shader, are familiar with GLSL or are interested in OpenGL filters can study it.

By the way, I recently read a project called Android-GpuImage-Plus, which is mainly about the filter implemented by Native layer. I have some good ideas for your reference.

Before, a friend posted an effect picture of an emoticon filter, which uses different emoticons to replace different pixels to generate an image composed of emoticons. The principle of the emoticon filter is the same as the character filter, but the character is replaced by the emoticon.

Because that sketch is not convenient to show, here is the realization principle of character drawing, using a shader to achieve character drawing effect.

Character painting filter principle

Character painting filter is actually the same principle as LUT filter, which is essentially lookup table, pixel replacement.

The first way to implement a character filter is to replace an image pixel by pixel with a character (a character is actually a small picture made up of multiple pixels).

There are two problems with per-pixel substitution:

A pixel has RGB 24 bits three channels, a total of 256×256×256 colors, so many colors to correspond with the character table is very troublesome;
Replacing characters pixel by pixel is equivalent to replacing one pixel of the original image with multiple pixels. For example, in the character table now used, the size of a character is 16×23 = 268 pixels, so the rendered image size becomes 268 times of the original, which is obviously unreasonable.

So character painting filters to achieve the correct idea is to put the original image to a greyscale image, so only 256 kinds of color types, and then do Mosaic, replace a character with a small grid, guarantee the aspect ratio the same as the character of small grid, to ensure that the replacement of characters don’t stretch, it renders the image size is the same as the original image.

Character painting filter principle is described in a word, the original image first grayscale Mosaic, and then replace the character with a small grid.

Character painting filter implementation

According to the principle described in the previous section, we first make grayscale Mosaic of the original image, and then directly transform the RGB component of the sampled pixels after obtaining grayscale value.

//RGB to gray formula Y = 0.299r + 0.587g + 0.114bCopy the code

The principle of Mosaic effect is to divide the image into many small areas, and the same color is taken in the small area. The color value can be the weighted average of some pixel values in the area. In this paper, the pixel value of the center point in the small rectangular area is taken.

The character table image used here is 128×69 with a total of 24 characters, each 16×23 pixels.

The realization of grayscale Mosaic.

// Grayscale Mosaic
#version 100
precision highp float;
varying vec2 v_texcoord;
uniform lowp sampler2D s_textureY;
uniform lowp sampler2D s_textureU;
uniform lowp sampler2D s_textureV;
uniform lowp sampler2D s_textureMapping;
uniform vec2 texSize;

vec4 YuvToRgb(vec2 uv) {
    float y, u, v, r, g, b;
    y = texture2D(s_textureY, uv).r;
    u = texture2D(s_textureU, uv).r;
    v = texture2D(s_textureV, uv).r;
    u = u - 0.5;
    v = v - 0.5;
    r = y + 1.403 * v;
    g = y - 0.344 * u - 0.714 * v;
    b = y + 1.770 * u;
    return vec4(r, g, b, 1.0);
}

const vec3  RGB2GRAY_VEC3 = vec3(0.299.0.587.0.114);
const float MESH_WIDTH = 16.0;// The width of one character
const float MESH_HEIGHT= 23.0;// The height of a character
const float MESH_ROW_NUM = 100.0;// Set the number of lines in the cell
void main(a)
{
    float imageMeshWidth = texSize.x / MESH_ROW_NUM;
	// Make the width ratio of the small cell consistent with the width ratio of the character to prevent the character from being stretched after the replacement
    float imageMeshHeight = imageMeshWidth * MESH_HEIGHT / MESH_WIDTH;

    vec2 imageTexCoord = v_texcoord * texSize;// Normalize coordinates to pixel coordinates
	
	// Take the pixel of the center point of the cell
    vec2 midTexCoord;
    midTexCoord.x = floor(imageTexCoord.x / imageMeshWidth) * imageMeshWidth + imageMeshWidth * 0.5;// Small cell center
    midTexCoord.y = floor(imageTexCoord.y / imageMeshHeight) * imageMeshHeight + imageMeshHeight * 0.5;// Small cell center
    vec2 normalizedTexCoord = midTexCoord / texSize;/ / normalization
    vec4 rgbColor = YuvToRgb(normalizedTexCoord);/ / sampling

    float grayValue = dot(rgbColor.rgb, RGB2GRAY_VEC3);// RGB to gray value
    gl_FragColor = vec4(vec3(grayValue), rgbColor.a);
}
Copy the code

Grayscale Mosaic effect.

After the grayscale Mosaic is completed, each cell replaces one character, and the 24 characters will be 0~~The gray value of 255 (normalized to 0~~1.0) It is divided into 24 levels, and the corresponding characters are selected according to the levels after the gray value is calculated.

Then according to the offset of the sampling coordinate in the small cell, the sampling coordinate of the character (including a small picture of a character) is calculated, and finally the character is sampled.

Character painting implementation of the complete shader.

/ / character painting
#version 100
precision highp float;
varying vec2 v_texcoord;
uniform lowp sampler2D s_textureY;
uniform lowp sampler2D s_textureU;
uniform lowp sampler2D s_textureV;
uniform lowp sampler2D s_textureMapping;// Character table texture
uniform float u_offset;
uniform vec2 texSize;// Original size
uniform vec2 asciiTexSize;// The size of the character table

vec4 YuvToRgb(vec2 uv) {
    float y, u, v, r, g, b;
    y = texture2D(s_textureY, uv).r;
    u = texture2D(s_textureU, uv).r;
    v = texture2D(s_textureV, uv).r;
    u = u - 0.5;
    v = v - 0.5;
    r = y + 1.403 * v;
    g = y - 0.344 * u - 0.714 * v;
    b = y + 1.770 * u;
    return vec4(r, g, b, 1.0);
}

const vec3  RGB2GRAY_VEC3 = vec3(0.299.0.587.0.114);
const float MESH_WIDTH = 16.0;// The width of one character
const float MESH_HEIGHT= 23.0;// The height of a character
const float GARY_LEVEL = 24.0;// There are 24 characters in the character table diagram
const float ASCIIS_WIDTH = 8.0;// The number of columns in the character table
const float ASCIIS_HEIGHT = 3.0;// The number of rows in the character table
const float MESH_ROW_NUM = 100.0;// Set the number of lines in the cell
void main(a)
{
    float imageMeshWidth = texSize.x / MESH_ROW_NUM;
	// Make the width ratio of the small cell consistent with the width ratio of the character to prevent the character from being stretched after the replacement
    float imageMeshHeight = imageMeshWidth * MESH_HEIGHT / MESH_WIDTH;

    vec2 imageTexCoord = v_texcoord * texSize;// Normalize coordinates to pixel coordinates
	
	// Take the pixel of the center point of the cell
    vec2 midTexCoord;
    midTexCoord.x = floor(imageTexCoord.x / imageMeshWidth) * imageMeshWidth + imageMeshWidth * 0.5;// Small cell center
    midTexCoord.y = floor(imageTexCoord.y / imageMeshHeight) * imageMeshHeight + imageMeshHeight * 0.5;// Small cell center
    vec2 normalizedTexCoord = midTexCoord / texSize;/ / normalization
    vec4 rgbColor = YuvToRgb(normalizedTexCoord);/ / sampling

    float grayValue = dot(rgbColor.rgb, RGB2GRAY_VEC3);// RGB to gray value
    //gl_FragColor = vec4(vec3(grayValue), rgbColor.a);

	// Calculate the offset within a character (a small image containing one character) based on the offset of the sampling coordinate within the cell
    float offsetX = mod(imageTexCoord.x, imageMeshWidth) * MESH_WIDTH / imageMeshWidth;
    float offsetY = mod(imageTexCoord.y, imageMeshHeight) * MESH_HEIGHT / imageMeshHeight;

    float asciiIndex = floor((1.0 - grayValue) * GARY_LEVEL);// Determine the number of characters according to the gray value
    float asciiIndexX = mod(asciiIndex, ASCIIS_WIDTH);
    float asciiIndexY = floor(asciiIndex / ASCIIS_WIDTH);

	// According to the character position and the offset within the character, calculate the character table texture sampling point coordinates
    vec2 grayTexCoord;
    grayTexCoord.x = (asciiIndexX * MESH_WIDTH + offsetX) / asciiTexSize.x;
    grayTexCoord.y = (asciiIndexY * MESH_HEIGHT + offsetY) / asciiTexSize.y;

    vec4 originColor = YuvToRgb(v_texcoord);// Sample the original texture
    vec4 mappingColor = vec4(texture2D(s_textureMapping, grayTexCoord).rgb, rgbColor.a);// Sample the character table texture

    gl_FragColor = mix(originColor, mappingColor, u_offset);// Make a final blend to preserve some of the original color
}
Copy the code

Character drawing effect.

Technical communication

Technical exchange/get source code can add my wechat: byte-flow

mo4tech.com (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.

Use OpenGL ES to make a character filter for video player and camera

Character painting filter principle

Character painting filter implementation

Technical communication

Use OpenGL ES to make a character filter for video player and camera

Character painting filter principle

Character painting filter implementation

Technical communication

Related Posts

The mi mix2 found itself not displaying text

From state management in Flutter to the use of providers

How does Android design user Session management?