blog.piasy.com
Blog.piasy.com/2017/10/06/…

I learned some basic concepts of OpenGL last June, organized a demo and two articles, and reviewed and revised some of them this June. Not long ago I further learned the four common 2D texture transformations from brother tie lei (as well as the rest of the summary text in this article), because brother tie lei is too busy to write quickly, so I will write for him here 🙂

For those of you who have not read the first two articles and are not familiar with the basic concepts of OpenGL, here is the link:

  • Android OpenGL ES 2.0: Basic concepts and Hello World
  • Android OpenGL ES 2.0 complete introduction (two) : rectangle, image, read video memory, etc

The overall train of thought

In basic concepts and Hello World we mentioned that the ultimate purpose of a Shader program is to determine the Vertex coordinates and Fragment colors of a graph. In fact, this is the most basic and core operation primitives provided by OpenGL. We want to use OpenGL to achieve any effect, whether it is static light and shadow, color, shape, or the physical effect of motion, particle effect, in the final analysis, it is to determine the vertex coordinates and the color of the slice according to the time and position. But it’s easy to say in the end, but there’s a lot of research on how to do it in the graphics-related field, and it’s a very broad field, and I don’t have the ability to do it here.

This article will expand on four common 2D texture transformations, the core idea is to adjust texture coordinates and vertex coordinates.

Ideally, our textures, graphics, and view ports should be the same size, so there should be no problem sticking the textures onto the image and drawing the graphics onto the viewport. In a suboptimal situation where the dimensions are different but the aspect ratio is the same, using the most basic scaling operation (OpenGL’s default width and height fill, FIT_XY) in both processes would not cause any problems.

For example, when we do a full-screen camera preview, the image captured by the camera (which will be the texture) is 640 by 480, but we want to display it on a 1920 by 1080 screen, and the image captured by an Android camera is usually horizontal, but we need a portrait preview. So that’s where we need to rotate and scale. Next I will share the idea of clipping, flipping, rotating, and scaling textures in turn.

tailoring

If we draw a “full-screen” rectangle (the vertices are in the range of [-1, 1]) and the texture is “full-screen” (the texture is in the range of [0, 1]), then we can attach the texture completely to the shape (the image will be rendered full-screen on the screen). If we want to crop out a certain percentage of pixels in each direction, we can move the texture coordinates in the corresponding direction closer to the midpoint (0.5). For example, we can render an 896*360 image onto a 640*360 screen without distortion by cutting out 128 pixels on each side.

Why is that? The graph below illustrates the principle of cutting 20% on the left and 20% on the right (128/640 = 0.2) :

flip

Continuing with the scenario proposed above, if we want to flip left to right (also known as horizontal flip), we can switch the two left points of the texture with the two right points, and if we want to flip up and down (also known as vertical flip), we can switch the top two points of the texture with the bottom two points.

I wanted to draw a schematic too, but I found it difficult, so you can imagine for yourself, for example, if you flip your phone left to right, does it go left to right? 🙂

rotating

If we number the vertices of the texture by lower left, lower right, upper left and upper right, the change of 90° counterclockwise is shown below:

For example, the bottom left -> bottom right -> top right -> top left of the original image is 0132, and after rotating it counterclockwise by 90°, it becomes 2013. The texture coordinates we pass to OpenGL are by position, and the rotation effect can be achieved by passing in different numbering sequences. However, it must be the number order that 0123 can be rotated, including 0132, 2013, 3201, and 1320.

So this idea can only achieve 90/180/270 rotation, rotation of any Angle needs to be combined with the projection matrix to achieve, but the effect of non-90/180/270 rotation will be very weird, generally not used. For example, the effect of 45° counterclockwise rotation is as follows:

You can see that the top left, top right and bottom right corner of the picture has become striped.

The zoom

There are usually three modes of scaling:

  • FIT_XY: The width and height are filled, if the width and height ratio is not consistent, deformation will occur;
  • CENTER_CROP: The short side is filled, the long side is scaled in proportion, and both ends of the part are cut off;
  • CENTER_INSIDE: the long side is filled, the short side is scaled in proportion, and the black side is left at both ends of the insufficient part;

FIT_XY is OpenGL’s default mode and we don’t need to do anything about it. CENTER_CROP/CENTER_INSIDE can be done by adjusting the vertex coordinates. CENTER_CROP is used to set the vertex coordinates beyond [-1, 1], which will be cut off by OpenGL. CENTER_INSIDE allows vertex coordinates to be less than [-1, 1], with no content, if we can set the color of the content-free region with a glClearColor call.

Cropping, flipping, and rotating don’t actually adjust what we draw (all “full-screen” rectangles), just how we paste the texture, while scaling doesn’t adjust how we paste the texture (” full-screen “textures), it adjusts what we draw.

Careful friends may notice that CENTER_CROP can be implemented using clipping, and indeed it is. CENTER_INSIDE, however, cannot be clipping. If the value range of texture coordinates exceeds [0, 1], duplicate edge pixels will appear, as shown in the following figure:

You can see that the images on the left and right have become striped.

About the striation effect

If we specify a texture coordinate value beyond the [0, 1] interval, how is the content of the excess part determined? All we see above are streaks, which look like the repetition of the edge pixels. Is that definitely the effect?

This problem is called texture wrapping in OpenGL. OpenGL specification defines four wrap modes: GL_REPEAT, GL_MIRRORED_REPEAT, GL_CLAMP_TO_EDGE, GL_CLAMP_TO_BORDER. The effect of these four modes is shown below (from open.gl) :

Since the output texture of Android camera is OES texture, and in OES texture, we can only use GL_CLAMP_TO_EDGE for wrap mode, so the preview collected by camera, It must be a duplicate effect of edge pixels (see section 3.7.14 of the OES_EGL_image_external documentation).

Show me the code

Talk is cheap, show me the code.

Next, I will modify the preview code of Grafika to add the above transformation function. The complete code can be obtained on my GitHub.

Grafika’s drawing logic is encapsulated in the Texture2dProgram#draw function. Its Shader code is very simple and is already covered by rectangles, images, reading video memory, and so on, except that it uses a different interface to draw rectangles. So we used gles20.gldrawelements (Gles20.gl_Triangles,,), and Grafika uses GLES20.glDRAwarrays (GLes20.GL_triangle_strip,,).

GlDrawElements are given a list of vertices and draw order (a list of vertex indexes) to draw, whereas glDrawArrays are given only a list of vertices. For very complex models that require a lot of triangles and a lot of repetition of vertices, the space taken up by vertex coordinates is much larger than that taken up by indexes, so glDrawElements space is more efficient.

In addition, glDrawElements and glDrawArrays both require a mode parameter to determine how the sequence of vertices is handled (the list of vertices given by glDrawArrays is the sequence of vertices, GlDrawElements requires the indexes in the index list to be replaced by vertices to obtain the vertex sequence), OpenGL defines three modes:

  • GL_TRIANGLES: Each of three vertices forms a triangle, i.e., vertex 012 forms a triangle, vertex 345 forms a triangle, and so on;
  • GL_TRIANGLE_STRIP: Any three adjacent vertices form a triangle, i.e. vertex 012 forms a triangle, vertex 123 forms a triangle, and so on;
  • GL_TRIANGLE_FAN: The first vertex does not move, and any subsequent adjacent two vertices form a triangle with the first vertex, i.e., vertex 012 forms a triangle, vertex 023 forms a triangle, and so on;

See the OpenGL glDrawElements document, The OpenGL glDrawArrays document, and the Triangle Primitives section of OpenGL Wiki for details.

Now let’s go back to Grafika’s drawing code.

In Grafika, the vertex coordinates and texture coordinates are all encapsulated in Drawable2d, so my main changes are in Drawable2d. In addition, I in webrTC-Android source guide (ii) : As mentioned in the preview implementation analysis, Grafika’s Shader code does not transform vertex coordinates, but texture coordinates. The logic of transformation is confusing, so I removed this logic. Use identity matrix).

Processing framework

Private static final float FULL_RECTANGLE_COORDS [] = {1.0 f, 1.0 f, / / 0 bottom left 1.0 f, 1.0 f, // 1 bottom right -1.0f, 1.0f, // 2 top left 1.0f, 1.0f, // 3 top right}; Private static final float FULL_RECTANGLE_TEX_COORDS[] = {0.0f, 0.0f, // 0 bottom left 0.0f, 0.0f, // 1 bottom right 0.0f, 1.0f, // 2 top left 1.0f, 1.0f // 3 top right}; public void setTransformation(final Transformation transformation) { if (mPrefab ! = Prefab.FULL_RECTANGLE) { return; } vertices = Arrays.copyOf(FULL_RECTANGLE_COORDS, FULL_RECTANGLE_COORDS.length); textureCoords = new float[8]; if (transformation.cropRect ! = null) { resolveCrop(transformation.cropRect.x, transformation.cropRect.y, transformation.cropRect.width, transformation.cropRect.height); } else { resolveCrop(Transformation.FULL_RECT.x, Transformation.FULL_RECT.y, Transformation.FULL_RECT.width, Transformation.FULL_RECT.height); } resolveFlip(transformation.flip); resolveRotate(transformation.rotation); if (transformation.inputSize ! = null && transformation.outputSize ! = null) { resolveScale(transformation.inputSize.width, transformation.inputSize.height, transformation.outputSize.width, transformation.outputSize.height, transformation.scaleType); } mVertexArray = GlUtil.createFloatBuffer(vertices); mTexCoordArray = GlUtil.createFloatBuffer(textureCoords); }Copy the code

Since GL_TRIANGLE_STRIP is used, the order of vertices is not lower left -> lower right -> upper right -> upper left, but lower left -> lower right -> upper left -> upper right. If we use bottom left -> bottom right -> top right -> top left, then 012 and 123 would be bottom left -> bottom right -> top right -> top left, unfortunately these two triangles don’t make a rectangle.

In addition, as mentioned above, all transformation operations are realized by modifying texture coordinates and vertex coordinates, and the order of each transformation is not important, for example, flip and crop first is also ok. Let’s look at the code for each transformation.

resolveCrop

private void resolveCrop(float x, float y, float width, float height) {
    float minX = x;
    float minY = y;
    float maxX = minX + width;
    float maxY = minY + height;

    // left bottom
    textureCoords[0] = minX;
    textureCoords[1] = minY;
    // right bottom
    textureCoords[2] = maxX;
    textureCoords[3] = minY;
    // left top
    textureCoords[4] = minX;
    textureCoords[5] = maxY;
    // right top
    textureCoords[6] = maxX;
    textureCoords[7] = maxY;
}
Copy the code

As you can see, the clipping effect is achieved by limiting the value range of texture coordinates.

resolveFlip

private void resolveFlip(int flip) { switch (flip) { case Transformation.FLIP_HORIZONTAL: swap(textureCoords, 0, 2); swap(textureCoords, 4, 6); break; case Transformation.FLIP_VERTICAL: swap(textureCoords, 1, 5); swap(textureCoords, 3, 7); break; case Transformation.FLIP_HORIZONTAL_VERTICAL: swap(textureCoords, 0, 2); swap(textureCoords, 4, 6); swap(textureCoords, 1, 5); swap(textureCoords, 3, 7); break; case Transformation.FLIP_NONE: default: break; }}Copy the code

When flipped horizontally, swap(0, 2) swaps the x coordinates of the lower left and right points, swap(4, 6) swaps the X coordinates of the upper left and right points, that is, the left and right points are swapped. Same thing with the vertical flip, you can do it yourself.

resolveRotate

private void resolveRotate(int rotation) { float x, y; switch (rotation) { case Transformation.ROTATION_90: x = textureCoords[0]; y = textureCoords[1]; textureCoords[0] = textureCoords[4]; textureCoords[1] = textureCoords[5]; textureCoords[4] = textureCoords[6]; textureCoords[5] = textureCoords[7]; textureCoords[6] = textureCoords[2]; textureCoords[7] = textureCoords[3]; textureCoords[2] = x; textureCoords[3] = y; break; case Transformation.ROTATION_180: swap(textureCoords, 0, 6); swap(textureCoords, 1, 7); swap(textureCoords, 2, 4); swap(textureCoords, 3, 5); break; case Transformation.ROTATION_270: x = textureCoords[0]; y = textureCoords[1]; textureCoords[0] = textureCoords[2]; textureCoords[1] = textureCoords[3]; textureCoords[2] = textureCoords[6]; textureCoords[3] = textureCoords[7]; textureCoords[6] = textureCoords[4]; textureCoords[7] = textureCoords[5]; textureCoords[4] = x; textureCoords[5] = y; break; case Transformation.ROTATION_0: default: break; }}Copy the code

We’re only going to analyze the case where we choose 90 degrees, and we’re going to look at the rest of the case. This code is the same as moving up left to bottom left, up right to top left, down right to top right, and down left to bottom right. Isn’t that the same as rotating 90 degrees counterclockwise?

resolveScale

private void resolveScale(int inputWidth, int inputHeight, int outputWidth, int outputHeight, int scaleType) { if (scaleType == Transformation.SCALE_TYPE_FIT_XY) { // The default is FIT_XY return; } // Note: scale type need to be implemented by adjusting // the vertices (not textureCoords). if (inputWidth * outputHeight == inputHeight * outputWidth) { // Optional optimization: If input w/h aspect is the same as output's, // there is no need to adjust vertices at all. return; } float inputAspect = inputWidth / (float) inputHeight; float outputAspect = outputWidth / (float) outputHeight; if (scaleType == Transformation.SCALE_TYPE_CENTER_CROP) { if (inputAspect < outputAspect) { float heightRatio = outputAspect / inputAspect; vertices[1] *= heightRatio; vertices[3] *= heightRatio; vertices[5] *= heightRatio; vertices[7] *= heightRatio; } else { float widthRatio = inputAspect / outputAspect; vertices[0] *= widthRatio; vertices[2] *= widthRatio; vertices[4] *= widthRatio; vertices[6] *= widthRatio; } } else if (scaleType == Transformation.SCALE_TYPE_CENTER_INSIDE) { if (inputAspect < outputAspect) { float widthRatio = inputAspect / outputAspect; vertices[0] *= widthRatio; vertices[2] *= widthRatio; vertices[4] *= widthRatio; vertices[6] *= widthRatio; } else { float heightRatio = outputAspect / inputAspect; vertices[1] *= heightRatio; vertices[3] *= heightRatio; vertices[5] *= heightRatio; vertices[7] *= heightRatio; }}}Copy the code

If inputWidth * outputHeight == inputHeight * outputWidth, then the three scaling modes actually work the same, so we don’t have to do anything.

CENTER_CROP, if inputAspect < outputAspect, that is, the inputAspect ratio is less than the outputAspect ratio, it indicates that the input width is the short side and the height is the long side. The width is full, that is, the x-coordinate value range of the vertex is full [-1, 1]. The height is scaled up, that is, the x-coordinate value of each vertex is multiplied by the magnification factor. The coefficients are outputAspect/inputAspect (the reasoning process is shown in the figure below). InputAspect > outputAspect indicates that the height is the long side and the width is the short side, and the processing logic is the same.

When CENTER_INSIDE, the calculation result of ratio is less than 1, that is, the value range of vertex coordinates is limited to be less than [-1, 1]. The logic is similar and you can deduce it by yourself.

conclusion

In this paper, ON behalf of Brother Tie Lei, I shared the realization ideas of four commonly used 2D transformation in OpenGL, and the realization code was based on Grafika project. I will summarize the related content of 3D transformation in the future, the time is not fixed, but brother Tie Lei will share the content of 3D coordinate transformation in OpenGL in the near future. Stay tuned!