First, graphics API

The first thing we need to know is what OpenGL is and what the other apis are.

OpenGL :(Open Graphics Library) is a cross-platform, kua programming language programming Graphics program interface, it abstracts the resources of the computer as a single OpenGL object, the operation of these resources abstraction into a single OpenGL instructions. [Graphics and image rendering processing for PC (Mac, Windows)]

OpenGL ES :(OpenGL for Embedded Systems) is a subset of the OpenGL 3d graphics API. Designed for embedded devices such as mobile phones, PDAs and game consoles, it removes many unnecessary and low performance apis. [For mobile terminal (iOS, Android)]

Metal: Apple has launched Metal, a new platform for game developers that can render 3D graphics 10 times better. Metal is Apple’s framework for 3D rendering. Apple bottom rendering is implemented by Metal.

DirectX: DirectX is not a pure graphics API. Most importantly, DirectX is a multimedia processing framework under Windows. It does not support the use of platforms other than Windows, so it is not a cross-platform framework. According to the classification of nature, it can be divided into four parts: display part, voice part, input part and network part.

So, what problem is a graphics API supposed to solve?

Simply put, it is to achieve the bottom rendering of graphics. And: rendering issues.

• For example, the system renders buttons, images, views, and layers
• For example, in game development, the rendering of game scenes/characters
• For example, in audio and video development, data rendering after video decoding
• For example, in the map engine, rendering data on the map
• For example, in core animation, implement animation drawing operations (rotation, scaling, movement, layer effects)
• For example, in video processing, add a filter effect to the video

Knowledge development:

OpenGL vs. OpenCV: OpenGL is used for rendering (displaying bitmaps). OpenCV is used for recognition (face recognition/identity recognition/object recognition), mostly related to artificial intelligence. Face++ can also do recognition.

OpenGL/OpenGL ES with Metal:

Apple introduced Metal in 2014 to improve its rendering performance and gradually began to abandon the OpenGL/OpenGL ES syntax, but this does not mean that it cannot be used (similar to swift oc is still available).

Before 2018, CoreAnimation was based on OpenGL ES to encapsulate the upper framework. In 2018, apple made the layout and migrated the bottom rendering to Metal.

Reasons for migration: 1. Many projects use the third party OpenGL ES. In order to meet the needs of developers to be able to customize and iterate, Apple has launched Metal because it is impossible to use the important rendering core framework by the third party all the time. 2. Before Metal, OpenGL ES could only use GPU to process graphics, and could not schedule GPU to carry out a large number of customized concurrent operations in a high degree. Therefore, Apple provided such an entrance in Metal that GPU could be fully invoked to complete this task.

The essence of OpenGL/OpenGL ES/Metal problem solving in any project is to use GPU chips to efficiently render graphics images. (Graphics apis are the only way iOS developers can approach gpus.)

Ii. Noun analysis

Context: OpenGL contextIt’s a giant state machine, and records hold some states)

• Before an application invokes any OpenGL instruction, an OpenGL context is created. This context is a very large state machine that holds the various states in OpenGL and is the basis for the execution of OpenGL instructions
• OpenGL functions no matter in which language, are similar to C language process-oriented functions, essentially are OpenGL context this huge state machine in a state or object operation, before using the object to set the current object. Therefore, by encapsulating OpenGL instructions, it is possible to encapsulate the relevant calls of OpenGL into an object-oriented graphics API

* Since OpenGL context is a large state machine, switching contexts tends to be expensive, but different drawing modules may need to use completely separate state management. So you can create multiple different contexts in your application, use different contexts in different threads, and share textures, buffers, and so on. This approach is much more efficient than repeatedly switching context or changing render state a lot.

2. State machine (theoretically a machine)

We can understand it as follows: State machine describes various states experienced by an object in its life cycle, transitions between states, motivations for transitions, conditions and activities performed during transitions. In other words, a state machine is a behavior that describes the sequence of states an object experiences in response to events in its life cycle and its response to those state events. Therefore, it has the following characteristics:

• Memory (ability to remember current state)
• Can receive input (according to the input content and their original state, modify their current state, and can have corresponding output)
• When entering the special state (stop state), it will no longer receive input and stop working

State machine in OpenGL

• OpenGL can record its own status (such as: the current use of color, whether to enable what function, etc.)
• OpenGL can accept input (when we call OpenGL, we can actually view OpenGL as receiving input from us. For example, we call glColor3f, which means that OpenGL will change its “current color” state after receiving this input.
• OpenGL can go into a stop state and no longer accept input. OpenGL always stops working before the program exits

3, apply colours to a drawing

The process of converting pictures, graphics, buttons, videos and other data into 2D spatial images and rendering them to a screen

VertexArray and VertexBuffer

All images in OpenGL are composed of primitives. In OpenGL ES, there are three primitives: point, line and triangle. Vertex data storage: stored in memory (vertex array) stored in GPU pre-allocated video memory (vertex buffer) (this method is more efficient)

Extension: bitmap (texture) we usually common JPG, PNG and other pictures belong to compressed pictures, these compressed pictures to our computer inside need to decompress into bitmap. Suppose an image has a resolution of 120 by 120. So it has 14400 pixels, each pixel is stored by RGBA, each color in RGBA takes 8 bits, (1 byte =8 bits) so it takes 4 bytes, so the bitmap size of this image is 14400 * 4. So bitmap is relatively large, direct transmission, storage computers may appear some problems, so there are compression, decompression

5, pipeline

A pipeline is an abstract concept. Rendering a graph in OpenGL goes through a pipeline node by node.

It can be understood as an assembly line, which has a fixed sequence of operations and needs to be executed one by one in order

6. Fixed pipeline/storage shaders

Shader is a wrapped shader program that only needs to pass in specific parameters to perform the rendering operation. Fixed pipeline: no custom programming. Programmable pipeline: customizable. Currently, OpenGL has only two programmable programs: vertex shader and chip shader

7. Shader program

Before OpenGL calls the draw function, you need to specify a shader program compiled by the shader. Common shaders:

• Vertex shader,
• Slice shader/pixel shader (just a different name in OpenGL and DX),
• Geometric shaders,
• Surface subdivision shader.

Usually we write methods/functions (code snippets) for CPU use. The shader, then, is for the GPU. Fixed shader: API provided by Apple, calling custom shader directly by passing arguments: custom code snippet written based on GLSL syntax

8. VertexShader

Code for dealing with vertices; A vertex shader is a vertex-by-vertex program that executes a vertex shader once for each vertex, in parallel, without access to other vertex data. This is where the transformation of the vertex coordinates from its own coordinate system to its normalized coordinate system takes place.

• Determine the position of vertices
• Handle transformation of each vertex of graph (rotation, translation, projection, etc.)
• The screen of the mobile phone is actually 2D. If 3D is displayed, 3D graphic data needs to be converted into 2D for projection conversion

9. FragmentShader

Slice: is the pixels in the screen so slice shader is to process the pixels in the screen one by one. It is a pixel-by-pixel program that executes the pixel shader once for each pixel, also in parallel. (For example, a 120 x 120 image is computed 14400 times, which is a lot of times for CPU, but is trivial for GPU, because GPU is parallel computation)

** Example: ** How is image saturation adjusted?

Pixel by pixel changes are made in the slice shader

10, GLSL (OpenGL Shading Language)

OpenGL shader language, custom shader programming. Follow OpenGL’s standard scheduling GPU to do calculations

11. Rasterization

Not programmable. In the process of pixel assembly, through two parts of work (1, fixed pixel range of graphics; 2, attach the color.) Generates slices.

Specific description:

• Rasterization: The process of converting vertex data into slices. (No element in the slice corresponds to a pixel in the frame buffer)
• Rasterization: The process of programming a two-dimensional image with a geometric element. (This process includes the two processes described above)
• Rasterization: The process of converting analog signals into discrete signals. The mathematical description of the object and the color information related to the object are converted into pixels for the corresponding position on the screen and colors for filling the pixels.

12, texture

You can think of it as a picture. When rendering graphics, you need to fill the image in its code, which can make the scene more realistic. In OpenGL, we tend to call images textures.

13. Blending (Blending)

For example: there are two layers of different colors, the two layers are superimposed together, color mixing will occur, resulting in a new color. This color mixing behavior is calculated by OpenGL’s mixing algorithm, can also be implemented through the chip shader, of course, the performance will be worse

14. Transformation

It is used when the graph wants to translate, scale, rotate and other transformation operations

15. Projection Matrix

Used when converting 3D coordinates to 2D screen coordinates, the actual lines will also be drawn in 2D coordinates

16. Render the screen/SwapBuffer (SwapBuffer)

• Render buffers typically map system resources, such as Windows. If you render the image directly to the window’s corresponding render buffer, you can display the image to the screen
• However, if each window has only one buffer, then the screen is refreshed during the drawing process and the window may not display the full image
• To solve these problems, regular OpenGL programs will have at least two buffers. What is displayed on the screen is called the screen buffer, and what is not displayed is called the off-screen buffer. After a buffer is rendered, the image is displayed on the screen by swapping the screen buffer with the off-screen buffer
• As a result of the display refresh is generally by rows, so the screen up and down in order to prevent the exchange buffer area belong to two different frames, so the exchange generally will be waiting for the signal to the display refresh to complete, the display refresh intervals exchange twice, the signal is known as the vertical sync signal, the technology is called vertical synchronization.
• With the use of double buffers and vSYNC, the frame rate is not quite as high as the hardware allows because you always have to wait for the buffer to be swapped before rendering the next frame. To solve this problem, a three-buffer technique was introduced, in which two off-screen buffers are rendered back and forth alternately while waiting for vSYNC, while vsync occurs, the screen buffer is swapped with the recently rendered off-screen buffer to make full use of the hardware performance