1. Introduction

IOS Audio and Video is a series that will document some of the bloggers’ learning notes and pitfalls in iOS audio and video, so that you can learn more about the past.

This series of articles includes but is not limited to:

  1. IOS Audio and Video: Introduction to Common OpenGL terms
  2. .

This article, the first in a series, is intended to give you a quick introduction to OpenGL.

2. Introduction of OpenGL

2.1 What is OpenGL

OpenGL (Open Graphics Library) is a cross-language, cross-platform application programming interface Library for rendering 2D and 3D vector Graphics.

It is a graphical API library that abstracts computer resources into OpenGL objects and operations on these resources into OpenGL instructions. Since it only provides rendering functions (operating on GPU chips) and has nothing to do with windowing systems, audio, printing, keyboard/mouse or other input devices, it can be cross-platform (mainly running on PC, such as Mac OS, Linux, Windows, etc.).

OpenGL ES, Metal, DirectX, etc. The main differences between them are:

  • OpenGL ESOpenGL for Embedded Systems: YesOpenGLIs designed for embedded devices such as mobile phones, PDAs and game consoles, removing many unnecessary and low performance apis.
  • Metal: Is a platform technology launched by Apple, mainly running on various Apple platforms (macOS, iOS, tvOS). The technology is designed for multithreading and provides excellent tools to integrate everything into XCode. After optimization,MetalmakeCPUandGPUCan work together to achieve optimal performance (it can increase rendering performance by 10 times for 3D images).
  • DirectX: is a multimedia programming interface created by Microsoft. It consists of a number of apis (not just graphical apis) and is limited to Windows platforms (currently not supported outside of Windows). According to its nature, it can be divided into four parts, namely, display part, sound part, input part and network part.

DirectX will not be covered in this series because the bloggers are mainly working on iOS development.

2.2 What problems does OpenGL solve

As a graphics API library, the essence of OpenGL, OpenGL ES and Metal to solve problems in any project is to use GPU chips to efficiently render graphics images. Using these graphics API libraries is also the only way iOS developers can get close to a GPU.

Therefore, graphics API libraries are often used in the following scenarios:

  • In game development, the rendering of the game scene
  • Audio and video development, the video decoded data rendering, to the video filter processing
  • Map development, rendering of map data
  • Animation, the realization of animation drawing
  • Aerospace, medical, etc

2.3 The problem of choice

Apple introduced Metal at WWDC 2014, but it wasn’t until WWDC 2018 that Apple completed the internal transition from OpenGL ES to Metal. It also announced the abandonment of OpenGL/OpenGL ES/OpenCL on Apple devices (macOS Mojave, iOS 12, tvOS 12). Developers working on graphics apis need to consider which one to start with from their own perspective, and consider the following aspects:

  • OpenGL/OpenGL ESCross-platform, andMetalApple only.
  • Apple’s own system has gone fromOpenGL/OpenGL ESMigration toMetalIt took a lot of time (4 years or so) to address the underlying API dependencies of Apple’s internal systems,OpenGL/OpenGL ESThis becomes a third-party graphics API library.
  • Most of the timeOpenGL/OpenGL ESThe project team is very large (such as Baidu Map, Autonavi map, most of the audio and video project team) and has not been completed yetMetalMigration work of. At this point onlyMetalIs not enough.

The so-called art does not pressure the body, alongOpenGL -> OpenGL ES -> MetalThe full mastery of the route is also an option.

3. OpenGL common terms

3.1 OpenGL State Machine

A state machine is a theoretical machine that describes the various states an object goes through in its life cycle, the transitions between the states, the reasons for the transitions, the conditions, and the activities performed during the transitions. In other words, a state machine is a behavior that describes the sequence of states an object experiences in response to events in its life cycle and its response to those state events. Therefore, it has the following characteristics:

  • Have memory function, can remember its current state;
  • Can receive input, according to the input content and their original state, modify their current state, and can have the corresponding output;
  • When entering a special state such as shutdown state, it will no longer receive input and stop working.

OpenGL itself is a huge state machine, which also:

  • Can record their own status (such as the current use of color, whether the mixed function is enabled, etc.);
  • OpenGLCan receive input (when calledOpenGLDelta function, you can actually view it as delta functionOpenGLAfter receiving our input), according to the content of the input and their own state, modify their own state, and can get the output (such as we callglColor3f,OpenGLAfter receiving this input, it changes its current color state. We callglRectf,OpenGLOutputs a rectangle);
  • OpenGLYou can enter a stopped state and no longer receive input. Before the program exits,OpenGLAlways stop working first.

It is important to note that each state change is global, so after completing a function in a state, the state needs to be turned off/switched back.

If you can use the glColor function to select a color, all objects drawn in the future will be that color, unless the glColor function is reset again. Similarly, you can use the glTexCoord function to set a texture coordinate, which will be used for all objects drawn until the glTexCoord function is used again.

Basically, OpenGL is a state machine that holds its state until the user enters a command to change it.

Such as:

// Get whether to deep test/mix
glIsEnabled(GL_DEPTH_TEST);
glIsEnabled(GL_BLEND);

// Enable/disable depth testing
glEnable(GL_DEPTH_TEST);
glDisable(GL_DEPTH_TEST);

// Turn on/off blending
glEnable(GL_BLEND);
glDisable(GL_BLEND);
Copy the code

3.2 OpenGL Context

OpenGL is procedurally oriented, and it requires a Context to record all the information and states required for rendering, namely the OpenGL Context. Before an application can invoke any OpenGL instruction, it first needs to create an OpenGL context.

  • OpenGL context is closely related to OpenGL state machine. OpenGL context can be regarded as a group of OpenGL state machines.

  • OpenGL adopts client-server mode. GPU is equivalent to a Server, which can correspond to multiple clients, namely contexts, and a Client maintains a group of state machines. Most OpenGL instructions are asynchronous, not executed immediately, but the context sends commands to the server (there are also apis for synchronization).

  • OpenGL context is a thread-local variable, which means that if we draw in a thread, we need to specify a context for each thread, and multiple threads cannot specify the same context at the same time.

  • Since OpenGL contexts are a large set of OpenGL state machines, switching contexts is often expensive, but different drawing modules may need to use completely separate state management. As a result, you can create multiple different contexts in your application, use different contexts in different threads, and share textures, buffers, and so on. This approach makes more sense and more efficient than repeatedly switching context or changing render state a lot.

Figure 3.3 yuan

Primitive, “Primitive,” is short for basic graphic elements. In OpenGL/OpenGL ES, any image is made up of primients.

  • OpenGLElements of: point, line segment, triangle, quadrilateral, polygon
  • OpenGL ESPrimitives of: point, line segment, triangle

3.4 Vertex arrays and vertex buffers

When drawing an image, the vertex position data of the image is the vertex data.

When I call the OpenGL draw method,

  • If the vertex data is passed in from memory, it is usually stored in a block of memory in the form of an array, which is calledVertex array(Vertex Array);
  • A more efficient way to do this is to pre-allocate a piece of video memory to pre-store the vertex data into video memory, which is called video memoryVertex buffer(Vertex Buffer).

3.5 Rendering

In OpenGL, everything is in 3D space and the screen/window is in 2D. To convert raw graphics/image data into 3D spatial images and render them on 2D screens/Windows.

There are two main rendering processes, which are:

  • Vertex rendering: The process of forming the final shape of vertex data through a series of operations, such as transformation, filtering, interpolation, etc.
  • Pixel rendering: Fill the shape with color. In this process, the colors to be filled can come from vertex colors, textures, or even colors calculated by some number (such as lighting).

3.6 the line

Graphics pipelines, or pipelines for short, describe the process of rendering Graphics. Rendering graphics does not happen overnight. The whole process goes through stages, similar to an assembly line in a factory.

A pipeline is an abstract concept, called a pipeline because the graphics card processes data in a fixed and strict order (like water flowing from one end of a pipe to the other, this order cannot be broken).

The pipeline can be divided into several phases, each of which will take the output of the previous phase as input. All of these phases are highly specialized (they all have a specific function) and are easy to execute in parallel.

Pipelines can be divided into fixed pipelines and programmable pipelines:

  • Fixed lineIs a solidified rendering process that only requires the developer to be inCPUCall the function to complete the rendering operation by entering the parameters/data required for rendering and specifying a specific switch. It does not require or allow developers to customize the specific logic of rendering.
  • Programmable pipelineThe rendering logic must be implemented by the developer, otherwise the final image cannot be rendered. Developers can write the specific logic of vertex rendering and pixel rendering according to their specific needs, which can simplify the logic of rendering pipeline to the greatest extent to improve rendering efficiency, or they can implement their own specific algorithms and logic to render effects that cannot be rendered by fixed pipelines. With high customizability, but also put forward higher requirements for developers.

3.7 shader

Shaders are programs that run on the GPU to implement rendering. These small programs run for a particular part of the pipeline (turning input into output). OpenGL also needs to specify a shader program before actually calling the draw function.

  • A shader is just a program that converts input to output, and it is a very independent program because they cannot communicate with each other. The only communication between them is through input and output.

  • The most common shaders are vertex shaders and chip shaders. There are also some other shaders (geometry shaders, surface subdivision shaders, etc.), but the first two shaders are not as common (until OpenGL ES 3.0, programmable shaders are only vertex shaders and chip shaders).

3.7.1 Vertex shaders

Vertex shaders are used to manipulate Vertex data (rotation, translation, projection, etc.). Vertex shaders are vertex-by-vertex programs, that is, each vertex data is executed once, in parallel and without access to other vertex data.

3.7.2 Slice shader

A Fragment Shader is a program that calculates the fill color of each pixel. It is a per-pixel program, that is, each pixel executes a chip shader once, which is of course parallel and independent.

Consider: Why does OpenGL use a GPU instead of a CPU?

Some books refer to a fragment Shader as a Pixel Shader, or a fragment Shader, but developers need only know that the three are the same thing (they’re just called different things, and a fragment Shader is called a fragment function in Metal).

3.7.3 GLSL

GLSL (OpenGL Shading Language) is a kind of C Language for writing shaders. GLSL is tailored for graphical computing and contains some useful features for vector and matrix operations.

3.7.4 Rendering process of shaders

Shaders are compiled, linked, and eventually generated into a shader program (glProgram), which must contain both the vertex shader and the chip shader’s operational logic, while other shaders are optional (such as the subdivision shader).

A brief description of the shader’s rendering process is shown below:

Description:

  1. inOpenGLDuring rendering, vertex shaders first perform operations on vertex data, and then transform vertices into primitives through pixel assembly
  2. Next comes rasterization, where graph metadata is converted to rasterized data
  3. Finally, the rasterized data is rendered through a pixel shader (per-pixel, and determines the fill color of the pixel).

Note: This is just an overview of the flow (from a shader’s perspective).

3.8 rasterizer

Rasterization is the process of converting vertex data into slices, which has the function of converting a graph into an image composed of a grid, characterized by each element corresponding to a pixel in the frame buffer.

  • Rasterization is actually a way of puttingThe geometric primitivesinto2 d imageIn the process. The process consists of two parts, the rasterization process is producedfragment
    • The first part of the work: determine which integral grid areas in window coordinates are occupied by basic primitives;
    • Part two: Assign a color value and a depth value to each area.
  • The input for rasterization reception isThe geometric primitivesAnd its output ispixel(refer to the shader rendering process)pixelated

3.9 the texture

Textures can be understood as pictures (essentially bitmaps), and images often need to be filled when rendered. The image here is actually a texture, and in OpenGL, we prefer to call it a texture.

  • Common image file formats (BMP, TGA, JPG, GIF, PNG)
  • Common texture formats (R5G6B5, A4R4G4B4, A1R5G5B5, R8G8B8, A8R8G8B8, etc.)

3.10 hybrid

Blending refers to Blending the original colors of a pixel position with colors to be drawn on them in a manner (Blending algorithm) to achieve a particular effect. Mix two/more colors together.

  • Hybrid algorithms can passOpenGLFunction to specify. butOpenGLThe number of hybrid algorithms provided is limited. If a more complex hybrid algorithm is needed, it can generally passChip shaderImplementation, of course, the performance will be worse than the native hybrid algorithm.

3.11 matrix

In OpenGL, matrices are often used for auxiliary operations, such as:

  • Transformation matrix(Transformation) for the translation, scaling, rotation Transformation of the graph;
  • Projection matrixProjection is used to convert 3D coordinates to 2d screen coordinates in which actual lines will also be drawn.

And so on.

3.12 the frame cache

Frame Buffer, also known as Frame cache or video memory, is the Buffer that receives the rendering results and specifies the area for the GPU to store the rendering results.

Frame caching is described as follows:

  • All graphics images share the same memoryThe frame buffer.
  • The frame bufferIn real time:The frame bufferStorage is frame by frame, render completed image, graphics card will keep refreshingThe frame bufferEach frame that is not captured is discarded.
  • The frame bufferEach frame is an explicit message: suppose the resolution is750 x 1334, then each frame is saved750 x 1334Pixels (each pixel has a color value).

The Chinese translation of Buffer comes from the fact that when a high-speed part of a computer communicates with a low-speed part, the output of the high-speed part must be temporarily stored somewhere to ensure that the high-speed part matches the low-speed part. This meaning was later extended to mean “temporary storage area”.

4. Think about:

4.1 according to the GPU?

Consider: Why does OpenGL use a GPU instead of a CPU?

1. To answer this question, understand the difference between a GPU and a CPU. Let’s take a look at the two designs, which are roughly as follows:

The image is from the Nvidia CUDA documentation. The green ones are the computing units, the orange-red ones are the storage units, and the yellow ones are the control units.

As you can see from the picture,

  • CPUwith
    • A powerful arithmetic operation unit (ALU) that can perform arithmetic operations in very few clock cycles;
    • Large cache, which can reduce latency;
    • Complex logic control units (LCS), which reduce latency by providing branch prediction when a program has multiple branches.
  • GPUwith
    • A large number of arithmetical units, which are computationally intensive;
    • Small cache, the purpose of the cache is not to store the data that needs to be accessed later, but forThreadService, this point andCPUDifferent. If there are many threads that need to access the same data, the cache merges those accesses and then accesses them againDRAM(Because the data that needs to be accessed is saved inDRAMInstead ofcacheInside), after obtaining the datacacheWill forward this data to the corresponding thread, in other words,GPUThe small cache serves as a data forwarding role.
    • A simple control unit that combines multiple accesses into fewer accesses.

In addition, although the CPU is called multi-core, but the total number did not exceed double digits; The number of cores of GPU is far more than that of GPU, which is called multi-core.

In general, CPUS are good at logical control and serial computing, while Gpus are good at large-scale concurrent computing. OpenGL uses a GPU instead of a CPU to render a graphic image, which is often accompanied by massive computations (such as the same coordinate transformation for each vertex, color calculation for each slice, etc.).

5. Links

  • OpenGL proper noun analysis (with popular examples and personal understanding)
  • The frame buffer
  • CPU and GPU differences

Please give me a thumbs up if you found this article helpful

PS: Please indicate the source, thank you!