This article is shared from Amoy front-end intelligent team – Fu Hang

sequence

In my last article, I talked about my transition from the client side to the front end. In just one year, I opened the door of the front end world, and there were simply endless things to play with. I used to love Java and finally realized the power of JavaScript, which can write everything, not only Web, client, back-end and system applications. Can also be in the neural network, the Internet of things, and even embedded can be, is simply an all-purpose language, can be said to be able to program the place theoretically can use JS to write! After switching to the front end, I found a lot of new and interesting things, such as WebGL and WebAssembly(I will return to this series later). There are still such things in the front end, which was completely unexpected to me. I always thought that the performance of Web was poor, so I always play on Native. Before I played client, in order to pursue the ultimate performance, I coincidentally did the following two things. The first is to learn ARM by myself and use assembly to realize my functions (which is consistent with WebAssembly); the second is to learn OpenGLES by myself and use GPU to accelerate the optimization of my code (which is consistent with WebGL). Now that I’m in front Land, why don’t I play with these two things? Because usually work is too busy, can not suddenly liver out of a big dry goods, and this goods is not simple, an article also learn not to go deep, so they plan to separate a few play together to play this goods. In fact, I also just learned, dare not preach what, that is, with the help of off-duty spare time by virtue of interest to learn to play, summed up the experience of just, and everyone exchange exchanges. I have written a red and blue 3D player and implemented the danmu SDK on Android using OpenGLES, so I will take this as the goal, learn WebGL, and then write a web version of the red and blue 3D player and implement the danmu SDK, although I don’t know whether it can be achieved, but in theory it is possible. Just go in that direction and try.

What is WebGL?

So what exactly is WebGL? When we want to learn or understand something, the first thing we usually do is to use a search engine to find information. Here’s an excerpt from the encyclopedia:

WebGL (Full Write Web Graphics Library) is a 3D Graphics protocol that allows JavaScript to be combined with OpenGL ES 2.0. By adding a JavaScript binding to OpenGL ES 2.0, WebGL can provide hardware 3D accelerated rendering for HTML5 Canvas, so that Web developers can use the system graphics card to more smoothly display 3D scenes and models in the browser, as well as create complex navigation and data visualization. Obviously, the WebGL technology standard eliminates the need to develop web-specific rendering plugins that can be used to create web pages with complex 3D structures, even to design 3D web games, and so on.

At first glance, who doesn’t know JavaScript? Canvas is also available on the client side. All that is left is OpenGL ES 2.0, so you just need to learn this. Look again at the encyclopedia description:

WebGL 1.0 is based on OpenGL ES 2.0 and provides an API for 3D graphics. It uses HTML5Canvas and allows the use of the Document Object Model interface. WebGL 2.0 is based on OpenGL ES 3.0, ensuring that many selective WebGL 1.0 extensions are provided and new apis are introduced. Automatic memory management can be implemented using partial Javascript.

So WebGL is based on OpenGLES, that’s great, I used to learn OpenGLES, right? That’s twice the result with half the effort ha ha ha! My learning style is slowly and step by step, from the most basic understanding of the introduction, to the simplest HelloWorld, and finally to practice with a practical example. Instead of coming up to teach you how to use it directly, maybe I will be slower, but I think the understanding is really the most profound, and for me without foundation, it is easier to get started, and will not give up the entry! After all, I think it’s important to develop an interest in this thing.

Graphics probe,

In fact, when I learn OpenGLES, it is a little painful, obviously I am a computer major, how can I so difficult to understand? How are so many concepts different from what I learned before? You really need to overturn your old mindset, and it’s better to throw it away and relearn it. Or step by step, first understand what is graphical programming.

Understand graphic programming

We know the development history of computer, from the most simple adder, to the Turing machine, the von neumann only by computer, and then to today’s smartphones, and I do believe you are quite obsession, two very idol, Bill Gates and Steve jobs, and read all the stories about them, presumably you clearly know the importance of graphics development. The gorgeous UI of MacOS, Windows, and today’s gorgeous pictures, games, and so on. Most simply, we know that when assembling a computer, we need to choose a good graphics card to play better games. In fact, we don’t know enough about computers. Haven’t written a line of code I have played countless beautiful screen of the game, however, when I learn programming, the most want to decrypt is a software, a such a cow game screen, in the end is how to write through these codes? Remember the first code you learned in C? Look at the console and see the output “Hello World!” Everyone said they would cheer for this great achievement. However, this is decades old technology; I want to know how to write a graphical interface instead of writing text on the console!

#include<stdio.h>
void main(a)
{
 printf("Hello World! \n");
}
  admindeMacBook-Pro- 191.:develop fuxing$ gcc hello_world.c admindeMacBook-Pro- 191.:develop fuxing$ ./a.out Hello World!  Copy the code

No matter what language we learn, C, Java, or JS, we actually program for the CPU, and all the logical computation is done by the CPU. When we look at how computers are made, we know that these calculations are actually the CPU’s instruction set operating on registers. For the output of the result display, basically are text letters, simple CRT display can be displayed. The operation of the CPU and the display of the output results, all of which are completed by the operating system and the compiler. Simply speaking, the compiler compiles the high-level language into 01 instructions, while the operating system manages the underlying hardware and encapsulates it into system interface calls. We only need to call the system interface. So let’s call printf() and we can display it on the monitor.



(Photo from Internet)



In fact, the system for most of the hardware operations are completed through the driver, each hardware has a corresponding driver, the driver to follow the unified specification and interface, and then realize. So every time we reinstall the system, we have to install various drivers. For the development of hardware, is the field of embedded, although not in-depth discussion, but can understand from here, the hardware is also a processing unit!



(Photo from Internet)



We finally know that the graphics card is actually a graphics processing Unit, also known as GPU (Graphic Process Unit), and THE same meaning of CPU, used for rendering pictures. So the question is, how is the graph drawn? In fact, we all know that pixel, we are in the age of electronic explosion, who doesn’t know resolution? A display has a resolution of 1920×1080, which means 1920 pixels horizontally and 1080 pixels vertically, and we can understand that a pixel is a very, very small point of light, so the screen is actually a super dense collection of pixels, and of course, this has material science implications, like LCD screens and so on, We’re just abstracting to understand it. After learning physics, we also know the three primary colors, as long as red, green and blue can be all colors, that is to say, in fact, each pixel has a different amount of red, green and blue. Of course, the display is certainly not so simple, the original data is YUV, which involves in-depth graphics, we still want to abstract understanding now.



In computers, a byte is 8 bits, and the value range is 0 to 255. If a byte is used to represent the value range of a primary color, then red, green and blue are three bytes, plus a transparency Alpha, RGBA is exactly four bytes, usually represented by an integer int. This allows you to quantify a figure into a number, and since it’s already digitized, it can be processed by a computer. Now we can understand that, in fact, a two-dimensional image is a two-dimensional shaping matrix, and this is all logic that we can manipulate in both CPU and memory. Finally, the operating system swaps the data to the graphics card, usually copied to video memory, and gives it to the GPU for rendering. Essentially, the GPU processes the data and uses it to control the display to emit different measurements of primary color at different points, so that the graphics can be displayed.

Graphical programming API


From the history above, it is clear that the graphics field is very important and scalable, between the operating system and hardware (drivers) layer, can do a lot of things, providing important graphical programming interfaces to facilitate the development of 2d and 3D graphics. So, the world basically has two big formations again, OpenGL and DirectX.



(Photo from Internet)



DirectX too familiar, we play games, must install this thing, childhood ah! It’s game development for Windows, rule the world! OpenGL is cross-platform and can be used anywhere (otherwise there would be no WebGL today)! And not just for game development, but for almost anything. Of course, very few people would use these two things to develop games directly, but rather to develop game engines, and then develop games based on those engines, like Unity3D!



For OpenGL, we used Shading Language, which is just a new Language. We just need to learn the syntax, but what we need to understand is how the rendering process is. Otherwise we wouldn’t know how to write in a coloring language.



Since both OpenGL and DirectX are used for graphics programming, why do we write so much UI in our daily development and never get involved in either? Let’s think about what UI is involved in everyday development. Image, Button, Text, EditText, InputText, etc., are basically assembled using these components, which seem to involve very little complex graphics, at most some animation effects. In fact, these components, or our daily use these apis, are already sealed the bottom how to render, for us, don’t need to know how to get to the bottom, perhaps are on the CPU for processing, then has carried on the shows from memory replication to memory, and may open hardware acceleration, In fact, all rendering processing is left to the GPU. Besides, in our daily development, we don’t need such high performance, there is no need to write anything GL at all.



If you’ve ever played a game on Windows without DirectX, you can actually play it, but not as smoothly, which means you don’t call the DirectX API, you build your game on top of the game engine. If you have DirectX, you have hardware acceleration. If you don’t, Will not run the game, or can use the CPU for rendering processing.



When we found that image-related CPU processing was no longer available in daily development and performance became a bottleneck, we had to implement the underlying rendering logic by ourselves, which was the time to write GL. For example, when playing a video, some pixels need to be processed in real time for each frame, which will be slow if processed by CPU. We can use GPU to process pixel logic, while CPU can only handle playback synchronization logic.

What is OpenGL ES

We have learned that to learn WebGL is actually to learn OpenGLES, which can be understood as WebGL is to call OpenGLES API with JS, so what is OpenGLES? Do WE need to learn OpenGL before we learn this? The answer is no. Due to the rapid development of mobile devices, there is a subset of APIS for these Embedded devices, OpenGL for Embedded Systems. Obviously, because it’s a subset, it’s a little bit of a clipping of OpenGL. And then OpenGL ES has now evolved to version 3.0, and each version is a huge leap forward. The big difference between 1.0 and 2.0 is that 2.0 supports custom rendering pipeline programming, which means that in addition to calling OpenGL’s API in a high-level language, we can also write coloring language programs in the pipeline! This is the biggest difference between 2.0, and 3.0 does not have huge changes based on 2.0, so the learning of the two can be progressive, there is no need to change.

OpenGL ES 1.0 rendering pipeline


A GPU has many parallel processing units inside it that process graphics signals, so it is much more efficient than a CPU’s serial execution. Our understanding of CPU power is that there are many instruction sets, and if we want to call these instruction sets, we need to write assembly language, and the API called at the high-level language level is the system or platform for us to encapsulate the capabilities. On the other hand, GPU will also have corresponding instructions. Hardware developers will develop corresponding drivers and provide standard APIS for system calls. OpenGL and DirectX provide rendering capabilities based on this. In fact, they are the standard ones, and backwards hardware provides rendering capabilities. A rendering pipeline is actually a series of drawing processes that are handled by a processing unit. ** Pipelines are shown as follows:





(Photo from Internet)

Key points to understand:

  • What is a graph element, in fact, is an image unit; When OpenGL draws a graph, it has a combination of elements. Drawing methods point, line and triangle, respectively corresponding to the three elements.
  • What is rasterization? A pixel is a continuous quantity in mathematics, but in a display it is a discrete pixel, so rasterization is the process of converting vertex data into pixels.
  • What is a slice and why not a pixel? A pixel is a point on a screen, that’s two dimensional, but a pixel on a screen in three dimensions might cover many pixels, so it’s not called a pixel in three dimensions, it’s called a slice.

OpenGL ES 2.0 rendering pipeline


The rendering pipeline for 2.0 is shown below:



(Photo from Internet)

The biggest difference of 2.0 is that there are many vertex shaders and chip shaders, which is convenient for programmers to develop, need to learn the coloring language, while 1.0 can only call the upper API.

Vertex shader


What is a vertex? A triangle has three vertices, a rectangle has four, and a line segment has only two. Vertex Shader is a programmable processing unit that transforms each Vertex of a graph into a texture coordinate, color, point position, and other Vertex attributes. Working principle diagram is as follows:



(Photo from Internet)



Attribute is the variable to which different information of each vertex belongs. It generally contains vertex coordinates and vertex texture coordinates and is transmitted through high-level language. Uniform are global variables and are data transmitted through high-level languages. Varying is the data that is output to the slice shader, usually the texture coordinates. Then there are the temporary variables you write code for and the built-in variables for GL_xxx.



An example vertex shader would look like this:

uniform mat4 uMVPMatrix; // Total transformation matrix
attribute vec3 aPosition;  // Vertex position
attribute vec2 aTexCoor;    // Texture coordinates
varying vec2 vTextureCoord;  // A variable to be passed to the chip shader
void main(a) {  gl_Position = uMVPMatrix * vec4(aPosition,1); // Calculate the vertex position according to the total transformation matrix  vTextureCoord = aTexCoor;// Pass the received texture coordinates to the slice shader } Copy the code

As you can see, GL is not complex and is similar to C. Uniform, attribute, and VARYING are all keywords that identify the source of data. Mat4, VEC3, and VEC2 are data types corresponding to 4-dimensional matrix, 3-dimensional vector, and 2-dimensional vector respectively.

Chip shader


The Fragment Shader will be executed once for each pixel after rasterization. The main function is to sample texture and summarize color.



What is texture, direct understanding is graphic surface, skin and so on, namely image, color, pattern and so on. For example, in Android, a Bitmap of an image is mapped directly to OpenGLES to become a texture, when the texture is an image, the Bitmap can be recycled, has been transferred to video memory.



Working principle diagram is as follows:



(Photo from Internet)



The data Varying comes from the vertex shader. Uniform is usually texture data, and gl_FragColor is the output result.

An example fragment shader code is as follows:

precision mediump float;// tell the precision is float
varying vec2 vTextureCoord; // Receives arguments from the vertex shader
uniform sampler2D sTexture;// Texture content data
void main(a)                         
{           
 // Sample the color value from the texture for this pixel  gl_FragColor = texture2D(sTexture, vTextureCoord); } Copy the code

In fact, draw a rectangle is by drawing two triangle synthetic, namely has six vertices, each vertex performs a vertex shader, however the vertex shader output for passed to the fragment shader the coordinates of the variable is not directly passed to the fragment shader, but after the rasterizer, through interpolation calculation, The coordinate of each slice is obtained and passed to the slice shader, so the slice shader performs the processing of each slice (pixel). For example, if you draw an image all over a 1920×1080 screen, the pixel shader is executed once for each pixel.

Shading language

This is basically how OpenGLES works, the shader language is another big piece of knowledge, you can write code on the above two shaders, it’s not possible to list everything here, learning this is like learning a language, and it’s not difficult to understand. First, we simply know several vector types, MAT4 is a four-dimensional matrix, VEC4 is a four-dimensional vector, VEC3 is a three-dimensional vector, vec2 is a two-dimensional vector, and then we can learn and summarize again.

conclusion

This play to learn so much, mainly introduce and understand the WebGL, temporarily don’t write code, even if there are some example code, mainly is given priority to with reading comprehension, to feel this historical process, cultivate interest, one step at a time, slowly learning, now focus on learning about shader first, later if you want to write advanced functionality, There are all kinds of transformations, math to learn, and simple examples to learn together next time. The above is only a personal learning experience, unavoidably there will be some mistakes, to understand the wrong place welcome to point out.