In order to use WebGL direct GPU rendering capability, many students’ first reaction is to use off-the-shelf open source projects, such as the famous Three.js. But is simply applying them the only option? Where do you start if you want to dig deeper into the WebGL foundation or even build your own wheels? This paper hopes to popularize some basic graphic library design knowledge based on the author’s own practical experience.
background
Not long ago, we added the ability to edit 3D text on the Web side of draft design. You can take text that is confined to two dimensions and add texture to it, like this:
The WebGL rendering of 3D text uses our own Beam base library. It is not a magic fork of an open source project, but is being implemented from scratch. As the author of Beam, the experience of pushing a new wheel to the ground in a production environment has definitely given me a lot of lessons to share. Let’s start with the basics and talk about the WebGL base library.
Rendering engine and WebGL base library
When WebGL is mentioned, people often think of open source projects like Three.js, which is a bit of a “React bucket for the Web front end.” In fact, Three is already a pretty advanced 3D rendering engine. There are some rough similarities and differences between it and a basic WebGL library like Beam:
- The rendering engine needs to mask the internal concepts of the rendering side such as WebGL, and the base library needs to open them up.
- The rendering engine can also have multiple renderers such as SVG/Canvas, while the base library focuses on one.
- The rendering engine is designed for specific scenes such as 3D or 2D; the base library does not assume this.
- Rendering engines are usually heavier in size, and base libraries are noticeably lighter.
In fact, I prefer to compare Three and Beam to React and jQuery: one tries to minimize the complexity of the renderer side, while the other tries to simplify the API of directly manipulating the renderer side. Given the flexibility of the graphics rendering pipeline, and the significant difference in weight (Three is already over 1M in size, and Tree Shaking is not very impressive), I believe the WebGL base library can be used in any scenario where control is desired.
The fact that the community has a popular WebGL base library like Regl proves that a similar wheel is not a bogus requirement.
WebGL is abstract
Before we can design the actual API of the base library, we need to at least understand how WebGL works. WebGL code has a lot of trivialities, and diving into it can make it hard to see the forest for the trees. According to the author’s understanding, the concepts we operate in the whole WebGL application are basically the following:
- Shader Shader is an object that stores graphics algorithms. Instead of JS code being executed in a single thread on the CPU, shaders are executed in parallel on the GPU, calculating the individual colors of millions of pixels per frame.
- Resource Is an object that stores graph data. Just as JSON becomes Web App data, resources are data passed to shaders, including large arrays of vertices, texture images, and global configuration items.
- Draw is a request to run the shader after selecting the resource. To render a realistic scene, it usually takes multiple sets of shaders and resources to draw back and forth multiple times to complete a frame. Before each drawing, we need to select the shader, associate different resources with it, and also start a graphics rendering pipeline.
- Command Indicates the configuration before drawing. WebGL is very stateful. The state machine must be handled carefully before each drawing. These state changes are implemented through commands. Beam greatly simplifies manual command management based on conventions, but you can also customize your own commands.
How do these concepts work together? See the picture below:
Buffers/Textures/imaginative are typical resources. There may be multiple draws in a frame, and each draw requires a shader and corresponding resources. Between draws, we manage WebGL state through commands. This is the thinking model I established for WebGL when DESIGNING Beam.
It’s important to understand this mental model. Beam’s API design is based entirely on this model. Let’s take a closer look at an actual scenario:
In the picture we draw many spheres with different textures. The rendering of this frame can be deconstructed into these concepts as follows:
- The shader is undoubtedly the rendering algorithm of the sphere texture. For classic 3D games, it is often necessary to switch shaders to render objects with different textures. But now that physics-based rendering algorithms are popular, it’s not hard to render these spheres using the same shader.
- Resources include large sections of sphere vertex data, texture image data, lighting parameters, transformation matrix and other configuration items.
- The drawing is done in stages. We chose to draw one sphere at a time, and each draw also starts a graphics rendering pipeline.
- Commands are those state changes performed between adjacent spheres drawn.
How do you understand state changes? Think of WebGL as a device with lots of switches and interfaces. Each time you press the start button (to perform drawing). You have to configure a bunch of switches, connect a line to the color machine, and connect a bunch of lines to the resource, like this:
It is also important to note that although we already know that a frame can be generated by multiple draws, each draw corresponds to an execution of the graphics rendering pipeline. But what is a graphical rendering pipeline? This corresponds to this picture:
The rendering pipeline generally refers to the process from vertex data to pixels on such a GPU. With modern programmable Gpus, certain stages in the pipeline are programmable. In the WebGL standard, this corresponds to the blue vertex shader and slice shader stages in the diagram. You can think of them as two functions that you have to write. In general, they do the following:
- The vertex shader takes in the original vertex coordinates and outputs the transformed coordinates according to your needs.
- The chip shader takes in a pixel position and outputs the pixel color calculated according to your needs.
These are the basic WebGL concepts I see from the perspective of the base library designer.
API Basic Design
Although the previous chapters didn’t cover code at all, once the concepts are clear enough, coding comes naturally. Because commands can be automated, I only defined three core apis when designing Beam, namely
- beam.shader
- beam.resource
- beam.draw
They correspond to managing shaders, resources, and paints, respectively. Let’s see how we can draw the Hello World triangle in WebGL based on this design:
Examples of Beam code are as follows:
import { Beam, ResourceTypes } from 'beam-gl'
import { MyShader } from './my-shader.js'
const { VertexBuffers, IndexBuffer } = ResourceTypes
const canvas = document.querySelector('canvas')
const beam = new Beam(canvas)
const shader = beam.shader(MyShader)
const vertexBuffers = beam.resource(VertexBuffers, {
position: [
- 1.- 1.0.// vertex 0, bottom left
0.1.0.// vertex 1, top middle
1.- 1.0 // vertex 2, bottom right].color: [
1.0.0.// vertex 0, red
0.1.0.// vertex 1, green
0.0.1 // vertex 2, blue]})const indexBuffer = beam.resource(IndexBuffer, {
array: [0.1.2]
})
beam
.clear()
.draw(shader, vertexBuffers, indexBuffer)
Copy the codeHere are some important API fragments, one by one. Beam is initialized with Canvas:
const canvas = document.querySelector('canvas')
const beam = new Beam(canvas)
Copy the codeWe then instantiate the shader with beam.shader, MyShader, which we’ll talk about later:
const shader = beam.shader(MyShader)
Copy the codeOnce the shader is ready, it’s time to prepare the resources. To do this we need to use the beam.resource API to create triangle data. These data are stored in different buffers, and Beam uses the VertexBuffers type to express them. The triangle has three vertices, and each vertex has two attributes, position and color, each corresponding to an independent Buffer. This makes it easy to declare the vertex data using a normal JS array (or TypedArray). Beam uploads them to the GPU for you:
Note the distinction between vertex and coordinate concepts in WebGL. A vertex can contain not only coordinate attributes of a point, but also normal vectors, colors, and other attributes. These attributes can be entered into the vertex shader for calculation.
const vertexBuffers = beam.resource(VertexBuffers, {
position: [
- 1.- 1.0.// vertex 0, bottom left
0.1.0.// vertex 1, top middle
1.- 1.0 // vertex 2, bottom right].color: [
1.0.0.// vertex 0, red
0.1.0.// vertex 1, green
0.0.1 // vertex 2, blue]})Copy the codeBuffers that hold vertices usually use very compact data sets. We can define a subset or superset of this data for actual rendering to reduce data redundancy and reuse more vertices. To do this we need to introduce the concept of IndexBuffer from WebGL, which specifies the vertex subscripts to use when rendering:
In this case, each subscript corresponds to three positions in the vertex array.
const indexBuffer = beam.resource(IndexBuffer, {
array: [0.1.2]})Copy the codeFinally, we are ready for rendering. Clear the current frame with beam.clear, then pass in a shader object and any number of resource objects for beam.draw:
beam
.clear()
.draw(shader, vertexBuffers, indexBuffer)
Copy the codeOur beam.draw API is very flexible. If you have more than one shader and more than one resource, feel free to combine them to chain the rendering and render complex scenes. Something like this:
beam .draw(shaderX, ... resourcesA) .draw(shaderY, ... resourcesB) .draw(shaderZ, ... resourcesC)Copy the codeDon’t forget to leave out one thing: how to determine the rendering algorithm for triangles? This is specified in the MyShader variable. It’s actually a shader’s Schema, like this:
import { SchemaTypes } from 'beam-gl'
const vertexShader = ` attribute vec4 position; attribute vec4 color; varying highp vec4 vColor; void main() { vColor = color; gl_Position = position; } `
const fragmentShader = ` varying highp vec4 vColor; void main() { gl_FragColor = vColor; } `
const { vec4 } = SchemaTypes
export const MyShader = {
vs: vertexShader,
fs: fragmentShader,
buffers: {
position: { type: vec4, n: 3 },
color: { type: vec4, n: 3}}}Copy the codeThe shader Schema in Beam consists of vertex shader strings, slice shader strings, and other Schema fields. Very roughly speaking, a shader executes once per vertex, while a fragment shader executes once per pixel. These shaders are written in the GLSL language of the WebGL standard. In WebGL, the vertex shader outputs gl_Position as a coordinate position, while the fragment shader outputs gl_FragColor as a pixel color. There is also a variable named vColor, which is passed from the vertex shader to the chip shader and interpolates automatically. Finally, the two attribute variables position and color correspond to the key in vertexBuffers. This is Beam’s convention for automating commands.
Advanced use of API
I’m sure many of you have some doubts about the usability of this design, because even if you can render triangles according to these rules, it doesn’t necessarily prove that it is suitable for more complex applications. Beam has already been used in various scenarios inside of us, so here are some further examples. Detailed descriptions of these examples can be found in my Beam documentation.
Render 3D objects
The triangle we just rendered is just a 2D shape. How do you render cubes, spheres, and more complex 3D models? It’s not that hard, just a few more vertices and shaders. Take rendering this 3D sphere with Beam as an example:
3D graphics are also made up of triangles, and triangles are still made up of vertices. Previously, our vertices contained position and color properties. For 3D spheres, we need to use the position and normal attributes. This normal is the normal vector, which contains the surface orientation of the sphere at the vertex position, which is very important for lighting calculation.
Not only that, but to convert vertices from 3D space to 2D space, we need a “camera” of matrices. We need to apply these transformation matrices to each vertex passed to the vertex shader. These matrices are globally unique to shaders running in parallel. That’s an imaginative concept in WebGL. Imaginative is also a resource type within Beam, containing different global configurations in the shader, such as camera position, line color, effect strength, etc.
So to render the simplest ball, we can reuse the slice shader from the previous example by updating the vertex shader to look like this:
attribute vec4 position;
attribute vec4 normal;
// Transform matrix
uniform mat4 modelMat;
uniform mat4 viewMat;
uniform mat4 projectionMat;
varying highp vec4 vColor;
void main() {
gl_Position = projectionMat * viewMat * modelMat * position;
vColor = normal; // Visualize the normal vector
}
Copy the codeBecause we had already added uniform variables to the shader, the Schema needed to add an IMAGINATIVE field accordingly:
const identityMat = [1.0.0.0.0.1.0.0.0.0.1.0.0.0.0.1]
const { vec4, mat4 } = SchemaTypes
export const MyShader = {
vs: vertexShader,
fs: fragmentShader,
buffers: {
position: { type: vec4, n: 3 },
normal: { type: vec4, n: 3}},uniforms: {
// The default field is handy for reducing boilerplate
modelMat: { type: mat4, default: identityMat },
viewMat: { type: mat4 },
projectionMat: { type: mat4 }
}
}
Copy the codeThen we can continue with Beam’s concise API:
const beam = new Beam(canvas)
const shader = beam.shader(NormalColor)
const cameraMats = createCamera({ eye: [0.10.10]})const ball = createBall()
beam.clear().draw(
shader,
beam.resource(VertexBuffers, ball.data),
beam.resource(IndexBuffer, ball.index),
beam.resource(Uniforms, cameraMats)
)
Copy the codeThe code for this example can be found in the Basic Ball.
Beam is a WebGL library that is not designed with 3D in mind, so geometric objects, transformation matrices, cameras, and other concepts are not part of it. The Beam example includes some Utils code for ease of use, but don’t expect too much of it.
Add animation
How do I move objects around in WebGL? You can of course calculate the new position after exercise and update the Buffer, but this can be slow. Another way is to update the transformation matrix mentioned above directly. These were tall, stylish, easy-to-update sources.
Using the requestAnimationFrame API, we can easily make the above sphere move:
const beam = new Beam(canvas)
const shader = beam.shader(NormalColor)
const ball = createBall()
const buffers = [
beam.resource(VertexBuffers, ball.data),
beam.resource(IndexBuffer, ball.index)
]
let i = 0; let d = 10
const cameraMats = createCamera({ eye: [0, d, d] })
const camera = beam.resource(Uniforms, cameraMats)
const tick = (a)= > {
i += 0.02
d = 10 + Math.sin(i) * 5
const { viewMat } = createCamera({ eye: [0, d, d] })
// Update uniform resources
camera.set('viewMat', viewMat) beam.clear().draw(shader, ... buffers, camera) requestAnimationFrame(tick) } tick()// Start Render Loop
Copy the codeThe camera variable here is an imaginative resource instance of Beam, and its data is stored in key-value format. You are free to add or higher different Uniform keys. When beam.draw is triggered, only uniform data that matches the shader is uploaded to the GPU.
The code for this example can be found in Zooming Ball.
Buffer resources can also be updated using a similar set() method, although this may be slower for heavier loads in WebGL.
Rendering image
We have seen three resource types of VertexBuffers/IndexBuffer/imaginative. If we want to render images, then we need the last key resource type, Textures. The simplest example of this is a 3D box with textures like this:
For graphs that require a texture, we need an additional texCoord attribute in addition to position and normal to align the image to the appropriate position in the graph. This value will also be interpolated and passed into the slice shader. Take a look at the vertex shader:
attribute vec4 position;
attribute vec4 normal;
attribute vec2 texCoord;
uniform mat4 modelMat;
uniform mat4 viewMat;
uniform mat4 projectionMat;
varying highp vec2 vTexCoord;
void main() {
vTexCoord = texCoord;
gl_Position = projectionMat * viewMat * modelMat * position;
}
Copy the codeAnd new slice shaders:
uniform sampler2D img;
uniform highp float strength;
varying highp vec2 vTexCoord;
void main() {
gl_FragColor = texture2D(img, vTexCoord);
}
Copy the codeNow we need to add the Textures field for Schema:
const { vec4, vec2, mat4, tex2D } = SchemaTypes
export const MyShader = {
vs: vertexShader,
fs: fragmentShader,
buffers: {
position: { type: vec4, n: 3 },
texCoord: { type: vec2 }
},
uniforms: {
modelMat: { type: mat4, default: identityMat },
viewMat: { type: mat4 },
projectionMat: { type: mat4 }
},
textures: {
img: { type: tex2D }
}
}
Copy the codeFinally, the render logic:
const beam = new Beam(canvas)
const shader = beam.shader(MyShader)
const cameraMats = createCamera({ eye: [10.10.10]})const box = createBox()
loadImage('prague.jpg').then(image= > {
const imageState = { image, flip: true }
beam.clear().draw(
shader,
beam.resource(VertexBuffers, box.data),
beam.resource(IndexBuffer, box.index),
beam.resource(Uniforms, cameraMats),
// The 'img' key is used to match the shader
beam.resource(Textures, { img: imageState })
)
})
Copy the codeThis is the basic texture used in Beam. Since we can directly control the image shader, it is easy to add image processing effects on top of this.
The code for this example can be found in the Image Box.
Why don’t you change createBox to createBall?
Render multiple objects
How do you render multiple objects? Let’s take a look at the flexibility of the Beam.draw API:
To render multiple spheres and cubes, we only need two sets of VertexBuffers and IndexBuffers, one for balls and the other for cubes:
const shader = beam.shader(MyShader)
const ball = createBall()
const box = createBox()
const ballBuffers = [
beam.resource(VertexBuffers, ball.data),
beam.resource(IndexBuffer, ball.index)
]
const boxBuffers = [
beam.resource(VertexBuffers, box.data),
beam.resource(IndexBuffer, box.index)
]
Copy the codeThen in the for loop, we can easily draw them in different uniform configurations. By updating modelMat before beam.draw, we can update the object’s position in the world coordinate system to make it appear in different locations on the screen:
const cameraMats = createCamera(
{ eye: [0.50.50].center: [10.10.0]})const camera = beam.resource(Uniforms, cameraMats)
const baseMat = mat4.create()
const render = (a)= > {
beam.clear()
for (let i = 1; i < 10; i++) {
for (let j = 1; j < 10; j++) {
const modelMat = mat4.translate(
[], baseMat, [i * 2, j * 2.0]
)
camera.set('modelMat', modelMat)
const resources = (i + j) % 2? ballBuffers : boxBuffers beam.draw(shader, ... resources, camera) } } } render()Copy the codeHere the render function starts with beam.clear, followed by the complex beam.draw rendering logic.
The code for this example can be found in Multi Graphics.
Off-screen rendering
The Framebuffer object can be used in WebGL for off-screen rendering to render the output onto a texture. Beam currently has a corresponding OffscreenTarget resource type, but note that this type cannot be thrown into beam.draw.
For example, the default render logic would look like this:
beam .clear() .draw(shaderX, ... resourcesA) .draw(shaderY, ... resourcesB) .draw(shaderZ, ... resourcesC)Copy the codeWith the optional offscreen2D method, this rendering logic can be easily nested in the function scope as follows:
beam.clear() beam.offscreen2D(offscreenTarget, () => { beam .draw(shaderX, ... resourcesA) .draw(shaderY, ... resourcesB) .draw(shaderZ, ... resourcesC) })Copy the codeThis redirects the output to the off-screen texture.
The code for this example can be found in the Basic Mesh.
Other rendering techniques
For real-time rendering, physics-based rendering (PBR) for standardizing texture and Shadow Mapping for rendering shadows are the two main advanced rendering techniques. I also implemented examples of both in Beam, such as the PBR material sphere shown above:
These examples omit some of the trivia and focus more on the readability of the code. Check it out here:
- The Material Ball shows a rendering of the base PBR Material Ball.
- Basic Shadow shows an example of Shadow Mapping.
As mentioned above, Beam’s PBR capability is also used to implement the 3D text feature of the current draft Web version. 3D text like this:
Or this:
They are all rendered using Beam. Of course, Beam is only responsible for rendering that is directly related to WebGL, and on top of that is the 3D text renderer that we used to embed in the plane editor after customization, as well as algorithms related to text geometry transformation. This code covers some of our patents and will not be open-source with Beam. In fact, Beam makes it easy to implement your own custom dedicated renderers to optimize for specific scenarios. This is what I expect from such a WebGL base library.
Examples of Beam based implementations are also shown in the examples that come with Beam:
- Object Mesh loading
- The texture configuration
- Classical illumination algorithm
- Tandem image filters
- Deep texture visualization
- Basic particle effects
- WebGL extension configuration
- Upper Renderer encapsulation
Beam is already open source, and you are welcome to provide new examples
Acknowledgments and summary
Beam implementation process, and AT industrial gathering in API design level of communication gave the author a lot of inspiration. The guidance of many senior colleagues inside and outside the company is also very helpful when the author is facing key decisions. Finally, the implementation of this program is inseparable from the support of the front end students in the group, the most important, everyone’s large amount of detail work.
In fact, I had no more complex WebGL experience than drawing a bunch of cubes before taking on the need for 3D text. But by learning from the basics, in just a few months, you can get familiar with WebGL while meeting your product needs and precipitate the wheels. So there’s no need to limit yourself to a comfort zone by saying, “This is beyond me.” There’s so much more we can do as engineers!
As for the necessity of Beam’s existence, at least in China, I really haven’t found any open source product that is more in line with the ideal design of the WebGL base library. This is not to say that the domestic technical strength is not good, like Shen Yi Da Shen’s ClayGL and Xie Guanglei’s G3D are very good. The difference is that they address a higher level of problems than Beam and are more relevant to the average developer. Comparing them to the Beam is like comparing the Vue to the simplified React Reconciler.
The more I did it, the more I realized it was a fairly niche field. This means that it may be difficult for such technical products to gain the mainstream community’s attempt and recognition.
But if there’s a last resort, someone has to do it.
I’m primarily a front-end developer. If you are interested in Web structured data editing, WebGL rendering, Hybrid application development, or computer hobbyist trivia, please follow me or my official account color-album