Case background

Recently in the development of map rendering based on WebGL API, implement a custom grid layer (such as map segmentation for big square, and the images stitching rendering), in order to save the texture upload overhead, tiles concentrated rendering the grid to a texture, then drawing according to tile the texture coordinates from their texture, probably schematic diagram is as follows:

Tiles are drawn on the large texture in the order in which they are loaded, with the same space width and vertical arrangement. For example, if the size of the tile is 256px, the position of tile 1 is {x:0, y:0}, and the position of tile 2 is {x:0, y:256}.

Then there are a series of problems: 1. Tile confusion: the position of tile 1 shows the contents of tile 4; 2. Invert tile contents.

Problem analysis

According to the debugging positioning, the root cause of the problem was found to be the Y-axis reversal.

Question 1: What is the y-flip? Why flip it?

The basic vertex model we use to draw the tile is a square centered at the origin. For each vertex coordinate, we need to map to a texture coordinate (left), pass it to the pixel shader, and use texture2D() to take the texture pixel. In this case, the texture coordinates for the top left vertex (-1,1) are (0,0).

The texture coordinate system has a different Y-axis orientation from the vertex coordinate system, which makes it inconvenient to map coordinates, so it can be easier to map coordinates by flipping the Y-axis of the texture coordinate system (above right).

WebGL also provide the corresponding interface to implement this function, WebGLRenderingContext. PixelStorei () is used to describe the pixels in WebGL storage mode function, including UNPACK_FLIP_Y_WEBGL turn whether can be used to set the Y:

Gl.pixelstorei (gl.unpack_flip_y_webgl, 1);Copy the code

Question 2: Why does the Y-axis reversal cause tile dislocation?

As mentioned above, first create a large texture using texImage2D and then draw tiles onto the large texture using texSubImage2D:

// x, y represents the offset gl.texsubimage2d (gl.texture_2d, 0, x, y, gl.rgba, gl.unsigned_byte, image);Copy the code

This interface is used to change the texture, the data of the stator area, by analogy in CanvasRenderingContext2D drawImage (), we are in the top left corner when using drawImage usually offset for the origin, so the imagined texture is shown in the figure below, The upper left corner of tile 1 corresponds to texture coordinates (0, 1), the lower left corner to (0, 0.75), and so on.

When texImage2D is used to create a large texture, the pixel storage mode of the image is already determined. When texSubImage2D is executed, the pixel storage position of the image is reversed as follows:

So the big texture should actually look like this:

So when we use the texture coordinates top left (0, 1)+ bottom left (0, 0.75), we get the texture of tile 4, resulting in tile dislocation.

Question 3: Why are the tiles upside down?

After obtaining the texture coordinates correctly, a new problem occurs:

The tiles appear upside down on the screen, and this only happens in Chrome/Firefox because in both browsers createImageBitmap is used to convert bloB images into bitmaps. In Safari (createImageBitmap is not supported) we changed the BLOB format to an Image object and ended up with this difference, so we started with ImageBitmap to locate the cause of the problem.

ImageBitmap represents a bitmap Image and is used to draw an Image in the canvas. Compared to Image, it has a lower latency because it is also converted to ImageBitmap when texSubImage2D is executed to draw an Image to the texture:

Whether it is drawing 2d images in canvas or creating textures in WebGL, the browser will decode the images when using them. This decoding is to unify the original format of the image (such as JPEG, PNG, etc.) into a bitmap, that is, each pixel is described using RGB or RGBA. When the image size is relatively large, decoding will also have a certain cost, and this time is synchronous. – the high-performance WebGL – use of ImageBitmap texture performance (www.jiazhengblog.com/blog/2019/0…

At the same time, the WebGL specification has some special description for ImageBitmap, when introducing pixelStorei’s three parameters: UNPACK_FLIP_Y_WEBGL, UNPACK_PREMULTIPLY_ALPHA_WEBGL, and UNPACK_COLORSPACE_CONVERSION_WEBGL clearly indicate that ImageBitmap is invalid. This can only be set when creating an ImageBitmap:

If the TexImageSource is an ImageBitmap, then these three parameters will be ignored. Instead the equivalent ImageBitmapOptions should be used to create an ImageBitmap with the desired format.

So it’s a safe guess that the pixel storage mode specified by pixelStorei is actually a pre-processing process that decodes images into bitmaps. There is no preprocessing when we draw the bitmap directly to the texture, so the UNPACK_FLIP_Y_WEBGL parameter is invalid.

summary

  1. UNPACK_FLIP_Y_WEBGLThis parameter is used to set whether to flip the Y-axis in texture pixel storage mode, depending on the coordinate system orientation of your vertex model, as appropriate. In our application scenario, the vertex model and the image coordinate system are opposite, so we need to set this parameter to 1.
  2. usetexSubImage2DThe same applies when uploading imagesUNPACK_FLIP_Y_WEBGLInfluence of parameters.
  3. If the uploaded image isImageBitmapObject is passed when it is createdImageBitmapOptionsIn theimageOrientation,premultiplyAlpha,colorSpaceConversionThree parameters make it equal topixelStoreiThe parameters set in the.

Finally, using a custom raster layer to overlay the hand-drawn image onto the map, the result is as follows: