QZone has been with us for nearly 12 years. Among them, THE QQ album, along with everyone, recorded the growth of many Chinese people.

Up to now, QQ album has stored more than 2 trillion pictures, with a physical storage of 300PB. An average of 300 million images are added every day, and users view them 50 billion times a day. Compared with the industry, it can be said that it is the largest album in China and one of the largest picture products in the world.

With the development of the business, not only the page style that users can see is changing, but also the photo album system developed and operated by Tencent architecture Platform Department has undergone constant optimization. Let’s introduce the background structure of QQ album.

If we build a small photo album system from scratch, the architecture we usually think of is like this.

The index system stores the user’s album list and the list of pictures in each album. The file system holds the actual image files.

In fact, the earliest QQ album system, is directly using MySQL to store user index, Linux file system to store picture files. This is the simplest system, able to meet a small number of users and basic functions. In the face of the rapid growth of album users and requests, the department specifically developed the underlying TFS storage system family KV engine to support.

The index

TFS has launched a distributed K-V storage platform TSSD based on large-capacity SSD for album users, frequent reads and writes, small operation data, and smooth expansion. The TSSD ensures data reliability through multiple replicas and eliminates random write performance bottlenecks through self-developed persistent storage engines. Combined with the inherent random read advantages of SSDS, TSSDS provide excellent mixed read and write performance, with a peak read and write IOPS of 100,000 + times per unit. While supporting high concurrent read and write, the unique I/O delay burr control technology enables 99.9% read and write access delay to be less than 10ms in operation.

In addition, by analyzing user behavior, it is found that a large number of user operations just pull album and picture list, without looking at the detailed information of the picture. In this way, we separate the user index and store the key information required by the list and other detailed information separately, which greatly reduces the average size of index data blocks read and written for each operation.

The compression

QQ album supports JPG, PNG, GIF and other mainstream picture formats, and under the premise of ensuring the quality of pictures, constantly try and use new formats with higher compression ratio. For example, WebP is already fully supported, saving 30% more storage and bandwidth than JPG. Currently, we are promoting the use of sharpP, a picture format developed by the company, which reduces the volume by more than 15% compared with WebP.

In order to cope with the large concurrent compression requests, we set up thousands of machine scale compression clusters. Moreover, Docker technology is mostly used in this part of compressed cluster, which is mixed with existing business machines and significantly saves costs.

For time-delay sensitive scenarios, we also promote the use of FPGA hardware compression card, which only consumes 10% of CPU machine. This capability is also currently available through Tencent Cloud.

The distribution of

1 billion + album users are logically divided into different parks, each with a complete set of services. Each zone can be smoothly expanded, and zones can be added as needed.

Upload to the nearest

To ensure the consistency of index information, the complete index information of each user is saved only in a specific campus. The user’s park attribution information is predetermined. For example, if a user’s index information is in Shanghai and he is on a business trip in Shenzhen, if he needs to visit the server in Shanghai for each upload, the delay will be very high.

To this end, we have reformed the background to upload pictures nearby. When users upload pictures, they will access the nearest park according to the region and operator, and land the picture data nearby. The index attribution information is synchronized by our server in the background. It greatly improves the success rate of upload and reduces the response delay.

Different ground disaster

Data and indexes can be written to a backup point when the program perceives a network failure in a park. A circular backup relationship is formed between multiple parks.

Download speed

More than 30 CDN acceleration points have been set up nationwide to provide users with download services nearby. There are also acceleration points in Hong Kong and the United States to provide acceleration for overseas users.

The cache in advance to push

Based on an analysis of user behavior, recently uploaded images tend to be viewed the most. In order to provide users with better download experience, after users upload albums, we will send instructions to CDN acceleration points in advance to inform them to pull pictures for caching. When friends of subsequent users visit, the photos can be loaded quickly without returning to the source.

In the future, we will continue to make efforts to improve the quality and stability of our service, so that users can use QQ album to record their life in a secure and convenient way.

Article source: Tencent Architect