QCon will be held in Shanghai Marriott Hotel from October 17 to 19, 2017. Technical topics such as “Fast Changing Internet Architecture” were set up in this conference to give an in-depth interpretation of the various innovations and attempts of ultra-fast growing companies in the Internet circle in business architecture. Among them, Mr. Fu Qiang, the backstage development Leader of Tencent Qzone, was invited to share the architectural Design and Practice of Hundreds of millions of pictures Service Platform — Qzone Massive Photo Service Platform.

We took this opportunity to interview Teacher Fu Qiang, who brought us his thoughts and experience on the design of QQ Space mass system, hoping to inspire you. If readers want to know more about the design process and summary of qZONE image platform architecture, welcome to participate in QCon Shanghai station and further exchange with Teacher Fu Qiang.

Fu Qiang, Tencent Qzone backstage development Leader.

Responsible for the architecture design and development of the largest UGC image platform in China. With years of experience in mass system design and optimization in mobile environment, I have a profound understanding of distributed database architecture, high-availability and high-concurrency system design, and the total number of platform photos is nearly 1 trillion at present.

Responsible for several technical optimization projects, saving more than 100 million yuan per year. He is currently committed to promoting machine learning in the field of images.

First of all, we will introduce our qzone background development team. There are 5 teams related to the whole qzone album background link:

  • Photo album background team responsible for photo album business development and overall optimization;

  • Storage development team responsible for image processing;

  • The data development team responsible for the development of the underlying structured database;

  • A group responsible for cold data backup research;

  • O&m team responsible for business o&M and machine management

In addition, there are two groups for IOS, AND front-end H5 AND WEB development, which are responsible for the promotion of specific projects.

At present, our team is mainly responsible for the structure optimization of album, the performance improvement and cost optimization of the whole system, the development of product features, the mining of historical data of album, and the exploration and application of machine learning in related scenes of album photos.

There are two main ways of division of labor and cooperation between different technical teams:

  • One is cooperation on specific projects. This is generally a specific project promotion with a relatively clear goal, and each team will participate in it together. Specific project leaders and project managers will divide the functions and manage the project schedule.

  • The other is based on the functional division of each group to do some long-term exploration and technical pre-research. For example, in-depth pre-research on storage, research and promotion of new image formats, low-cost data backup technology, unified operation management platform, etc. Many technical achievements are simultaneously reused and promoted to internal products such as wechat, Wechat cloud, mailbox and Tencent cloud.

In Tencent, as long as you have a clear goal and enough challenges, the promotion of cooperation in this aspect is usually more smooth. For example, the TPG image flow optimization project we cooperated with audiovisual in the first half of the year involved multiple teams from four departments. Overall mobile traffic optimization dropped by 60%. Save more than 90% traffic for GIF images.

Qzone has been an “old” product for 12 years. After 12 years of iteration and evolution, it can be said that Qzone is one of the most complex products at present. There was a time when we were struggling with development efficiency and maintenance costs because of the breadth of the business, the number of entrances and scenarios, and the heavy historical burden. Our current approach to this problem is as follows:

First, cut into and optimize more from the product level, including the clearing and offline of the historical logic (business).

At the same time, for the new business logic, the decoupling of logic and the built-in offline logic should be fully considered at the beginning of the design. It turns out that optimizations based on product-level business logic have a direct contribution to clearing up historical baggage, which is often overlooked by developers.

Second, the code architecture design mainly considers the extensibility and function reuse.

This part of our approach is to do more of the underlying general standardized component framework, such as our existing CGI components, Server development components, mobile terminal access components and so on, which can be standardized with a lot of business logic processing, saving a lot of development and maintenance costs.

Even so, with the constant change of business, our code still has great redundancy and needs to be optimized. The contradiction here is that business reconstruction and optimization are endless, and for business development, resources and time are very short.

The lesson here is that combining code refactoring with requirements development, adding some refactoring to our business development process, or combining architectural with cost optimization, can better facilitate refactoring.

At the user level, as businesses get bigger, the problems they face get exponentially bigger.

The relatively small size for the business system, one over ten thousand probability of system problems can need not too concerned about, but when the business is worth trillions of level, even one over one million, one over ten million probability problems may also be because the system scale extreme amplifier, because will affect quite a base into had to optimization problem by the user.

At the same time, considering from the business level, the business cost and operating cost will become a constant consideration when your size increases.

The first architectural optimization of our business was upgrading from MySQL cluster to TFS (Tencent File System), which supports user data scale up to EB level. At that time, it was basically equivalent to equal expansion of unlimited capacity of the whole System. This stage solved the capacity problem at one fell swoop and laid the foundation for the explosion of business volume.

The second phase of our optimization mainly focuses on solving user access problems, including ultra-fast upload, cross-operator access and download acceleration. After optimization, it brings a great improvement in user experience and a large increase in data volume.

With the emergence of new storage media and users’ requirements for photo capacity and performance, in the third stage, our optimization focus is mainly on the redesign of TFS, which mainly supports unlimited capacity of single user, and adopts TFS based on SSD media (TSSD) and high-speed index upgrade version. At the same time, special processing of cold data to reduce costs.

The fourth stage of architecture optimization is mainly for mobile terminal and weak network optimization. At present, our optimization direction mainly focuses on high performance, low cost and stronger data processing capacity.

In terms of the storage capacity of the existing system, our actual capacity currently reaches 0.4EB. From the perspective of capacity alone, the current architecture can fully meet the development in the next ten years.

In the whole architecture iteration process, we still maintain high performance. On the one hand, the main design idea of our architecture design is simplicity, including the simplification of business logic and processing process. Another point is that we are currently using structured storage, which can be targeted to optimize the system, which also contributes to high performance.

Finally, we will consider whole-end optimization of the whole system, which can be optimized from the client to the storage to ensure high performance. At present, the main problem of our storage architecture is that the flexibility of structured storage is relatively poor. Here we are also drawing on some ideas from the industry.

In fact, Tencent has very rich internal components and open source software.

In terms of storage, TSSD is a massive storage system based on KEY-value SSD, CKV is a hybrid storage system based on memory and SSD, TFS is a massive file system based on directory management, and BTFS is a cold storage system.

It can be said that the open source components used in mainstream Internet companies can be found in Tencent’s basic target products and solutions. At present, we use a relatively large proportion of self-developed components, the main reason is that these components have a special internal maintenance team, and basically through the test and optimization of large-scale online business, reliability and practicality are relatively guaranteed.

At the same time, many open source components are used in our project, such as Redis, Kafka, Ngnix, raft and so on. At present, we tend to adopt open source framework for generally independent modules and small and medium-sized systems, with good flexibility. At present the idea of Tencent itself is to open source the internal good components as far as possible.

Machine learning is currently applied in three directions: picture (computer vision), voice, text. We have amassed an enormous amount of valuable image data. At the same time, the Youtu laboratory developed from our department has accumulated deep technical reserves in machine learning. Finally, based on our product form, the results of machine learning can be well combined with business scenarios to maximize the product value.

Therefore, in general, we have three factors of “timing”, “geography” and “people and people”, and it is quite natural to carry out a project combining AI with pictures. In the first half of this year, we mainly worked on two projects combining pictures:

The first one is to use machine learning to identify the content of pictures and the identity of users. We mainly use the idle time of online machines to run the trained model for image processing. At present, it should be the largest image processing system in the industry, processing tens of billions of pictures in peak hours every day.

The other is to use the GPU and CPU capabilities of the client to apply super-resolution images. This project has also been launched, which can fully exploit the processing capabilities of the client and save 75% of the user’s bandwidth while ensuring clarity. After extreme optimization, it can run the trained model on more than 80% of mobile phones.

Both programs have had good results. As the performance of mobile terminals is getting stronger and stronger, including GPU gradually becoming the standard of mobile phones, and combined with the continuous accumulation of data and breakthroughs in algorithm research, AI-related applications should be a general direction in the future, and we also hope to make more efforts in this area.

With storage today, data size and performance are no longer the biggest challenges. Both should be ahead of the business by now. At present, the challenges are mainly focused on the data processing capabilities of storage systems, including intelligent data processing and the price requirements of large-scale storage. Currently we are focusing on NewSQL and JBOD, which have emerged in recent years and are expected to be introduced into our system.

As a background development Leader, the biggest feeling in these years should be not to limit myself to the background field. In fact, front-office knowledge, business-related knowledge, and general knowledge including process management can help you grow well.

At present, there are many distributed database architectures, and new architectures are constantly emerging. If you want to learn this knowledge, it is recommended to combine your own projects or tools to introduce an open source framework for in-depth study, and it is best to be driven by problems encountered in the application process. As you dig deeper, you will find that many of the design ideas of highly concurrent systems are the same.

QCon: Thank you, Mr. Fu Qiang, for accepting our interview. We look forward to your sharing of the Architecture Design and Practice of Hundreds of millions of Photos Service Platform — QZone Massive Photo Service Platform at QCon Shanghai.

If you are interested in Mr. Fu qiang’s speech, please click “Read the original article” below to learn more.

Today’s recommendation,

Click on the image below to read it

Why don’t tech companies have Ctos?