Tencent worker bee Git

code.tencent.com

Quote:

Recently, the 2018 DevOps China Salon was held at Tencent Tower in Shenzhen. The salon invited a number of guests to share their DevOps practices and insights. At the meeting, Mr. Luo Qi, senior engineer of Tencent and head of worker Bee system architecture, shared his experience of “Revealing the Core of Tencent code Management: Worker Bee Git System Architecture”, and expounded the origin, development and future planning of Tencent Worker Bee.

1 Tencent worker bee

Tencent Worker Bee is the enterprise code management collaboration solution created by Tencent after 10 years of accumulation and exploration. With code review, branch management, conversational development, integrated customization, review and monitoring enterprise R & D management system features, adhering to the forefront of R & D thought and advanced R & D concept, help enterprises throughout the R & D process, so that the development and R & D management more agile and efficient.

Structure of worker bee system

Since 2012, Git has gradually matured. Because Git is decentralized, fast pull branch and easy to use, it is favored by many developers, and the use of Git is gradually popular in China. At this time, Tencent is also preparing to build a Git code hosting system, and plans to gradually switch the original SVN to Git.

We adopted our own approach because it fits the growing code hosting needs of Tencent and fits well with the company’s corporate management, security and internal open source culture. In terms of architecture, facing the size of Tencent, we need the ability of distributed cluster and highly available solutions, and we also need to strengthen the backstage management, monitoring, operation and maintenance capabilities. Moreover, self-research has a good controllability, can do a lot of enterprise customized functions.

In the choice of system architecture, we chose microservices in order to cope with Tencent’s total code base of hundreds of tons and the constantly generated platform access requirements, such as automatic construction, automatic code scanning, etc. Combined with container deployment solutions, microservices have features such as rapid horizontal expansion, pluggability, and incremental distribution, which can well meet high availability and high concurrency scenarios.

Routing is at the heart of all services, and all requests for code warehouse operations are addressed through routing, which ultimately makes business calls transparent and less intrusive in a faceted manner.

Server, the code base on the right, uses the geo-redundant disaster recovery solution.

3 What capabilities the architecture provides

Horizontal scaling, which is more convenient because services are deployed in containers, coupled with stateless underlying services. Even if a large number of services are suddenly connected, the system can be easily expanded by adjusting the number of instances at any time to respond quickly. Independence between services. If one link is abnormal, the other links still work properly. For example, the network where the database is located is affected, and the Web service is temporarily abnormal. However, because the HTTP proxy has cached routing data, the most basic client operations such as Pull/Push can still run normally, and the basic code operations of developers are not affected. Provide circuit breaker and lossy service to prevent avalanche effect caused by high instantaneous load. High availability of services. We provide high availability and disaster recovery (Dr) backup capabilities by using the two-site, three-center, active-write and active-active modes, cross-room and periodic incremental backup.

4. Problems encountered in actual operation

With the increase of the user quantity of worker bee system, new problems emerge gradually in the operation of the system, such as slow access of remote projects, increasing number of large library projects and timeout of Merge operation of large library. To solve these immediate problems as quickly as possible, we decided to perform a new version of the evolution of the worker bee system.

Evolution of worker bee system architecture

Below is the architecture flow chart of the new version, with the new features added to the system in yellow. We introduce IDC selector to solve the problem of remote deployment and LFS to solve the problem of large library, so as to achieve fast access to remote project and fast response to large library operation respectively.

  • In the actual operation of LFS, a large number of users reported the problem of slow loading. Through scanning, it was found that the projects of such users contained a large number of dependent package files, which were also associated with a large number of history. In addition, some game business units have projects with more than 50 gigabytes of data, including a large number of images and videos.

The following is an architectural flowchart for an LFS deployment. We deployed an LFS agent and a dedicated LFS server. When users perform LFS operations, Git intelligent cooperation will push text Pointers of files such as codes, pictures and videos to the code repository through HTTP proxy, and then push file resources to the LFS proxy. The LFS proxy will first search for routes, conduct permission authentication, and finally push file resources to the LFS server. In addition, in order to give the user unified control in the background, we also added background configuration in the HTTP proxy to intercept a single file, a single commit, and then inform the user to use LFS for conversion.

  • Remote collaborative development We found that the slow access to remote projects occurred in scenarios where the same department employees were distributed in Shenzhen and Beijing, and they all needed to use the worker bee system. The Beijing team has independent projects, while shenzhen and Beijing staff have public projects. When Clone and Push operations for large libraries occur in the project, the system response will be very slow. As the same department, with public projects, it is impossible to deploy two completely different environments. How do you solve this problem?

Assume that IDC1 is Shenzhen and IDC2 is Beijing. We can solve these problems simply by adding an IDC coordinator to a complete system in Shenzhen, and then deploying the most basic HTTP proxy, routing, IDC selector, and a new Server cluster for reading and writing in Beijing. When the colleague in Beijing uses the client, the access request will be allocated to the corresponding place by IDC selector, so as to realize the nearest access. In terms of collaborative development, the Web and database in Beijing and Shenzhen adopt the same system, so the data are visible to each other, and colleagues can carry out collaborative development through Fork public projects.

The ability to improve through evolution

In the previous version, we have sharded the storage data and micro-services, so that the overall storage capacity can be freely expanded horizontally, and finally realize the overall storage capacity without upper limit. With LFS, individual libraries can now be theoretically unlimited. Now it can not only easily cope with the requirements of overall project migration in the game department, but also achieve unified configuration and management of rules for intercepting large files in the background.

Through IDC selector, the worker bee system can be visited and co-developed nearby. On the basis of ensuring data unification and easy maintenance of the system, it provides better use experience for colleagues in the branch.

Computation and storage separate MySQL sharding support, so that data capacity and API calls can also be sharded operation, thus providing massive data support.

Provides Open APIS and OAuth functionality for better integration with third-party applications

7 expectations for the future

As a code hosting system, worker bee system is an important part of DevOps development tool chain. The worker bee system now connects to other systems through Web Hooks, apis, and OAuth, connecting requirements, code hosting, and construction. In the future, we will focus more on the needs of the enterprise, opening up more capabilities of the worker bee system and integrating services on DevOps more closely, ultimately forming a closed loop of the R&D tool chain.