Yu Zhenhua, senior architect of The Software Development Department of The Bank of Beijing, has been engaged in the development and planning of the bank’s core system for a long time. He has participated in the construction of many core information systems, including the first and second generation payment system, the construction of the fourth generation bank core system, the construction of distributed core system and other enterprise-level projects. At present, the main research and development direction is focused on building advanced, efficient, OLTP-oriented banking transaction system, and improving the service capacity of banking information system.

This article is compiled from the transcript of Professor Yu Zhenhua’s speech at TiDB DevCon 2019, the topic of which is “RESEARCH and Practice of TiDB in the core financial field of banks”.

Today I attended TiDB DevCon 2019 to exchange TiDB practice with so many friends from all walks of life. This is a rare opportunity, because usually there is one-way communication between our technical team and TiDB team, and there are few opportunities for horizontal communication between customers. As the teachers said just now, I think they are very interesting, but also hope that through our conference, we can rub out a different spark.

Bank of Beijing has conducted in-depth cooperation with PingCAP team. At present, several sets of important real-time transaction systems have been connected, including the important Online system, UnionPay card-free payment, financial interconnection service platform, etc. Now how to evaluate whether a product is stable or not depends to a large extent on whether this product is applied in finance, especially in the core financial scene, and whether it can support the requirements of the financial scene. We started production in March, May and June of 2018. After more than half a year, we see TiDB supporting financial scenarios as well. From the side, distributed database technology, has indeed reached a certain maturity.

I. Background introduction

I believe that in the past few years, especially in the past three or four years, you should all feel that. Both the way of work and the way of life have undergone great changes. All kinds of information and scientific and technological products have emerged. Some people say that this change is called industrial scientific and technological revolution 4.0. I don’t know if this is accurate or not, but this change does pose a relatively big challenge to our banking system.

In Figure 1, I listed several requirements, such as high concurrency, that require you to scale quickly. Product release, for example, requires that you have the ability to quickly release and we should do have a lot of products, do the implementation team, everyone should have a lot of feelings, may also neglected the product the day before, for example, may be sold very hot, the next day to each project is urgent project, ask you to release in the fastest time. Of course, there are some common problems, such as the cost of traditional architecture is difficult to control, and there is an urgent need to tackle autonomous control. In fact, in the traditional closed-source ecology, it is difficult for us to achieve the requirements of autonomous control.

Second, system analysis

In this context, we from the global perspective, the bank of systemic analysis has been done for the previous technology form, shown in figure 2 lists some typical architecture form, there are some in the bank now architecture inside still exist, such as the application of monomer, such as the traditional database again, now use most DB2 and Oracle, There are traditional stand-alone or cluster deployment models, waterfall development models, and of course operations for traditional architectures.

Today we are going to talk about distributed database, which is a new technology, but we can not say that the past technology architecture is negated. Was the form of technology good in the past? Frankly, I think it’s good, it’s not good to be able to sustain the financial business for so many years, but at a point like today, there are problems. As mentioned earlier, high concurrency requirements, scalability, cost, and product delivery have their drawbacks.

In this case, we started a new round of structural transformation of Bank of Beijing, and the distributed database has also been included in our work scope. We have been in contact with PingCAP for a long time. In the process of more than a year’s work, there will be a lot of technical details, technical solutions, work procedures and so on to be discussed. If I really summarize how this work is done, I will summarize the following three aspects. It may sound silly at first, but if you actually try to do this, you might have the same feeling.

The first is pragmatic. Architectural transformation is not about technology for technology’s sake, new product for new product’s sake, but really improving the efficiency of your business development, development, operation and maintenance.

The second, and I think probably the most important, is to win fast. No matter what kind of enterprise you are in to do technology upgrade, technology transformation, more or less will encounter some resistance, especially in the traditional enterprise. To win quickly, to release value quickly, to let people around you, to let your team, to let your organization, to see its value quickly, will make your future work smoother.

The third is “full stack”. Because it is an overall architectural transformation, we want to build a platform that can unlock the value of the whole, rather than focusing on the gains and losses of one city or one pool. Today, I originally wanted to introduce the application architecture and distributed database architecture of Bank of Beijing, but due to time constraints, I will only talk about the construction of distributed database.

Third, progress

Before introducing the specific content, let’s keep pace with the progress of our current work. In March 2018, we launched the industry’s first distributed database for core financial services, using a two-site, three-center and five-copy architecture. Based on the distributed database, we put into operation in May the Online payment clearing platform, which is also a very important real-time transaction system with capital business, and put into operation the UnionPay card-free payment platform in June. This chart (FIG. 3) may be a little old, but now we also put into production the financial Interconnection service platform, IFRS9 impairment system. In fact, what we will do in the future is quite consistent with what Liu Qi said just now, including HTAP, including these solutions of container cloud and so on, which are also our most urgent needs at present.

3.1 Special Evaluation

In retrospect, bank of Beijing started the construction of distributed database very early in the industry. It was also because we started this work relatively early that we encountered many difficulties and puzzles in the whole process. The technical strength of the line concentrated on DB2, Oracle may be more, for the master of distributed database, there needs to be a cycle. The first step we took was to build an evaluation system for financial business in order to ensure the availability of the product.

In the upper left corner of Figure 4 are tests for this functionality, such as high availability of the database, linear scaling, and online upgrade capabilities, which are our test points. In the lower left corner of Figure 4, which is performance-oriented testing, we did not use tools that were already on the market, such as TPCC, Sysbench, and so on. Because our actual analysis down think the market has some of these tools and our financial scene have some distance, use them to test may not have great reference significance, so we research the distributed database oriented financial performance evaluation system, can let us clear the distributed database can be used in the financial scene, and for the function and performance, So you can have a measurable tool.

In the process, thanks to the payment and settlement association, mail tunnels courtyard and other higher level units and organizations to give us help, in addition, we also and Intel hardware vendors to compare the depth of cooperation, such as this year (2018) new hardware platform, we also do a special test of distributed database, our hardware architecture for the future selection provides effective reference.

3.2 Deployment Mode

As for the technical level of distributed database, several lecturers have introduced a lot just now, so I would like to talk about some different and leading parts of Beijing bank. As you can see in Figure 5, this architecture is the architecture of the data storage layer of Bank of Beijing. The architecture of Bank of Beijing adopts the mode of deployment of two places, three centers and five duplicates.

It is a great challenge to build a distributed database across city and long distance. For example, Beijing and Xi ‘an are about 1,000 kilometers away from each other, and the delay is relatively high. Our measured delay is about 17 milliseconds. The 17 milliseconds, if put in a SQL, come and go 30 milliseconds, such a delay we certainly cannot accept. Therefore, in this case, we used a five-copy mode: two IDCs in Beijing are placed with two copies each, and one IDC in Xi ‘an is placed with one copy, adopting the mode of 2:2:1. The advantage of this is that after the current application request comes, it does not need to use the network from Beijing to Xi ‘an. If three of the four copies in Beijing are successful, the front-end can return in real time, and some instances in Beijing are allowed to fail. Do so SQL average delay, about 1.2 milliseconds,.95 delay about 5 milliseconds, this is a relatively good result (network, unionpay business is actually much more complex than the Internet business).

Here is a small story we actually encountered in the production process. At noon on a Saturday, I received a phone call from our operation and maintenance staff, who said that a TiKV storage server was broken. The first thing I asked that day was whether the broken one affected the service. He said there was no impact on service, which was normal. I say get the hardware manufacturer to fix the machine. I felt quite happy at the time, and I accidentally verified it in the production system. At noon the next Sunday, he called me again to say that another server had crashed. At that time, I was worried about whether there was something wrong with the hardware server we purchased. When I thought of this, I immediately made an investigation. Of course, the first question I asked was whether the service was affected, but he said it was not. In this way, the failure of two storage servers for two consecutive days does not affect the service, which also proves the effectiveness of the multi-copy scheme.

3.3 Two places and three centers

Figure 6 shows the entire deployment pattern including applications, F5 through TiDB, PD, TiKV, and so on. At present, we have two relatively large systems, Namely Netpay and UnionPay. The business volume of these two systems is relatively large, with one or two million transactions per day. In Xi ‘an, we also deployed a slave cluster. What is this slave cluster for? This secondary cluster is a case of docking with some OLAP or large reports to avoid too much impact on the load of the primary cluster.

4. Application and practice

4.1 Existing problems

Someone said, “When you have a hammer, everything looks like a nail.” We look forward to moving from traditional databases to distributed databases, where everything can be solved. But the truth is, there is certainly no one-size-fits-all technology solution. In the lower right corner of Figure 7, I list some of the problems or episodes that have occurred since the beginning of our project.

For example, we just introduced the row of DB2, Oracle applications are more. DB2 and Oracle used to use READ COMMITTED isolation levels, so Repeatable READ of TiDB may need to be adapted. We also had this problem in the early stages of construction: Insert data on one side, but not on the other, because TiDB is the isolation level for such snapshots.

There is also the problem that the index of the execution plan is not selected, which is also encountered in our actual production process. There is an index, but the index is not selected precisely. Cause SQL to run very slowly, memory to eat more. I think this problem can be solved. The temporary solution is to manually force Hint. I believe TiDB will also consider this in the future version upgrade to make the execution plan more accurate.

There is also the problem of hot data, which relies on databases to solve, which is not possible at this stage. Both traditional database and distributed database, to introduce the application of other cache components that can be solved, the traditional scheme, technical scheme also has a lot of, we do like a more traditional way of hash, hash out to solve the hot data, now have a cache, you can cache is applied to deal with this matter.

Our application architecture adopts the form of microservice, which has higher requirements for database compared with the form of single application. Because the traditional application of monomer, the transaction is more, the number of SQL into micro service, whether the application logic, or database processing logic, will be more fine-grained, transaction commit frequency doubling, submitted for MVCC optimistic model has certain pressure, in the process of we measured, the more fine-grained may appear in poor performance.

These are some of the small episodes in our practice.

4.2 Differences with Internet industry in application practice

Today, many friends from Internet enterprises also share their experience. What is the difference between the financial industry and the Internet industry to do distributed database landing?

First of all, the development period of banks is different from that of many emerging Internet technology companies. Banks have very mature hardware system, deployment mode, software design mode, development mode, operation and maintenance mode. It will be more difficult to implement new technologies on such platforms. Why do you get this conclusion? Because now there are a lot of software manufacturers, a lot of product people, everyone wants to do new systems. But it is certainly not a one-size-fits-all solution to move large historical systems, because the costs are too high. So you have to run it in parallel, and for this kind of parallelism of old and new architectures, a lot of times you don’t have a solution, you can’t do it. In fact, we are working on this right now, doing an elegant parallelism of old and new systems, including parallelism of business logic, and parallelism of business data. If you are interested, you can also talk to us privately about this, which I think is very important.

The second point is the difference in organizational structure. Take micro services as an example. For the development of single applications for so many years, it is very clear who is in charge of technology, who is in charge of business and which department for each application. In many cases, it is difficult to determine the rights and responsibilities if microsertization and separation are carried out, and it is not realistic for the enterprise organizational architecture to adapt to the system architecture. Of course, historical assets, business scenarios and Internet enterprises are also different. Banks have more historical assets in informatization, and their businesses are more complex than the Internet.

4.3 New Architecture

Figure 9 is part of our system construction architecture diagram. At the bottom is the distributed NewSQL database base platform, and at the top is the application system. Currently, the traditional architecture and the new microservice architecture coexist.

5. Future prospects

Finally, we will introduce the direction of our future construction.

First, after phased practice, the new architecture still needs to be verified in various aspects to ensure its advantages in high availability, scalability and cost. In the next stage, we hope to expand the scope of application, and gradually migrate to the system with fast business development, large scale and high requirements for concurrency.

Second, we need to establish a set of application specifications, or guidelines for financial level development for TiDB. We are currently working on this, including best practices for r&d applications and parallel solutions for old and new architectures. It is a very important work for us to build the middleware of heterogeneous database transmission between traditional database and TiDB. After this part is finished, I believe it will be more beneficial to our application expansion.

Third, we also need to do HTAP, which may be more consistent with what Liu Qi talked about just now. I have read the design concept and design method of TiFlash before, and I think it is a relatively novel way, which is much better than some current data analysis schemes that still need T+1, with more integrated technical architecture and smoother business process. In addition, we have been working on improving performance and reducing network dependency.

Finally, we hope to share more of the experience of Bank of Beijing with you, so that you will no longer encounter the problems and troubles in our construction process and carry out the structural transformation work more smoothly.

That’s all FOR today. Thank you.