Review version 1.0

We review the 1.0 version of the content, after analysis of the requirements, the final 1.0 version is just to do an MVP – minimum feasible product, only to complete the most simplified core process, namely: registration – > login – > money – > transaction – > money. In terms of architecture design, from API design to key process design, to database design, and finally to the design of the server side, basically take saving development cost as the consideration and adopt the lowest cost design scheme.

In general, the MVP version is designed to separate the front and back ends. API uses HTTP communication protocol + JSON transmission protocol, and TLS to encrypt the transmitted data. User authentication adopts JWT scheme, which can be stateless. A total of 19 API interfaces are defined, which are divided into four modules according to business fields: user, account, transaction and market. GET method is used for read request of query class, and POST method is used for non-query class request. Implementation process, also did not use cache, MQ and other middleware, the database only used MySQL, transaction matching is also adopted to achieve a simple database matching scheme. Two third-party platforms are connected. One is Ali Cloud email push platform, which is used to send emails. One is the Infura platform for docking with the Ethereum blockchain. The database design aspect also proposed some design norms and principles, according to the DDD design idea, finally designed nine tables. The server is a single application. It adopts a simple three-layer architecture, which is divided into API layer, Service layer and DAO layer.

The overall architecture diagram is as follows:

With this 1.0 release, the professional can see that there are some important issues that need to be addressed:

  • Some service functions are missing, including retrieving passwords, changing passwords, logging out, KYC authentication (real-name authentication), and managing the background.

  • There is still work to be done, including adding more trading pairs and more cycles of K-chart data.

  • Interface polling request market data, and read from the database, high delay, low efficiency, poor performance.

  • The performance of database matching is poor.

Next, let’s look at design solutions to these problems.

Iterating business requirements

For the missing service functions, retrieving the password, changing the password, and logging out are simple to add API interfaces and implement the service logic. KYC authentication requires the uploading of a photo and access to a third-party file storage service. At present, the more important functions of the management background should include: user management, KYC audit, withdrawal audit, order management, transaction pair management and so on.

As for the business functions to be improved, more support for transaction pairs is mainly to connect with several mainstream blockchain apis. Now mainstream blockchain basically has mature third-party apis, and access is relatively easy. It is also easy to add k-line graph data of more cycles, which can be calculated and recorded according to the calculation formula of different cycles.

Below, there are some other important design points that need to be added.

Complementing password-related design, what are the best practices for ensuring password security throughout the process, from user input on the client to network transmission to server data storage? Here are a few points I’ve made:

  1. TLS is the foundation;

  2. The password is encrypted separately. The encryption algorithm must be asymmetric, such as RSA and ECC.

  3. If the user login password is incorrect, do not directly prompt “incorrect password”, you need to provide a general message, such as “incorrect username or password”.

  4. If the number of consecutive incorrect passwords exceeds N, for example, six times, the user is locked for a period of time.

  5. The database is stored in the scheme of slow hash + Salt. Different users use different Salt values. The slow hash algorithms mainly include Argon2, Scrypt, Bcrypt and PBKDF2.

  6. Add multiple verification, such as login device detection, fingerprint recognition, face recognition, mobile phone verification code, etc.

Some people think that TLS can be used to encrypt passwords separately, but this is not true. TLS only ensures that packets captured by a third party during transmission are ciphertext, but it does not prevent hackers from intercepting data on the client and server. Data interception on the server is difficult, but interception on the client is relatively easy. In the simplest case, if you visit well-known websites such as Zhihu and JINGdong in the browser and use the packet capture tool to capture the request, you will find that the data is in plain text, not ciphertext, although the request is HTTPS. There are many attacks against HTTPS, such as degraded attack and man-in-the-middle attack. Therefore, HTTPS alone is not enough for protection.

Why is asymmetric encryption recommended for passwords, rather than one-way hashing or symmetric encryption? If a one-way hash is used, such as MD5/SHA, the actual password for the server is the hash value, not the user’s original password, and it is very troublesome to upgrade the encryption algorithm in the future. That for the hacker, there is no need to crack the user’s original password, directly with hashed password to the server request can pass verification. If symmetric encryption is used, such as AES/DES, then the client needs to keep the encryption key securely, which is very difficult. With asymmetric encryption, the public key is stored on the client and is not compromised.

In terms of database storage, the former direction of thinking is how to prevent data leakage, but now more consideration is how to prevent data from being restored after leakage. That means we have to make it so that even with all the data and code stolen, it’s still hard to crack the original code. To achieve this goal, the main idea is to increase the cost of cracking, so that the cost of cracking far exceeds the benefit, so that there is no point in cracking. But the slow hash + Slat scheme can achieve this goal, because each user’s Salt value is different, can not use the rainbow table for batch cracking; Add slow hashing and the time cost of brute force increases exponentially.

Now wechat, Alipay, and many financial applications only use 6-digit payment passwords. The reason why it is safe is also because of the multi-verification mechanism. Assuming that each layer of checksum has a 30% chance of being cracked alone, the probability of being cracked with three layers of checksum becomes 30% * 30% * 30% = 2.7%, which greatly improves security.

So going back to our trading system, in this iteration, these features should be added and the development cost should not be very high. In terms of multiple verification, the second verification function of mobile phone verification can be added during login and payment. This scheme is the simplest to implement and also increases security to a certain extent.

Finished talking about the design of password security, and then the design of the management background to do some explanation.

The management side and the client side have very different functions and features, and a lot of business logic is also different. The users on the management side have more data rights than the users on the client side. Therefore, the management side and the client side are usually separated into different services, but the underlying database is the same. After adding the management background, the overall structure of the whole trading system is roughly as follows:

Optimization problem

In fact, our market problem can be divided into two problems, one is the problem of the client to obtain market data, the other is the problem of reading market data from the database. We can solve these two problems separately and then combine them to solve the whole problem.

Client gets market data

The client to obtain market data is essentially a problem of instant messaging between the Web and the server. There are actually four ways to achieve instant messaging on the Web: polling, long polling, long connection and WebSocket.

Polling (Polling), also called short Polling, requires that the client periodically sends requests to the server and the server returns response data. Therefore, the data obtained is not real-time and has a delay. In addition, because the server constantly requests, in many cases there is no new data update, so most requests are invalid requests, which is a serious waste of resources for the server. The advantage of polling is that it is simple, easy to understand, and easy to implement, which is why we chose polling for the first release.

Long Polling (Long Polling) is also initiated by the client. Different from short Polling, the server does not immediately return a response after receiving the request, but suspends the request until the data is updated. As soon as the client receives the response, it sends the next request, so it’s still essentially polling. Compared with short polling, long polling significantly reduces many invalid requests and saves resources. The downside is that a hung connection can lead to a waste of resources.

Long connection (SSE) is essentially different from short and long polling in that it allows the server to push data to the client. The process goes like this: the client sends a request, the server blocks upon receiving the request, and holds the connection. When the server has data to respond to, it responds with the held connection, and holds the connection. The long connection applies to the scenario of one-way push from the server to the client. However, the downside of the long connection is that it only works with advanced browsers, not IE.

WebSocket is completely different. In addition to the INITIAL connection using HTTP, the other times are directly based on TCP protocol for communication, which can achieve full-duplex communication between the client and the server, with high performance and low cost. It is the best choice for instant messaging on the Web. And not just on the Web side, but also on the App side. Therefore, our client to obtain market data optimization scheme, is also the most appropriate use of WebSocket.

However, WebSocket is much more complex to implement than HTTP, and is different in design from THE HTTP API. If poorly designed, it can still be a waste of resources, so it’s worth noting.

First, after the client establishes a connection with the server, the client notifies the server of the data it needs by sending a subscription message, such as this:

{ "subscribe": "market.ethusdt.kline.1min" }
Copy the code

This is a message to subscribe to one-minute K line data of an ETH/USDT transaction pair. After receiving this message, the server continuously pushes the updated data to the client as long as the one-minute K line data of the transaction pair is updated until the user cancels the subscription or disconnects. It’s also easy to unsubscribe, and the client sends another message like this:

{ "unsubscribe": "market.ethusdt.kline.1min" }
Copy the code

However, subscription messages are only used to receive subsequent updates of data, known as incremental data. However, the client sometimes needs to obtain the full data, especially during initialization. In this case, it can send a single request message, similar to the HTTP request. For example, we need to obtain the initial full data of the 1-minute K-chart of the ETH/USDT transaction pair.

{ "request": "market.ethusdt.kline.1min" }
Copy the code

After receiving this message, the server returns the full data all at once.

Ideally, the connection between the client and the server will remain as long as the two parties do not voluntarily disconnect. But in practice, one end can be disconnected abnormally for a variety of reasons, and the other end doesn’t know it. More often than not, the client is abnormally disconnected, and the server does not know it, but still pushes data to the client, and the data will be lost. To deal with this situation, a mechanism is needed to check whether the two ends are still connected, and this mechanism is heartbeat detection. Heartbeat detection means that one end sends a packet (also called a heartbeat packet) to the other end periodically (for example, every five seconds). After receiving the packet, the other end sends a packet back to check whether the connection between the two ends is normal. The implementation of heartbeat packet is also simple. The initiator only needs to send a ping message and the other end replies with a Pong message, like this:

{ "ping": 1606972817326 }
{ "pong": 1606972817326 }
Copy the code

The following string of numbers can be the time stamp when the heartbeat packet was sent. The reply message uses the same time stamp to know which heartbeat packet was replied.

If one end fails to receive a heartbeat packet more than three times, the other end is considered disconnected. In this case, if the client is still online, the client needs to initiate a reconnection.

The database reads market data

What are the main problems with reading quotation data directly from the current MySQL database?

Because we are polling request of market data, so will continuously produce read requests to the database, and as more and more online users, concurrent read it to the database will be more and more, but also continuously, so the database can easily achieve the performance bottlenecks, and achieve performance bottlenecks, can also affect the written request.

To solve the problem of database read performance bottleneck, most people think of the first solution is read and write separation. Read/write separation actually means that the database is divided into master library and slave library, read request to slave library, master library handles write request, after writing data, then copy to slave library. In this way, the burden of heavy read operations is transferred to slave libraries. If a single slave library cannot support a large number of read requests, multiple slave libraries can be deployed to achieve load balancing. MyCat is commonly used for read/write separation.

However, with read/write separation, there are also problems with master/slave data consistency. Data consistency between master and slave is required so that correct data can be read from slave. Therefore, there is a mechanism of data synchronization (replication) between master and slave. However, there are three primary/secondary replication mechanisms: asynchronous replication, full synchronous replication, and semi-synchronous replication. But regardless of which replication scheme you use, because databases have become clustered, it is difficult to achieve a trade-off between high performance and consistency. Asynchronous replication can maintain high performance, but cannot ensure data consistency. Fully synchronous replication ensures consistency but severely compromises performance. Semi-synchronous replication is a compromise, a balance between the two.

In fact, to solve our problem, using read-write separation is not the only solution, let alone the best solution. In addition to read/write separation, Redis caching can be used, and MongoDB can also be used. Redis cache should be preferred because it has the highest read/write performance.

Therefore, after choosing Redis, we need to think about what kind of data structure is more appropriate to store various market data. Our market data includes deep data, transaction records, K line data, Ticker data.

Deep data mainly contains the price and quantity of buying and selling orders, which need to be sorted by price and will change frequently. Therefore, sorted set is more suitable for saving. Buy order and sell order are stored separately with different keys, score is set as price, value is set as a binary of price and quantity. In addition, depth data is characterized by only one data for each price, so multiple records of the same score cannot exist. Therefore, to update the number of lots at a certain price, the score record corresponding to the price should be deleted first and then inserted.

Transaction records are added over time, Ticker data is updated based on the latest transaction records, and K line data is calculated based on cumulative transaction records. Transaction records are better stored in message queues (MQ), which can be listened on by Ticker and K line processing threads to obtain the latest transaction and update their data. Redis 5.0+ stream is a good fit for this scenario.

K-line data includes the opening price, closing price, high price, low price and trading volume of each time period, and the records also increase over time. However, in the latest time period, that is, the latest record needs frequent modification, so the more suitable storage structure is list.

Ticker data mainly displays the opening price, highest price, lowest price, latest price and trading volume of the day. There is no need to have multiple records, but only one record is needed. Part of the data will be updated according to the result of the transaction record.

Consolidation solves market problems

Then, the design of the two separate problems is combined to form an integrated solution.

In fact, it’s easy to start with transaction records, which can be a starting point for data flow. Each matched a successful transaction record, saved to Redis transaction record MQ, and increase the MQ monitoring thread, after listening to the new transaction, there are mainly four operations:

  1. Push the latest transaction record to the subscribed client through WebSocket;

  2. Update the latest K line data and cache, and then push the latest K line record to the client that subscribes to the corresponding K line data through WebSocket;

  3. Update Ticker data and cache, and then push the latest Ticker information to the clients subscribing to Ticker data through WebSocket;

  4. Update depth data, subtract the volume of transactions.

There are several places that trigger the change of depth data. In addition to transaction records and successful withdrawal of orders by users, the number of corresponding prices of depth data will be reduced. In addition, after the user places an order, the remaining quantity of the order that is not immediately closed will increase the quantity of the corresponding price.

In addition, WebSocket push depth data is not triggered by a certain event, but a timed push, usually every 1 second.

Finally, it can be found that, in fact, the whole market module is relatively independent, can be removed into an independent service. And, on the other hand, the quotes API, which is going to be open to the outside world, becomes open API, so in order not to affect the internal API services, it must also be independent services. In this case, early separation into a separate service, the subsequent changes to the minimum. When removed, the overall structure of the trading system looks like this:

Upgrade matching technology

The performance of database matching is generally around 100 TPS. The performance is very low. The main reason is that there are too many interactions with the database, I/O is also a lot, and the transaction logic of the database will be constrained. Database matching system, want to improve the performance, can only rely on upgrading the hardware, database hardware configuration is very high, the cost is also very high. Moreover, because of the matching rules, the matching can only be serial, not parallel, so there is no way to use the horizontal expansion of the server to improve the overall performance.

The matching logic is based on the principle of “price first, time first”. The orders that cannot be matched immediately will be placed in the transaction Orderbook (Orderbook). Orderbook is essentially two queues of buy and sell, each of which is ordered by price first and time first. The so-called price first, time first, that is to say: the order of the selling order queue is in order from low to high price, the order of the buy queue is on the contrary, in order from high to low price; Orders for the same price are ordered in order of time.

As shown in the figure above, each small box represents an order, H is the order at the head, N is the order at the same price as H but after H in the order time, S is the first order at the next price. It is obvious from the figure that orders are sorted by time in the horizontal direction and by price in the vertical direction.

When matching, it is to take out H power of attorney and new power of attorney to match. If the new order is buy, H order of sell order queue will be matched. If the new order is a sell order, the order H of the buy queue is obtained. If all H orders match, the order marked N becomes the new H order. If all the orders in the first row are matched, the S order will become the new H order.

Using the database matching technology, there is only the order table which saves all the orders, and Orderbook cannot be saved into the above data structure. Every time when matching judgment, it is necessary to query the order H from the whole table according to the complex query conditions, which is very time-consuming and the performance naturally fails.

So, now the industry basically no longer use the database match, and switch to memory match. Memory matching performance, a very ordinary server can easily reach 10 thousand TPS, high memory configuration even reach 100 thousand TPS is not difficult.

There are two key reasons why memory matching can achieve such a high performance improvement:

  1. Caches the entire Orderbook in memory and manipulates data directly in memory, which is very fast.

  2. It is easier to store Orderbook in memory with the data structure shown in the figure above, so that the header order can be quickly fetched each time the order is matched, without the need for a full table query.

In terms of specific implementation schemes, there are actually two approaches: one is to use Redis to save Orderbook; The second is to store Orderbook directly with objects in the programming language. In fact, the industry said memory matching, is achieved with the second scheme, but some people will understand the memory matching memory database matching, so it produced the first scheme.

However, the disadvantage of memory matching is the volatile memory. When the server fails and stops, all transaction data will be lost, and the reliability and consistency of the system will be greatly reduced accordingly. There are complex and simple solutions to this problem. The complex scheme mainly uses multi-machine hot backup technology to ensure reliability and replication state machine technology to ensure consistency. The simple solution is simply to restart the server, query the order from the database during initialization and reload it into memory. At present, we are more suitable to use the simple scheme first, because the implementation cost of the complex scheme is too high, and the input-output ratio is not high at the current stage.

Finally, the use of memory matching, which relies heavily on server memory, is not suitable for sharing memory with other services, so it should also be separated into a separate matching engine service. This decoupling is also simple, because the input to the matchmaking engine is a ordered order queue, and the output is also a queue. As a result, the overall architecture of our trading system became the following:

conclusion

At this point, several important issues left over from 1.0 have been resolved, and with those issues resolved, we are essentially moving to 2.0. Improve the business function, optimize the market services, upgrade the matching technology. The whole system is also divided into market services, client services, matching services, management services, but it is not micro service architecture, just four monomers. In the next article, we will analyze and see how the next stage should evolve.

Previous articles:

Evolution of Trading System Architecture (I) : version 1.0