Matching engine development: The beginning
Match engine development: MVP version
Matching engine development: data structure design
Matching engine development: black box docking
Match engine development: decrypt black box process
The business process
The previous articles have talked about some of the internal design of the black box, including the core software structure, data structure, directory structure, etc. Starting in this section, we will go deeper and decrypt more of the design and implementation details inside the black box.
The first step in decrypting a black box is to understand how the data is processed inside it. The same is true when designing a new system. The first step is to clarify the business process and data flow. For matching engine, is to understand: from input to output, the middle has gone through what processing process.
As mentioned in the previous article, this matchmaking engine defines three inputs: start matchmaking, process orders, and close matchmaking. Let’s look at the flow behind each of these three inputs.
Opening match
Open set is open (to) a trade mark of the matching engine, not open set of trade mark is unable to process orders, and have opened a set of trade mark will not open again, or it will appear at the same time, there are two engines handle orders with a trade mark, this is unreasonable, the same trade mark order only by a engine serial processing.
Why not parallel? If orders for the same transaction object can be processed in parallel by multiple engines, there are at least a few problems:
- ** Which shall prevail in the transaction price? ** In theory, there can only be one transaction price at a time, and then after parallel, there will be multiple transaction prices, and then the transaction price is difficult to determine.
- ** How to maintain a unified entrusted ledger? ** Theoretically, each trading object has a single proxy ledger that holds all the orders, so how can this unified ledger be maintained across multiple engines after parallelization? If the unified maintenance of the database, it will undoubtedly reduce matching performance; If it is divided into multiple sub-books, it is difficult to guarantee the principle of price first and time first.
Both of these problems are difficult to solve, so all orders are ordered first and then thrown into the engine for serial processing.
Speaking of ordering, it is natural to need a ordering queue, so when enabling matching, it is necessary to initialize the order ordering queue corresponding to the transaction object. After initializing the sequencing queue, you can actually start the engine corresponding to the transaction object. In the Go program, the engine for each trading object runs as a separate Goroutine; In other languages, such as Java, it runs in separate threads.
After the engine is started, the transaction order book needs to be initialized to hold the order. It then waits for orders to be placed in the queue to be processed one by one.
And consider another scenario, what happens when the matchmaker restarts? For the opening of the matching trading object, after the restart of the need to restore it? If so, how to recover? The simplest solution is of course to use the cache, with Redis to enable the matching of the trading object cache, restart from Redis to load and restart these trading objects.
Therefore, there are actually two scenarios for triggering the matchmaking. One is triggered by the active invocation of the interface, and the other is automatically loaded from the Redis cache after the program restarts.
Finally, the result of enabling matchmaking is returned synchronously, so it has no asynchronous output.
In summary, the internal process of matching is roughly as follows:
Process orders
Once matchmaking is enabled, you can receive input to process the order. When the matchmaker receives the request to process the order, the first step needs to do some checks, including whether each parameter is valid, whether the order is repeated or exists, whether the engine corresponding to the transaction object is started, etc. After passing the check, the entire order can be cached in Redis and then added to the sequencing queue of the corresponding transaction object, waiting for the engine of the corresponding transaction object to consume it for matching processing. The process is shown below:
When the order is successfully added to the sequencing queue, the interface can synchronously return a successful response. Subsequent processing results are output via asynchronous MQ. After receiving the order, the engine of the trading object will produce different output results according to different situations.
As we know, there are two actions to handle an order: place the order and cancel the order. The business logic of order cancellation is very simple. It is to query the existence of the order from the transaction entrustment account, delete the order from the entrustment account if there is, and then output the result of successful order cancellation. If it does not exist, the result of a fallback order failure is displayed. The order business logic is more complex, but also according to different order types for different processing. The matchmaker version at the time of writing supports six different types, including two limit and four market types. Here are the results of different order types under different conditions.
- Limit: normal price limit. When there is an order in the entrusting book that can match the transaction of the order, one or more transaction records may be generated, and each transaction record will produce asynchronous output; When there are no matching orders in the entrusting ledger, the order (all or the rest) is added to the entrusting ledger, and no output is generated.
- Limit-ioc: IOC limit price – immediate transaction remaining cancellation. When there is an order in the entrusting book that can match the transaction of the order, one or more transaction records may be generated, and each transaction record will produce asynchronous output; When there is no matching order in the entrusting book, the order (all or the remaining quantity) will be cancelled, and a successful cancellation output will be generated.
- Market: Default market price – immediate cancellation of the remaining transaction. Like IOC price limit, when an order exists in the order queue (also known as the counterparty) in the entrusted ledger in the opposite direction of the order, one or more transaction records may be generated, and each transaction record will produce asynchronous output; When the counterparty has no order in the entrusting book, the order (all or the remaining quantity) will be cancelled, and then an output of successful cancellation will be generated. Different from IOC limit price: IOC limit price order is the commission price specified by the user, while the market price does not need to specify the commission price, and will be directly concluded with the head order of the counterparty, until the order has been completed or the counterparty has no order.
- Market -top5: market-top5 instant trading remaining cancellation. Market can conclude transactions with orders of all price stalls of the counterparty. However, market-Top5 can only conclude transactions with orders of five price stalls of the counterparty at most, and orders beyond five stalls will not be concluded. The remaining unsold orders will be processed and an output of successful withdrawal orders will be generated.
- Market -top10: market-top10 instant trading remaining cancellation. Orders within 10 price ranges of the counterparty will be closed at most.
- The market-opponent’s best price. If the counterparty has no order, the order will be cancelled directly and an output of successful cancellation will be generated. If the counterparty has an order, it will only close one order at most. If there is still an unclosed quantity, it will be converted into a limit order at the price of the counterparty’s first order and added to the entrusted account, and no output will be generated at this time.
It can be expressed as follows:
In addition, every request to process an order — either an order or a cancellation — is also cached in Redis and updated when changes are made. In this way, the order can be restored after the program restarts.
Close match
When an underlying trade is ready to be taken down, cancelled or suspended, the engine needs to be turned off. Before shutting down the engine, it is a good idea for the upstream service to stop calling the interface that processes the order, otherwise unexpected errors may occur, even though the program is fault-tolerant.
When closing the engine, there are also some simple judgments, such as whether the engine of the trading object has been started. The engine that is not started cannot be shut down naturally.
If there are unprocessed orders in the sequencing queue when the engine is shut down, you should wait until those orders are processed before actually shutting down the engine.
Finally, the cache is also cleared, removing all orders for that transaction object from the cache.
The result of shutting down the engine is also returned synchronously, so there is no asynchronous output.
The flow chart is also relatively simple:
summary
This section explains the core business process inside the matchmaking black box, including opening matchmaking, processing orders, closing three inputs of their internal logic. With these processes understood, the next article will look at code implementation.
The convention leaves a few questions to ponder: Is it easy to have a problem turning off matchmaking and having concurrent requests for orders? If so, where? What is it? How can it be solved?
The author’s personal blog
The author’s official account: Keegan