Matt Heath and Suhail Patel, two senior engineers at UK digital bank Monzo, shared their lessons on how to manage 1600 back-end microservices at a seminar in London. The five-year-old British bank, which has more than 4 million financial users, entered the U.S. market last September and is also developing digital banking services for businesses.
All of Monzo’s financial services are provided through mobile apps, which is why they decided to build a distributed architecture from the beginning, rather than a huge core banking system. Mesos was initially used to build container clusters, but in 2016 it was switched to K8s to build a platform for implementing various financial microservices. Monzo was one of the few early adopters of K8s and also migrated its infrastructure to AWS to reduce operations manpower.
Monzo uses Cassandra, a NoSQL database that is easy to expand horizontally. The main development language of the back-end system is simple Go. Their reason is that this language ensures downward compatibility. Simply recompile with the new version of Go and execute it, allowing you to use the new functionality to improve memory management efficiency.
This is a company like your development tools, if has the need to connect the third party system or payment platform, Monzo as homemade development integrated mechanism, to improve efficiency, avoid the use of third party integration tools, and brought into a lot of extra code, you don’t need they developed an interactive tool, for example, to connect the AWS and K8s environment, Allow developers to quickly deploy or restore old deployments with a pull Reques directive.
In terms of microservice design, Monzo runs each microservice in a Docker container. The code for containerizing microservices is divided into two layers. The first Layer is the Shared Core Library Layer that all microservices must have built in. The code base includes RPC, Cassandra, locking mechanism, Log recording mechanism, Monitoring Matrics, Queuing and other six categories. The other layer is the business layer, which is the code to be put into the micro-server.
Monzo’s principle of breaking down the granularity of microservices is to break it down to the smallest possible degree. They explain that the more detailed the breaking down, the risk of change can be minimized, for example, updating a single function can reduce the impact on other microservices. However, the smaller the granularity of microservices, the cost is the generation of a large number of microservices. In the first year, there were only a few hundred microservices, but by early November 2019 there were 1,500, and by December last year there were more than 1,600, According to Monzo. There were more than 9,300 non-repeating calls between these microservices.
Because all services are online, Monzo wants to implement zero-trust security system as far as possible. Therefore, whitelist method is adopted to control the list of other micro-services that each micro-service can call. At first, the number of microservices was small and the whitelist was maintained manually, but when the number reached thousands, or even nearly 10,000, the maintenance was very complicated, so Monzo decided to develop an automated maintenance tool.
Monzo, accustomed to developing his own tools, first selected a microservice with the most stringent security control, service.ledger, and established a call whitelist in the configuration file by using the network policy resources of K8s. It’s a microservice that moves money across accounts.
Then, Monzo developed a microservice communication profiling RPCMap, which can automatically analyze every Go program to find the source of all calls to the service. Ledger microservice to build a whitelist.
After having the list, Monzo next uses the NetworkPolicy resource of K8s to perform filtering. The NetworkPolicy is established in the ledger service configuration file of service. Only the network traffic sources with admissible labels can be allowed. Put in a whitelist file of authorized sources. When another development team wants call privileges, it updates the whitelist file and recreates the Ledger service (because the files in the code have changed) to take effect. The Ledger development team can review the new external calls during the build phase.
However, after testing, several problems were found, which increased the burden of manually reviewing calls and increased the risk that microservices would revert to older versions. In addition, the development team was used to manually editing the configuration file, so that everyone could modify the call whitelist and it was difficult to control. Later Monzo decided to import the open source K8s network security control project Calico, in each micro-service to establish a micro firewall function, to manage access. In addition, Monzo also makes great efforts to improve the information transparency of micro-service management, such as self-made micro-service analysis tool, which is convenient for the development team to query the list and status information of apis used by micro-service after each code checkout, and a large number of visual monitoring tools are used to track traffic and consumption.
In addition to the tools management mechanism, Monzo also produced a back-end engineer 101 guide, which requires development teams to follow the first time they write a back-end application, covering new service establishment, RPC handler import, information query, such as publishing and using information through Firehose, how to write unit tests, and how to deploy. Both have detailed instructions and regulations, and a Slack channel to discuss getting started with this set of back-end application specifications. Monzo requires that every developer follow this specification to develop back-end microservices.
Monzo explanation, the service of the fine granularity, although helps improve elasticity, but need to matching application architecture and tools, through standardized engineers focus on business problems, continuous improvement tools and functions, can quickly make a series of iterative modifications, to break the complexity of the large financial applications, but also reduce risk. (Source: K8S Chinese Community)
Reference: www.ithome.com.tw/news/136462