More excellent articles.
“Microservices are not all, just a subset of a specific domain”
Selection and process should be careful, otherwise it will be out of control.
Out of all the monitoring components, there’s always one for you
What are we Developing with Netty?
This is probably the most pertinent Redis specification
Portrait of a Programmer: A Decade of Ups and Downs
Most useful series:
The Most Common Set of “Vim” Techniques for Linux Production
The most Common set of “Sed” Techniques for Linux Production
The Most Common Set of “AWK” Tips for Linux Production
Everyone is learning SpringCloud, seems to learn SC is awesome coax, feel like. But microservices are only a small part of the overall enterprise application landscape. Microservices introduce more problems than they solve, and you run into all sorts of bottlenecks.
Microservices solve the problem of compute nodes, but the root cause is storage nodes. As businesses get bigger and bigger, storage, coding, and management become problems.
Here are some universal truths that don’t need to be labeled “XX company best practices.”
Below is a simple but not simple diagram of microservices related to data expansion. Small and medium sized companies can play well if they have these elements; In larger companies, because of their size, every component will encounter bottlenecks, so called specific optimization is not divorced from its essence.
So here we go.
Note that this graph is only a subset of the main data paths, the others including CDN, communication layer, etc., are not in this column.
This diagram does not contain a domain-specific architecture, but rather a general overview. Let’s start with the database capacity bottleneck and take a look at the percentage of inservers.
The database
For user data to be stored, there is a database. Over the years, NoSQL hasn’t been able to quell the fear of developers, so the likes of MySQL remain the preferred storage for most companies.
Assuming your business is growing well, this is a lot more interesting. At the start of the project, the more 6 SQL you play, the more holes you will bury for future generations. Because THE SQL function is too rich, accidentally, show off skills. You will find that the larger the forest, the higher the standard requirements for SQL. Some official features are strictly prohibited within the company.
The market is doing well, and it’s finally coming home to roost. What used to be a skill is now a liability. Slow query, full – text scanning, recruiting killed. Want to add cache, only to find no way to start; Want to separate the database and table, the results found that the table relationship is complex.
Small watch and wide watch
So the first step is to fill the hole. A joint query business with more than 3 tables is most likely not reasonable. Before adding caches, libraries and tables, we need to redesign the table.
Forget the database paradigm, we will have two types of tables: small and wide.
Small tables provide the most basic data, which can be done with a simple KV. Some joint queries, is directly in the program can be circular splicing. It’s much better to loop 1000 10-millisecond queries in your program than to run a single query in 6 seconds. This is the nature of distributed systems, where small batch queries are more viable than hang.
The wide table provides the analysis data commonly used for an important function through redundancy. This kind of table generally has a large number of fields. When writing, redundant data is obtained through splicing, and the scenario of reading more and writing less is generally used.
Once this is done, the rest of the work can proceed.
Depots table
In the “sub-database sub-table”? Selection and process should be careful, otherwise it will be out of control “, a detailed description of the sub-database sub-table selection, here is a brief talk about the process.
Split database and table is likely to introduce some kind of middleware, because just splitting the database is not enough. HA and FailOver are required at the same time.
The sub-database is divided into vertical sub-and horizontal sub-. Vertically oriented is the business separation, that is, part of the table according to the business logic independent to other libraries; The horizontal approach is capacity, which means that the data has a way to expand through the schema of separate databases and tables.
The data must have a measurable shard dimension, otherwise it will be too scattered, or too skewed, affecting subsequent processing.
Data synchronization
For example, some reporting services require a full amount of data.
Sharing data between different businesses by sharing databases is a pretty stupid idea to say the least. This is where some data synchronization tools are needed.
A data synchronization component is an essential component of a company. There are high-latency synchronization tools based on the last update time and low-latency binlog-based synchronization tools. Some companies also have what is called multi-room synchronization for stability.
Data synchronization is most afraid of exceptions because most synchronizations have sequential requirements. When everything is going well, everyone is happy. Once an exception occurs, additional means are required to ensure data synchronization and delay during the exception.
It’s all dirty work, automation sometimes backfires, and monitoring comes first.
Hierarchical data storage
Predictably, even if you split up your repositories and tables, you can still quickly reach a bottleneck. After dividing the database and table, some of your statistical functions may not be used, which is a serious injury in some traditional management systems.
A layered data storage layer is necessary. Some of your businesses, maybe one branch is MySQL, another condition is ES.
Different DBS do different things. RDBMS only do the original data storage and query, is flat fast data channel; Specific single-machine high performance DB, do some convergence and scientific calculation; Distributed RT-like storage for some medium-size data and provide some medium-delayed search capabilities; Massive storage system, storage system all historical records, and provides offline analysis function.
Who guarantees hierarchical data storage design? With the exception of a portion of the business that distributes data through MQ, it is our data synchronization component.
The cache
But the pressure on DB was so great that we had to consider caching. The cache should not be used arbitrarily. There are two principles: one is that the cache should not invade the business, that is, it should not contain business logic. One is cache hit ratio to be high, otherwise counterproductive. Caching is a complement to high concurrency and high speed interfaces, and is a necessary but not sufficient condition for system stability.
In addition to external cache clusters such as Redis, in-JVM caching is also an important place. Caches exist because OF the slowness of I/O devices, are usually stored in memory and disappear after a power failure.
Caching involves synchronizing data between the source database and the cached database. Usually, when updating the source repository, the relevant data in the cache will be deleted, so that the latest data can be read the next time.
modular
It’s time to modularize the project, because hundreds of programmers sharing a code base is risky enough.
Modularity is typically split along lines of business. For example, the separation of payment module and reporting module.
After modules are split, similar modules share the database. But it’s mostly done with redundant data, which decouples the business so that one part of the business is broken and the other part is fine. Like if there’s a murder next door and you go to work the next day.
Find a way to interact with each other using HttpClient, OkHttp, etc. The important thing is unification, and with unification comes a fancy name: RPC.
A small module can grow into a big line of business, or no one will buy it.
MQ
Another way to share data or interact with data between modulars is MQ. More than peak-cutting, MQ is a paradigm of interaction, a decoupling of the business.
Kafka is used by almost every company and has a maximum throughput of several hundred thousand. RabbitMQ and RocketMQ are used in scenarios with high reliability requirements, but are machine intensive.
MQ resources tend to be extremely reliable, and as infrastructure, if something goes wrong, it can be very dangerous. The design takes into account the flow of data processing in the event of an exception and the compensation strategy for MQ recovery.
The MQ cluster is designed to be small to prevent messages of different services and reliability levels from influencing each other. MQ is business – and functionally isolated to a minimum set of services.
To avoid adverse impact on services caused by MQ downtime, MQ messages on non-critical links cannot block services. Such messages are usually sent through asynchronous threads.
Micro service
We’ve used messaging and modularity to split the system into projects. Managing these projects in a unified way, unifying their interaction patterns and governance above, is the category of microservices.
Microservices are a process of formalizing multi-module projects. Non-standard service and microservice system, is to coexist for a period of time, how to ensure the replacement of the old and new services, is a management problem.
Functional components
According to SpringCloud, in order for a service to be discovered, it needs to register itself in a common registry, and other services can get an instance of it from the same place and call it.
The function that actually makes the call is RPC’s function. RPC takes into account a number of features such as timeouts, reloads, and fuses. On some very heavily trafficked nodes, you may also want to consider preheating.
RPC should be able to generate some statistical data, such as TPS, QPS, TP values, etc., which is obviously lacking in SpringCloud, so we need to rely on external systems for analysis.
External requests need to pass through a layer of gateways before they can flow internally. General operations, such as permissions, limiting, greyscale, etc., can be handled at the gateway layer.
Service governance
The most important feature of microservices is its governance function. Service governance is based on monitoring information. By measuring the size, duration, and distribution of each invocation, the service topology can be obtained.
Generally, the following information is most useful: 1. QPS, QPS distribution of time series, highest interval QPS 2. Average response time, average interface response time, maximum and minimum time 3
The above information can be used to sketch the service. It is the data basis for expansion, reduction and special management.
Another issue raised by microservices is the call chain, the actual path of a request. Troubleshooting problems in a distributed environment can be very difficult. Call chains can help developers quickly locate problems and understand the flow of business data.
The purpose of service governance is to find unreasonable requests and distribution, such as an interface taking too long; An interface has a large number of requests and needs to be cached. The dependency chain of a function is too long, requiring service optimization.
Service governance relies on a large number of external analysis tools, more common business models and the support of big data platforms.
We also put monitoring/alerting in the service governance section. In “Out of all the monitoring components, there is always one for you”, we discuss the technical options for monitoring in detail.
The log
The log aggregation function is to gather scattered logs into one place, and its main challenge is the amount of data.
Logs are divided into two parts. One part is full and can be backed up to the HDFS or log bastion machine through periodic synchronization. One part is the filtered log, such as some abnormal information, centralized in a certain processing platform for alarm.
Many developers like to output user behavior data to log files, which can be collected for streaming or offline computing to generate recommendations and models. Log information has entered the big data processing category, we will not describe more.
Continuous integration
If the technical team in a moderately sized company has a system worth working on, a release system is one. Is it So Hard to Release a system? A possible model is discussed.
Publishing systems are a convenient cover for a bunch of scripts. Workflow tools, release validation, and CI/CD capabilities are easy to add to your own release system.
Many micro service promotion articles, talking about virtualization (Docker), in fact, is not necessary. Virtualization reduces service orchestration time and facilitates capacity expansion and reduction. However, virtualization has high requirements on monitoring, log collection, and network topology. Recommendations are the last step in the system, not the first.
Whether your system is flexible also depends on the cultural environment of the company. If the last line takes a week or two to go through the approval process, then doing an agile continuous integration system is not necessary.
infrastructure
Infrastructure refers more to the operation and maintenance system, which is the cornerstone supporting the healthy development of the entire system. I prefer to make a distinction between basic operations and infrastructure because their models and cultures are the cornerstones of a company’s R&D environment.
END
The system seems simple and has a fixed solution. But the problem is that many companies, from startup to failure, play for so many years and still do not play well.
It’s pathetic.
