If you happen to be in a small to mid-sized development team…

There are many small and medium-sized r&d teams, but there is little discussion in the community about the architectural practices of small and medium-sized r&d teams. Small and medium-sized r&d teams, especially those of 50 to 200 people, focus more on business logic and rapid iteration to validate business models in the early stages of business exploration, and less on technical architecture.

At this time, if we continue to follow the original structure and research and development mode, there will be a lot of problems, and we can no longer play. Is there a set of middleware and architecture upgrade solutions that can be directly implemented, based on open source, low cost and can be built quickly?

As an IT veteran with more than 10 years of experience, I have led the upgrading of the technology architecture of two companies, and I am here to discuss this issue with you.

I’ll be publishing this series over the next few days.

According to our past experience, the business research and development can quickly enter the project practice after the sharer gives a lecture for about an hour. New team members can also learn quickly on their own through the WIKI. This is our previous requirements for ourselves, as far as possible to reduce the tool’s requirements on personnel, simple and practical, reduce the cost.

Some of the Demo in this article uses C# language, but at the framework or architecture level, there is not much direct relationship with the language itself. Examples include RabbitMQ, Job, Redis and centralized logging, all of which are deployed on the same server side with slightly different client language versions.

All Demo can run directly, service address and management background can also be accessed directly. Due to the cost involved in the deployment in public cloud, I plan to continue until the end of March next year.

These small basic work, I hope to help small and medium-sized RESEARCH and development teams, to solve the practical problems encountered in everyone’s project. I am willing to grow up with you. Your sharing and praise are my motivation for this effort. Thank you!

The article series is divided into three parts, including framework, architecture, and common applications.

  • Frameworks are the use of middleware or tools such as caches, message queues, centralized logging, metrics, microservices frameworks, etc.
  • The architecture part is mainly about the improvement of design ideas, including the overall architecture of the enterprise, the architecture design of a single project, and the unified application stratification.
  • Common Applications is a combination of business and technology, with single sign-on and enterprise payment gateways.

The following is a detailed introduction to the chapter:

To do a good job, you must sharpen your tools

If operation and maintenance is the foundation, then the frame is the load-bearing wall. Rural housing is built brick by brick to build, and urban House is to lay the foundation first, then build load-bearing wall, and finally build brick, so the construction and introduction of middleware is the premise of the construction of high availability, high performance, easy to expand and scalable large and medium-sized system.

Each of the framework chapters consists of four main parts: what it is, how it works, usage scenarios, and a straightforward debug Demo. Among them, Demo and middleware have been tested by the two companies for four years, involving hundreds of applications, more than 100 databases with more than 10,000 tables, daily orders from tens of thousands to hundreds of thousands, and annual GMV from billions to tens of billions.

All of the middleware and tools are open source, and in the early days we also developed some of our own, such as centralized logging and metrics frameworks. Later, in the second company, in order to build quickly, reduce the cost, easy to maintain and expand, all changed to open source. This is not only conducive to personal learning and growth, knowledge reuse and career, but also conducive to team building and talent introduction.

Centralized cache Redis

Caching is one of the challenges of computers, and so is distributed caching. Redis looks very simple, but it affects system efficiency, performance, and data consistency.

It is not easy to use well, and the problems involved include: cache duration (complex multidimensional calculation), cache invalidation (active update), cache keys (Hash and facilitate manual intervention), selection of cache content and data structure, handling of cache avalanche, handling of cache penetration, etc.

In addition to caching, Redis has other features such as Lua computing power, Limit and Session time Windows, distributed locks, etc.

Message queue RabbitMQ

Message queue, like Gezhouba, has the capacity to accumulate a large amount of data, and then asynchronously output it reliably. It is the core of EDA’s event-driven architecture and the key to synchronizing data with CQRS. The reason for choosing RabbitMQ over Kafka is that business systems have high reliability requirements for messages and complex features such as message acknowledgement Ack.

Centralized log ELK

Logs are classified into system logs and application logs. How do you locate a problem in a cluster of hundreds of servers? How do you track the gigabytes or even terabytes of data generated every day? Centralized logging is a solution to such problems.

In the early days, we used Log4Net+MongoDB, which was developed independently, to collect and retrieve log information. However, as the amount of data increased, the query speed became slower and slower. ELK, which was later converted to open source, has lost some of its ease of use, but it supports large amounts of data and language-independent features. Below is an architecture diagram of ELK.

Task scheduling Job

Task scheduling jobs, like database jobs or Windows scheduled tasks, are key to asynchronous and batch processing in distributed systems. Jobs are divided into WinJobs and HttpJobs. Winjobs are scheduled tasks at the operating system level and implemented using Quartz.NET. The HttpJob is independently developed and implemented, which uses URL to periodically invoke microservices.

HttpJob cleverly solves the single point and publishing problems of WinJobs with clustering, and centrally manages all the scheduling rules, both simple rules and Cron expressions. HttpJob is easy to use, but the interval should not be less than 1 minute. After all, scheduling by URL is not efficient. The following figure shows the HttpJob management background.

Apply monitoring Metrics

“There is no improvement without measurement”, measurement is the basis of improvement and optimization, is a prerequisite to do a good system. Zabbix is typically used for system-level monitoring, while Metrics is used for business-application level monitoring.

Business applications are black boxes that collect real-time status of applications from buried data points and display it on a large screen or kanban board. It is the basis of alarm systems and digital management, and can be combined with centralized logging to quickly locate and find problems. Our business monitoring system uses Metrics.NET+InfluxDB+Grafana.

Microservices framework MSA

Microservices are the reuse of fine-grained business behaviors that need to match business capabilities and business phases. Microservice framework is a key component to realize microservice and distributed architecture. Our microservice framework is implemented based on open source Service Ack.

It is easy to use, good performance, automatic document generation, easy to debug testing, debugging tool Swagger UI, automatic interface testing tool SoapUI. The interface of micro-service is open by adopting the micro-service gateway independently developed by us, which can be simply configured through the governance background. The gateway uses NIO and IOCP to achieve high concurrency. Its main functions include authentication, timeout, current limiting, fusing, and monitoring. The following figure shows Swagger UI debugging tool.

Solr

The associated query after the sub-table, the fuzzy query of a large section of text, how to achieve these? Obviously the traditional database does not have a very good solution, then you can use professional retrieval tools.

Full-text retrieval tool Solr is not only simple and easy to use, but also supports high concurrency of massive data. It only needs to realize quasi-real-time or periodic synchronization of data on both sides of the system. Here’s how Solr works.

More tools

  • Distributed coordinator ZooKeeper ZK working principle, configuration center, Master election, Demo, a sufficient article.
  • Dapper.NET, an ORM framework, is a database access tool suitable for Internet systems with simple syntax, fast running speed, independent of database and controllable SQL writing.
  • EmitMapper and AutoMapper Object mapping tools EmitMapper has high performance, while AutoMapper is easy to use.
  • IoC framework Inversion of control IoC lightweight framework Autofac.
  • DLL package management NuGet, the internal DLL package management tool of the company, can solve the problems of centralized storage, update, reference, and dependency of DLLS.
  • Jenkins one-click compilation, publishing, automated testing, one-click rollback, high efficiency and low failure.

Structure – Thought enhancement

Being able to use the above frameworks does not necessarily make a good architect, but a good architect must be able to use frameworks. In addition to tool skills, architects also need design thinking enhancement and performance tuning skills.

This article takes the real project as the background, the thought method pursues simple and effective, the main content includes the enterprise overall architecture, the single project architecture design, the unified application layer, the debugging tool WinDbg.

Overall Enterprise Architecture

When we have hundreds of thousands of applications, we need not only the architecture design of a single project, but also the overall architecture of the enterprise for top-level thinking and guidance. Large companies have the same business mentality as small vendors, but it is more difficult for large companies to see the whole picture and the essence of the business. While small companies lack customer flow and middleware application scenarios, medium-sized companies have both, so the overall enterprise architecture is relatively easy to land.

The overall enterprise architecture, which includes business architecture, application architecture, data architecture, and technical architecture, needs to be easily switched among technology, business, and management. The attached file is a real case of removing sensitive information, with reference to TOGAF standard. However, the content is time oriented and time oriented, including enterprise business model, architecture status, architecture planning and architecture implementation.

Single project architecture design

The architectural design of a single project, like construction drawings, can directly guide the implementation of the engineering code. The first link is functional requirements, the next link is code implementation, and this is where the value of architectural design lies. From functional requirements to use cases, to use case activity diagrams, to domain diagrams, architectural layering, to core code, they are all linked together.

A poor domain diagram can result from a poor use case activity diagram, as the use case activity diagram is the next step on the domain diagram. Focus on responsibilities, boundaries, application relationships, storage, and deployment are at the heart of the architecture design, as illustrated in the figure below.

Unified application stratification

Layering applications is simple, but getting hundreds of applications across a company to adopt a uniform layering structure is not. It should be large or small, simple and easy to use, and support a variety of scenarios. We use THE IPO method: I represents Input, O represents Output, P represents Process, one in, one out, one processing. The essence of the application system is a machine, is a processing equipment, but also a in a out a processing, IPO is more simple and practical compared with DDD.

Debugging tool WinDbg

WinDbg or GDB is a good solution to the occasional problem that occurs in a production environment. The debugging tool WinDbg is like a doctor’s stethoscope. It is a reverse analysis tool to diagnose problems when the system is sick. The Dump file is similar to the black box of an airplane and records the running status of programs in the production environment.

This section describes how to use the debugging tool WinDbg and the packet capture tool ProcDump, and shares a real case. Someone wrote the code years ago, causing the occasional CPU spike every month or two.

We use ProcDump to grab the Dump file of the abnormal process in the production environment, and then analyze it with WinDbg command without knowing the code, and finally locate the line of code with the problem.

Common Applications

First tools, then frameworks, then architectural design, and finally into common applications. Because public applications are closely combined with business systems, but have certain independence, they are generally developed independently, and it is not convenient to open source without using open source. Public applications mainly include single sign-on (SSO), enterprise payment gateway, CTI communication gateway (SMS mail wechat), single sign-on (SSO) and enterprise payment gateway.

Single sign-on (sso)

Applications always come together when they are split. Split is the implementation level, compositing is the user level, and compositing must solve authentication and navigation problems. Single sign-on (SSO), which requires only one login and can be accessed everywhere, is based on the user system, permission system, authentication system and enterprise portal. Our credentials data tokens use the JWT standard to address security issues across languages, clients, and Webapis.

Enterprise Payment Gateway

The enterprise payment gateway centralizes and encapsulates the company’s major payments, such as Alipay, Tenpay, wechat, and Advance payment. It unifies the way that business system calls each payment interface and simplifies the interaction between business system and payment system.

It will unify various payment interfaces into payment, withholding, distribution, refund, return distribution, difference, transfer, freezing, unfreezing, advance payment, etc., only need to select the payment type when calling. The enterprise payment gateway integrates the design, development, deployment, monitoring, and maintenance of major payment systems, and provides unified encryption and decryption, serialization, logging, and security isolation.

The list of topics covered in this series (not published in that order) is as follows. If you are interested, please follow:

  • The opening
  • Cache Redis
  • Message queue RabbitMQ
  • Centralized log ELK
  • Task scheduling Job
  • Apply monitoring Metrics
  • Microservices framework MSA
  • Solr
  • Distributed coordinator ZooKeeper
  • Gadgets: Dapper.NET/EmitMapper/AutoMapper/Autofac/NuGet
  • Release tool Jenkins
  • Overall architecture design
  • Single project architecture design
  • Unified application stratification
  • Debugging tool WinDbg
  • Single sign-on (sso)
  • Enterprise Payment Gateway
  • “Article

The authors introduce

Zhang Huiqing, an IT veteran of more than 10 years, has successively served as architect of Ctrip, chief architect of Gooda Group, CTO of Zhongqing E-Travel and led the upgrading and transformation of the technical architecture of the two companies. Focus on architecture and engineering efficiency, technology and business matching and integration, technology value and innovation.

Thanks to Yuta Tian guang for correcting this article.