The architecture of a high – performance, high – availability, scalable, scalable distributed web site is briefly described, and an architecture reference is given. Part book notes, part personal experience summary. It has a good reference value for large distributed website architecture.
First, the characteristics of large websites
Many users, widely distributed
Heavy traffic and high concurrency
Massive data, high availability of services
Poor security environment and vulnerable to network attacks
Multiple functions, fast change, frequent release
From small to large, gradual development
User-centric
Free service, paid experience
Two, large site architecture goals
High performance: Provides fast access experience.
High availability: The site service is always accessible.
Scalability: increase/decrease processing power through hardware.
Security: provide website security access and data encryption, secure storage and other policies.
Extensibility: Easily add/remove new features/modules by adding/removing them.
Agility: on demand, quick response;
Three, large website architecture model
Layer: generally can be divided into application layer, service layer, data layer, management layer, analysis layer;
Segmentation: Generally, the application layer is divided into home page and user center based on service/module/function features.
Distributed: Applications are deployed separately (for example, on multiple physical machines) and work together through remote calls.
Cluster: Multiple applications, modules, and functions (for example, multiple physical machines) are deployed to provide external access through load balancing.
Cache: Data is stored closest to the application or user to speed up access.
Asynchrony: Asynchronizes synchronous operations. The client sends a request without waiting for a response from the server. After the server processes the request, the client notifies the requester in the form of notification or polling. Generally refers to: request – response – notification pattern.
Redundancy: Add copies to improve availability, security, and performance.
Security: Effective solutions to known problems, discovery and defense mechanisms for unknown/potential problems.
Automation: Repetitive tasks that do not require human involvement are done by means of tools, using machines.
Agility: Actively accept changing requirements and quickly respond to business development requirements.
4. High-performance architecture
User-centric, fast web access experience. The main parameters are shorter response time, larger concurrent processing capacity, higher throughput and stable performance parameters.
It can be divided into front-end optimization, application layer optimization, code layer optimization, storage layer optimization.
Front-end optimization: the part of the site before the business logic;
Browser optimization: reduce the number of Http requests, use browser caching, enable compression, Css Js location, Js async, reduce Cookie transfer;
CDN acceleration, reverse proxy;
Application layer optimization: servers that handle web site business. Use caching, asynchrony, clustering
Code optimization: reasonable architecture, multi-threading, resource reuse (object pool, thread pool, etc.), good data structure, JVM tuning, singleton, Cache, etc.
Storage optimization: cache, SSD, optical fiber transmission, optimized read and write, disk redundancy, distributed storage (HDFS), NOSQL, etc.
High availability architecture
Large sites should be accessible at all times. Normal provision of external services. Because of the complexity of large sites, distributed, cheap servers, open source databases, operating systems and other characteristics. It is difficult to ensure high availability, which means that site failures are inevitable.
How to improve usability is an urgent problem to solve. First, usability needs to be considered at the architecture level when planning. The industry generally uses several nines to represent usability metrics. For example, four nines (99.99) allow 53 minutes of unavailability over the course of a year.
Different layers use different policies. Generally, redundancy and failover are used to solve high availability problems.
Application layer: Generally designed to be stateless, it does not matter which server is used to process each request. Generally, load balancing technology (Session synchronization needs to be solved) is used to achieve high availability.
Service layer: load balancing, hierarchical management, fast failure (timeout setting), asynchronous invocation, service degradation, idempotency design, etc.
Data layer: redundancy backup (cold, hot backup [synchronous, asynchronous], warm backup), failover (confirmation, failover, recovery). The famous theoretical basis for high availability of data is THE CAP theory (persistence, availability, data consistency [strong consistency, user consistency, and ultimately consistency])
6. Scalable architecture
Scalability is the ability to increase/decrease the processing power of a system by adding/removing hardware (servers) without changing the original architectural design.
Application layer: The application is segmented vertically or horizontally. Load balancing is then performed for a single function (DNS,HTTP[reverse proxy],IP, link layer).
Service layer: similar to application layer;
Data layer: sub-library, sub-table, NOSQL, etc.; Common algorithm Hash, consistent Hash.
Extensible architecture
It can easily add/remove functional modules, providing good expansibility at the code/module level.
Modularization, componentization: high cohesion, internal coupling, improved reusability, expansibility.
Stable interface: Defines a stable interface. The internal structure can be changed “at will” without changing the interface.
Design pattern: apply object-oriented thinking, principles, using design pattern, code level design.
Message queues: Modular systems that interact through message queues to decouple dependencies between modules.
Distributed service: public module servitization, provide other systems to use, improve reusability, expansibility.
Viii. Security Architecture
Effective solutions to known problems, discovery and defense mechanisms for unknown/potential problems. For security issues, first of all, we should improve security awareness, establish an effective mechanism for security, and guarantee it from the policy level and organizational level. For example, the server password can not be leaked, the password is updated every month, and three times can not repeat; Weekly security scans, etc. Strengthen the construction of safety system in an institutionalized way. At the same time, attention should be paid to all aspects related to safety. Security issues cannot be ignored. Including infrastructure security, application system security, data security and so on.
Infrastructure security: hardware procurement, operating systems, network environment security. Generally, regular channels to buy high-quality products, choose a safe operating system, timely repair vulnerabilities, install anti-virus software firewall. Virus protection, back door. Configure firewall policies, establish DDOS defense systems, use attack detection systems, and perform subnet isolation.
Application security: In the development of the program, the known common problems, using the right way, at the code level to solve. Protects against cross-site scripting attacks (XSS), injection attacks, cross-site request forgery (CSRF), error messages, HTML comments, file uploads, path traversal, etc. You can also use Web application firewalls (such as ModSecurity) to scan for security vulnerabilities to enhance application-level security.
Data security: storage security (exist in reliable equipment, real-time, scheduled backup), storage security (important information encryption and preservation, select appropriate personnel complex preservation and detection, etc.), transmission security (prevent data theft and data tampering);
Common encryption and decryption algorithms (monomial hash encryption [MD5,SHA], symmetric encryption [DES,3DES,RC]), asymmetric encryption [RSA], etc.
Nine, agility
Website architecture design, operation and maintenance management to adapt to change, to provide high scalability, high scalability. Convenient to respond to rapid business development, sudden increase in traffic access requirements.
In addition to the architecture elements described above, agile management and agile development ideas need to be introduced. Make business, product, technology, operation and maintenance unified, on demand, quick response.
Examples of large architectures
The above uses a seven-layer logical architecture, the first layer is customer layer, the second layer is front-end optimization layer, the third layer is application layer, the fourth layer is service layer, the fifth layer is data storage layer, the sixth layer is big data storage layer, and the seventh layer is big data processing layer.
Client layer: supports PC browser and mobile APP. The difference is that the mobile APP can be accessed directly through IP access, reverse proxy server.
Front-end layer: DNS load balancing, CDN local acceleration and reverse proxy service;
Application layer: website application cluster; Vertical split by business, such as merchandise applications, member centers, etc.;
Service layer: provide public services, such as user services, order services, payment services, etc.
Data layer: support relational database cluster (support read/write separation), NOSQL cluster, distributed file system cluster; And distributed Cache;
Big data storage layer: supports log data collection at the application layer and service layer, structured and semi-structured data collection of relational database and NOSQL database;
Big data processing layer: Perform offline data analysis or Storm real-time data analysis through Mapreduce, and store the processed data into relational database. (In practice, offline data and real-time data are classified according to business requirements and stored in different databases for use by the application layer or service layer).
Spring source code analysis, Dubbo, Redis, Netty, ZooKeeper, Spring Cloud, distributed information
Also welcome to pay attention to my public number: Java large website architecture
