Hello. I’m here to update your profile again. Today we continue to update bytedance’s recruitment profile. (I’ve found that I especially like to organize bytes…)
This article is mainly to share with friends who have interview needs. I sorted out the interview questions of many Internet companies for many years, the interview data of many big names, as well as the Java learning materials. Friends in need can click into access. Code word: nuggets.
The interview questions
The interview lasts 2 hours. Application + principle understanding; Java backend recruitment +3 years of experience
1. Introduce yourself
2. JVM components
(1) the heap
(2) Method area (metadata area)
(3) Program counter
(4) Local method stack
(5) Virtual machine stack
3. Difference between virtual stack and local method stack (thread private, one per thread)
(1) The local method stack serves the same function as the virtual machine stack, which describes the memory model of method execution;
(2) That is, each method execution will create a stack frame on the stack, storing the information of the method entry and exit, local variable table, operands and so on. When a method starts and finishes, the corresponding stack frame is pushed and removed;
4. Can you use heap instead of stack?
(1) Stack is a memory model describing the method currently executing on the thread, and the heap stores the objects created when the program is running;
(2) Heap life cycle is consistent with the program, thread sharing, stack life cycle is consistent with the execution thread, and only for the current thread, so the memory size is much smaller than the heap;
(3) Stack memory address must be continuous and uniformly allocated. And stack last in first out, the method is completed stack frame pop up memory recycling in time. Ensure stack read speed;
(4) The heap memory address is not necessarily continuous, and the memory collection is not real-time collection by GC;
(5) When the stack memory is put into the heap, the short execution cycle of most threads will cause a large number of garbage objects or long-term objects will also have the problem of memory leakage, and have been migrated with garbage collection;
(6) Therefore, the memory management mode of heap and stack is different. Replacing stack with heap will increase the complexity of heap memory reclamation and make it easier to manage their respective memory separately
(7) The two memory functions are different. Stack overflow will prompt stack overflow error, and heap overflow will prompt memory overflow error. If put together, the error cannot be clearly located
The significance of the separation is that the two memory release timing requirements will lead to different memory management methods, which will increase the complexity of the existing garbage collection. In addition, the stack memory access rate and isolation requirements are inconsistent, and the use of heap storage also increases the complexity of heap memory management and reduces the efficiency of thread execution.
Why does the JVM have a garbage collector
The JAVA language does not provide a way to manually release object resources. The memory space is released by the JVM, so the garbage collector is provided to help us release memory resources. This also avoids memory leaks if we forget to release memory after allocating memory.
6. How to locate garbage and which nodes are ROOT nodes
(1) Objects referenced by class constants and static properties (static,final);
(2) objects referenced in the virtual machine stack and local method stack (local variable table);
7. How to solve the problem that Java programs occupy 100% of the CPU
CPU usage of 100 indicates that a computationally intensive type of thread is currently executing and in a state that cannot exit/performs computationally difficult (such as an infinite loop and GC scan phase when a large number of objects are not dead)
(1) Check whether the GC is normal. When a large number of small objects exist, the GC is very resource-intensive;
(2) Check the thread information that occupies resources and locate code problems according to the thread execution information;
(3) View the process with the highest occupation: top View the process with the highest occupation PID.
(4) Check the suspicious threads of the process: top-h -p PID Check the threads that occupy the highest proportion in the current process.
Printf “%x\n” thread PID or echo “obase=16; Thread PID “| BC;
Printing process thread (6) and the corresponding thread information query: jstack pid | grep – A num hexadecimal pid
(7) Check the thread stack information call path and the current method, check the code logic;
Blocking threads will cause CPU100
Spikes in CPU usage are more likely to result from performing computationally intensive tasks, rather than from threads in a blocked state that have not yet executed in a wait-time slice
9. Deadlock causes CPU usage of 100
(1) When lock competition occurs, the spin lock will execute the lock acquisition operation continuously until the number of retries or lock is obtained, making the CPU into busy waiting state;
(2) Applicable to operations with short service execution time;
while(! lock.lock()){
/ /...
}
Copy the code10, Connection=keep-alive
(1) EACH HTTP request will perform TCP connection handshake operation, which consumes resources and the performance is not very good, so the client tells the server to establish a long connection mode for long connection reuse. The next request of the client can be sent using the long connection.
(2) HTTP1.0 protocol requires the client to add Connection=keep-alive indicates a long Connection request, the server also needs to return Connection=keep-alive indicates support, and then both parties establish a long Connection, the client can reuse the Connection request;
Http1.1 default Connection=keep-alive; http1.1 default Connection=keep-alive; http1.1 default Connection=keep-alive;
(4) Long connection can ensure that the client can reuse the connection, but can not make the server actively send requests to the client (reuse is the Socket connection of the underlying TCP, but it needs to be separated from TCP keep-alive)
11, why use Websocket to do real-time communication connection, JSONP understand?
(1) Websocket is a full-duplex communication based on TCP. It can send information between the client and the server in real time and mainly solves the problem that the server actively pushes information to the client
(2) One client can establish a Websocket connection, without creating a large number of HTTP connection requests, reducing the pressure on the server
(3) Reduce the transmission of a large number of repeated request headers and reduce network bandwidth pressure
(4) the json
12, Long polling mechanism can not do communication
(1) Long polling requires different request connections for different real-time information transmission, and requires the server to HOLD the request, which will occupy a large number of service resources; Websocket can reuse a TCP channel to transmit multiple types of information, which is distinguished by text protocol.
(2) The Http request header needs to carry a lot of information every time, occupying a lot of network resources
(3) Unable to solve the problem of the server side actively sending messages
(4) Advantages and disadvantages of short/long polling and WebSocket:
- Short polling
Definition: Ordinary polling. At a specified interval (e.g., every second), the browser sends an HTTP request to the server, and the server returns the latest data to the client’s browser.
Application scenario: Traditional Web communication mode. Background processing data, it takes a certain amount of time, the front end want to know the processing results of the back end, it is necessary to send requests to the back end irregularly to get the latest situation.
Advantages: The front and back end of the program is easier to write.
Disadvantages: most of the requests are useless, difficult to maintain, waste bandwidth and server resources; The result of the response is not sequential (because it is asynchronous, subsequent requests are sent when the sent request does not return a result. If the subsequent request returns the result before the previous request, the result data will already be stale and invalid.
Example: Suitable for small applications.
- Long polling
Definition: A client sends an Ajax request to the server. The server receives the request and holds the connection until it receives a new message and closes the connection. The client processes the response and then sends a new request to the server.
Advantages: Less frequent requests without messages and less resource consumption.
Disadvantages: server hold connection will consume resources, return data sequence is not guaranteed, difficult to manage and maintain.
- WebSocket
Definition: Websocket is based on THE HTTP protocol. After establishing a link with the server, the server will actively push data changes to the front end.
Advantages: Fast response and no waste of resources. (traditional HTTP requests, the concurrent ability is dependent on A multiple TCP connections at the same time access to the server implementation (hence the concurrency is limited by the browser to allow the number of simultaneous connections), whereas the websocket allows us to concurrent multiple requests on A ws connection, A response after A request has not yet arrived, you can continue to B requests. Due to the slow start characteristic of TCP (it takes time to get up the new connection speed) and the handshake loss of the connection itself, this feature of WebSocket protocol has a great efficiency improvement. HTTP headers are too big and most of the hundreds or thousands of bytes of headers per request are duplicated. Websockets reuse long connections without this problem.
Disadvantages: The versions of Web sockets supported by mainstream browsers are inconsistent. There is no standard API on the server side. The following is a list of webSocket compatibility for the major players.
Example: Implement instant messaging, such as stock market analysis, chat rooms, online games, and so on, instead of polling and long polling
Resolution: Resolves two problems with the HTTP protocol:
① Passivity of the server. The HTTP protocol responds only after the client asks. Fixed synchronization delays
② Solve the problem of resource consumption on the server
13. How to guarantee Kafka sequential consumption
(1) Only one partition is created for a queue to ensure sequential consumption
(2) Message delivery on a queue uses the hash result of key to ensure partial order
Kafka Partion and Consumer
(1) A Partion can only be consumed by one consumer in one consumer group
(2) A consumer can consume multiple partions in the queue
15, Redis distributed lock SETNX first, not enough time to set the expiration time down how to do
(1) Set key value nx ex seconds
(2) LUA scripts guarantee atomic operations
What happens when Redis runs out of memory
(1) Implement elimination strategy. The default is return error; There are also random elimination and least recently used elimination strategies (be careful to set maxMemory, otherwise 64-bit machines will run out of machine memory)
(2) Use cluster mode to store data in fragments (middleware solves sharding/business keyhash implementation)
(3) Reduce the generation of large keys and optimize the query of large keys
17, How does Redis check big keys
(1) redis-cli –bigkeys; Based on the scan command, you don’t have to worry about blocking. Count bytes for String types and elements for collection types.
(2) Debug object key View the serialized length of the key
(3) Manually run the BGsave command to generate an RDB file, and use redis RDB Tools to analyze the RDB file
(4) View the memory usage of the key (version 4.0+)
18. Do you know about LUA scripts
– the LUA tutorial
19. What if a large number of Keys fail at the same time
The main purpose of caching is to speed up queries and protect scarce resources such as databases from being overwhelmed. The most common problems of cache are cache penetration, breakdown and avalanche. Under high concurrency, a large number of requests fall into the database, resulting in full database resources and database failure.
(1) Concept
- The cache to penetrate
Under high concurrency, the cache will not be hit when a non-existent value is queried, causing a large number of requests to fall directly on the database, such as a non-existent activity in an active system.
- Cache breakdown
Under high concurrency, a particular value is queried, but at this time the cache expires, the cache does not hit, resulting in a large number of requests directly to the database, such as the active system to query the activity information, but in the process of the activity of the active cache suddenly expired.
- Cache avalanche
Under high concurrency, a large number of cache keys fail at the same time, resulting in a large number of requests falling on the database. For example, there are many activities going on in the active system at the same time, but at some point all the active caches expire.
(2) Common solutions
- NULL values are cached directly
- Current limiting
- Cache warming
- Hierarchical caching
- The cache never expires
(3) Layering-cache practice
Layering-cache is a combination of cache NULL values, cache warming, limiting, hierarchical caching, and indirectly implementing “never expire” cache penetration, breakdown, and avalanche.
- NULL values are cached directly
The most effective way to deal with cache penetration is to cache NULL values directly, but do not cache NULL values for too long or NULL data will not be updated for too long or too short to prevent cache penetration. In layering-cache, NULL values are not allowed to be stored in level-1 caches. Level-2 caches can be configured to allow NULL values to be stored in level-2 caches. If NULL values are allowed to be stored in level-2 caches, the framework can also configure the expiration time multiplier between non-null values and NULL values. This allows us to precisely control the NULL expiration time of each cache, with very fine granularity of control. When the NULL cache expires, I can also use traffic limiting, cache warming, etc., to prevent penetration.
Example:
@cacheable (value = "people", key = "#person.id", depict = "firstCache ", timeUnit = TimeUnit.MINUTES), secondaryCache = @SecondaryCache(expireTime = 10, timeUnit = TimeUnit.HOURS, isAllowNullValue = true, magnification = 10))
public Person findOne(Person person) {
Person p = personRepository.findOne(Example.of(person));
logger.info("Is the id and key is :" + p.getId() + "The data is cached.");
return p;
}
Copy the codeIn this example, isAllowNullValue = true means that nullvalues are allowed to be swapped. Magnification = 10 means that the time multiplier between NULL and non-null values is 10.
- Current limiting
One of the common methods to deal with cache penetration is traffic limiting. Common traffic limiting algorithms include sliding window, token bucket algorithm and leaky bucket algorithm, or directly using queues and locks, etc. In layering-cache, I mainly use distributed locks for traffic limiting.
Layering-cache data reading process:
Here is the core code for reading data:
private <T> T executeCacheMethod(RedisCacheKey redisCacheKey, Callable<T> valueLoader) {
Lock redisLock = new Lock(redisTemplate, redisCacheKey.getKey() + "_sync_lock");
// The same thread loops through the cache 20 times, waiting 20 milliseconds each time, if there is still no data to execute the cached method directly
for (int i = 0; i < RETRY_COUNT; i++) {
try {
// Get the lock from the cache
Object result = redisTemplate.opsForValue().get(redisCacheKey.getKey());
if(result ! =null) {
logger.debug("Redis cache key= {} cache hit, no need to execute cached method", redisCacheKey.getKey());
return (T) fromStoreValue(result);
}
// Obtain the distributed lock to query data
if (redisLock.lock()) {
T t = loaderAndPutValue(redisCacheKey, valueLoader, true);
logger.debug("Redis cache key= {} after fetching data from database, wake up all waiting threads", redisCacheKey.getKey());
// Wake up the thread
container.signalAll(redisCacheKey.getKey());
return t;
}
// The thread waits
logger.debug("Redis cache key= {} data from the database did not obtain the lock, enter the wait state, wait {} milliseconds", redisCacheKey.getKey(), WAIT_TIME);
container.await(redisCacheKey.getKey(), WAIT_TIME);
} catch (Exception e) {
container.signalAll(redisCacheKey.getKey());
throw new LoaderCacheValueException(redisCacheKey.getKey(), e);
} finally {
redisLock.unlock();
}
}
logger.debug("Redis cache key={} wait {} times, total {} milliseconds, no cache, directly execute the cached method", redisCacheKey.getKey(), RETRY_COUNT, RETRY_COUNT * WAIT_TIME, WAIT_TIME);
return loaderAndPutValue(redisCacheKey, valueLoader, true);
}
Copy the codeWhen need to load cache, need to get to the lock to have access to the backend to load the cached data, or you will wait for (the same thread loop query cache 20 times, each time for 20 milliseconds, if still no data directly to execute the cached method, the main is to prevent access to lock and load the cached thread out of the question, No return resulting in a deadlock). When the thread that acquired the lock completes execution, it puts the acquired data into the cache and wakes up all waiting threads. Thread.sleep() is not used to wait for a Thread. When using Spring Redis Cache, I found that when the number of concurrent requests reached 300, the Cache would expire and cause a deadlock. When there are too many threads waiting, a lot of context switching will occur. As a result, the thread that has acquired the lock cannot obtain the execution right of the CPU, resulting in a deadlock. In layering-cache, we use locksupport. parkNanos, which frees CPU resources. Because we are using redis distributed locks, we cannot use wait-notify.
- Cache warming
One effective way to deal with cache breakdowns and avalanches is cache preloading.
@cacheable (value = "people", key = "#person.id", depict = "firstCache ", timeUnit = TimeUnit.MINUTES), secondaryCache = @SecondaryCache(expireTime = 10, preloadTime = 2,timeUnit = TimeUnit.HOURS,))
public Person findOne(Person person) {
Person p = personRepository.findOne(Example.of(person));
logger.info("Is the id and key is :" + p.getId() + "The data is cached.");
return p;
}
Copy the codeIn layering-cache, the level 2 cache is configured with two times, expireTime is the cache expiration time, and preloadTime is the cache refresh time (preloadTime). Each time the level 2 cache is hit, it checks whether the past time of the cache is less than the refresh time. If it is less than the refresh time, an asynchronous thread will be started to pre-update the cache and put the new value into the cache, effectively ensuring that the hot data “never expires”. Not all threads will drop into the library to refresh the cache. If they do not acquire the lock, they will terminate the current thread.
/** * Refresh the cached data */
private <T> void refreshCache(RedisCacheKey redisCacheKey, Callable<T> valueLoader, Object result) {
Long ttl = redisTemplate.getExpire(redisCacheKey.getKey());
Long preload = preloadTime;
// If NULL values are allowed to be cached, the automatic refresh time is also divided by a multiple
boolean flag = isAllowNullValues() && (result instanceof NullValue || result == null);
if (flag) {
preload = preload / getMagnification();
}
if (null! = ttl && ttl >0 && TimeUnit.SECONDS.toMillis(ttl) <= preload) {
// Determine whether to force a refresh before starting the refresh thread
if(! getForceRefresh()) { logger.debug("Redis cache key={} soft refresh cache mode", redisCacheKey.getKey());
softRefresh(redisCacheKey);
} else {
logger.debug("Redis cache key={} Strong refresh cache mode", redisCacheKey.getKey()); forceRefresh(redisCacheKey, valueLoader); }}}/** * Hard flush (executes cached methods) **@param redisCacheKey {@link RedisCacheKey}
* @paramValueLoader Data loader */
private <T> void forceRefresh(RedisCacheKey redisCacheKey, Callable<T> valueLoader) {
// Start as few threads as possible because the thread pool is limited
ThreadTaskUtils.run(() -> {
// Add a distributed lock and place only one request to flush the cache
Lock redisLock = new Lock(redisTemplate, redisCacheKey.getKey() + "_lock");
try {
if (redisLock.lock()) {
// After acquiring the lock, determine the expiration time to see if data needs to be loaded
Long ttl = redisTemplate.getExpire(redisCacheKey.getKey());
if (null! = ttl && ttl >0 && TimeUnit.SECONDS.toMillis(ttl) <= preloadTime) {
// Load the data and place it in the cache
loaderAndPutValue(redisCacheKey, valueLoader, false); }}}catch (Exception e) {
logger.error(e.getMessage(), e);
} finally{ redisLock.unlock(); }}); }Copy the codeIf the cache fails at the same time, it will be a very slow process to warm up the cache, such as restarting the service or launching a new cache. At this time, we can use the flow cutting method to slowly warm up the cache. For example, we can cut 10% of the flow at the beginning, and then cut 30% of the flow after observing that there is no anomaly. After observing that there is no anomaly, we can cut 60% of the flow again, and then cut the full volume. Although this way is a little tedious, once we encounter abnormalities, we can quickly cut back the flow, so that the risk is controllable.
(4) Summary
In general, layering-cache is more about cache penetration, breakdown, and avalanche prevention than recovery. In fact, there is no perfect solution to these problems of cache, only the most suitable for their business systems. Currently, the layering-cache framework can handle most of the caching problems
20, Redis Pipeline
(1) Redis Pipeline principle
- The principle of
Most students have always had a misunderstanding about the Redis pipeline. They think it is a special technology provided by the Redis server that can speed up the Redis access efficiency. But in fact Redis Pipeline itself is not directly provided by Redis server technology, this technology is essentially provided by the client, with no direct relationship to the server. Let’s take a closer look at this.
- Redis message interaction
When we use the client to perform an operation on Redis, as shown in the figure below, the client sends the request to the server, which then sends the response back to the client. It takes a network packet time to get back and forth.
If multiple instructions are executed consecutively, it can take multiple network packets to and from each other. As shown in the figure below.
Back to the client-side code level, the client went through read – write – read – write four operations before fully executing two instructions
Now if we adjust the order of read and write to write – write – read – read, these two instructions can also be completed normally
Two consecutive writes and two consecutive reads take only one network trip in total, just as sequential write operations merge and sequential read operations merge.
- Pipeline pressure test
Let’s practice the power of pipes. Redis comes with a pressure test tool, Redis-Benchmark, that you can use to test pipelines.
First we pressure a common set instruction, QPS about 5W /s.
redis-benchmark -t set -q
SET: 51975.05 requests per second
Copy the codeWe added the pipe option -p parameter, which represents the number of parallel requests within a single pipe. See P=2 below, QPS reaches 9W /s.
redis-benchmark -t set -P 2 -q
SET: 91240.88 requests per second
Copy the codeIf P is 3, QPS goes to 10w/s
SET: 102354.15 requests per second
Copy the codeHowever, if we continue to improve P parameter, WE find that QPS cannot go up. Why is that? Because the CPU capacity here has reached a bottleneck, Redis single-threaded CPU has reached 100%, so there is no further improvement.
- Deep understanding of pipeline nature
Let’s take a closer look at the flow of a request interaction. The truth is it’s complicated because it goes through the network protocol stack, and it goes deep into the kernel.
① The client process invokes write to write the message to the send Buffer allocated by the operating system kernel for the socket.
② The client operating system kernel sends the buffer to the nic.
Network adapter Hardware sends data to the server’s network adapter via “Internet routing.”
③ The server operating system kernel puts the nic data into the RECV Buffer allocated by the kernel for the socket.
④ The server process calls READ to fetch the message from the receive buffer for processing.
⑤ The server process calls write to write the response message to the send Buffer allocated by the kernel for the socket.
⑥ The server operating system kernel sends the buffered content to the nic.
Nic hardware sends data to the client’s nic via “Internet routing”.
⑦ The client operating system kernel puts the data of the network card into the recV buffer allocated by the kernel for the socket.
The client process calls read to retrieve the message from the receive buffer and return it to the upper-layer business logic for processing.
Pet-name ruby end.
Steps 5 to 8 and 1 to 4 are the same, but the direction is reversed, one is request, the other is response.
We initially thought that the write operation would wait until the other party received the message before returning, but this was not the case. The write operation only writes data to the send buffer of the local operating system kernel and then returns. The operating system kernel does the rest to asynchronously send the data to the target machine. But if the send buffer is full, then you need to wait for the buffer to empty space, which is the real time of write IO operations.
We initially thought that the read operation was pulling data from the target machine, but this is not the case. The read operation simply fetches data from the local operating system kernel’s receive buffer. But if the buffer is empty, then you have to wait for the data to arrive, which is the real time of read IO operations. So for a simple request like value = redis.get(key), write takes almost no time, and writes directly to the send buffer and returns, while read takes more time. It waits for a response message to be routed through the network to the target machine for processing and then sent back to the current kernel read buffer before it can return. This is the real cost of a round-trip network.
For pipes, sequential write operations take no time at all, after which the first read waits for a network overhead, and all response messages are sent back to the kernel’s read buffer, and subsequent reads are immediately returned from the buffer.
- summary
This is the nature of the pipe, which is not a server feature, but a huge performance improvement on the client side by changing the order of reads and writes. When writing Redis, we usually write a data, wait until the server returns ok, we write a data, so naturally slow, but pipeline is: Write data as long as the local cache is ok, keep writing until the cache is full, of course, the data is also constantly going to the Redis server, to clear the client’s cache, this of course is faster, because there is no longer need to wait for the server to return.
(2) Redis: Pipeline details
- The background of pipeline
The Redis client executes a command in four stages: send command — > queue command — > execute command — > return result
This process is called Round trip time(RTT for short), mGET Mset effectively saves RTT, but most commands (such as HGEtall, but no MHGEtall) do not support batch operation, which requires N RTT consumption. Pipeline is needed to solve this problem at this time
- The performance of the pipeline
① Did not execute N commands using pipeline
② Use pipeline to execute N commands
Summary: This is a group of statistical data, the execution speed of Pipeline is faster than that of line by line, especially the greater the network delay between client and server, the more obvious the performance. The test code is posted below to analyze the performance difference between the two:
@Test
public void pipeCompare(a) {
Jedis redis = new Jedis("192.168.1.111".6379);
redis.auth("12345678");// Authorization password Specifies the requirepass password of redis.conf
Map<String, String> data = new HashMap<String, String>();
redis.select(8);// use the eighth library
redis.flushDB();// Clear all data in database 8
// hmset
long start = System.currentTimeMillis();
/ / hmset directly
for (int i = 0; i < 10000; i++) {
data.clear(); / / empty map
data.put("k_" + i, "v_" + i);
redis.hmset("key_" + i, data); // Loop 10000 data inserts into redis
}
long end = System.currentTimeMillis();
System.out.println("Co-insert :[" + redis.dbSize() + The article "].. ");
System.out.println("1. Time spent setting values in batches without using PIPE" + (end - start) / 1000 + "Seconds..");
redis.select(8);
redis.flushDB();
// Use pipeline hmset
Pipeline pipe = redis.pipelined();
start = System.currentTimeMillis();
//
for (int i = 0; i < 10000; i++) {
data.clear();
data.put("k_" + i, "v_" + i);
pipe.hmset("key_" + i, data); // Encapsulates the value into the PIPE object, which is not executed at this point and remains on the client side
}
pipe.sync(); // Send the encapsulated PIPE to Redis once
end = System.currentTimeMillis();
System.out.println("PIPE insert :[" + redis.dbSize() + The article "].. ");
System.out.println("2. Time spent using PIPE to set values in batches" + (end - start) / 1000 + "Seconds..");
//--------------------------------------------------------------------------------------------------
// hmget
Set<String> keys = redis.keys("key_*"); // query all result keys with the values set above
// Use Jedis hgetall directly
start = System.currentTimeMillis();
Map<String, Map<String, String>> result = new HashMap<String, Map<String, String>>();
for (String key : keys) {
// Set keys to 10000, loop 10000 times
result.put(key, redis.hgetAll(key)); // Use the redis object to evaluate by key value and put the result into the result object
}
end = System.currentTimeMillis();
System.out.println("Common values :[" + redis.dbSize() + The article "].. ");
System.out.println("3. Batch value time without using PIPE" + (end - start) / 1000 + "Seconds..");
// Use pipeline hgetall
result.clear();
start = System.currentTimeMillis();
for (String key : keys) {
pipe.hgetAll(key); // Use PIPE to encapsulate the key that needs to be specified
}
pipe.sync(); // Submit to Redis for query
end = System.currentTimeMillis();
System.out.println("PIPE :[" + redis.dbSize() + The article "].. ");
System.out.println("4. Time spent using PIPE to batch value" + (end - start) / 1000 + "Seconds..");
redis.disconnect();
}
Copy the code
- Native batch commands (Mset, MGET) compared to pipeline
① The native batch command is atomic, pipeline is non-atomic
Atomic concept: A transaction is an indivisible minimum unit of work that either succeeds or fails. Atomic operations mean that one of your business logic must be indivisible. ② Native batch commands have multiple keys in one command, but pipeline supports multiple commands (transactions exist), non-atomic ③ Native batch commands are server implementation, and pipeline needs server and client to jointly complete
- Correct use of pipeline
The number of commands used for pipeline assembly should not be too large, otherwise the data volume is too large, increasing the waiting time of the client, and may cause network congestion. A large number of commands can be split into several small pipeline commands to complete.
We all know that REDis provides mSET, MGET methods, but does not provide mDEL methods, if you want to achieve, you can use pipeline implementation.
② Pipeline use steps in Jedis: get Jedis object (generally from the connection pool) — > get Jedis object pipeline object — > add instructions — > execute instructions
Test class methods:
@Test
public void testCommond(a) {
// Initialize the tool class
JedisUtils jedis = new JedisUtils("192.168.1.111".6379."12345678");
for (int i = 0; i < 100; i++) {
/ / set value
jedis.set("n" + i, String.valueOf(i));
}
System.out.println("keys from redis return =======" + jedis.keys("*"));
}
// Use pipeline to batch delete
@Test
public void testPipelineMdel(a) {
// Initialize the tool class
JedisUtils jedis = new JedisUtils("192.168.1.111".6379."12345678");
List<String> keys = new ArrayList<String>();
for (int i = 0; i < 100; i++) {
keys.add("n" + i);
}
jedis.mdel(keys);
System.out.println("after mdel the redis return ---------" + jedis.keys("*"));
}
Copy the codeMdel method under JedisUtils:
/** * delete multiple string keys and release connections **@param keys*
* @returnValue is returned on success. Null */ is returned on failure
public boolean mdel(List<String> keys) {
Jedis jedis = null;
boolean flag = false;
try {
jedis = pool.getResource();// Borrow the Jedis object from the connection
Pipeline pipe = jedis.pipelined();// Get the pipeline object of the Jedis object
for(String key:keys){
pipe.del(key); // Put multiple keys into the PIPE delete directive
}
pipe.sync(); // Execute the command exactly when the pipeline object is called remotely
flag = true;
} catch (Exception e) {
pool.returnBrokenResource(jedis);
e.printStackTrace();
} finally {
returnResource(pool, jedis);
}
return flag;
}
Copy the codeSubmit all actions using pipeline and return execution results:
@Test
public void testPipelineSyncAll(a) {
// Initialize the tool class
Jedis jedis = new Jedis("192.168.1.111".6379);
jedis.auth("12345678");
// Get the pipeline object
Pipeline pipe = jedis.pipelined();
pipe.multi();
pipe.set("name"."james"); / / tone value
pipe.incr("age");/ / since the increase
pipe.get("name");
pipe.discard();
// Combine the different types of action commands submitted and return the action operations as a list
List<Object> list = pipe.syncAndReturnAll();
for (Object obj : list) {
// Print the operation result
System.out.println(obj);
}
// Disconnect to release resources
jedis.disconnect();
}
Copy the code- Redis transactions
Pipeline is a combination of multiple commands, and redis provides a simple transaction to ensure its atomicity.
① Simple transactions in Redis
A set of commands that need to be executed together is placed between multi and exec, where multi represents the start of a transaction and exec represents the end of a transaction.
② Stop transaction discard
③ The command is incorrect and the syntax is incorrect, causing the transaction to fail
④ Run error, correct syntax, but type error, transaction can end normally
⑤ Watch command: After using watch, multi becomes invalid and transaction becomes invalid
The mechanism of WATCH is as follows: When the transaction EXEC command is executed, Redis will check the WATCH key. The WATCH key will be executed only when the WATCH key has not changed since the start of WATCH. If the WATCH key changes between the WATCH command and the EXEC command, the EXEC command returns a failure. Summary: Redis provides simple transactions and does not support transaction rollback
21, Linux SSH connection disconnected (window), how to let the process continue to execute
(1) Nohup **** &
(2) Solve the problem of automatic stop of programs or services running after Linux closes the terminal window, and solve the problem of automatic stop of programs running after Linux closes the terminal
Httpcookie client can be modified, the server can be modified. What to do
(1) Cookie is the parameter in the Http request header, which is stored in the client. The main information includes KV pair, expiration time, domain information and path information, and whether it is read-only
(2) Cookie can be tampered by the client (overwriting the old value, and the expiration time is set to 0 if deleted), so most of the non-sensitive data or login information is stored now. Of course, if the user sets the password to save, some will also be saved in the cookie. Check the user’s login status at request time.
(3) Cookie can be modified by clients and servers (set-cookie creates a new override/validity period is set to 0 to delete it), but when users know the role of cookie data, it is also dangerous to modify and try to obtain other people’s information, so sensitive cookie data should be encrypted to prevent users from parsing/enabling The modification does not take effect.
(4) Cookie size and cookie quantity limit of domain name different browsers have different restrictions, independent of protocol
23, WHAT does Http connection Time wait mean
(1) I don’t know that. I don’t have any. Can someone explain? I know the TCP connection is in TIME_WAIT state
(2) After the communication parties establish a TCP connection, the party who actively closes the connection will enter the TIME_WAIT state to receive the information that may not have been sent during the packet life cycle
(3) Quote: Look at the TCP/IP protocol suite we know, this is done to let the remnants of TCP packets disappear in the network, that is to say, if we don’t have to wait until the time to let OS release this port to other process, other processes are likely would receive a session on the remnants of the TCP packets, it will appear a series of unpredictable errors
(4) Reference:
① The HTTP client waits indefinitely to return the result
② Principle of TIME_WAIT status
③TCP three-way handshake and four-way wave
At the network layer, communication between two hosts can be realized. But this is not specific, because the actual entity that communicates is the process in the host, a process in one host exchanging data with a process in another host. Although THE IP protocol can send data packets to the destination host, it is not delivered to the specific application process of the host. End-to-end communication should be between application processes.
UDP: No connection needs to be established before data transmission, and the remote host does not need to acknowledge receiving UDP packets. Although UDP does not provide reliable delivery, it is precisely because of the savings and a lot of overhead, which makes it relatively fast. For example, some services that require high real-time performance are often used UDP. The protocols at the application layer include DNS,TFTP,DHCP,SNMP, and NFS.
TCP provides connection-oriented services. A connection must be established before data is transmitted and released after data is transmitted. Therefore, TCP is a reliable transportation service, but because of this, it inevitably adds a lot of overhead, such as validation, flow control, etc. The protocols at the application layer include SMTP,TELNET,HTTP, and FTP.
- Common familiar port number
- An overview of TCP
TCP regards the connection as the most basic object. Every TCP connection has two endpoints. This breakpoint is called a socket, and it is defined as a socket formed by concatenating the port number to an IP address. The resulting socket is 192.3.4.16:80.
- TCP header
1. Write the source port and destination port to the source port and destination port respectively.
2. Serial number, accounting for 4 bytes, each byte in the byte stream transmitted in the TCP connection is numbered sequentially. For example, if the serial number field of a packet is 301 and the data carried has 100 fields, it is obvious that the serial number of the next packet segment (if any) should start from 401.
3. The confirmation number, which is 4 bytes, is the serial number of the first data byte of the next packet expected to be received from the other party. For example, B receives the packet from A, whose serial number field is 501 and the data length is 200 bytes. This indicates that B correctly receives the data from A up to the serial number 700. Therefore, B expects the next data number from A to be 701, so B sets the confirmation number to 701 in the confirmation message segment sent to A.
4, data offset, accounting for 4 bits, it indicates how far the TCP packet data distance from the beginning of the TCP packet segment;
5, reserved, accounting for 6, reserved for future use, but should be 0;
6. Emergency URG. When URG=1, it indicates that the emergency pointer field is valid. Tell the system that there is urgent data in this message segment.
7. Confirm ACK. The confirmation number field is valid only when ACK=1. According to TCP, ACK must be set to 1 for all packets transmitted after the connection is established.
8. Push PSH. When two application processes communicate interactively, sometimes the application process at one end wants to receive the response immediately after typing a command, so PSH=1;
9. Reset RST. When RST=1, a serious error occurs in the TCP connection, and the connection must be released and re-established.
Syncing SYN, used to synchronize sequence numbers when a connection is established. If SYN=1, ACK=0, it indicates a connection request packet. If SYN=1, ACK=1 is accepted, the response packet should be SYN=1, ACK=1.
11. Terminate the FIN to release the connection. If the FIN value is 1, the sender of the packet has finished sending data and wants to release the data.
12, the window, accounting for 2 bytes, refers to the notification receiver, send this paper how much space you need to accept;
13, check and, accounting for 2 bytes, check the head and data of these two parts;
14. Emergency pointer, accounting for 2 bytes, indicates the number of bytes of emergency data in this paragraph;
15, options, variable length, define some other optional parameters.
- TCP Connection establishment (three-way handshake)
(1) THE TCP server process first creates the transfer control block TCB, always ready to accept the client process connection request, at this time the server enters the LISTEN state;
(2) The TCP client process creates the TCB and sends a connection request packet to the server. SYN=1 in the packet header and seq= X are selected. Then the TCP client process enters the SYN-sent state. According to TCP, the SYN segment (SYN=1) cannot carry data, but must consume a sequence number.
③ After receiving the request packet, the TCP server sends an acknowledgement packet if it agrees to the connection. In the acknowledgement packet, ACK=1, SYN=1, ACK= X +1, and seq= Y are initialized. Then, the TCP server process enters the SYN-RCVD state. This message also does not carry data, but again consumes a serial number.
(4) The TCP client process sends an acknowledgement to the server after receiving the acknowledgement. Confirm the ACK=1, ACK= y+1, and seq= X +1 of the packet. In this case, the TCP connection is ESTABLISHED and the client enters the ESTABLISHED state. According to TCP, AN ACK packet segment can carry data, but does not consume serial numbers if it does not.
⑤ After receiving the confirmation from the client, the server enters the ESTABLISHED state, and the two parties can communicate with each other.
- Why does the TCP client send an acknowledgement at the end?
In short, the main purpose is to prevent invalid connection request packets suddenly sent to the server, resulting in errors.
If you are using two handshake connection is established, suppose there is such a scenario, the client sends the first request connection and not lost, just because in the network node in the retention time is too long, as a result of the TCP client has not received the confirmation message, thought the server did not receive, at this time to send this message to the server, The client and server then complete the connection with two handshakes, transfer data, and close the connection. This message should be invalid. However, the two-handshake mechanism will allow the client and server to establish a connection again, which will lead to unnecessary errors and waste of resources. If the three-way handshake is used, even if the invalid packet is sent, the server receives the invalid packet and replies with an acknowledgement message, but the client does not send an acknowledgement again. Since the server does not receive an acknowledgement, it knows that the client has not requested a connection.
- TCP connection release (four waves)
After data transfer is complete, both parties can release the connection. Initially, both the client and the server are in an ESTABLISHED state, and then the client is actively shut down and the server is passively shut down.
① The client process sends a connection release packet and stops sending data. Release the header of the data packet, FIN=1, whose sequence number is SEq = U (equal to the sequence number of the last byte of the previously transmitted data plus 1). At this point, the client enters the fin-WaIT-1 state. According to TCP, FIN packets consume a sequence number even if they do not carry data.
(2) After receiving the connection release packet, the server sends an ACK=1, ACK= u+1, and carries its serial number seq= V. Then the server enters the close-wait state. The TCP server notifies the higher-level application process that the client is released from the direction of the server. This state is half-closed, that is, the client has no data to send, but if the server sends data, the client still accepts it. This state also lasts for a period of time, i.e. the duration of the close-wait state.
③ After the client receives the confirmation request from the server, it enters the fin-WaIT-2 state and waits for the server to send the connection release packet (before receiving the final data from the server).
④ After the server sends the final data, it sends a connection release packet to the client with FIN=1 and ACK = U +1. The server is in the half-closed state and probably sends more data. Assume that the serial number is SEQ = W.
⑤ After receiving the connection release packet from the server, the client must send ACK=1, ACK= w+1 and its sequence number is SEq = U +1. In this case, the client enters the time-wait state. Note that the TCP connection is not released at this time, and the client can enter the CLOSED state only after 2 ∗ * ∗MSL (maximum packet segment life) and the corresponding TCB is revoked. ⑥ The server enters the CLOSED state as soon as it receives the confirmation sent by the client. Similarly, revoking the TCB terminates the TCP connection. As you can see, the server ends the TCP connection earlier than the client.
- Why does the client end up waiting for 2MSL
MSL (Maximum Segment Lifetime) : TCP allows different implementations to set different MSL values.
First, ensure that the client sends the final ACK packet to reach the server, because the ACK packet may be lost, standing in the server’s perspective, I have already sent the FIN + ACK message request disconnect, the client also did not give me response, should be I send the request of the disconnect message it did not receive, then the server will send a again, The client receives the retransmitted message within the 2MSL period, responds with a message, and restarts the 2MSL timer.
Second, prevent “invalid connection request message segment” as mentioned in “three-way handshake” from appearing in this connection. After the client sends the last acknowledgement message, in this 2MSL time, all the message segments generated during the duration of the connection can be removed from the network. In this way, the new connection does not contain the request packets of the old connection.
Why is establishing a connection a three-way handshake, but closing a connection a four-way wave?
During connection establishment, the server receives a SYN packet in LISTEN state and sends the ACK and SYN packets to the client. And close connection, the server receives the other side of the FIN message, just said to each other can no longer send data but also receives the data, and the oneself also is not necessarily all data are sent to each other, so their can immediately shut down, also can send some data to each other, then send the FIN message now agreed to close the connection to the other side, therefore, The ACK and FIN are usually sent separately, resulting in an extra ACK.
- What if the connection has been established, but the client suddenly fails?
TCP also has a keepalive timer, so obviously if the client fails, the server can’t wait forever and waste resources. The server resets this timer every time it receives a request from the client, usually for two hours. If it does not receive any data from the client within two hours, the server sends a probe segment, which is then sent every 75 seconds. If there is no response after 10 probe packets are sent, the server assumes that the client is faulty and closes the connection.
24. Does the Integer assignment address change
Here is to write an algorithm for the convenience of passing the parameter is passing the value and passing the address to ignore, the code does not have to run.
Integer is passed as a parameter. The original variable is not modified because the address is passed. Reassigning a parameter changes the address of the variable, leaving the address unchanged
25. Algorithm: Calculate the number of least squares that make up N
- The squares are repeatable
Given a positive integer n, find several perfect squares (e.g. 1, 4, 9, 16…). So that their sum is equal to n. You want to minimize the number of perfect squares that make up the sum. Given an integer n, return the minimum number of perfect squares that sum to n. A perfect square is an integer whose value equals the square of another integer; In other words, its value is equal to the product of an integer. For example, 1, 4, 9, and 16 are perfect squares, but 3 and 11 are not.
- Example 1:
Input: n = 12 Output: 3 Description: 12 = 4 + 4 + 4
- Example 2:
Input: n = 13 Output: 2 Description: 13 = 4 + 9
- Tip:
1 <= n <= 104
Algorithm: find the deepest level of child nodes and
It’s the sum of all the children at the deepest level, not the sum of all the children.
Given the root node of a binary tree, return the sum of the deepest leaf nodes.
- Example 1:
Input: root = [1, 2, 3, 4, 5, null, 6, 7, null, null, null, null, 8] output: 15
- Example 2:
Input: root = [6,7,8,2,7,1,3,9, null, 1, 4, null, null, null, 5] output: 19
- Tip:
The number of nodes in the tree is in the range [1, 104]. 1 <= Node.val <= 100
SQL > select * from employee where salary = NTH
select salary from efm group by salary order by salary desc limit n-1, 1
Write in the last
This article ends here! In! Close to the end. Thank you for your patience (tears……) If this article helps you. Don’t forget to like, retweet, and bookmark. Your likes are my motivation to keep updating. Thank you for watching.
