A month to meet several big factories summed up the 5W word after the end face (continuous update)

Click like to see again, form a habit, wechat search [Three prince Aobin] pay attention to the tool of the Internet to ignoble life.

This article has been included on GitHub github.com/JavaFamily.

preface

Some time ago, Aobing is not in review, a lot of partners also want my review route, and some knowledge points in my own notes, well, the third party spent a month, a whole month ah, for you to sort out.

I’m going to start with a big trick okay, my review of the brain map, can be said to be all no, in order to prevent stolen pictures, I added watermark.

You will find this episode very hardcore and I will continue to update it without saying anything, for the sake of my staying up late for a month with acne, you can like it lol

I don’t know why the gold is so small, but everything else is big

Note: If the picture is compressed, you can go to the public number [Three Prince Aobin] reply [review] to get the original picture

Spring

The seven modules of the Spring framework

Spring Core: The basic part of the framework that provides an IoC container for bean management.

Spring Context: Inherits BeanFactory, provides Context information, and extends JNDI, EJB, email, internationalization, and more.

Spring DAO: Provides an abstraction layer for JDBC and also provides a declarative transaction management approach.

Spring ORM: Provides ORM mapping layers such as JPA, JDO, Hibernate, and MyBatis.

Spring AOP: Integrates all AOP capabilities

Spring Web: Provides basic Web development context information, and existing Web frameworks such as JSF, Tapestry, Structs, etc., provide integration

Spring Web MVC: Provides a full-featured implementation of Model-view-Controller for Web applications.

Beans define five scopes

Singleton Prototype Request Session Global Session

Spring IOC initialization process?

Resource location is to find user-defined bean resources, BeanDefinition can be loaded with beanDefinition by ResourceLoader using the unified Resource interface (Resource interface) BeanDefinition registration loaded into ioc (maintaining BD via HashMap) registers these BeanDefinitions with the IOC container via BeanDefinitionRegistery

BeanDefinition loading process?

Define BeanDefinitionReader parsing XML document BeanDefinitionDocumentReader parsing the document into a beanDefinition

DI dependency injection process? (Instantiate, handle dependencies between beans)

After Ioc initialization, the dependency injection process is triggered when the user first asks the Ioc container for the Bean

• If lazy-init=true, the bean will be initialized when the first getBean is created. If lazy-init=false, the bean will be initialized when the container starts.

• Call beanFactory.getBean () to generate the bean’s;

• The bean generation process uses the decorator pattern to produce beans that are BeanWrappers (enhancements of beans);

  How does dependency injection handle bean dependencies?

  When beanDefinition is loaded, if the bean has dependencies, use a placeholder instead. When getBean is called, if the placeholder is encountered, get the bean from ioc and inject it into the instance

The lifecycle of the Bean?

• Instantiate beans: The Ioc container instantiates by getting information from the BeanDefinition object, which is wrapped in a BeanWrapper object
• Set object properties (DI) : Attribute dependency injection is done through the BeanWrapper interface for setting properties.
• Inject the Aware interface (BeanFactoryAware, which can be used to get other beans, ApplicationContextAware) : Spring checks to see if the object implements the xxxAware interface and injects the relevant xxxAware instance into the bean
• BeanPostProcessor: Custom processing (pre-processing and post-processing)
• InitializingBean and init-method: Execute our own defined initialization methods
• use
• Destroy: Destroy the bean

IOC: Inversion of control: the creation rights of objects are managed by Spring. DI (dependency injection) : When Spring creates an object, it injects the properties that the object depends on into the class.

Spring’s IOC injection approach

Constructor injection setter method injection annotation injection interface injection

How do I check if a cyclic dependency exists?

The Bean can be marked when it is created, and if the recursive call comes back and it finds that it is being created, it indicates a circular dependency.

How can Spring solve the Bean cycle dependency problem?

The cyclic dependency scenarios in Spring are:

• Cyclic dependency of the constructor
• Property
• SingletonObjects: The first level of caching, which contains instantiated singletonObjects; EarlySingletonObjects: Level 2 cache containing pre-exposed singletons; SingletonFactories: a third-level cache that holds object factories for objects to be instantiated
• Spring first obtains the bean from the level 1 cache singletonObjects when creating the bean. If not, and the object is being created, then get it again from the level-level cache earlySingletonObjects, or if not, from the level-level cache singletonFactories. (After the Bean calls the constructor to instantiate, even if the properties are not yet filled, You can use the level 3 cache to pre-expose the dependent reference value (pre-exposure), which can locate the object in the heap according to the object reference, based on Java reference passing), move from level 3 cache to level 2 cache and place yourself in level 1 cache for other use after being fully initialized.
• Because adding singletonFactories to the three-level cache requires that the constructor is executed, the constructor’s cyclic dependency cannot be resolved.
• Solution to constructor cyclic dependencies: Lazy loading using the @lazy annotation in the constructor. When you inject dependencies, you inject proxy objects first, and then you create them when you first use them. This is why they are called three maps instead of three levels of cache.

What design patterns are used in Spring?

• Factory pattern: BeanFactory in Spring is the embodiment of a simple factory pattern, which obtains bean objects based on the unique identity passed in;
• Singleton pattern: provides global access to BeanFactory;
• Proxy pattern: The principle of AOP functionality uses proxy pattern (1, JDK dynamic proxy. 2, CGLib bytecode generation technology proxy.
• Decorator pattern: Dependency injection requires BeanWrapper;
• Observer Pattern: A common place for the Observer pattern in Spring is the implementation of a listener. Such as ApplicationListener.
• Policy pattern: Bean instantiation determines how the Bean instance is initialized (reflection or CGLIB dynamic bytecode generation)

Core AOP Concepts

1. Aspect: A class is an abstraction of object features, and a aspect is an abstraction of crosscutting concerns

2. Crosscutting concerns: What methods to intercept and what to do with them. These concerns are called crosscutting concerns.

Joinpoint: the point that was intercepted. Since Spring only supports join points of method types, Spring refers to the method that was intercepted. In fact, join points can also be fields or constructors.

Pointcut: A definition that intercepts join points

5. Advice: Advice refers to the code to execute after intercepting the join point. There are five types of advice: pre, post, exception, final, and surround.

6. Target object: The target object of the proxy

Weave: The process of applying an aspect to a target object and resulting in the creation of a proxy object

8. Introduction: An introduction allows you to dynamically add methods or fields to a class at runtime without changing the code.

Explain AOP

Traditional oop development code logic from the top down, the process will produce some cross-cutting issues, our main business logic relation with these problems, will be scattered in various parts of the code, difficult to maintain, aop idea is to separate the business logic and the problem of crosscutting, achieve the goal of decoupling, enhance code reusability and development efficiency;

The main application scenarios of AOP are:

• log
• Monitor performance
• Access control
• Transaction management

AOP source analysis

• @ EnableAspectJAutoProxy to container (the beanFactory) registered in a AnnotationAwareAspectJAutoProxyCreator object;

• AnnotationAwareAspectJAutoProxyCreator proxy objects created on the target object, the object, is to encapsulate the JDK and additional two technology, realize the dynamic proxy objects created (in the process of creating a proxy object will create a proxy factory first, Get all the enhancers (notification methods), inject them and the target class into the agent factory, and then use the agent factory to create the object.

• The proxy object executes the target method, obtains the interceptor chain of the target method, and uses the interceptor chain mechanism to enter each interceptor in turn for execution

  AOP Application Scenarios

  • logging
  • Transaction management
  • Thread pool shutdown and so on

Which dynamic proxies does AOP use?

• When the bean implementation has an interface or a subclass of Proxy, — JDK dynamic Proxy; There is no interface, and Spring uses CGLIB to generate proxy objects;
• The JDK dynamic Proxy mainly involves two classes in the java.lang.Reflect package: Proxy and InvocationHandler.
• Proxy uses the InvocationHandler interface (which defines crosscutting logic) to dynamically create Proxy objects for the target class.

JDK Dynamic proxy

• Bind is used to establish the relationship between the Proxy and the real object, and proxy.newProxyInstance (target) is used to generate the Proxy object
• Proxy objects invoke real object methods by reflecting the Invoke method

Dynamic proxy is different from static proxy

• Static proxy, where the.class file for the proxy class exists before the program runs;
• Dynamic proxy: Dynamically create proxy objects using reflection while the program is running.

CGLIB differs from JDK dynamic proxies

• The Jdk must provide an interface to use;
• C does not. A non-abstract class can implement dynamic proxies

SpringMVC

For springMVC process:

(1) : The user request is sent to the DispatcherServlet, the DispatcherServlet calls the HandlerMapping processor mapper;

(2) : HandlerMapping finds the corresponding processor according to XML or annotations, generates the processor object and returns it to the DispatcherServlet;

(3) : The DispatcherServlet will call the corresponding HandlerAdapter;

(4) : The HandlerAdapter calls the specific processor after adaptation to handle the request, generates ModelAndView and returns it to the DispatcherServlet

(5) : DispatcherServlet sends ModelAndView to ViewReslover for parsing and returns the generated View to DispatcherServlet;

(6) : DispatcherServlet renders the View according to the View;

->DispatcherServlet->HandlerMapping->Handler ->HandlerAdapter ->ModelAndView ->DispatcherServlet->ModelAndView->ViewReslover->View -> Return to customer

Mybatis

Mybatis principle

• SqlsessionFactoryBuilder generates sqlsessionFactory (singleton)
• The factory mode generates a SQLSession to execute SQL and control transactions
• Mybatis enables the Mapper (SQL Mapper) interface to run by dynamic proxy, that is, generating proxy objects for the interface to map the SQL query results to POJOs

SqlSessionFactory build process

• Parse and read the XML in the Configuration to create a Configuration object (singleton)
• Use Configruation class to create sqlSessionFactory (Builder mode)

Mybatis level-1 cache and level-2 cache

Level 1 cache is enabled by default and cannot be turned off.

• Level 1 cache refers to the principle of SqlSession level cache: The data structure used is a map. If two commit operations (modify, add, or delete) occur in between, the level 1 cache area in this SqlSession is cleared
• Level 2 caches are caches that can span a SqlSession. Is mapper level caching; How it works: This is implemented through CacheExecutor. A CacheExecutor is actually an Executor proxy object

Zookeeper+eureka+springcloud

SpringBoot startup process

• New springApplication object, using the mechanism of spi load applicationContextInitializer, applicationLister interface instance (the meta-inf/spring. Factories);

• The run method is used to prepare the Environment, load the applicationContext, and publish events

• Create the Spring container, refreshContext (), automate the starter configuration, load the spring.factories, and instantiate the beans

  This section describes how SpringBoot automatic configuration works

  • EnableAutoConfiguration Find the meta-INF /spring.factories (with the beans you need to create) configuration file
  • Read the spring.factories file in each starter

Spring Boot’s core annotations

The core annotation @SpringBootApplication is made up of three

• @SpringBootConfiguration: Combines @Configuration annotation to realize the function of Configuration file.
• EnableAutoConfiguration: Enables automatic configuration.
• @componentscan: Spring ComponentScan.

What are the common SpringBoot starter types

Spring-boot-starter – Web-Web and RESTful applications; Spring-boot-starter – test-unit and integration tests; Spring-boot-starter – JDBC – traditional JDBC; Spring-boot-starter -security – Use SpringSecurity for authentication and authorization. Spring-boot-starter – data-jPA – Spring Data JPA with Hibernate; Spring-boot-starter -data-rest – Use Spring Data REST to expose simple REST services

The core configuration file for Spring Boot

(1) : applica. yml is generally used to define a single Application level, if used with spring-cloud-config

(2).bootstrap.yml (loaded first) system level parameter configuration, these parameters are generally invariable

Zuul is different from Gateway

(1) zuul is a Netflix project integrated in Spring-Cloud, and Gateway is a sub-project of Spring-Cloud;

(2) zuul does not provide asynchronous support for flow control, etc., which is supported by Hystrix. Gateway provides asynchronous support, abstract load balancing, and abstract flow control. In theory, gateway is better suited to improve system throughput (but not necessarily better performance), and the final performance needs to be determined by rigorous pressure testing

(3) : The underlying implementation of both are servlets, but the Gateway has nested a layer of WebFlux framework

(4) : Zuul can be used in other microservice frameworks, and there is no internal flow limiting and load balancing; Gateway can only be used in springcloud;

Zuul principle analysis

(1) : Zuulservlet has a zuulRunner object that initializes RequestContext (which stores the requested data), RequestContext is shared by all ZuulFilters;

(2) : There is a FilterProcessor (manager of ZuulFilter) in zuulRunner, which obtains zuulFilter from filterLoader;

(3) : With these filters, the Pre-> route-> POST filters are executed by Zuulservlet. If there is an error in the execution of these filters, the error filter will be executed and the result will be returned to the client after the execution.

Gateway Principle Analysis

(1) : When the request reaches the DispatcherHandler, the DispatchHandler instantiates the HandlerMapping interface in the IOC container during initialization

(2) : Use handlerMapping to match the corresponding Route according to the request URL, and then have the corresponding filter to make the corresponding request and forward the final response back

Zookeeper Working Principle (to be found)

At the heart of Zookeeper is atomic broadcasting, a mechanism that ensures synchronization between servers. The protocol that implements this mechanism is called the Zab protocol. The Zab protocol has two modes: recovery mode and broadcast mode.

Zoo is different from eur

• Zookeeper ensures cp (consistency)
• Eureka guarantees AP
• Zoo’s registration service was disabled during the election and was not available during the election
• Each EUR node has an equal relationship. As long as there is one eur node, the service can be guaranteed. However, the queried data may not be the latest
• Zoo has the leader and follower roles, and EUR is equal to each other
• Zoo uses the half survival principle (avoid split brain) and EUR uses self-protection mechanism to solve the partitioning problem
• ZooKeeper is based on CP and does not guarantee high availability. If ZooKeeper is being selected as the master or more than half of the machines in the ZooKeeper cluster are unavailable, then the data will not be available. Eureka is based on AP, which ensures high availability, even if all the machines are down, you can still get the data from the local cache. As a registry, the configuration changes infrequently, except for releases (releasing new versions) and machine failures. CP is not appropriate for infrequent configuration changes, and AP can sacrifice consistency to ensure availability when problems occur, both returning old data and caching data. So in theory Eureka is a better place to be a registry. In the real world, most projects will probably use ZooKeeper because the cluster is not large enough and there are few cases where more than half of the machines used as the registry are down. So it’s not really a big deal.

Hystrix Principle (to be checked)

By maintaining its own thread pool, it starts service degradation when the thread pool reaches a threshold, returning fallback defaults

Why is the Hystrix fuse needed

To prevent avalanches, release resources in a timely manner, and prevent more cascade failures in the system, you need to isolate faults and delays to prevent the failure of a single dependency from affecting the entire application.

Advantages and Disadvantages of Microservices

• Each service is highly cohesive, loosely coupled, and programmed for interfaces;
• The communication cost between services, data consistency, and difficulty in operation and maintenance of multiple services are increased. The transmission efficiency of HTTP is not as good as THAT of RPC

Eureka self-protection mechanism

• Eur does not remove services that have not received a heartbeat for a long time and should expire
• New service registrations and query requests are still accepted, but are not synchronized to other nodes (high availability)
• When the network is stable, the new registration information of the current instance will be synchronized to other nodes (final consistency).

MQ contrast

ActiveMQ: A message queue product produced by Apache. It has been used for a long time, and the latest version is relatively slow to update. RabbitMQ: Erlang language development, support many protocols, very heavy weight, more suitable for enterprise development. The performance is good, but it is not conducive to secondary development and maintenance. RocketMQ: Ali open source messaging middleware, pure Java development, with high throughput, high availability, suitable for large-scale distributed system applications, distributed transactions. ZeroMQ: Claims to be the fastest message queuing system, especially for high-throughput scenarios, implemented in C. The message queue selection depends on the application, ZeroMQ is small and beautiful, RabbitMQ is large and stable, Kakfa and RocketMQ are fast and powerful

JAVA based

The difference and relation between AVL tree and red-black tree (R-B tree)

• AVL is a strictly balanced tree, so when adding or removing nodes, the number of rotations is more than the red-black tree, depending on the situation.
• Red-black trees use non-strict balance to reduce the cost of rotation times when adding or deleting nodes.
• So simply, select AVL tree for query, and select red black tree for query update times
• The AVL tree has about 20% performance advantage when it is inserted and deleted sequentially, and the red-black tree has about 15% performance advantage when it is random operation. Of course, the real application is generally random, so the red-black tree has a wider application. The index is B+ tree and the Hashmap is red-black tree

Why redis Zset uses skip lists instead of red black trees

• Skiplist has the same complexity as a red-black tree and is much simpler to implement.
• Redblack trees use the rebalance when inserting and deleting objects in a concurrent environment.

JAVA basic data types

(1 byte is 8 bits) Integer: byte (1 byte), short (2 bytes), int (4 bytes), long (8 bytes) Floating point: float (4 bytes), double (8 bytes) Boolean: Boolean (1 byte) Character: char (2 bytes)

IO and NIO

Includes classes File, outputStream, inputStream, Writer, ReaderSeralizable (5 classes 1 interface)

NIO’s three core elements selector, channel, and buffer

NIO is different from IO. IO is stream oriented, NIO is buffer oriented. I/O blocking, NIO not blocking

Exception class

< span style = “line-height: 20px;” style = “line-height: 20px;”

LVS (4 and 7 layers) principle

• It consists of a front-end virtual load balancer and a back-end real server farm.
• After the request is sent to the virtual server, the virtual server forwards the request to the real server based on the packet forwarding policy and load balancing algorithm
• Layer 4 (LVS, F5) is IP+ port-based load balancing. Layer 7 (NGINX) is load balancing based on APPLICATION-layer information such as URL

The StringBuilder and StringBuffer

• The StringBuilder faster;
• StringBuffer is thread safe

Interrupt/isInterrupted/interrupt

• The thread calling this method is in a state that will be set to “interrupt” (set operation)
• Isinterrupted () is whether the interrupt signal is true or false (the get operation) for the thread corresponding to the thread object calling this method. For example, we can call the isInterrupted method in thread A and look at thread A
• Interrupted () is a static method: the internal implementation is called isInterrupted () for the current thread and resets the interrupted status of the current thread (getAndSet)

Sleep is different from wait

Sleep belongs to thread class, wait belongs to object class; Sleep does not release locks

The difference between CountDownLatch and CyclicBarrier

• Con is used when the main thread waits for all subthread tasks to complete before executing. Cyc is used when a group of threads wait for each other to reach a certain state before executing simultaneously.
• CountDownLatch is not reusable, CyclicBarrier is

Terminating thread method

• Use the exit flag to say the thread exits normally.
• Stopping the use of the String pool as the lock Object by declaring this.Interrupted () throw new InterruptedException () will result in two threads holding the same lock, while the other thread does not execute and uses something else such as new Object ()

The principle and application of ThreadLocal

Principle:

The thread creates a copy and accesses its own internal copy, which is a member variable called ThreadLocalMap called threadLocals, where key is itself and value is a copy of the variable where the actual value is stored

Application:

• To solve the database connection, store the connection object, different threads store their sessions;
• Solve the thread safety problem of simpleDateFormat;
• Memory leaks will occur, explicitly remove.. Don’t work with thread pools, because workers tend not to exit;

ThreadLocal memory leak problem

For a strong reference, set TL =null, but the reference to the key will still point to the ThreadLocal object, so there will be a memory leak, whereas a weak reference will not. However, there is still a memory leak. The ThreadLocal is reclaimed and the key becomes null, causing the entire value to be unreachable. Solution: At the end of use, call threadLocal. remove to free the reference to its value;

What if we wanted to get the ThreadLocal value of the parent thread

ThreadLocal is not inherited, so it is not retrievable, but we can do this using InteritableThreadLocal. InteritableThreadLocal inherits from ThreadLocal, overrides createdMap, and uses interitableThreadLocals instead of threadLocals.

This variable, when the thread is initialized (init method), determines if the parent’s interitableThreadLocals variable is null, and if not, puts it in the child, but it doesn’t really matter. After the parent creates the child, If changes to the parent thread are not synchronized to the child thread… Similarly, it is useless to assign a value to a child thread after it has been created

Thread state

Thread pools come in five states: running, haggis, stop, Tidying, and TERMINATED.

• Running: The thread pool is in the running state and can accept and execute tasks. This is the default state for creating threads

• Haggis: Calls to haggis () do not accept new tasks, but slowly work through the accumulated ones.

• Stop: calls the showdownnow () function, does not accept new tasks, does not process existing tasks, will interrupt existing tasks.

• Tidying: Becomes Tidying when the thread pool status is on or stop and the number of tasks is 0. The hook function terminated () is called at this time.

• TERMINATED: execution completed.

In the thread pool, an atomic class is used to record the information about the thread pool, with the top 3 bits of int representing the state and the next 29 bits representing the number of threads in the pool.

How are thread pools implemented in Java?

• The thread in the thread is abstracted as the static inner class Worker, which is implemented based on AQS and stored in HashSet;
• The thread to be executed is stored in BlockingQueue;
• The basic idea is to extract tasks to be executed from the workQueue and put them in the worker.

What happens if a thread in the thread pool runs with an exception

If the task is submitted using submit, the returned feature will contain exception information, but if execute, the exception stack will be printed. But it does not affect other threads. The thread pool then deletes the thread and adds a new worker.

Thread pool Principle

• When the number of viable core threads in the pool is less than the corePoolSize, the pool creates a core thread to process the submitted task
• If the number of core threads in the thread pool is full, that is, the number of threads is equal to the corePoolSize, a new submitted task will be placed in the workQueue for execution.
• When the number of threads in the thread pool is equal to the corePoolSize and the workQueue is full, determine whether the number of threads has reached maximumPoolSize, that is, whether the maximum number of threads is full. If not, create non-core threads to execute the submitted tasks.
• If the current number of threads reaches maximumPoolSize and new tasks come along, the rejection policy is applied directly.

Rejection policies

• AbortPolicy directly throws an exception to prevent the thread from running;

• CallerRunsPolicy If the discarded thread task is not closed, execute the thread;

• DiscardOldestPolicy Indicates that the earliest thread in the queue attempts to submit the current task

• DiscardPolicy Discards the current task

NewFixedThreadPool (thread pool with fixed number of threads)

• The blocking queue is an unbounded queue LinkedBlockingQueue
• Suitable for CPU intensive tasks, suitable for long-term tasks

NewCachedThreadPool (pool of cacheable threads)

• The blocking queue is SynchronousQueue
• Suitable for executing a large number of short – term small tasks concurrently

NewSingleThreadExecutor (single-threaded thread pool)

• The blocking queue is LinkedBlockingQueue
• It is applicable to the scenario where tasks are executed in serial, task by task

NewScheduledThreadPool (timed and periodically executed thread pool)

• The blocking queue is DelayedWorkQueue
• A scenario where tasks are executed periodically and the number of threads needs to be limited

Java related lock

Synchronized implementation principle

ContentionList (queue of thread requesting lock) entryList (queue of qualified candidates) waitSet (queue blocking after wait) onDeck (contentionList) ower (contention to lock thread)! Ower (state after successfully releasing the lock); Synchronized is an unfair lock.

Synchronized attempts to spin the lock when a thread enters the ContentionList. If it fails to acquire the lock, it enters the ContentionList, which is unfair to threads already queued. Another unfair thing is that the thread that spin acquired the lock may also directly preempt the lock resource of the OnDeck thread.

The underlying layer is implemented by a pair of Monitorenter and MonitoreXit directives (monitor lock)

Each object has a monitor lock. When the monitor is occupied, it is locked, and the thread attempts to acquire ownership of the monitor when it executes the Monitorenter instruction:

• If the entry count of monitor is 0, the thread enters monitor, sets the entry count to 1, and the thread is the owner of Monitor.
• If a thread already owns the monitor and just re-enters, the number of entries into the monitor is increased by one.
• If another thread has already occupied Monitor, the thread blocks until the number of monitor entries is zero, and then tries again to acquire monitor ownership.

How does ReentrantLock achieve reentrancy?

Internal custom synchronizer Sync, lock through the CAS algorithm, the thread object into a two-way list, each time to obtain the lock, look at the current maintenance of the thread ID and the current request thread ID is the same, the same can reentrant;

How to avoid deadlock with ReentrantLock?

• Response interrupt lockInterruptibly ()
• Polling lock tryLock ()
• TryLock (long time)

The difference between tryLock and Lock and lockInterruptibly

(1) : tryLock returns true if tryLock is acquired, false if tryLock is not,

TryLock (long timeout, TimeUnit unit) : tryLock (long timeout, TimeUnit unit) : tryLock (long timeout, TimeUnit unit) : tryLock (long timeout, TimeUnit unit)

(3) : return true if lock can be acquired, or wait until the lock is acquired

(4) : Lock and lockInterruptibly. If two threads execute the methods separately, but interrupt both threads at this point, Lock will not throw an exception, while lockInterruptibly will throw an exception.

What is the difference between CountDownLatch and CyclicBarrier

CountDownLatch is used only once while waiting for another thread to reach a certain point before continuing to execute logic (the child thread will not be blocked, but will continue to execute). The most common is the join form, the main thread to wait for the child thread to complete the task, in the main thread to get the result of the way (of course not), the internal is to use the counter subtracting implementation (yes, it is AQS), AQS state assume the function of the counter, initialize, use CAS assignment, The main thread calling await () is added to the shared thread waiting queue, and the child thread calling countDown uses spin to subtract 1 until it reaches 0.

The CyclicBarrier is basically waiting for a group of threads to end up in the same state, and then release the brake. CyclicBarrier can also be passed a Runnable object that can be executed when the CyclicBarrier is released. CyclicBarrier is loopable. When calling await, it resets the state if count becomes 0. To reset the state, CyclicBarrier adds a field called parties, which holds the initial value, and reassigns it when count becomes 0. Another difference is that CyclicBarrier is not AQS based, but RentrantLock based implementation. Wait queues are stored in the form of condition variables.

Synchronized is different from ReentrantLock

• Both are reentrant locks; R is to show acquire and release locks, s is implicit;
• R is more flexible and can tell whether or not the lock has been successfully acquired. You can define read and write locks at the API level and s at the JVM level.
• R can define fair locking; Lock is the interface and S is the keyword in Java

What is a Semaphore

A semaphore is a concurrency toolkit with fixed resource limits, based on an AQS implementation, that is set at construction time to a value that represents the number of resources. The semaphore is used to control the number of concurrent threads (druid’s database connection number). The semaphore is also divided into fair and unfair cases. The basic way is similar to reentrantLock, when a task is called to request a resource, the semaphore will be spin-reduced by 1. If it succeeds, it succeeds. If it fails, the number of resources becomes 0, and it will be added to the queue to wait. When release is called, it increments, replenishes resources, and wakes up the wait queue.

Semaphore application

• Acquire () release () can be used for building object pools, resource pools, such as static global object pools, database connection pools;
• Can create S with a count of 1 as a mutex (binary semaphore)

Reentrant locking concept

(1) : a reentrant lock means that the same thread can acquire the same lock multiple times without blocking because it has been acquired before and has not been released;

(2) both reentrantLock and synchronized are reentrantLock

(3) : One of the advantages of reentrant locks is that deadlocks can be avoided to some extent

ReentrantLock Principles (CAS+AQS)

CAS+AQS queue to implement

(1) : First try to acquire the lock through CAS. If a thread has occupied the lock by this time, it will join the AQS queue and be suspended;

(2) : When the lock is released, the thread at the top of the queue will be awakened by CAS to try to acquire the lock again.

(3) : If the lock is not fair, and there is another thread trying to acquire the lock may let this thread grab the lock;

(4) : If it is a fair lock, it will be queued to the end of the queue, and the first thread of the queue will get the lock.

AQS principle

A synchronization queue consisting of the internal classes of a Node, which is a two-way linked list structure. The state of a lock is determined by controlling (volatile int) the state of the lock. If the non-reentrant state is not 0, the lock is blocked.

If the reentrant lock is 0, then the lock is executed. If the lock is not 0, then the current thread is the thread that acquired the lock. If the reentrant lock is 5 times, then state=5. When releasing a lock, it also needs to be released 5 times until state=0 before any other thread is eligible to acquire the lock

AQS Two resource sharing methods

• Exclusive: Exclusive, which only one thread can execute, such as ReentrantLock

• Share: Multiple threads can execute Semaphore, CountDownLatch, ReadWriteLock, and CyclicBarrier simultaneously

The CAS theory

The memory value V, the old expected value A, the new value B to be modified, when A=V, change the memory value to B, otherwise do nothing;

Disadvantages of CAS:

(1) ABA problems; (2) : If CAS fails, spin will put pressure on the CPU; (3) : atomicity of only one variable can be guaranteed, i++ is not guaranteed

Application of CAS in Java:

(1) : Atomic series

Fair lock and sub-fair lock

(1) : Fair lock means to check whether there are threads waiting in a queue to acquire the lock before allocating the lock. The thread with the longest queue time is allocated preferentially, and it is not fair to directly attempt to acquire the lock. (2) : Fair lock requires maintenance of one more thread queue, which is inefficient; Default unfairness

Exclusive and shared locks

(1) : ReentrantLock is an exclusive lock (pessimistic locking policy). (2) : The read lock in ReentrantReadWriteLock is a shared lock. (3) : JDK1.8 StampedLock. In this way, we do not match the data may be read, so need a little extra code to determine whether there is in the process of reading writing, the reading is a kind of optimistic locking, lock optimistic locking concurrency is more efficient, but once there’s a small chance to lead to read data inconsistencies, need to be able to detect, read it again

There are four lock states

• unlocked

• When a thread accesses a synchronized block of code to acquire the lock, the thread ID of the lock bias is stored in the object header and the stack frame record. When the thread enters the synchronized block again, no CAS operation is required to lock. If the bias lock is not enabled, the new object will be an ordinary object (i.e. no lock, if there is a slight race, it will become a lightweight lock). If the bias lock is enabled, the new object will be an anonymous bias (bias lock). Mark Word (a marker field that stores the runtime data of the object itself), class Pointer (a Pointer to the metadata of the object’s class)

• Lightweight lock (spin-lock) (1) : Allow a thread to spin and wait for a period of time before blocking. It may be that another thread has been unlocked during the waiting period. In this case, there is no need for the thread to perform the blocking operation, avoiding switching from user mode to kernel mode. (Adaptive spin time is the time for a thread context switch)

• (2) : Deadlock may be caused when using spin lock, and deadlock may be caused when recursion call

• (3) : The underlying spin Lock is achieved by pointing to the Lock Record in the thread stack

• Heavyweight lock

The difference between lightweight and biased locking

(1) : Lightweight locking uses CAS to avoid entering costly mutually exclusive operations

(2) : Bias locking completely eliminates synchronization in a non-competitive scenario, even CAS is not executed

The spin lock is upgraded to the heavyweight lock condition

(1) : a thread spins more than 10 times;

(2) : The number of waiting spin threads exceeds half of the number of system cores;

Read and write lock understand, know how to implement read and write lock

Commonly used read-write lock ReentrantReanWritelock, this actually and already a similar, is also based on AQS, but this is based on the Shared resources, not mutually exclusive, the key lies in the processing of the state, the read-write lock high 16. Remember to read status, low 16 for a state, parted, The read case is read lock reentrant, read/write read/write is mutually exclusive, just judge the lower 16 bits.

Zookeeper implements distributed locks

(1) : The node name is unique. When the node is locked, all clients create the node together. Only one successful creation obtains the lock, and the node is deleted when the node is unlocked.

(2) : Using temporary sequential nodes, all clients create temporary sequential nodes when locking, create the lock with the smallest node sequence number, otherwise monitor nodes smaller than their own sequence number to wait

(3) : The advantage of solution 2 to 1 is that when ZooKeeper breaks down, temporary sequential nodes automatically delete and release locks, which does not cause lock wait.

(4) : Option 1 creates a stampede effect (when there are many processes waiting for the lock, many processes will come to fight for the lock when it is released).

(5) : Performance is not as good as redis lock due to frequent creation and deletion of nodes

Volatile variables

(1) : visibility of variables

(2) : prevent reordering of instructions

(3) : the atomicity of single read and write operations of variables is guaranteed, but the atomicity of i++ operation cannot be guaranteed, because the essence is two read and write operations

How does volatile ensure visibility between threads and prevent reordering of instructions

Volatile visibility is guaranteed by the atomicity of instructions. The JMM defines eight types of atomicity instructions, such as write, Store, read, and Load. Volatile, on the other hand, requires write-store and load-read operations to be atomic. This ensures that all reads are read from main memory and that all writes are synchronized to main memory (which is exactly a memory barrier). Reordering of instructions is guaranteed by a memory barrier.

• One is the compiler barrier: it prevents the compiler from reordering and ensures that instructions before the optimization barrier are not executed after the optimization barrier when compiling the program.
• The second is the CPU barrier: sfence guarantees writes, lfence guarantees reads, and lock acts like a lock. Java executes an extra “load addl $0x0, (%esp)” operation, which is equivalent to a lock instruction, adding a full memory barrier instruction.

JVM

Relationship between JRE, JDK, and JVM:

The JDK is the smallest development environment, consisting of jre++ Java tools.

The JRE is the minimal environment in which Java runs and consists of the JVM + core class libraries.

The JVM is the virtual machine, the container in which Java bytecode runs. If you have only a JVM, you cannot run Java because of the lack of core class libraries.

JVM memory Model

(1) : heap < object, static variable, shared

Java8 has removed PermGen and replaced it with Metaspace. Java8 has removed PermGen and replaced it with Metaspace

(3) : the virtual machine stack < < the thread executes the method when the internal store local variables will store the heap object address and so on data >

(4) : local method stack < store the local variable table of various native methods and other information >

(5) : program counter < record which byte code instruction position the current thread executes to >

Object has four kinds of references

(1) : strong (main cause of memory leak)

(2) : soft (only soft reference, insufficient space will be reclaimed), suitable for caching

(3) : weak (only, GC will recycle)

(4) : Virtual references (used to track GC status) are used to manage out-of-heap memory

The composition of an object:

An object is divided into three areas: object header, instance data, and alignment population

Object header: it contains two parts: 1. It stores its own runtime data such as hash code, generational age, and lock flag (but not absolute). Is not fixed.) 2. Metadata pointer to class. It is also possible that there is a third section, the array type, with an extra block to record the length of the array (since the length of an array is not determined by the JVM, the JVM only has metadata information).

Instance data: will be determined according to the vm allocation policy. In the allocation policy, all types of the same size will be grouped together in the order in which they are defined (including variables of the parent class).

Aligned padding: This doesn’t make a lot of sense, mainly in the virtual machine specification that objects must be 8-byte integers, so when objects don’t meet this condition, placeholders are filled

If an object is alive:

Generally, there are two algorithms to judge whether an object is alive, one is reference counting, the other is reachability analysis. In Java, it’s mostly the second

Java performs reachability analysis based on:

According to GC ROOTS. GC ROOTS can include: reference objects in the virtual machine stack, class variables in the method area, constant references in the method area, and object references in the local method stack.

JVM class loading order

(1) : Load the binary byte stream of the acquisition class and convert its static storage structure into the runtime data structure of the method area

(2) : verification file format verification, metadata verification, bytecode verification, symbol reference verification

(3) : Prepare to allocate memory for class static variables in the method area and set default initial values for the class variable data types, excluding instance variables, which will be allocated to the Java heap along with the object when it is instantiated

(4) : Resolve the process of replacing symbolic references in the constant pool with direct references

(5) : Assign the correct initial values to static variables that are initialized to the class (values that are explicitly assigned in Java code)

The JVM has three kinds of loaders

(1) : Start the class loader (home) to load the JVM core class libraries, such as Java.lang.*, etc

(2) : extension class loader (ext), parent is the start class loader, from jre/lib/ext load class library

(3) : Application class loader (user classpath path) the parent loader is an extension class loader, which loads classes from environment variables

Parent delegation mechanism

(1) : The classloader receives the class loading request

(2) : delegate the request to the parent loader and up until the class loader is started

(3) : The initiator loader checks whether it can be loaded, and if it can be loaded (end); Otherwise, an exception is thrown to tell the child loader to load

(4) : ensure the uniqueness and security of the class and ensure the priority of the JDK core class loading

What does the parent delegate model do:

Ensure that the Java base Class remains the same in different environments to avoid security problems caused by overwriting the base Class by a custom Class. You can also avoid repeated loading of classes.

How to break the parent delegation model?

(1) : custom ClassLoader, inherit ClassLoader class override loadClass method;

(2) : SPI

How Tomcat breaks the parent delegate model:

Tomcat has its own particularity. It needs to accommodate multiple applications, achieve application-level isolation, and reduce repetitive loading, so it is divided into: /common container and application shared class information,/ Server container itself class information,/ share application common class information,/WEB-INF/lib application level class information. The whole can be divided into: BoostrapClassLoader ->ExtensionClassLoader->ApplicationClassLoader->CommonClassLoader->CatalinaClassLoader (loader of the container itself) /Shar EClassLoader (shared) ->WebAppClassLoader. Although the parent delegate model is satisfied at first glance, it is not, because the parent delegate model is submitted to the parent class first for loading, and Tomcat is the first to determine whether it is responsible for the location of the file to load.

SPI: (Service Provider Interface)

(1) : Service provision interface (service discovery mechanism) :

(2) : Automatically load the classes defined in the file by loading META_INF/services in the ClassPath

(3) : through the ServiceLoader. The load/Service. Through reflection method will get the implementation class instance

SPI application?

(1) : the application of the JDBC database driver connection process is the application of this mechanism

(2) : The earliest Common-logging provided by Apache only has an interface. Not implemented.. Providers that discover logs use SPI to specifically find log provider implementation classes

Defects in parent delegation?

(1) : The core of parent delegate is that the more basic classes are loaded by the higher level loader, the basic class is always called as the API called by the calling code, it is impossible to realize the basic class to call the user’s code… .

(2) : JNDI service its code is loaded by the startup class loader, but it needs to adjust the application implemented by the independent vendor, how to solve? The Thread Context ClassLoader is used by the JNDI service to load the required SPI code. That is, the parent class loader requests the child class loader to complete the class loading action. All SPI loading actions in Java are basically in this way, such as JNDI, JDBC

Causes of fullGC

(1) : Lack of space in the old age

(2) : the space of permanent generation (method area) is insufficient

(3) : explicitly call system.gc ()

Advantages and disadvantages of out-of-heap memory

Some versions of Ehcache, various NIO frameworks, Dubbo, Memcache, etc., use bytebuffers in NIO packages to create out-of-heap memory, which is memory that is not controlled by the JVM.

There are several advantages over in-heap memory:

Reduced garbage collection because garbage collection suspends other work. It speeds up the replication. Because heap internal flush to remote is first copied to direct memory (non-heap memory) and then sent; Out-of-heap memory eliminates the copying. Can be expanded to larger memory Spaces. Like more than a terabyte or more than main memory.

The disadvantages are summarized as follows:

-xx: MaxDirectMemerySize specifies that when the threshold is reached, system.gc is called to perform a full GC. Generally simple objects or flat is more suitable for jSTAT to view the overview of memory reclamation, real-time view of the allocation of each partition reclamation, JMAP to view the memory stack, view the size of the object in memory, jStack to view the thread stack, deadlock, performance bottlenecks

The JVM has seven garbage collectors

(1) : Serial collector replication algorithm, single thread, Generation)

(2) : ParNew Collector (copy algorithm, multithreading, Generation)

(3) : Parallel Scavenge (multithreaded, copy algorithm, Generation, high throughput)

(4) : Serial Old collector (mark-sorting algorithm, Old age)

Be insane. Jdk8 adopts the application of the Parallel Scavenge + Parallel Old collector. Be insane.

(6) : CMS collector (mark-clean algorithm, old, garbage collection thread can almost do at the same time with the user thread, low throughput, memory fragmentation) at the expense of throughput to obtain the shortest collection stop time -XX: Be insane. +UseConcMarkSweepGC JDk1.8 default garbage collector Parallel Scavenge +Parallel Old JDK1.9 default collector G1

Usage scenario:

(1) : Applications are sensitive to pauses

(2) : IN the JVM, having a relatively large number of longer-lived objects (older objects) is more suitable for using CMS

CMS garbage collection process:

(1) : initial identifier < find gcroot (STW) >

GC Roots are as follows:

1: Objects loaded by the system class loader

2: Indicates the thread in the active state

3: Objects in the JNI stack

4: various lock objects being used for synchronization

5: Objects held by the JVM itself, such as system class loaders, etc.

(2) : concurrent marking (three-color marking algorithm) The three-color marking algorithm processes the change of object reference in concurrent marking: black: self + child object marking completion gray: self completion, child object unfinished white: no marking; Incremental update (incremental update) : incremental update (incremental update) : incremental update (incremental update) : incremental update (incremental update) : incremental update (incremental update) : incremental update (incremental update) : incremental update So CMS must remark again (after jdk1.9)

G1 solution:

SATB (Snapshot at the begining) puts white on the stack, and the marking process is run concurrently with the application (no stop-the World is required). This method causes some garbage objects to be considered alive, so G1 makes the memory used more than it actually needs. The next time we collected the ZGC processing: color Pointers 2*42 squared =4T

(3) : Re-marking (STW)

(4) Concurrent clearing

Note: Re-marking prevents the object from being referenced after being marked as garbage

(5) : G1 collector (young + old, good performance in multi-cpu and large memory scenarios) G1 is the default garbage collector in java9, which is the replacement of CMS logical generation. Using the idea of regioning (2048 by default), STW provides a garbage collector to solve the problem of CMS algorithm generating space debris HotSpot, which is enabled by -xx: +UseG1GC

There are three modes garbage collection modes provided in G1

(1) : Young GC (when the Eden region is exhausted and cannot allocate memory)

(2) : Mixed GC (triggered when the old age size as a percentage of the total heap size reaches this threshold)

(3) : Full GC (object memory allocation speed is too fast, mixed GC will not be able to collect, and the old memory will be full)

(8) : ZGC and Shenandoah (Oracle production charges) no STW

Arthas monitoring tool

(1) : Dashboard command to view the overall RUNNING status of the JVM

(2) : JVM Displays detailed information about the JVM

(3) : thread Displays information about all threads in the JVM (similar to jstack)

(4) : sc * display all classes (search class)

(5) : Trace tracing method

Location frequent full GC, heap memory full oom

Step 1: the JPS acquisition process The second step: jmap – histo pid | head – 20 that have an object in the continuously create note: jmap if online server heap memory, particularly large, getting stuck to heap archived (usually said pressure measured in the test environment, export archived) – XX: + HeapDumpOnOutOfMemoryError or jmap – dumpLformat = b, File = XXX PID Roll out the file for analysis (Arthas did not implement the jmap command) heapdump –live/XXX /xx.hprof Export file

G1 Garbage Collector (Emphasis)

Collection process (1) : Young GC (Young generation collection) — When the Eden zone of the young generation is exhausted — STW First stage, the root is scanned. The root is the second phase of updating RS (Remembered Sets) by objects to which static variables point, local variables in the chain of method calls being executed, and so on. Process cards in the dirty Card queue and update RS. After this phase is complete, RS can accurately reflect the old reference to the object in the memory segment. Identify objects in Eden that are pointed to by older objects. These objects in Eden that are pointed to are considered to be living objects. The fourth stage is to copy the object. In this stage, the object tree is traversed, and the surviving objects in the memory segment of the Eden region are copied to the hollow memory segment of the Survivor region. In the fifth stage, references are processed. Handles Soft, Weak, Phantom, Final, JNI Weak and other references.

When The heap memory usage reaches a certain value (45% by default), there is no need for stop-the-world and a young GC is performed before concrruent marking

The end of the concurrent marking process is followed by a mixed gc process. Mixed recycling means that the young and the old are recycled at the same time

(4) : Full GC? Full GC means that the preceding method does not work properly. G1 stops application execution and uses the single-threaded memory collection algorithm for garbage collection, resulting in poor performance and long application pause time. To avoid Full GC, you need to adjust it once it happens.

When does Full GC happen?

For example, if the heap memory is too small, G1 will fall back to full GC when no empty memory segments are available to copy the living objects. This situation can be solved by increasing the memory

Although G1 heap memory is still generational, memory in the same generation is no longer a contiguous memory structure

The young generation is divided into Eden and Survivor, while the Old generation is divided into Old and Humongous

The newly allocated object will be allocated to the memory segment in Eden

The Humongous Region is used to hold large objects, if an object occupies more than half of a segmented Region;

If the size of an object exceeds the size of one or even several segments, the object is allocated across multiple Humongous segments that are physically contiguous.

Humongous objects take up a lot of memory and are continuously reclaimed first

In order to reclaim a single memory segment without scanning for objects in the entire heap memory (objects in a single memory segment may be referenced by objects in other memory segments), RS data structures are introduced. RS improves performance by allowing G1 to recycle younger generations without having to scan older objects. Each memory segment corresponds to an RS that holds references to that segment from objects in other segments

The JVM records and processes each application reference assignment statement object.field= Object, updating the reference to the RS. But this RS update is not real time. G1 maintains a Dirty Card Queue

So why not update RS directly at the reference assignment statement?

This is for the sake of performance, which is much better with queues.

TLAB: Thread Local Allocation Buffer?

Since heap memory is shared by the application, multiple threads of the application need to lock for synchronization when allocating memory. To avoid locking and improve performance, each application thread is assigned a TLAB. The memory in TLAB comes from memory segments in the young generation of G1. When the object is not a Humongous object and TLAB can be installed, the object is assigned first to the TLAB of the thread that created it. This allocation is fast, because TLAB is threaded, so no locks are needed

PLAB： Promotion Thread Local Allocation Buffer

G1 copies (” promotes “) the objects in Eden to Survivor, and the objects in Survivor to Old during the young generation collection. The collection process in G1 is multithreaded. To prevent multiple threads from copying to the same memory segment, the replication process also needs to be locked. To avoid locking, each thread in G1 is associated with a PLAB, which eliminates the need for locking

OOM problem locating method

(1) : jMAP-heap 10765 as shown in the figure above, you can view the allocation size and usage of the new generation and old generation heap memory;

(2) : jstat to view the GC collection

(3) : jmap-dump: live, format=b, file= local

(4) : Open the analysis through the MAT tool

DUBBO

Dubbo process

(1) : The Provider starts and registers with the Register

(2) : The Consumer subscribes, and the registry notifies the Consumer

(3) : Consumers consume from producers

(4) : Monitor statistics of producers and consumers

What serialization frameworks does Dubbo recommend to use, and what else?

Hessian serialization is recommended, as well as Duddo, FastJson, and Java’s own serialization

What communication framework does Dubbo use by default, and what else?

The default use of the Netty framework is also recommended, and the content also includes Mina and Grizzly.

What are the load balancing policies available in Dubbo and which is the default?

(1) : random call < default >

(2) : weight polling

(3) : Minimum number of active

(4) : Consistency Hash

RPC process

(1) The consumer invokes the service to be consumed,

(2) : The client stub serializes the method, input parameter and other information to the server stub

(3) : The deserialization operation of the server stub calls the local service for related processing according to the decoding result

(4) : The local service executes the specific business logic and returns the processing result to the server stub

(5) : Serialization of server stub

(6) : deserialization of client stub

(7) : The service consumer gets the final result

Implementation of RPC Framework The goal of the IMPLEMENTATION of THE PC framework is to wrap the process of invocation, encoding/decoding so that users feel like they are calling a remote service as if they were calling a local service

Service exposure, Service Reference, Service invocation (TODO)

Redis

Why is redis single threading so fast?

(1) : pure memory operation, avoid a large number of database access, reduce the direct reading of disk data, Redis will store data in memory, read and write data are not limited by the disk I/O speed, so the speed is fast

(2) : Single-threaded operation, avoid unnecessary context switch and race conditions, there is no multi-process or multi-threaded switchover caused by CPU consumption, do not need to consider various lock problems, there is no lock release operation, no performance consumption due to possible deadlock

(3) : Non-blocking I/O multiplexing mechanism is adopted

Underlying implementation of Redis data structure

String:

(1) The data structure of Simple Dynamic String (SDS)

struct sdshdr{

 // Record the number of bytes used in the buf array

 // is equal to the length of the SDS stored string

 intLen.

 // Record the number of unused bytes in the buf array

 int free;

 // An array of bytes to hold strings

 charBuf [];

}

Copy the code

Its advantages: (1) there is no memory overflow problem caused by string changes

(2) The time complexity of obtaining the string length is 1

(3) Space is pre-allocated. Lazy space releases the free field. By default, sufficient space is reserved to prevent multiple memory reallocation

Application scenario: String Cache structure user information, count

Hash:

Based on the array + linked list, some rehash optimization is carried out; 1.Reids Hash uses the chip-address method to handle collisions, and it does not use red-black tree optimization.

2. Hash table node adopts single linked list structure.

3. Rehash optimization (using the idea of divide and conquer to divide the huge migration workload into each CURD to avoid service busywork)

Application scenario: You can partially obtain the structure information without serializing all the fields

The List:

Application scenarios: (1) : Such as Twitter list, fan list and so on can use the Redis list structure to achieve

(2) : The implementation of list as a two-way linked list, that is, can support reverse lookup and traversal

Set:

The internal implementation is a HashMap with a value of null, which actually computes the hash to quickly rearrange, which is why sets provide the ability to determine whether a member is in a set. Application scenario: Deduplicated scenario, intersection (Sinter), union (Sunion), difference set (Sdiff), such as common concerns, common preferences, second-degree friends and other functions

Zset:

Internal use of HashMap and SkipList to ensure the storage and order of data, HashMap is the member to score mapping, while the SkipList is stored in all members, sorting is based on the score stored in HashMap, using the structure of the SkipList can obtain high lookup efficiency. And in the implementation is relatively simple. Hop table: Each node maintains multiple Pointers to other nodes so that nodes can be quickly accessed. Application scenario: Implement delay queue

Redis transactions

(1) : Multi starts transactions

(2) : Exec executes the commands within the transaction block

(3) : Discard cancellations

(4) : Watch monitors one or more keys. If the key is changed before transaction execution, the transaction will be interrupted

Implementation characteristics of Redis transactions

(1) : All commands will be executed sequentially and serialized. During transaction execution, Redis will not provide any service for other client requests, thus ensuring that all commands in the transaction are executed atomically.

(2) : If a command fails to execute in a Redis transaction, subsequent commands will still be executed

(3) : Before the transaction is started, if there is a communication failure between the client and the server and the network is disconnected, all subsequent statements to be executed will not be executed by the server. However, if the network interrupt event occurs after the client executes the EXEC command, then all commands in the transaction are executed by the server

(4) : When the append-only mode is used, Redis will call the system function write to write all the write operations in the transaction to disk in this call.

However, if a system crash occurs during the write process, such as a power failure, then only part of the data may be written to the disk and the rest may be lost.

The Redis server performs a series of necessary consistency checks upon restart, and if a similar problem is found, it exits immediately with an error message. At this point, it’s time to take advantage of the Redis toolkit’s Redis-check-Aof tool, which can help you locate data inconsistencies and roll back parts of the data that have been written. After the fix we were able to restart the Redis server again

Redis synchronization mechanism?

(1) : full copy, 1. When the slave is first started, it connects to the Master and sends the PSYNC command.

2. The master will run the bgsave command to generate the RDB file, and all write commands during this period will be written to the buffer.

3. After master BGsave is executed, RDB files are sent to slaveCopy the code
    


After receiving the RDB file, the slave discards all the old data and starts loading the RDB file


After RDB file synchronization is complete, the slave executes all write commands sent from the master buffer.


     

Every time the master executes a write command thereafter, it sends the same write command to the slave.
    

(2) : If an incremental copy occurs, such as intermittent network disconnection or command loss, the secondary node saves the offset of its replication and the running ID of the primary node

7. The primary node sends the data in the replication backlog buffer to the secondary node based on the offset to ensure that the primary/secondary replication is in a normal state.

   Redis cluster mode performance optimization

   (1) Master is best not to do any persistence work, such as RDB memory snapshots and AOF log files

   (2) If the data is important and AOF is enabled for a Slave, the policy is set to synchronize data once per second

   (3) For the speed of master-slave replication and the stability of connection, Master and Slave should be on the same LAN

   (4) Try to avoid adding slave libraries to the stressed master database

   Master < -slave1 < -slave2 < -slave3… Master < -slave1 < -slave2 < -slave3… Such a structure is convenient to solve the single point of failure and realize the replacement of Slave to Master. If the Master fails, you can immediately enable Slave1 to be the Master.

   Redis cluster solution

   (1) : indicates the official cluster scheme

   (2) : twemProxy

   The twEMProxy solution is a single point and can easily put a lot of pressure on it, so keepalived is often used to implement the high availability of TwemProy

   (3) : CODIS is sharded based on the client

Cluster is unavailable

(1) : The master fails and the current master has no slave

(2) : More than half of the master clusters fail, regardless of whether any slave clusters enter the Fail state

Redis is best suited to the scene

(1) : Session cache

(2) : Leaderboard/counter ZRANGE

(3) publish/subscribe

Cache elimination strategy

(1) : First-in, First-out algorithm (FIFO)

(2) : Least Frequently Used (LFU)

(3) : Least Recently Used (LRU)

LRU is efficient when hot spot data is present, but occasional, periodic batch operations can result in a sharp drop in LRU hit ratio and severe cache contamination

Redis Expiring key deletion policy

(1) : Lazy deletion is CPU-friendly but wastes CPU resources

(2) : delete periodically (uncommon)

(3) : Periodically delete, CPU friendly, save space

Cache avalanches and how to handle them

A large number of caches are invalid at the same time.

Treatment method:

(1) : The expiration mark is added to cache data

(2) : Set different cache expiration times

(3) : The two-layer cache policy C1 is short-term and C2 is long-term

(4) : Update the policy periodically

Cache breakdown reasons and handling methods

Frequently request to query the data that does not exist in the system.

Treatment method:

(1) : cache NULL policy. If the query feedback result is NULL, the null result is still cached and the expiration time is set to no more than 5 minutes

(2) : Bloom filter, all possible data is mapped to a sufficiently large bitmap. Google Bloom filter: based on memory, restart failure does not support large amount of data, and cannot be used in distributed scenarios. Redis Bloom filter: scalable, no restart failure, requires network I/O, and the performance is lower than Google

Redis blocking cause

(1) : Unreasonable use of data structure BigKey

(2) : THE CPU is saturated

(3) : persistent blocking, RDB fork subthread, Aof flushing per second, etc

Solution Cluster access skewed due to hot Keys

(1) : Use the local cache

(2) : Use the feature of sharding algorithm to break up the key (add a prefix or suffix to the hotkey, and change the number of hotkeys into a multiple M of the number of Redis instances N, so that the access to a Redis key becomes N * M Redis keys)

Redis distributed locks

After version 2.6, the lua script ensures that setnx and Setex are atomized (after setNx, there is no setex, the service is suspended, and the lock is not released). A obtains the lock, and when the expiration time exceeds, the lock is automatically released. B obtains the lock and executes the lock. Solution: Determine value before remove (value may be modified under high concurrency, should use lua to ensure atomicity)

How does Redis persist

Bgsave to do full image persistence, Aof to do incremental persistence. Because BGSave takes a long time, is not real-time enough, and will cause a large amount of data loss during downtime, AOF is needed to cooperate with it. When the Redis instance is restarted, memory is rebuilt using bgSave persistent files, and aof is used to replay recent operations to achieve a complete recovery of the state before the restart.

What would happen if the power suddenly went out?

Depending on the configuration of the SYNC property of the Aof log, if performance is not required, the disk is synced with each write instruction, so that data is not lost. However, in the context of high performance, it is not realistic to sync each time. In general, timing sync is used, such as 1S1 times. At most 1 second of data will be lost at this time.

Redis lock renewal issue?

(1) : Distributed reentrant lock RLock based on Redis Redission, and lock in Java collection;

(2) : Redission internally provides a watchdog that monitors locks and continuously extends the lock validity period. The default timeout period for checking locks is 30 seconds

(3) : The problem with this scheme is that if you write the value of myLock to a Redis master instance, it will be asynchronously copied to the corresponding master and slave instances. However, once the Redis Master breaks down during this process, the master/slave switchover occurs, and the Redis slave becomes the Redis Master.

This would result in client 2 trying to lock on the new Redis Master and client 1 thinking it had successfully locked. The solution: simply make the new Redis instance unavailable to the client for a TTL period, during which time all client locks will be invalid or automatically released.

How does BGSave work?

The fork and the cow. Fork means that Redis performs BGsave operations by creating child processes, and cow means copy on write. After the creation of child processes, the parent process shares data segments, and the parent process continues to provide read and write services. The page data written to redis will gradually be separated from the child process.

RDB is different from AOF

(1) : THE compact format of R file facilitates data recovery. When the RDB file is saved, the parent process will fork out the child process to complete the specific persistence work, maximizing the performance of Redis and faster recovery of large data sets. The backup operation is triggered only when the save command is manually submitted or the command is closed.

(2) : A records each write operation to the server (once written for 1s by default) to save more complete data. When redis restarts, these commands will be replayed to recover data, which has high operation efficiency, less data loss during faults, but larger file volume;

Of the 100 million keys, there are 10w keys that start with some fixed, known prefix, and if you find all of them, right?

Use the keys command to scan the list of keys in the specified mode. If the redis is serving an online business, what’s the problem with using keys? Redis is single threaded. The keys command will cause the thread to block for a period of time, and the online service will stop until the command is completed. In this case, you can use the SCAN instruction. Scan instruction can extract the key list of the specified mode without blocking, but there is a certain probability of repetition. It is ok to do it once on the client, but the overall time will be longer than using keys directly.

How to use Redis for asynchronous queues?

Generally, the list structure is used as a queue, with Rpush producing messages and LPOP consuming messages. When lPOP has no messages, sleep properly for a while and try again.

How about instead of sleep?

The list also has a directive called blpop, which blocks when there is no message until it arrives.

Can we produce and consume multiple times?

A 1: N message queue can be implemented using the PUB/SUB topic subscriber pattern.

What are the disadvantages of pub/ SUB?

In the case of consumers going offline, produced messages are lost and professional message queues such as RabbitMQ are used.

How does Redis implement delayed queues?

Using the sortedset, the timestamp of the time you want to execute is used as the score, the message content is used as the key to call Zadd to produce the message, and the consumer uses the ZrangebyScore directive to get the data up to N seconds old for polling processing.

Why does Redis Zset use skip lists instead of red black trees?

(1) : Skiplist has the same complexity as a red-black tree and is much simpler to implement.

< br style = “color: RGB (51, 51, 51); color: RGB (51, 51, 51);

MYSQL

Database three normal forms

One: Ensure the atomicity of each column

Two: non-primary key columns do not have partial dependency on primary keys (requires each table to only describe one thing)

Three: satisfies the second normal form, and the columns in the table have no transition-dependent non-primary key columns

Principle of primary/secondary database replication

(1) : update events (UPDATE, INSERT, DELETE) of the primary database are written to binlog

(2) : The master library creates a binlog dump thread and sends the binlog contents to the slave library

(3) : The slave library creates an I/O thread that reads the binlog from the master library and writes it to the relay log.

(4) : The slave library will also create an SQL thread to read from the slave log and write to the slave DB.

Replication Mode Classification

(1) : Asynchronous replication (default) After the primary library writes the binlog log, it can return to the client successfully. There is no need to wait for the binlog log to be passed to the secondary library. However, once the primary library breaks down, data may be lost.

(2) semi-synchronous replication :(after version 5.5) (install the semi-synchronous replication plug-in) ensure that the slave receives the binlog passed from the master and writes it to its relay log before notifying the waiting thread at the master. If the wait times out, the semi-synchronous replication is turned off and automatically switched to asynchronous replication until at least one slave informs the master that the binlog message has been received

The storage engine

Myiasm is the default storage engine for mysql. It does not support database transactions, row-level locking, and foreign keys. Insert update requires lock table, low efficiency, query speed, Myisam uses non-clustered indexes

(2) innoDB supports transaction, B+ tree implementation, suitable for handling multiple concurrent update operations, ordinary select is snapshot read, snapshot read is not locked. InnoDb uses clustered indexes

Clustered index

(1) : a clustered index is an index created with a primary key

(2) : Each table can only have one cluster index, because the records in a table can only be stored in one physical order, and the actual data pages can only be sorted by a B+ tree

(3) : Table records are arranged in the same order as the index

(4) : Clustered index storage records are physically continuous

(5) : The primary key insertion speed of a clustered index is much slower than that of a non-clustered index

(6) : The clustering index is suitable for sorting, while the non-clustering index is not suitable for sorting, because the leaf nodes of the clustering index themselves are placed together with the indexes and data in the same order. The index order is the data order, and the data order is the index order, so it is fast. Non-clustered index leaf nodes reserve a pointer to the data. The index itself is sorted, of course, but the data is not sorted. Data query consumes more I/O, so it is slow

(7) : Updating clustered index columns is expensive because innoDB is forced to move each updated row to a new location

Nonclustered index

(1) : indexes other than primary keys

(2) : The leaf node of clustered index is the data node, while the leaf node of non-clustered index is still the index node, and a link pointing to the corresponding data block is reserved

(3) : Clustering index is suitable for sorting, while non-clustering index is not suitable for sorting

(4) clustered indexes store records that are physically continuous, while non-clustered indexes store records that are logically continuous.

Why queries are faster using clustered indexes?

Once the row containing the first value is found using the clustered index, you can ensure that rows containing subsequent index values are physically adjacent

What should I do with clustered indexes?

Do not include frequently modified columns in the cluster index, because after the code value is changed, the data rows must be moved to the new position, and the index will be rearranged, which will cause a great waste of resources

What is InnoDB table primary key generation policy?

If there is no Unique key, InnoDB will add a hidden column named row_id as the primary key. If there is no Unique key, InnoDB will add a hidden column named row_id as the primary key.

What is the maximum number of non-clustered indexes?

You can create up to 249 non-clustered indexes per table. Non-clustered indexes require large amounts of disk space and memory

What is the difference between BTree and Hash indexes?

(1) : The BTree index may need to use half search multiple times to find the corresponding data block (2) : The HASH index uses the HASH function to calculate the HASH value and find the corresponding data in the table (3) : A large number of different data are queried accurately with equivalent value. The efficiency of the HASH index is usually higher than that of B+TREE (4) : HASH indexes do not support fuzzy queries, range queries, and leftmost matching rules in the federated indexes, which are all supported by Btree indexes

Advantages and Disadvantages of database indexes

(1) : The fields that need to be queried, sorted, grouped and combined are suitable for indexing

(2) : The more indexes, the slower the data update table, try to use the field with a large proportion of different field values as the index, the joint index is more efficient than multiple independent indexes

(3) : Frequently query data and create indexes. If data needs to be frequently changed, it is not recommended to use indexes

(4) : When the data in the table is added, deleted and modified, the index also needs dynamic maintenance, which reduces the maintenance speed of data.

The underlying implementation of the index is B+ tree, why not use red black tree, B tree?

(1) : B+Tree non-leaf nodes only store key value information. When the height of B+Tree is lowered, there is a chain pointer between all leaf nodes and data records are stored in leaf nodes

(2) : In the structure of red-black Tree, H is significantly deeper and its efficiency is significantly worse than that of B-tree

(3) : B+ trees also have the disadvantage of taking up more space because the keys are repeated. However, the spatial disadvantage is often acceptable compared to the performance advantage, so B+ trees are more widely used in databases than B trees

Index failure condition

(1) : if you want to make the or condition valid, add an index to each or field

(2)

(3) : If the column type is a string, make sure the data is quoted in the condition, otherwise the index is not used

(4) : where index column uses a function contingent operation

Database transaction Characteristics

ACID Atomicity, consistency, isolation, permanentness

How is the database transaction theory implemented?

(1) : Through the pre-write log implementation, redo and undo mechanism is the foundation of the database implementation of transactions

(2) : Redo log is used to relive the process of rewashing data in case of power failure/database crash etc., rewashing data in redo log to database, ensuring the Durability of transactions.

(3) : The undo log is to undo the operation on the database when the transaction fails, ensuring the atomicity of the transaction

Database transaction isolation level

(1) : read uncommitted– dirty, no repeat read — fantasy read A read B’s uncommitted transaction, B rollback, A dirty read;

(2) : read-committed (B); (3) : read-committed (B); (4) : read-committed (B);

(3) : repeatable read -read< default >– Magic read transaction is enabled, and UPDATE modification operation of other transactions A is not allowed to read the committed transaction of B, but B inserts A new row in the table, and then A has an extra row when reading, which causes magic read;

(4) : serializable

Seven transactional communication behaviors

(1) Propagation.REQUIRED< Default > If there is a transaction, join it, if there is no transaction, create a new transaction.

(2) Propagation.SUPPORTS If there is a transaction, join the transaction; If no transaction currently exists, it continues to run non-transactionally.

(3) Propagation.MANDATORY If there is a transaction, join this transaction. If no current transaction exists, an exception is thrown.

(4) Propagation.REQUIRES_NEW Creates a new transaction, and postpones the current transaction if it exists.

(5) Propagation.NOT_SUPPORTED runs in non-transactional mode. If the current transaction exists, pause the current transaction.

(6) Propagation.NEVER in non-transaction mode, if there is a transaction, throw an exception.

(7) To create a new transaction; If so, other transactions are nested within the current transaction.

The four conditions necessary to create a deadlock

(1) : mutually exclusive: resource X can only be owned by one thread at any time (2) : owned and wait: Thread 1 owns resource x and waits for resource Y, but does not release x (3) : non-preempt: Once resource X is owned by thread 1, no other thread can preempt x (4) : loop wait: Thread 1 holds x and waits for y, and thread 2 holds y and waits for X

Used for mining the word limit of 2W words, the original you go to my Git or public number to see, I pinpointed the 2W word

conclusion

The content is too hardcore, resulting in a lot of typography detail, I can’t do it as well as other issues, forgive me.

There are a lot of content and things involved, may be a lot of point to the end, there are a lot of incomplete, there are also a lot of wrong points, has been nearly 3W word, I verify is really difficult, I will put on GitHub, you can update this article with me, for the benefit of future generations.

Maybe the next time I need to see it, I’ll have to look at this review.

I am Aobin, a tool of living on the Internet.

The more you know, the more you don’t know. The three lines of talents are the biggest driving force for the creation of C-C. See you next time!

Note: If this blog has any mistakes and suggestions, welcome to comment talent, you say something!

---

The article continues to update, can wechat search “three Prince Aobin” the first time to read, reply to [data] [interview] [resume] I prepared for the first line of big factory interview data and resume template, this article GitHub github.com/JavaFamily has been included, there are big factory interview complete test point, welcome to Star.