1. Problem description

The operation manager responded that the number of redis client connections exceeded the default maximum number of 1W. Run the./redis-cli -h host -p port info clients command on the redis server. You can see that a large number of redis client connections are alive, but most of the connections are idle.

I logged in a service instance to perform scheduled tasks. Basically, there is no traffic in this service instance, but there are still 40 Redis connections. It is certain that some of our connections were created and saved to the present. Since we are using uniform Redis components and configurations, we are only adding special configuration parameters on individual services. At present, there are about more than one hundred docker container instances in the whole service field, which means there are thousands of idle connections, so the number of idle connections will definitely affect the performance of the server. Incorrect terminal output may cause thread information loss. You are advised to export the output to a file for further analysis.

jstack  pid > stack.log

cat stack.log | grep redis/reidsson/jedis
Copy the code

You can see a lot of information about redisson connection threads, and you can probably guess that the Redisson client is holding these connections. This is just a guess, depending on the actual environment on the line.

2. Analyze the problem

We wrap the redis client component, which USES a variety of redis client (jedis/redisson/redisTemplate), if we can directly find the connection object is very easy to locate problem here. For faster problem finding in online environments, it is recommended that the arthas tool use OGNL expressions to view the instance’s member variables. Sc and orgNL need to work together as shown in the following example:

Sc - d org. Apache. Dubbo. Config. Spring. The extension. After obtaining the corresponding instances of this SpringExtensionFactor execution ognl members can see variable name ognl 18 b4aac2 - c  '#context=@org.apache.dubbo.config.spring.extension.SpringExtensionFactory@CONTEXTS.iterator.next, #context.getBean("redisson").getConnectionManager()'Copy the code

2.1 jedis connection

The connection management of Jedis is relatively simple. The framework uses JedisPool(apache’s GenericObjectPool is used at the bottom) to manage the connection. The general connection usage flow is as follows:

JedisPool(GenericObjectPool) JedisFactory

-> Jedis

-> BinaryJedis

-> Client

-> socket

So we get the GenericObjectPool object pool of the Jedis instance object to get the total connection information. See that there are only 10 objects in the connection pool (object pool), so these 30 + connections can only be redisson clients.

 ognl -c 18b4aac2 '#context=@org.apache.dubbo.config.spring.extension.SpringExtensionFactory@CONTEXTS.iterator.next, #context.getBean("jedisPool").internalPool' 
Copy the code

2.2 redisson connection

Compared with the simple Jedis framework, Redisson framework is based on the Netty communication framework based on NIO and provides a series of distributed transaction operations and data structures. Redisson’s framework source code is based on AIO programming model to achieve, read up or some difficulty, the whole framework code amount is more, we only focus on the redis connection related to the class information.

Redisson’s network connection management is implemented through ConnectionManager, where Map<RedisClient, MasterSlaveEntry> client2Entry stores connection pool information. IdleConnectionWatcher connectionWatcher starts a scheduled task to scan and remove useless or idle connections, so we can see all the connections of the current client by fetching one of the two. We can see the current connection information by executing arthas’s command to get the client2Entry list. The results are shown below. These connections are created by Redisson.

ognl -c 18b4aac2 '#context=@org.apache.dubbo.config.spring.extension.SpringExtensionFactory@CONTEXTS.iterator.next, #context.getBean("redisson").getConnectionManager().getEntrySet().iterator().next().masterEntry.allConnections'
Copy the code

The Redisson client uses SingleServer mode, and the initial call process for the connection is shown below.

MasterSlaveConnectionManager#initSingleEntry()

->MasterSlaveEntry#setupMasterEntry

->MasterConnectionPool#add()

-> ConnectionPool#add()

-> ConnectionPool#initConnection()

-> ConnectionPool#createConnection()

ConnectionPool# the createConnection () in the initialized connection operation, here minimumIdleSize corresponding is exposed connectionMinimumIdleSize attributes, If we make this configuration property smaller, we control the connection initialization. The connection pool of the framework starts with 32 connections by default. Here minimumIdleSize connectionMinimumIdleSize attribute of the corresponding is exposed, if we connect the configuration properties become smaller, control of initialization, this framework of connection pooling enabled by default is 32.

private void createConnection(boolean checkFreezed, AtomicInteger requests, ClientConnectionsEntry entry, RPromise initPromise, int minimumIdleSize, AtomicInteger initializedConnections) { ..... acquireConnection(entry, new Runnable() { @Override public void run() { RPromise promise = new RedissonPromise(); // Call RedisClient#connectAsyncI(); promise.onComplete((conn, e) -> { .... // Get the number of connections left to initialize, Here can see minimumIdleSize helped int totalInitializedConnections = minimumIdleSize - initializedConnections. The get (); . int value = initializedConnections.decrementAndGet(); If (value == 0) {// Initialize connection completed} else if (value > 0 &&! Initpromise.isdone ()) {if (requests. IncrementAndGet () <= minimumIdleSize) {// Continue to call createConnection(checkFreezed, requests, entry, initPromise, minimumIdleSize, initializedConnections); }}}); }});Copy the code

3. Solutions

In order to reduce the free connection to the redis server resource utilization, increase in the components of Redisson connectionMinimumIdleSize and connectionPoolSize two attributes configuration logic, back online after execution, effective configuration, number of idle connections back to normal.

4. To summarize

In the project, jStack and Arthas (ogNL is really cool) were used to gradually locate and analyze the cause of the redisL overconnection problem. In this process, I also learned the relevant implementation principles of Jedis and Redisson, and gained a lot. Often used to clients such as redis/mysql/dubbo/zookeeper, etc., actually is very exquisite, connection configuration parameters need to be done according to the different operating conditions and performance requirements and reasonable configuration.

The resources

Jstack related usage

Zshell. Cc / 2017/09/24 /…