How to quickly implement a connection pool?

In practice, we often use various connection pools. For example, a connection pool needs to be created because the number of connections to the FTP server is limited. The number of connections to the database is limited and a connection pool is required. So how do we quickly implement a connection pool?

Whether it’s an FTP connection pool, or a database connection pool, we’ll see that they all have the same thing in common: lifecycle management, connection creation management, and so on. If we had implemented these features from scratch, it would have taken us a long time! Is there a generic library that can quickly implement a thread pool?

Thanks to Java’s well-established ecology, a common library was developed for this purpose: Apache Commons Pool (hereafter referred to as ACP). In essence, the ACP library provides the general ability to manage object pools, as well as connection pools!

What is ACP?

The ACP library provides a complete set of apis for object pooling, as well as several distinct object pooling implementations. Currently, the most commonly used version is version 2.0, which is not a simple upgrade to version 1.x. Version 2.0 is a complete rewrite of the object pooling implementation, with significant performance and scalability improvements, and includes robust instance tracking and pool monitoring.

Apache Commons Pool — Overview Apache Commons Pool — Overview

How to use ACP?

To implement a thread pool using ACP, you first need to import the ACP dependency package. Here is Maven as an example.

Mons < dependency > < groupId > org.apache.com < / groupId > < artifactId > Commons - pool2 < / artifactId > < version > 2.0 < / version > </dependency>Copy the code

To implement an object pool using ACP, there are roughly three steps:

• Create object Factory: tells ACP how to create the object you want.
• Create an object pool: Tell ACP what kind of object pool you want to create.
• Use object pools: ACP tells you how to use your objects.

Creating an object factory

The object factory tells the ACP how to create, activate, passivate, and destroy your objects. Creating an object factory is as simple as implementing the PooledObjectFactory interface of ACP. The PooledObjectFactory interface is defined as follows:

public interface PooledObjectFactory<T> {
  PooledObject<T> makeObject() throws Exception;
  void destroyObject(PooledObject<T> p) throws Exception;
  boolean validateObject(PooledObject<T> p);
  void activateObject(PooledObject<T> p) throws Exception;
  void passivateObject(PooledObject<T> p) throws Exception;
}
Copy the code

More often, however, we will inherit the BasePooledObjectFactory class to implement the object factory. Because the BasePooledObjectFactory class is the base implementation class for PooledObjectFactory, using it can save us a lot of trouble. By inheriting this abstract class, we only need to implement two methods: create() and wrap().

// Tell ACP how to create an object public abstract create() throws Exception; Public abstract PooledObject<T> wrap(T obj);Copy the code

The create() method defines your object initialization process and returns the initialized object. For example, if you want to define an SFTP connection, you first need to define a JSch object, then set the account password, then connect to the server, and finally return a ChannelSftp object.

Public ChannelSftp create() {// SFTP connection creation process}Copy the code

The wrap() method defines the object you want to return, which is essentially a ChannelSftp object for an SFTP connection pool. DefaultPooledObject class DefaultPooledObject class DefaultPooledObject

@Override
public PooledObject<Foo> wrap(Foo foo) {
    return new DefaultPooledObject<Foo>(foo);
}
Copy the code

Creating an Object Pool

After you have created the object factory, ACP already knows how to create the objects you need. Next, you need to create an object pool based on your actual needs. In ACP, we create an object pool with GenericObjectPool and GenericObjectPoolConfig.

// Declare an object pool private GenericObjectPool<ChannelSftp> sftpConnectPool; GenericObjectPoolConfig poolConfig = new GenericObjectPoolConfig(); poolConfig.setEvictionPolicyClassName("tech.shuyi.javacodechip.acp.SftpEvictionPolicy"); poolConfig.setBlockWhenExhausted(true); poolConfig.setJmxEnabled(false); poolConfig.setMaxWaitMillis(1000 * 10); poolConfig.setTimeBetweenEvictionRunsMillis(60 * 1000); poolConfig.setMinEvictableIdleTimeMillis(20 * 1000); poolConfig.setTestWhileIdle(true); poolConfig.setTestOnReturn(true); poolConfig.setTestOnBorrow(true); poolConfig.setMaxTotal(3); AbandonedConfig AbandonedConfig = new AbandonedConfig(); abandonedConfig.setRemoveAbandonedOnMaintenance(true); abandonedConfig.setRemoveAbandonedOnBorrow(true); this.sftpConnectPool = new GenericObjectPool<>(sftpConnectFactory, poolConfig, abandonedConfig);Copy the code

In the code that created the SFTP connection pool above, we configured some thread pool parameters and set the discard policy. The discard policy is very important. If it is not set, invalid connections will be obtained and files will fail to be retrieved. Abandoning strategy is through poolConfig setEvictionPolicyClassName to set, we set is SftpEvictionPolicy class here, the content of the code is as follows:

@Slf4j @Component public class SftpEvictionPolicy implements EvictionPolicy<com.jcraft.jsch.ChannelSftp> { @Override public boolean evict(EvictionConfig config, PooledObject<com.jcraft.jsch.ChannelSftp> underTest, Int idleCount) {try {// If (! underTest.getObject().isConnected()) { log.warn("connect time out, evict the connection. time={}",System.currentTimeMillis() - underTest.getLastReturnTime()); return true; } }catch (Exception e){ return true; } return false; }}Copy the code

At this point, the code to create the thread pool is finished, and the SftpConnectPool file is as follows:

@Slf4j public class SftpConnectPool { private GenericObjectPool<ChannelSftp> sftpConnectPool; Public SftpConnectPool(SftpConnectFactory SftpConnectFactory) {GenericObjectPoolConfig poolConfig = new GenericObjectPoolConfig(); poolConfig.setEvictionPolicyClassName("tech.shuyi.javacodechip.acp.SftpEvictionPolicy"); poolConfig.setBlockWhenExhausted(true); poolConfig.setJmxEnabled(false); poolConfig.setMaxWaitMillis(1000 * 10); poolConfig.setTimeBetweenEvictionRunsMillis(60 * 1000); poolConfig.setMinEvictableIdleTimeMillis(20 * 1000); poolConfig.setTestWhileIdle(true); poolConfig.setTestOnReturn(true); poolConfig.setTestOnBorrow(true); poolConfig.setMaxTotal(3); AbandonedConfig AbandonedConfig = new AbandonedConfig(); abandonedConfig.setRemoveAbandonedOnMaintenance(true); abandonedConfig.setRemoveAbandonedOnBorrow(true); this.sftpConnectPool = new GenericObjectPool<>(sftpConnectFactory, poolConfig, abandonedConfig); } public ChannelSftp borrowObject() { try { return sftpConnectPool.borrowObject(); } catch (Exception e) { log.error("borrowObject error", e); return null; } } public void returnObject(ChannelSftp channelSftp) { if (channelSftp! =null) { sftpConnectPool.returnObject(channelSftp); }}}Copy the code

I’ve also added the borrowObject and returnObject methods for ease of use, but these aren’t necessary. In these two methods, we called the borrowObject and returnObject methods of the GenericObjectPool class respectively. This is exactly what ACP provides, using thread pool objects, borrowing an object and returning it later.

Note: In this step, object pooling is already included. However, in practice, we often put the object pool declaration and use in the same class, so for the sake of explanation, there is no separation here. Therefore, the use of object pools below is essentially a further encapsulation of object pools.

Using object pools

Here we have created the SFTP object pool, is not very simple! But in the real world, we usually do some encapsulation on top of that. For our SFTP connection pool, we will directly provide external file download services, further encapsulate the SFTP object pool, do not need to care about how to obtain files.

public class SftpFileHelper { @Autowired private SftpConnectPool sftpConnectPool; public void download(String dir, String file, String saveUrl)throws IOException { ChannelSftp sftp = sftpConnectPool.borrowObject(); log.info("begin to download file, dir={}, file={}, saveUrl={}", dir, file, saveUrl); try { if (! StringUtils.isEmpty(dir)) { sftp.cd(dir); } File downloadFile = new File(saveUrl); sftp.get(file, new FileOutputStream(downloadFile)); }catch (Exception e){log.warn(" failed to download file ", e); }finally { sftpConnectPool.returnObject(sftp); } log.info("file:{} is download successful", file); }}Copy the code

Finally, let’s write a test case to see if it works.

@RunWith(SpringRunner.class) @SpringBootTest @Slf4j public class SftpFileHelperTest { @Autowired private SftpFileHelper sftpFileHelper; @Test public void testDownloadFtpFile() throws Exception { sftpFileHelper.download("dir", "fileName", "fileName"); }}Copy the code

If there are no accidents, you will see a green line, the file has been downloaded successfully!

conclusion

This article demonstrates the object Pool functionality most commonly used by the Apache Commons Pool library. After reading this article, we know that creating a thread pool requires three steps:

• Create object Factory: tells ACP how to create the object you want.
• Create an object pool: tell ACP what kind of object pool you want to create and set the expulsion policy.
• Use object pools: ACP tells you how to use your objects.

The code for this article is stored in the Github project: Java-code-Chip, which can be obtained by clicking on the address: Java code – chip/SRC/main/Java/tech/shuyi/javacodechip/acp at master chenyurong/Java code — chip

The ACP library allows readers to quickly create a pool of objects and focus more on business content. In fact, ACP offers much more than that, and much more advanced functionality.

For example, if multiple SFTP servers are connected, you need to obtain different connection objects using different IP addresses. The dumbest thing to do at this point is to copy one more copy of code for each different address and do it in a different way. But that’s a lot of work, and there’s a lot of duplicate code. This time you can use the BaseKeyedPooledObjectFactory to replace BasePooledObjectFactory, so as to realize through the key to realize the different address connection object management.

If you are interested in ACP, you can explore it by yourself.

Thank you for reading. If this article is helpful to you, click “like” or share it on Moments.

The resources

• Apache Commons: Pool- Ali Cloud Developer Community
• Apache Common Pool2 object pool application
• The Pool – the Project Information