A, the opening
Redis, as the current universal cache selection, is popular because of its high performance. The 2.x version of Redis only supports single-machine mode, with the introduction of cluster mode starting with version 3.0.
Redis Java ecosystem client includes Jedis, Redisson, Lettuce, different clients have different capabilities is the way to use, this article mainly analyzes Jedis client.
The Jedis client supports access modes in single-machine mode, sharded mode and cluster mode at the same time. The data access in single-machine mode is realized by constructing Jedis objects, the data access in ShardedJedis objects and the data access in cluster mode is realized by constructing JedisCluster objects.
Jedis client supports single command and Pipeline access to Redis cluster, which can improve the efficiency of cluster access.
The overall analysis of this article is based on Jedis version 3.5.0, and the relevant source code is referred to this version.
Ii. Comparison of Jedis access modes
Jedis client operation of Redis is mainly divided into three modes, which are single machine mode, sharding mode and cluster mode.
-
The single-machine mode mainly creates Jedis objects to manipulate single-node Redis and is only applicable to accessing a single Redis node.
-
ShardedJedis is to access multiple Redis nodes of ShardedJedisPool by creating ShardedJedisPool objects. It is a data distribution scheme implemented by clients before Redis has cluster function. In essence, clients realize distributed storage of data through consistent hash.
-
Cluster mode (JedisCluster) is mainly to create JedisCluster objects to access multiple Redis nodes in cluster mode. It is cluster access implemented by clients after the introduction of cluster mode in Redis3.0. In essence, data distributed storage is realized by introducing the concept of slot and CRC16 hash slot algorithm.
Stand-alone mode does not involve any sharding ideas, so we focus on the concepts of sharding and clustering.
2.1 Sharding Mode
-
Sharding mode essentially belongs to sharding based on the client, which realizes how to find the corresponding node in Redis cluster according to a key.
-
The client sharding mode of Jedis is implemented by consistent Hash. The advantage of consistent Hash algorithm is that when Redis nodes are added or deleted, only a small part of the data before and after the addition or deletion of nodes will be affected, which has a smaller impact on data compared with other algorithms such as modulus taking.
-
Redis is used as cache in most scenarios, so there is no need to consider the impact of cache penetration caused by data loss. When Redis nodes increase or decrease, there is no need to consider the problem that some data cannot be hit.
The overall application of the sharding mode is shown in the following figure. The core is the consistent Hash policy on the client.
(Quoted from: www.cnblogs.com)
2.2 Cluster Mode
Cluster mode essentially belongs to server sharding technology, sharding function is provided by Redis cluster itself, officially provided from Redis 3.0.
The principles of clustering are as follows: A Redis cluster contains 16384 Hash slots. Each key stored in Redis belongs to one of the 16384 Hash slots. The cluster uses the formula CRC16(key)%16384 to calculate which slot the key belongs to. The CRC16(key) statement is used to calculate the CRC16 checksum of the key.
Each node in the cluster is responsible for processing a portion of the hash slots. For example, a cluster can have three hash slots, where:
-
Node A handles hash slots 0 through 5500.
-
Node B handles hash slots 5501 through 11000.
-
Node C handles hash slots 11001 through 16383.
In cluster mode, Redis first performs CRC16 calculation on the corresponding key value to obtain the corresponding hash value, then maps the hash value to the total number of slots to the corresponding slot, and finally to the corresponding node for reading and writing. Take the command set(“key”, “value”) as an example, it will use CRC16 algorithm to calculate the key and get the hash value of 28989, then take the module of 16384 and get 12605, finally find the Redis node corresponding to 12605, and jump to this node to execute the set command.
The overall application of cluster mode is shown in the figure below. The core lies in the design of the cluster hash slot and the redirection command.
(Quoted from: www.jianshu.com)
The basic usage of Jedis
// Access to Jedis single-machine mode
public void main(String[] args) {
// Create a Jedis object
jedis = new Jedis("localhost".6379);
// perform the hmget operation
jedis.hmget("foobar"."foo");
// Close the Jedis object
jedis.close();
}
// Access to Jedis sharding mode
public void main(String[] args) {
HostAndPort redis1 = HostAndPortUtil.getRedisServers().get(0);
HostAndPort redis2 = HostAndPortUtil.getRedisServers().get(1);
List<JedisShardInfo> shards = new ArrayList<JedisShardInfo>(2);
JedisShardInfo shard1 = new JedisShardInfo(redis1);
JedisShardInfo shard2 = new JedisShardInfo(redis2);
// Create ShardedJedis object
ShardedJedis shardedJedis = new ShardedJedis(shards);
// Set with ShardedJedis
shardedJedis.set("a"."bar");
}
// Jedis cluster mode access
public void main(String[] args) {
// Build a redis cluster pool
Set<HostAndPort> nodes = new HashSet<>();
nodes.add(new HostAndPort("127.0.0.1".7001));
nodes.add(new HostAndPort("127.0.0.1".7002));
nodes.add(new HostAndPort("127.0.0.1".7003));
/ / create JedisCluster
JedisCluster cluster = new JedisCluster(nodes);
// Execute the method in the JedisCluster object
cluster.set("cluster-test"."my jedis cluster test");
String result = cluster.get("cluster-test");
}
Copy the codeJedis implements data access in single-machine mode by creating Jedis class objects, and implements data access in cluster mode by constructing JedisCluster class objects.
To understand the entire process of Jedis accessing Redis, you can first understand the access process in single-machine mode, and then analyze the access process in cluster mode.
4. Access to Jedis single machine mode
The overall flow chart of Jedis accessing single-machine mode Redis is shown below. It can be seen from the figure that the core process includes the creation of Jedis objects and the access of Redis through Jedis objects.
To be familiar with the process of Jedis accessing single-machine Redis is to understand the creation process of Jedis and the process of executing Redis commands.
-
The core of the Jedis creation process is to create the Jedis object and the Jedis internal variable Client object.
-
Jedis accesses Redis through a Client object inside Jedis.
4.1 Creation Process
The class diagram of Jedis itself is shown below, from which we can see that Jedis inherits from the BinaryJedis class.
In the BinaryJedis class, there is a Client object that connects to Redis. Jedis uses the Client object of the parent BinaryJedis class to read and write Redis.
The Jedis class creates the Client object from its parent BinaryJedis during creation, and understanding the Client object is key to further understanding the access process.
public class Jedis extends BinaryJedis implements JedisCommands.MultiKeyCommands.AdvancedJedisCommands.ScriptingCommands.BasicCommands.ClusterCommands.SentinelCommands.ModuleCommands {
protected JedisPoolAbstract dataSource = null;
public Jedis(final String host, final int port) {
// Create the parent BinaryJedis object
super(host, port); }}public class BinaryJedis implements BasicCommands.BinaryJedisCommands.MultiKeyBinaryCommands.AdvancedBinaryJedisCommands.BinaryScriptingCommands.Closeable {
// Access the redis Client object
protected Client client = null;
public BinaryJedis(final String host, final int port) {
// Create a Client object to access Redis
client = newClient(host, port); }}Copy the codeThe Client class diagram is shown below. The Client object inherits from the BinaryClient and Connection classes. In the BinaryClient class there are Redis access password and other related parameters, in the Connection class there are access Redis socket object and the corresponding input and output streams. Connection is essentially the core class that communicates with Redis.
The Client class initializes the core Connection object during creation, and Connection is responsible for communicating directly with Redis.
public class Client extends BinaryClient implements Commands {
public Client(final String host, final int port) {
super(host, port); }}public class BinaryClient extends Connection {
// Store information about connections to Redis
private boolean isInMulti;
private String user;
private String password;
private int db;
private boolean isInWatch;
public BinaryClient(final String host, final int port) {
super(host, port); }}public class Connection implements Closeable {
// Manage the socket information connected to Redis and the corresponding I/O streams
private JedisSocketFactory jedisSocketFactory;
private Socket socket;
private RedisOutputStream outputStream;
private RedisInputStream inputStream;
private int infiniteSoTimeout = 0;
private boolean broken = false;
public Connection(final String host, final int port, final boolean ssl,
SSLSocketFactory sslSocketFactory, SSLParameters sslParameters,
HostnameVerifier hostnameVerifier) {
/ / build DefaultJedisSocketFactory to create and Redis connection Socket object
this(newDefaultJedisSocketFactory(host, port, Protocol.DEFAULT_TIMEOUT, Protocol.DEFAULT_TIMEOUT, ssl, sslSocketFactory, sslParameters, hostnameVerifier)); }}Copy the code4.2 Access Process
Take Jedis executing the set command as an example. The entire process is as follows:
-
The SET operation of Jedis is implemented through the set operation of the Client.
-
The Client set operation is implemented through sendCommand of the parent Connection class.
public class Jedis extends BinaryJedis implements JedisCommands.MultiKeyCommands.AdvancedJedisCommands.ScriptingCommands.BasicCommands.ClusterCommands.SentinelCommands.ModuleCommands {
@Override
public String set(final String key, final String value) {
checkIsInMultiOrPipeline();
// The client performs the set operation
client.set(key, value);
returnclient.getStatusCodeReply(); }}public class Client extends BinaryClient implements Commands {
@Override
public void set(final String key, final String value) {
// Run the set commandset(SafeEncoder.encode(key), SafeEncoder.encode(value)); }}public class BinaryClient extends Connection {
public void set(final byte[] key, final byte[] value) {
// Send the set commandsendCommand(SET, key, value); }}public class Connection implements Closeable {
public void sendCommand(final ProtocolCommand cmd, final byte[]... args) {
try {
// socket connects to redis
connect();
// Send commands according to redis protocol
Protocol.sendCommand(outputStream, cmd, args);
} catch (JedisConnectionException ex) {
}
}
}
Copy the code5. Access to Jedis sharding mode
The access of Jedis sharding mode is understood based on the principle of consistent Hash of Redis sharding mode introduced previously.
About the concept of Redis sharding mode: Redis does not have the concept of cluster mode before version 3.0, which results in the limited amount of data that can be stored by a single node. Redis clients such as Jedis implement data sharding through consistent Hash algorithm on the client side.
In essence, Redis sharding mode has nothing to do with Redis itself, but only solves the problem of limited storage of single node data through the client.
The core of ShardedJedis accessing Redis is to initialize the consistent Hash object when constructing the object and build the mapping between the classic Hash value and node. After the mapping is completed, operations such as set are used to address Hash values to nodes. After the addressing is completed, operations on single nodes are performed directly.
5.1 Creation Process
The creation process of ShardedJedis lies in the initialization process related to consistent Hash in Sharded of the parent class. The core lies in the construction of consistent virtual nodes and the mapping between virtual nodes and Redis nodes.
The core part of the source code is based on the weight mapping into not 160 virtual nodes, through the virtual node to locate the specific Redis node.
public class Sharded<R.S extends ShardInfo<R>> {
public static final int DEFAULT_WEIGHT = 1;
// Save the mapping between virtual nodes and redis nodes
private TreeMap<Long, S> nodes;
/ / the hash algorithm
private final Hashing algo;
// Save connection information for redis node and Jedis accessing this node
private final Map<ShardInfo<R>, R> resources = new LinkedHashMap<>();
public Sharded(List<S> shards, Hashing algo) {
this.algo = algo;
initialize(shards);
}
private void initialize(List<S> shards) {
nodes = new TreeMap<>();
// Iterate over each redis node and set the hash mapping to the node
for (int i = 0; i ! = shards.size(); ++i) {final S shardInfo = shards.get(i);
// Map to no more than 160 virtual nodes according to the weight
int N = 160 * shardInfo.getWeight();
if (shardInfo.getName() == null) for (int n = 0; n < N; n++) {
// Build hash values and node mappings
nodes.put(this.algo.hash("SHARD-" + i + "-NODE-" + n), shardInfo);
}
else for (int n = 0; n < N; n++) {
nodes.put(this.algo.hash(shardInfo.getName() + "*" + n), shardInfo);
}
// Save access objects for each noderesources.put(shardInfo, shardInfo.createResource()); }}}Copy the code5.2 Access Procedure
The access process of ShardedJedis is the calculation process of consistent Hash. The core logic is as follows: Hash algorithm is used to calculate the Hash value of the accessed key, the corresponding Redis node is obtained according to the Hash value, and the corresponding access object Jedis is obtained according to the corresponding Redis node.
After obtaining the access object Jedis, command operations can be performed directly.
public class Sharded<R.S extends ShardInfo<R>> {
public static final int DEFAULT_WEIGHT = 1;
private TreeMap<Long, S> nodes;
private final Hashing algo;
// Save connection information for redis node and Jedis accessing this node
private final Map<ShardInfo<R>, R> resources = new LinkedHashMap<>();
public R getShard(String key) {
// Find the corresponding access object Jedis according to the redis node
return resources.get(getShardInfo(key));
}
public S getShardInfo(String key) {
return getShardInfo(SafeEncoder.encode(getKeyTag(key)));
}
public S getShardInfo(byte[] key) {
// Generate a hash value for the accessed key
// Find the corresponding redis node based on the hash value
SortedMap<Long, S> tail = nodes.tailMap(algo.hash(key));
if (tail.isEmpty()) {
return nodes.get(nodes.firstKey());
}
returntail.get(tail.firstKey()); }}Copy the codeSix, Jedis cluster mode access
Understand Jedis cluster mode access based on Redis cluster principle introduced earlier.
Jedis is able to locate keys and hash slots by mapping hash slots to Redis nodes, and this discovery process is based on the Cluster slot command of Redis.
Commands for Redis cluster operations: Redis via Cluster Slots returns the overall status of the Redis cluster. The information returned for each Redis node contains:
-
Hash slot start number
-
End hash slot number
-
The hash slot corresponds to the master node, which is represented by AN IP address /Port
-
The first copy of the master node
-
The second copy of the master node
127.0. 01.:30001> cluster slots
1) 1) (integer) 0 // Start slot
2) (integer) 5460 // End the slot
3) 1) "127.0.0.1" // Host of the master node
2) (integer) 30001 // Port of the master node
3) "09dbe9720cda62f7865eabc5fd8857c5d2678366" // Node code
4) 1) "127.0.0.1" // slave host of the slave node
2) (integer) 30004 // Slave node port
3) "821d8ca00d7ccf931ed3ffc7e3db0599d2271abf" // Node code
2) 1) (integer) 5461
2) (integer) 10922
3) 1) "127.0.0.1"
2) (integer) 30002
3) "c9d93d9f2c0c524ff34cc11838c2003d8c29e013"
4) 1) "127.0.0.1"
2) (integer) 30005
3) "faadb3eb99009de4ab72ad6b6ed87634c7ee410f"
3) 1) (integer) 10923
2) (integer) 16383
3) 1) "127.0.0.1"
2) (integer) 30003
3) "044ec91f325b7595e76dbcb18cc688b6a5b434a1"
4) 1) "127.0.0.1"
2) (integer) 30006
3) "58e6e48d41228013e5d9c1c37c5060693925e97e"
Copy the codeThe overall flow chart of Jedis accessing cluster mode Redis is shown below. It can be seen from the figure that the core process includes the creation of JedisCluster object and the access to Redis through JedisCluster object.
The core of creating a JedisCluster object is to create a JedisClusterInfoCache object and establish the mapping between slot and cluster nodes through cluster discovery.
JedisCluster accesses the Redis cluster by obtaining the Redis node where the key resides and accessing it through the Jedis object.
6.1 Creation Process
JedisCluster class relations as shown in the figure below, you can see in figure core variables JedisSlotBasedConnectionHandler object.
JedisCluster parent BinaryJedisCluster created JedisSlotBasedConnectionHandler object, the object is responsible for communicating and Redis cluster.
public class JedisCluster extends BinaryJedisCluster implements JedisClusterCommands.MultiKeyJedisClusterCommands.JedisClusterScriptingCommands {
public JedisCluster(Set<HostAndPort> jedisClusterNode, int connectionTimeout, int soTimeout,
int maxAttempts, String password, String clientName, final GenericObjectPoolConfig poolConfig,
boolean ssl, SSLSocketFactory sslSocketFactory, SSLParameters sslParameters,
HostnameVerifier hostnameVerifier, JedisClusterHostAndPortMap hostAndPortMap) {
// Access the parent class BinaryJedisCluster
super(jedisClusterNode, connectionTimeout, soTimeout, maxAttempts, password, clientName, poolConfig, ssl, sslSocketFactory, sslParameters, hostnameVerifier, hostAndPortMap); }}public class BinaryJedisCluster implements BinaryJedisClusterCommands.MultiKeyBinaryJedisClusterCommands.JedisClusterBinaryScriptingCommands.Closeable {
public BinaryJedisCluster(Set<HostAndPort> jedisClusterNode, int connectionTimeout, int soTimeout,
int maxAttempts, String user, String password, String clientName, GenericObjectPoolConfig poolConfig,
boolean ssl, SSLSocketFactory sslSocketFactory, SSLParameters sslParameters,
HostnameVerifier hostnameVerifier, JedisClusterHostAndPortMap hostAndPortMap) {
/ / create JedisSlotBasedConnectionHandler object
this.connectionHandler = new JedisSlotBasedConnectionHandler(jedisClusterNode, poolConfig,
connectionTimeout, soTimeout, user, password, clientName, ssl, sslSocketFactory, sslParameters, hostnameVerifier, hostAndPortMap);
this.maxAttempts = maxAttempts; }}Copy the codeJedisSlotBasedConnectionHandler core is to create and initialize JedisClusterInfoCache object, the object cache with Redis cluster information.
InitializeSlotsCache is used to initialize the JedisClusterInfoCache object, which is used to discover cluster nodes and slots.
public class JedisSlotBasedConnectionHandler extends JedisClusterConnectionHandler {
public JedisSlotBasedConnectionHandler(Set<HostAndPort> nodes, GenericObjectPoolConfig poolConfig,
int connectionTimeout, int soTimeout, String user, String password, String clientName,
boolean ssl, SSLSocketFactory sslSocketFactory, SSLParameters sslParameters,
HostnameVerifier hostnameVerifier, JedisClusterHostAndPortMap portMap) {
super(nodes, poolConfig, connectionTimeout, soTimeout, user, password, clientName, ssl, sslSocketFactory, sslParameters, hostnameVerifier, portMap); }}public abstract class JedisClusterConnectionHandler implements Closeable {
public JedisClusterConnectionHandler(Set<HostAndPort> nodes, final GenericObjectPoolConfig poolConfig,
int connectionTimeout, int soTimeout, int infiniteSoTimeout, String user, String password, String clientName,
boolean ssl, SSLSocketFactory sslSocketFactory, SSLParameters sslParameters,
HostnameVerifier hostnameVerifier, JedisClusterHostAndPortMap portMap) {
// Create JedisClusterInfoCache
this.cache = new JedisClusterInfoCache(poolConfig, connectionTimeout, soTimeout, infiniteSoTimeout,
user, password, clientName, ssl, sslSocketFactory, sslParameters, hostnameVerifier, portMap);
// Initialize Slot information for jedis
initializeSlotsCache(nodes, connectionTimeout, soTimeout, infiniteSoTimeout,
user, password, clientName, ssl, sslSocketFactory, sslParameters, hostnameVerifier);
}
private void initializeSlotsCache(Set<HostAndPort> startNodes,
int connectionTimeout, int soTimeout, int infiniteSoTimeout, String user, String password, String clientName,
boolean ssl, SSLSocketFactory sslSocketFactory, SSLParameters sslParameters, HostnameVerifier hostnameVerifier) {
for (HostAndPort hostAndPort : startNodes) {
try (Jedis jedis = new Jedis(hostAndPort.getHost(), hostAndPort.getPort(), connectionTimeout,
soTimeout, infiniteSoTimeout, ssl, sslSocketFactory, sslParameters, hostnameVerifier)) {
/ / found through discoverClusterNodesAndSlots clusters
cache.discoverClusterNodesAndSlots(jedis);
return;
} catch (JedisConnectionException e) {
}
}
}
}
Copy the codeThe JedisClusterInfoCache nodes function stores Redis cluster node information. The slots function stores slots and cluster node information.
Nodes and Slots maintain objects that are JedisPool objects, which maintain connection information to Redis. Cluster of discovery process by discoverClusterNodesAndSlots, essence is to find command execute Redis cluster cluster slots.
public class JedisClusterInfoCache {
// Save redis cluster node information
private final Map<String, JedisPool> nodes = new HashMap<>();
// Save the mapping between redis slots and Redis nodes
private final Map<Integer, JedisPool> slots = new HashMap<>();
// Take charge of cluster discovery logic
public void discoverClusterNodesAndSlots(Jedis jedis) {
w.lock();
try {
reset();
List<Object> slots = jedis.clusterSlots();
for (Object slotInfoObj : slots) {
List<Object> slotInfo = (List<Object>) slotInfoObj;
if (slotInfo.size() <= MASTER_NODE_INDEX) {
continue;
}
// Obtain the slot information of the redis node
List<Integer> slotNums = getAssignedSlotArray(slotInfo);
// hostInfos
int size = slotInfo.size();
for (int i = MASTER_NODE_INDEX; i < size; i++) {
List<Object> hostInfos = (List<Object>) slotInfo.get(i);
if (hostInfos.isEmpty()) {
continue;
}
HostAndPort targetNode = generateHostAndPort(hostInfos);
// Save the redis node information
setupNodeIfNotExist(targetNode);
if (i == MASTER_NODE_INDEX) {
// Save the mapping between slots and redis nodesassignSlotsToNode(slotNums, targetNode); }}}}finally{ w.unlock(); }}public void assignSlotsToNode(List<Integer> targetSlots, HostAndPort targetNode) {
w.lock();
try {
JedisPool targetPool = setupNodeIfNotExist(targetNode);
// Save the slot and the corresponding JedisPool object
for(Integer slot : targetSlots) { slots.put(slot, targetPool); }}finally{ w.unlock(); }}public JedisPool setupNodeIfNotExist(HostAndPort node) {
w.lock();
try {
// Produce the nodeKey of the redis node
String nodeKey = getNodeKey(node);
JedisPool existingPool = nodes.get(nodeKey);
if(existingPool ! =null) return existingPool;
// Produce the JedisPool corresponding to the redis node
JedisPool nodePool = new JedisPool(poolConfig, node.getHost(), node.getPort(),
connectionTimeout, soTimeout, infiniteSoTimeout, user, password, 0, clientName,
ssl, sslSocketFactory, sslParameters, hostnameVerifier);
// Save the redis node key and the corresponding JedisPool object
nodes.put(nodeKey, nodePool);
return nodePool;
} finally{ w.unlock(); }}}Copy the codeThe following figure shows the class relationship of JedisPool. The internal internalPool is pooled through the Apache Common Pool.
The JedisPool internalPool creates Jedis objects through the makeObject of the JedisFactory.
Each Redis node will correspond to a JedisPool object, through JedisPool to manage Jedis application release reuse.
public class JedisPool extends JedisPoolAbstract {
public JedisPool(a) {
this(Protocol.DEFAULT_HOST, Protocol.DEFAULT_PORT); }}public class JedisPoolAbstract extends Pool<Jedis> {
public JedisPoolAbstract(a) {
super();
}
}
public abstract class Pool<T> implements Closeable {
protected GenericObjectPool<T> internalPool;
public void initPool(final GenericObjectPoolConfig poolConfig, PooledObjectFactory<T> factory) {
if (this.internalPool ! =null) {
try {
closeInternalPool();
} catch (Exception e) {
}
}
this.internalPool = newGenericObjectPool<>(factory, poolConfig); }}class JedisFactory implements PooledObjectFactory<Jedis> {
@Override
public PooledObject<Jedis> makeObject(a) throws Exception {
// Create a Jedis object
final HostAndPort hp = this.hostAndPort.get();
final Jedis jedis = new Jedis(hp.getHost(), hp.getPort(), connectionTimeout, soTimeout,
infiniteSoTimeout, ssl, sslSocketFactory, sslParameters, hostnameVerifier);
try {
// Jedis object connection
jedis.connect();
if(user ! =null) {
jedis.auth(user, password);
} else if(password ! =null) {
jedis.auth(password);
}
if(database ! =0) {
jedis.select(database);
}
if(clientName ! =null) { jedis.clientSetname(clientName); }}catch (JedisException je) {
jedis.close();
throw je;
}
// Return the Jedis object wrapped as DefaultPooledObject
return newDefaultPooledObject<>(jedis); }}Copy the code6.2 Access Procedure
JedisCluster accesses Redis through JedisClusterCommand to achieve retry mechanism, and finally through Jedis object to achieve access. From the perspective of implementation, JedisCluster encapsulates a layer on top of Jedis for cluster node location and retry mechanism.
Taking the set command as an example, the entire access is achieved through JedisClusterCommand as follows:
-
Compute the Redis node where the key resides.
-
Get the Jedis object corresponding to the Redis node.
-
Set operations are performed on Jedis objects.
public class JedisCluster extends BinaryJedisCluster implements JedisClusterCommands.MultiKeyJedisClusterCommands.JedisClusterScriptingCommands {
@Override
public String set(final String key, final String value, final SetParams params) {
return new JedisClusterCommand<String>(connectionHandler, maxAttempts) {
@Override
public String execute(Jedis connection) {
returnconnection.set(key, value, params); } }.run(key); }}Copy the codeThe core of the Run method of JedisClusterCommand is to locate the Redis node where the key of Redis is located, and then access the Jedis object corresponding to the node.
When Jedis objects fail to access, JedisClusterCommand retries and executes the renewSlotCache method to rediscover nodes in the cluster.
public abstract class JedisClusterCommand<T> {
public T run(String key) {
// Compute slots for keys
return runWithRetries(JedisClusterCRC16.getSlot(key), this.maxAttempts, false.null);
}
private T runWithRetries(final int slot, int attempts, boolean tryRandomNode, JedisRedirectionException redirect) {
Jedis connection = null;
try {
if(redirect ! =null) {
connection = this.connectionHandler.getConnectionFromNode(redirect.getTargetNode());
if (redirect instanceofJedisAskDataException) { connection.asking(); }}else {
if (tryRandomNode) {
connection = connectionHandler.getConnection();
} else {
// Get the Jedis object according to slotconnection = connectionHandler.getConnectionFromSlot(slot); }}// Execute the real Redis command
return execute(connection);
} catch (JedisNoReachableClusterNodeException jnrcne) {
throw jnrcne;
} catch (JedisConnectionException jce) {
releaseConnection(connection);
connection = null;
if (attempts <= 1) {
// Ensure the last two opportunities to refresh the information corresponding to slots and nodes
this.connectionHandler.renewSlotCache();
}
// Retry the operation according to the number of retries
return runWithRetries(slot, attempts - 1, tryRandomNode, redirect);
} catch (JedisRedirectionException jre) {
// If the Move command is returned, the information about slots and nodes will be refreshed immediately
if (jre instanceof JedisMovedDataException) {
// it rebuilds cluster's slot cache recommended by Redis cluster specification
this.connectionHandler.renewSlotCache(connection);
}
releaseConnection(connection);
connection = null;
return runWithRetries(slot, attempts - 1.false, jre);
} finally{ releaseConnection(connection); }}}Copy the codeJedisSlotBasedConnectionHandler cache object maintains the slot and the mapping relationship between node, through getConnectionFromSlot method to obtain the corresponding Jedis slot object.
public class JedisSlotBasedConnectionHandler extends JedisClusterConnectionHandler {
protected final JedisClusterInfoCache cache;
@Override
public Jedis getConnectionFromSlot(int slot) {
// Obtain the JedisPool object corresponding to the slot
JedisPool connectionPool = cache.getSlotPool(slot);
if(connectionPool ! =null) {
// Get the Jedis object from the JedisPool object
return connectionPool.getResource();
} else {
// Refresh the slot information on failure
renewSlotCache();
connectionPool = cache.getSlotPool(slot);
if(connectionPool ! =null) {
return connectionPool.getResource();
} else {
//no choice, fallback to new connection to random node
returngetConnection(); }}}}Copy the codeSeven, Jedis Pipeline implementation
The technical core idea of a Pipeline is to send multiple commands to the server without waiting for a response, and finally to read the response in one step. The advantage of this mode is to save the network overhead of the request response mode.
The core difference between Redis common commands such as SET and Pipeline batch operation is that the operation of set command will directly send a request to Redis and wait for the result to return synchronously, while the operation of Pipeline will send a request but not immediately wait for the result to return synchronously. The specific implementation can be seen from Jedis source code.
Native pipelines Hash keys to the same node because only one of the Client objects can be connected.
In cluster mode, the key belonging to different nodes can use Pipeline, which needs to save the client object of the corresponding node for each key, and obtain the data at the end of the execution. In essence, a cluster Pipeline can be encapsulated on the basis of a single-node Pipeline.
7.1 Analysis of Pipeline usage
When a Pipeline accesses a single-node Redis, the Pipeline object is returned via the Pipeline method of the Jedis object, and other command operations are accessed through the Pipeline object.
Pipelines analyze from a usage perspective, sending multiple commands in batches and finally using syncAndReturnAll to return results all at once.
public void pipeline(a) {
jedis = new Jedis(hnp.getHost(), hnp.getPort(), 500);
Pipeline p = jedis.pipelined();
// Batch send commands to redis
p.set("foo"."bar");
p.get("foo");
// Wait for the response result synchronously
List<Object> results = p.syncAndReturnAll();
assertEquals(2, results.size());
assertEquals("OK", results.get(0));
assertEquals("bar", results.get(1));
}
public abstract class PipelineBase extends Queable implements BinaryRedisPipeline.RedisPipeline {
@Override
public Response<String> set(final String key, final String value) {
// Send the command
getClient(key).set(key, value);
// Pipeline's getResponse simply aggregates the requests to be responded into the pipelinedResponses object
returngetResponse(BuilderFactory.STRING); }}public class Queable {
privateQueue<Response<? >> pipelinedResponses =new LinkedList<>();
protected <T> Response<T> getResponse(Builder<T> builder) {
Response<T> lr = new Response<>(builder);
// Save to the response queue
pipelinedResponses.add(lr);
returnlr; }}public class Pipeline extends MultiKeyPipelineBase implements Closeable {
public List<Object> syncAndReturnAll(a) {
if (getPipelinedResponseLength() > 0) {
// According to the number of commands sent in batches, that is, the number of commands to be returned in batches, the client object reads the commands in batches
List<Object> unformatted = client.getMany(getPipelinedResponseLength());
List<Object> formatted = new ArrayList<>();
for (Object o : unformatted) {
try {
// Format each returned result and finally save it in the list for return
formatted.add(generateResponse(o).get());
} catch(JedisDataException e) { formatted.add(e); }}return formatted;
} else {
returnjava.util.Collections.<Object> emptyList(); }}}Copy the codeThe normal set command sends a request to Redis and immediately gets the response via getStatusCodeReply, so this is a pattern of request response.
GetStatusCodeReply forces the Redis server to flush() when obtaining the response result and then reads the response result.
public class BinaryJedis implements BasicCommands.BinaryJedisCommands.MultiKeyBinaryCommands.AdvancedBinaryJedisCommands.BinaryScriptingCommands.Closeable {
@Override
public String set(final byte[] key, final byte[] value) {
checkIsInMultiOrPipeline();
// Send the command
client.set(key, value);
// Wait for the request response
returnclient.getStatusCodeReply(); }}public class Connection implements Closeable {
public String getStatusCodeReply(a) {
// Send requests immediately with flush
flush();
// Process the response request
final byte[] resp = (byte[]) readProtocolWithCheckingBroken();
if (null == resp) {
return null;
} else {
returnSafeEncoder.encode(resp); }}}public class Connection implements Closeable {
protected void flush(a) {
try {
// Flush the output stream to ensure that packets are sent
outputStream.flush();
} catch (IOException ex) {
broken = true;
throw newJedisConnectionException(ex); }}}Copy the codeViii. Concluding remarks
Jedis as Redis official preferred Java client development package, support the vast majority of Redis commands, is also the daily use of Redis client.
After understanding the implementation principle of Jedis, it can not only support the daily operation of Redis, but also better deal with the additional operation of Redis, such as the technology selection in capacity expansion.
Through the introduction of Jedis for single machine mode, distribution mode, cluster mode three scenarios access, let everyone have a macro to micro understanding process, master the core idea of Jedis and better application in practice.
Author: Wang Zhi, Vivo Internet Server Team