Welcome to the column: Advances in Java Architecture Technology. There are a lot of batJ interview questions, and a variety of technical sharing, if you have a good article, welcome to submit.

Wechat official account: Architectural Notes of Murong Qianyu. Welcome to make progress together.

What is Redis?

Redis is a key-value storage system. Like Memcached, it supports storing a relatively large number of value types, including String (string), list(linked list), set(collection), zset(sorted set — ordered set), and hash (hash). As with Memcached, the data is cached in memory for efficiency. The difference is that Redis periodically writes the updated data to disk or the modified operation to the appended record file, and on this basis realizes master-slave synchronization.

Redis is a high performance key-value database. Redis largely compensates for memcached key/value storage, and in some cases complements relational databases. It provides Java, C/C++, C#, PHP, JavaScript, Perl, object-C, Python, Ruby, Erlang and other clients, easy to use. [1]

Redis supports master/slave synchronization. Data can be synchronized from the primary server to any number of secondary servers, which can be the primary server associated with other secondary servers.

The difference between Redis and Memcached

U Persistence:

L Redis can be used to do cache, also can do storage; ADF and RDB persistence modes are supported

L Memcached can only cache data

U Data structure:

L Redis has a wealth of data types: strings, linked lists, Hash, sets, ordered sets

Memcached is generally just strings and objects

(iii) Installation and configuration of Redis

1. Decompress: tar -zxvf redis-3.0.5.tar.gz

2. Make

3. Make PREFIX=/root/training/redis install

4. Cp ~ / tools/redis – 3.0.5 / redis. Conf/root/training/redis/etc /

The core configuration file of Redis: redis.conf

Redis command script:

Redis-benchmark Performance test tool

Redis-check-aof Checks AOF logs

Redis-check-dump checks RDB logs

Redis – CLI Starts command line client

Redis – sentinel

Redis-server Starts the Redis service

Start the Redis:

. / redis – server.. /etc/redis6379.conf

. / redis – server.. /etc/redis6380.conf

This starts an instance of Redis on port 6379 and port 6380 respectively. You can also run the ps command to view:

Start the Redis client: Redis -cli

L Port 6739 is connected by default. You can also specify the port number by using -p:

L./redis-cli –help Displays the help information

(iv) Operation of Redis

www.redis.cn/commands.ht…

Key Operation key of redis is stored in string. The common operation instructions of Redis are as follows:

1. Select db_name from db_name where db_name is used

2. Exits Key_name Checks whether the specified key exists

3. Get key_name Obtains the value of the specified key

4. mget key1 [key2..] Gets the value of the specified multiple keys,

Get a randomkey

6. Set key_name key_value Sets key-value

7. Getset key_name key_value Sets key-value and returns the old key value

8. Setnx key_name Value Specifies the key only when the key does not exist

9. Mset key1 val1[key1 val2…] Multiple key-values are set at the same time

10. Del key_name Deletes the specified key

11. Rename key_name new_key_name Renames a specified key

12. Type key_name Returns the value type of the specified key

13. Dump key_name serializes the specified key and returns the serialized value of the key

Keys pattern Searches for all keys that match the specified pattern. * is supported as a wildcard character.

Keys a* # find all keys that start with ‘a’

Keys * # find all keys

17. Move key_name db_name Moves the specified key-value to the specified database

Expire key_name seconds pexpire key_name milliseconds Sets the expiration time for the specified key, in seconds and milliseconds

19. Expireat key_name timestamp Sets the expiration of the specified key at the specified UNIX timestamp

20. TTL key_name PTTL key_name Returns the TTL (Time to Live) of the key in seconds and milliseconds.

21. Persist key_name Removes the expiration time of the specified key, which will persist

22. Flushdb Deletes all keys in the current database

23. Flushall Deletes all database keys

Other operations can be viewed in the link

The data type

(1) the string

(2) list

(3) the Hash

④ Disordered set

⑤ Ordered set

⑥ Redis data type case analysis: website statistics of user login times

  1. 100 million users, some who log in regularly, some who log in infrequently
  2. How to record user login information
  3. How do I query active users: for example, those who log in three times in one week

Solution 1: Use a relational database


Solution 2: Use Redis to store login information. You can use Redis setbit. Whether to log in or not is indicated by 1 and 0


25. Java client

① Basic operations

(2) the connection pool

③ Use Redis to realize distributed lock

Build projects using Maven:

redis.clients

jedis

2.9.0

(v) Transaction and message mechanism of Redis

1. Redis transactions

Redis transaction support is relatively simple at this point. Redis can only ensure that the commands in a transaction initiated by one client can be executed consecutively without inserting commands from other clients. This is easy to do because Redis handles all client requests in a single thread. Normally redis receives a command from a client and returns the result immediately. However, when a client issues a multi command in a connection, the connection enters a transaction context and subsequent commands are not executed immediately, but are placed in a queue. When an exec command is received from this connection, Redis executes all commands in the queue sequentially. The execution results of all commands are packaged together and returned to the client. The connection then terminates the transaction context.

Oracle database transactions compared to Redis transactions

2. Redis transaction example: bank transfer

Transfer $100 from Tom to Mike

  • set tom 1000
  • set mike 1000
  • multi
  • decrby tom 100
  • incrby mike 100
  • exec

3. Redis lock mechanism: Watch

L Example: Buying tickets

4. Transactions and locks in Java applications

(1) transaction

(2) the lock

5. Message mechanism of Redis: message publishing and subscription, suitable for online chat

Publish: Publishes the message

Publish channel name “Message content”

Subscribe: Subscribes messages

Format: Subscribe Channel name

Psubscribe: Uses wildcards to define messages

Format: pSUBSCRIBE Channel * name

Using Java programs to publish and subscribe messages, you need to inherit the JedisPubSub class

(6) Persistence of Redis

Redis provides several different levels of persistence:

  • RDB persistence can generate point-in-time snapshots of data sets at specified intervals.
  • ** AOF (append-only file) ** Persists a record of all write operation commands executed by the server and restores the data set by re-executing these commands when the server starts. All commands in the AOF file are saved in Redis format, and new commands are appended to the end of the file. Redis can also rewrite AOF files in the background so that the size of the AOF file does not exceed the actual size needed to store the state of the dataset.
  • Redis can also use both AOF and RDB persistence. In this case, when Redis restarts, it will use AOF files to restore the dataset in preference, since AOF files usually hold more complete datasets than RDB files.
  • You can even turn off persistence so that data only exists while the server is running.

1, RDB,

Working principle: Take a snapshot for the MEMORY at intervals and write the data in the memory to a file (RDB file).

  • RDB Sample test: You can use Redis-Benchmark for stress testing
  1. /bin/redis-benchmark -n 100000 Indicates that 100000 operations are performed
  • Disadvantages of RDB:
  1. Between snapshots, if a power outage occurs, data is lost
  2. Example: After generating an RDB, insert a new value. Sudden power outages can cause data loss

2. AOF: Log

  • How it works: Records commands for operations

  • Configuration parameters:

  • What is the rewriting of AOF: rewrite

  1. If you reverse generate a key in memory, as if you could, the AOF file would record 100 operations after 100 repetitions, which would cause the AOF file to be too large

Such as:

set age 0

incr age

incr age

. 100 times

And finally, age is 100

After rewriting, directly execute: set age 100

2. Check the size of aOF log files

3, Redis persistent attention to the problem

  • The RDB recovers quickly
  • If both RDB and AOF exist, AOF is used for recovery by default

(7) Clusters of Redis

1. The role of clusters

L Primary/secondary backup prevents host downtime

L Read and write tasks are separated and shared by the master

L Task separation, such as backup and computing work shared by slave servers

2. Two deployment modes of Redis cluster


3. Communication principle of Redis master-slave service


4. Configuring Redis clustering (master-slave mode)

  • Primary node: Disable RDB and AOF
  • Slave node: slaveof localhost 6379 Enables RDB and AOF

5. High availability of Redis cluster

  • Redis 2.4+ comes with an HA implementation, Sentinel

  • Configuration file: sentinel.conf

  • redis-sentinel .. /etc/sentinel.conf

  • View logs:

6. Implement proxy sharding for Redis

  • Twemproxy is a proxy sharding mechanism open-source by Twitter.
  • Twemproxy as a proxy, can accept access from multiple programs, according to the routing rules, forward to the background of each Redis server, and then back to the original route. This scheme solves the problem of carrying capacity of single Redis instance well.

  • The installation

    ./configure –prefix=/root/training/proxy

    make

    make install

  • The configuration file

  • Check whether the configuration file is correct./nutcracker -t conf/nutcracker

  • Start the proxy server./nutcracker -d -c conf/nutcracker. Yml

(8)RedisCluster**** Distributed solution

1. What is Redis Cluster?

Redis Cluster is a distributed solution of Redis, officially launched in Redis 3.0 version, effectively solve the needs of Redis distributed. Cluster architecture can be used to achieve load balancing when the bottleneck of single memory, concurrency, and traffic is encountered.

  • Bottlenecks in the use of Redis

    In our daily use of Redis, we often encounter some problems:

    (1) High availability, how to ensure the continuous high availability of Redis.

    (2) Capacity problem, single instance Redis memory can not be infinitely expanded, after 32G into the 64-bit world, performance decline.

    (3) Concurrency performance problems. Redis claims 100,000 concurrent instances, but there is an end.

  • Redis – the advantage of cluster

    (1) Official recommendation, no doubt.

    (2) Decentralization: the cluster can increase 1000 nodes at most, and the performance expands linearly with the increase of nodes.

    (3) Convenient management. Nodes can be added or removed, slots can be moved, etc.

    (4) Simple and easy to use.

2. Data distribution theory and data partitioning of Redis

Distributed databases primarily solve the problem of mapping the entire data set to multiple nodes according to partitioning rules, that is, the data set is divided into multiple nodes, each node is responsible for a subset of the entire data. Common partitioning rules are hash and sequential partitioning. Redis Cluster uses hash partitioning rules.

Virtual slot partitioning is a clever use of hash space, using well-distributed hash functions to map all data to a fixed range of integers defined as slots. For example, the Redis Cluster slot ranges from 0 to 16383. A slot is the basic unit for data management and migration in a cluster.

Redis Cluster uses virtual slot partitions. All keys are mapped from 0 to 16383 according to the hash function. Slot = CRC16(key)&16383. Each node is responsible for maintaining a portion of slots and the key data mapped to the slots.

3. Architecture of Redis Cluster

We take six nodes as an example to introduce the architecture of Redis Cluster. Three are master nodes and the other three are slave nodes.

4. Installation and deployment (Mode 1: Using a configuration file)

① Compilation and installation of Redis

  • Install Linux operating system and GCC compilation

  • Compile and install Redis

    Decompress: tar -zxvf redis-3.0.5.tar.gz

    make

    make PREFIX=/root/training/redis install

    Cp ~ / tools/redis – 3.0.5 / redis. Conf/root/training/redis/etc /

② Install the Ruby environment and Ruby Redis interface

This tool is located in the SRC folder of redis source code. It is a Ruby program. This program creates a new cluster by sending special commands to the instance, and checks the cluster. Or the cluster is reshared, so you need to install the Ruby environment and the corresponding Redis interface

Here is how you can install Ruby using yum:

[media]

Name =Red Hat Enterprise Linux 7.4

baseurl=file:///cdroom

enabled=1

gpgcheck=1

gpgkey=file:///cdroom/RPM-GPG-KEY-redhat-release

3 Install and deploy the Redis Cluster on six nodes

  • Primary nodes: 6379, 6380, 6381

  • Slave nodes: 6382, 6383, 6384

  • The parameters that need to be modified are as follows: The red font needs to be modified by each instance.

    daemonize yes

    port 6379

    cluster-enabled yes

    cluster-config-file nodes/nodes-6379.conf

    cluster-node-timeout 15000

    dbfilename dump6379.rdb

    appendonly yes

    appendfilename “appendonly6379.aof“

  • There are six configuration files:

    redis6379.conf

    redis6380.conf

    redis6381.conf

    redis6382.conf

    redis6383.conf

    redis6384.conf

④ Start the Redis instance

bin/redis-server conf/redis6379.conf

bin/redis-server conf/redis6380.conf

bin/redis-server conf/redis6381.conf

bin/redis-server conf/redis6382.conf

bin/redis-server conf/redis6383.conf

bin/redis-server conf/redis6384.conf

Run the ps command to view the process:

⑤ Use redis-trib.rb automatic deployment mode

Bin /redis-trib.rb create –replicas 1 192.168.56.72:6379 192.168.56.72:6380 192.168.56.72:6381 192.168.56.72:6382 192.168.56.72 192.168.56.72:6383:6384

Note:redis-trib.rbLocated in the SRC folder of Redis source codeCopy toBin directory.

– replicas 1 indicates that we want to create a slave node for each primary node in the cluster.

Start configuring the cluster:

⑥ Test the Redis Cluster

  • Login from a client: bin/redis-cli -c -p 6379

    -c indicates logging in to the cluster

    You can run the cluster nodes command to view nodes in the cluster

5. Installation and deployment (Mode 2: run the create-cluster command)

Copy the create-cluster source code from utils/create-cluster to the bin directory in the installation directory

Example Change the path of the create-cluster command

  • Start and create the Redis cluster
  1. bin/create-cluster start
  2. bin/create-cluster create
  • Test the cluster using the client

**6. Use Java program to test Redis Cluster

Jar package: Commons – pool2-2.3. Jar jedis – 2.7.0. Jar * *

7. Basic Redis Cluster management

Use the cluster command to manage the Redis cluster:

Command description: Cluster info


Command description: Cluster Nodes (you can use Excel to import command output, easy to view)


A few notes:

  • Ping-sent: Indicates the time when the last ping was sent. This time is a Unix millisecond timestamp. 0 indicates that no ping was sent.
  • Pong-recv: The last time pong was received, represented by a Unix timestamp.
  • Config-epoch: indicates the epoch value of the node (or of the current master if the node is a slave). Each time a node fails to switch, a new, unique, and incremental epoch is created. If multiple nodes compete for the same hash slot, the node with the higher EPOCH value will grab it.

Cluster slots


A few other commands:


Add and delete nodes

  • Adding nodes (2 at a time)

① Add new configuration files redis6385.conf and redis6386.conf

2 Modify parameters based on previous configurations

③ Start a new node

bin/redis-server conf/redis6385.conf

bin/redis-server conf/redis6386.conf

4 Add an active node

Bin/redis – trib. Rb add -node 192.168.56.72:6385 192.168.56.72:6379

Among them:

192.168.56.72:6385 is a newly added node

192.168.56.72:6379 is an old node in the cluster

⑤ Check the node information in the cluster

Bin/redis – trib. Rb check 192.168.56.72:6379

⑥ Adding a secondary node

Bin/redis – trib. Rb add – node, slave, master – id f27013eaebcf998078edf1621e4433534f8d8b11 192.168.56.72:6386 192.168.56.72:6379

Among them:

–slave: Indicates that a slave node is added

–master-id: indicates the master node of the slave node

192.168.56.72:6386: Newly added secondary node

192.168.56.72:6379: specifies an old node in the cluster

⑦ Check the cluster status again

Bin/redis – trib. Rb check 192.168.56.72:6379

8 Reassign slots

Bin/redis – trib. Rb reshard 192.168.56.72:6379

⑨ Check the information about the node slot in the cluster again. You can see the information about the newly added node.

cluster slots

Or use:

Bin/redis – trib. Rb check 192.168.56.72:6379

Remove nodes

① Delete the secondary node first:

Bin/redis – trib. Rb del -node 192.168.56.72:6385 21 e3e82610ce206708aa8f4189e2c9b2f3d1c540

② Delete the primary node

Bin/redis – trib. Rb del -node 192.168.56.72:6386 f757ce35102b3aaff3621b539e2dc675eafe3c01

③ If the following error occurs, clear (move) the slot on the node before deleting it

Bin /redis-trib.rb reshard 192.168.56.72:6386

Step 2: bin/redis – trib. Rb del -node 192.168.56.72:6386 f757ce35102b3aaff3621b539e2dc675eafe3c01

4 Check the cluster information again. The node has been deleted

Bin/redis – trib. Rb check 192.168.56.72:6379

8.Redis Cluster node migration fails

① First, to see the effect of the demonstration, clear the previous Settings, again using six nodes as an example

② Start each node and create a cluster

bin/redis-server conf/redis6379.conf

bin/redis-server conf/redis6380.conf

bin/redis-server conf/redis6381.conf

bin/redis-server conf/redis6382.conf

bin/redis-server conf/redis6383.conf

bin/redis-server conf/redis6384.conf

Bin /redis-trib.rb create –replicas 1 192.168.56.72:6379 192.168.56.72:6380 192.168.56.72:6381 192.168.56.72:6382 192.168.56.72 192.168.56.72:6383:6384

3 Check the cluster status

Bin/redis – trib. Rb check 192.168.56.72:6379

(4) Kill the primary node on 6379 and check the cluster status again

⑤ Wait for a while and check the cluster status again

⑥ If you restart the instance on 6379, it will become the slave node of 6382

Wechat official account: Architectural Notes of Murong Qianyu. Welcome to make progress together.