This is my 12th day of the August Challenge
Lettuce is a Redis connection pool. Unlike Jedis, Lettuce is an asynchronous connection pool implemented mainly based on Netty and ProjectReactor. Because it is based on ProjectReactor, it can be used directly for spring-WebFlux asynchronous projects, and of course, synchronous interfaces are provided.
In our microservices project, we use Spring Boot and Spring Cloud. And use spring-data-Redis as a library to connect to Redis. And the connection pool uses Lettuce. Meanwhile, our online JDK is OpenJDK 11 LTS, and every process has JFR records open. For JFR, refer to this series: JFR Full Solutions
After Lettuce 6.1, Lettuce also introduced JFR-based monitoring events. Reference: events. The flight recorder
1. Redis connection related events:
- ConnectEvent: This event is emitted before an attempt is made to connect to Redis.
- ConnectedEvent:Event that is emitted when a connection is establishedThe remote IP address, port, and Redis URI used to establish the connection
ChannelHandlerIn thechannelActiveThe event that is emitted at the beginning of the callback. - ConnectionActivatedEvent: After completing a series of initialization operations for the Redis connection (such as SSL handshake, sending PING heartbeat command, etc.), this connection can be used for events that are emitted when the Redis command is executed.
- ConnectionDeactivatedEvent: without any command being processed and
isOpen()Is false,The connection is not active, ready to be shut down. That’s when this event is emitted. - DisconnectedEvent: This event is emitted when the connection is actually closed or reset.
- ReconnectAttemptEvent: Redis connections in Lettuce are maintained as persistent. When connections are lost, they are automatically reconnected. This event is emitted when reconnection is needed.
- ReconnectFailedEvent: This event is emitted when a connection is reconnected and failed.
2. Redis cluster related events:
- AskRedirectionEvent: This event is emitted when the Redis slot is in the migrated state and ASK is returned.
- MovedRedirectionEvent: This event is emitted if the Redis slot is not on the current node and returns MOVED.
- TopologyRefreshEvent: This event is emitted when querying cluster topologies if the scheduled task for cluster topologies refresh is enabled. However, the task of checking cluster topologies periodically needs to be enabled in the configuration. For details, see cluster-topology-refresh
- ClusterTopologyChangedEvent: when Lettuce found Redis cluster topology changes, will send out the event.
3. Redis command
- CommandLatencyEvent: Lettuce measures the response time of each command and emits this event regularly. This, too, needs to be manually configured, and we’ll see how to enable it later.
- CommandStartedEvent: This event is emitted when a command is started.
- CommandSucceededEvent: This event is emitted when a command is successfully executed.
- CommandFailedEvent: This event is emitted when a command fails to execute.
The monitoring of Lettuce is designed based on event distribution and monitoring mechanism, and its core interface is EventBus:
EventBus.java
Public interface EventBus {// get Flux<Event> get(); Void publish(Event Event); }Copy the codeIts default implementation is DefaultEventBus,
public class DefaultEventBus implements EventBus { private final DirectProcessor<Event> bus; private final FluxSink<Event> sink; private final Scheduler scheduler; private final EventRecorder recorder = EventRecorder.getInstance(); public DefaultEventBus(Scheduler scheduler) { this.bus = DirectProcessor.create(); this.sink = bus.sink(); this.scheduler = scheduler; } @override public Flux<Event> get() {return bus.onBackPressureDrop ().publishon (scheduler); } @override public void publish(Event Event) {// Call the recorder recorder. Record (Event); Next (event); // After calling the Recorder record, publish the event sink. }}Copy the codeIn the default implementation, we found that publishing an event first calls the Recorder record and then puts it into FluxSink for event publishing. At present, the actual implementation of the Recorder is based on JFR JfrEventRecorder. View the source code:
JfrEventRecorder
public void record(Event event) { LettuceAssert.notNull(event, "Event must not be null"); Jfr.event jfrEvent = createEvent(Event); // Create JFR Event(Event); if (jfrEvent ! = null) { jfrEvent.commit(); }} private jdK.jfr.event createEvent(Event Event) {try {// Get the constructor, if the constructor is an Object constructor, The Constructor of the JFR Event corresponding to this Event was not found. > constructor = getEventConstructor(event); if (constructor.getDeclaringClass() == Object.class) { return null; } // use constructor to create JFR Event return (jdk.jfr.event) constructor. } catch (ReflectiveOperationException e) { throw new IllegalStateException(e); }} private final Map<Class<? >, Constructor<? >> constructorMap = new HashMap<>(); private Constructor<? > getEventConstructor(Event event) throws NoSuchMethodException { Constructor<? > constructor; // If there is a JFR Event constructor for this class in the cache Map, return the JFR Event constructor. Synchronized (constructorMap) {constructor = constructorMap.get(event.getClass()); } if (constructor == null) {// if (constructor == null) {// if (constructor == null) {// if (constructor == null) {// if (constructor == null); The constructor String jfrClassName = event.getClass().getPackage().getName() + ".jfr "+ returns Object if it does not exist event.getClass().getSimpleName(); Class<? > eventClass = LettuceClassUtils.findClass(jfrClassName); if (eventClass == null) { constructor = Object.class.getConstructor(); } else { constructor = eventClass.getDeclaredConstructors()[0]; constructor.setAccessible(true); } synchronized (constructorMap) { constructorMap.put(event.getClass(), constructor); } } return constructor; }Copy the codeI found that this code was not very good and needed to acquire the lock every time I read it, so I made some changes and made a Pull Request: reformat getEventConstructor for JfrEventRecorder not to synchronize for each read
We can tell if an Event has a CORRESPONDING JFR Event by looking at the same path as a class that starts with JFR followed by its own name. It can be found that:
io.lettuce.core.event.connectionPackage:ConnectedEvent->JfrConnectedEventConnectEvent->JfrConnectedEventConnectionActivatedEvent->JfrConnectionActivatedEventConnectionCreatedEvent->JfrConnectionCreatedEventConnectionDeactivatedEvent->JfrConnectionDeactivatedEventDisconnectedEvent->JfrDisconnectedEventReconnectAttemptEvent->JfrReconnectAttemptEventReconnectFailedEvent->JfrReconnectFailedEvent
io.lettuce.core.cluster.eventPackage:AskRedirectionEvent->JfrAskRedirectionEventClusterTopologyChangedEvent->JfrClusterTopologyChangedEventMovedRedirectionEvent->JfrMovedRedirectionEventAskRedirectionEvent->JfrTopologyRefreshEvent
io.lettuce.core.event.commandPackage:CommandStartedEvent– > noCommandSucceededEvent– > noCommandFailedEvent– > no
io.lettuce.core.event.metricsPackage:,CommandLatencyEvent– > no
As you can see, there is currently no JFR monitoring for instructions, but for us, instruction monitoring is the most important. We consider adding JFR corresponding events for instruction-related events
If for io.lettuce.core.event.com mand package under the instructions to generate the corresponding JFR, then the number of events is a bit too much (an application instance we may carry out hundreds of thousands of every second, Redis instruction). So we tend to add JFR events for CommandLatencyEvent.
CommandLatencyEvent contains a Map:
private Map<CommandLatencyId, CommandMetrics> latencies;
Copy the codeCommandLatencyId contains the Redis connection information and the commands that are executed. CommandMetrics are time metrics that include:
- Time indicator for receiving a response from the Redis server to determine whether the Redis server is responding slowly.
- The time taken to process the response from the Redis server may be because the application instance is too busy. Compare this with the time taken to receive the response from the Redis server to determine the time taken to process the response.
Both indicators contain the following information:
- The shortest time
- The longest time
- Percentile time, default is top 50%, top 90%, top 95%, top 99%, top 99.9%
MicrometerOptions:Public static final double[] DEFAULT_TARGET_PERCENTILES = new double[] {0.50, 0.90, 0.95, 0.99, 0.999};
We want to enable JFR to view response time statistics over a period of time for each command on a different Redis server. We can do this:
package io.lettuce.core.event.metrics; import jdk.jfr.Category; import jdk.jfr.Event; import jdk.jfr.Label; import jdk.jfr.StackTrace; @Category({ "Lettuce", "Command Events" }) @Label("Command Latency Trigger") @StackTrace(false) public class JfrCommandLatencyEvent extends Event { private final int size; public JfrCommandLatencyEvent(CommandLatencyEvent commandLatencyEvent) { this.size = commandLatencyEvent.getLatencies().size(); commandLatencyEvent.getLatencies().forEach((commandLatencyId, commandMetrics) -> { JfrCommandLatency jfrCommandLatency = new JfrCommandLatency(commandLatencyId, commandMetrics); jfrCommandLatency.commit(); }); }}Copy the codepackage io.lettuce.core.event.metrics; import io.lettuce.core.metrics.CommandLatencyId; import io.lettuce.core.metrics.CommandMetrics; import jdk.jfr.Category; import jdk.jfr.Event; import jdk.jfr.Label; import jdk.jfr.StackTrace; import java.util.concurrent.TimeUnit; @Category({ "Lettuce", "Command Events" }) @Label("Command Latency") @StackTrace(false) public class JfrCommandLatency extends Event { private final String remoteAddress; private final String commandType; private final long count; private final TimeUnit timeUnit; private final long firstResponseMin; private final long firstResponseMax; private final String firstResponsePercentiles; private final long completionResponseMin; private final long completionResponseMax; private final String completionResponsePercentiles; public JfrCommandLatency(CommandLatencyId commandLatencyId, CommandMetrics commandMetrics) { this.remoteAddress = commandLatencyId.remoteAddress().toString(); this.commandType = commandLatencyId.commandType().toString(); this.count = commandMetrics.getCount(); this.timeUnit = commandMetrics.getTimeUnit(); this.firstResponseMin = commandMetrics.getFirstResponse().getMin(); this.firstResponseMax = commandMetrics.getFirstResponse().getMax(); this.firstResponsePercentiles = commandMetrics.getFirstResponse().getPercentiles().toString(); this.completionResponseMin = commandMetrics.getCompletion().getMin(); this.completionResponseMax = commandMetrics.getCompletion().getMax(); this.completionResponsePercentiles = commandMetrics.getCompletion().getPercentiles().toString(); }}Copy the codeIn this way, we can analyze these events as follows:
First, in the event browser, select Lettuce -> Command Events -> Command Latency, right-click and use Events to create a new page:
In the event page created, group by commandType and display the indicators of interest in the chart:
I also submitted a Pull Request to the community in response to these changes: Fix #1820 add JFR Event for Command Latency
In Spring Boot (which adds the Spring-boot-starter -redis dependency), we need to manually open the collection of CommandLatencyEvent:
@ the Configuration (proxyBeanMethods = false) @ Import ({LettuceConfiguration. Class}) / / need compulsory in RedisAutoConfiguration automatically loaded @AutoConfigureBefore(RedisAutoConfiguration.class) public class LettuceAutoConfiguration { }Copy the codeimport io.lettuce.core.event.DefaultEventPublisherOptions; import io.lettuce.core.metrics.DefaultCommandLatencyCollector; import io.lettuce.core.metrics.DefaultCommandLatencyCollectorOptions; import io.lettuce.core.resource.DefaultClientResources; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import java.time.Duration; @configuration (proxyBeanMethods = false) public class LettuceConfiguration {/** * Collect command statistics * @return */ @bean public DefaultClientResources getDefaultClientResources() { DefaultClientResources build = DefaultClientResources. Builder (). CommandLatencyRecorder (new DefaultCommandLatencyCollector (/ / open CommandLatency event collection, DefaultCommandLatencyCollectorOptions empty after each collection and configuration data. The builder (). The enable () resetLatenciesAfterEvent (true). The build ())) CommandLatencyPublisherOptions (/ / once every 10 s acquisition order statistics DefaultEventPublisherOptions.builder().eventEmitInterval(Duration.ofSeconds(10)).build() ).build(); return build; }}Copy the codeWechat search “my programming cat” to follow the public account, every day, easy to improve technology, win a variety of offers:
