The background,

Hystrix is an open source Netlifx fault-resistant framework that protects against avalanches, including service degradation, service meltdown, dependency isolation, and monitoring (Hystrix Dashboard).

Despite the fact that Hystrix is officially no longer maintained and there are new framework options like Alibaba Sentinel, there are still many projects that continue to use Hystrix in terms of component maturity and application cases, and the project I worked on is one of them. Therefore, I would like to share my Hystrix experience with you.

Second, experience summary

2.1 Selection of isolation policies

Hystrix provides two resource isolation strategies, thread pools and semaphores. The similarities and differences between them are as follows:

​

When using caches (local memory caches are more suitable for this scenario, and network caches such as Redis need to be evaluated), we can use semaphore isolation strategies because these services are quick to respond, do not take too long for container threads, and also reduce some overhead of thread switching and improve service efficiency.

The specific policy needs to be evaluated based on service scenarios. In general, thread pool isolation is recommended.

2.2 Setting thread pool size and timeout period

Under the thread pool isolation policy, the setting of thread pool size and timeout time is very important, which directly affects the responsiveness of system services. For example, if the size of the thread pool is set too large, it will cause resource waste and overhead such as thread switching. If the Settings are too small to support user requests, resulting in request queuing. However, if the timeout time is set too long, some time-consuming requests will block the thread, causing other normal requests to queue up. If set too short, too many normal requests will be fuses.

Hystrix’s advice is as follows:

​

That is, convert to the following calculation formula:

• Thread pool size = Service TP99 response time (in seconds) * Requests per second + redundant cache value

• Timeout in milliseconds = 1000 milliseconds/number of requests per second

For example, a service TP99 receives 30 requests every second, and the response time of each request is 200ms, which can be calculated according to the above formula: Thread pool size = 0.2 * 30 + 4 (redundant cache value) = 10, and timeout = 300ms

2.3 Annotation Overlay

In real development, you might encounter situations where an external calling method has Hystrix annotations used with other annotations, such as query methods with cache annotations. At this point, pay special attention to the execution order between annotations to avoid unexpected results:

• Cache annotations are not in effect

The Hystrix comments section is performed in the outermost layer, Hystrix due to the internal implementation is through ProceedingJoinPoint getTarget () to obtain the target object, using reflection to invoke way of execution on the target object method directly, thus cause the loss among other annotation logic. Hystrix annotations can be executed at the innermost layer by specifying the annotation execution Order @order.

• The query method is fused because the cache is abnormal

If Hystrix annotation section execution is at the outermost layer, Hystrix fuse management method logic includes cache call logic in addition to third-party service remote calls. An exception in the cache call counts as a whole method exception, causing the whole method to be fused.

2.4 Service Exception Handling

Let’s take a look at the following code to see if there are any problems:

@HystrixCommand(fallbackMethod="queryUserByIdFallback") public User queryUserById(String userId) { If (stringutils.isEmpty (userId)) {throw new BizException(" invalid "); } Result<User> result; try { result = userFacade.queryById(userId); } catch(Exception e) { log.error("query user error. id={}", id, e); } if(result ! = null && result.isSuccess()) { return result.getData(); } return null; }Copy the code

Hystrix determines whether the fuse of each dependent command is turned on during operation based on the success or failure rate information of the call request. If opened, subsequent requests will be rejected. Thus, the control of exceptions has a big impact on the performance of Hystrix.

Looking back at the example above, you’ll see two exception handling problems:

• Exception handling when parameter verification fails

Abnormal failures of non-system calls such as invalid parameter verification should not affect the circuit breaker logic and should not be counted as failure statistics. Optimization Suggestions is the parameter calibration in the remote invocation encapsulation method outside, or encapsulated into HystrixBadRequestException thrown. Because in the internal logic Hystrix HystrixBadRequestException exception has been the default value is not count failure statistics scope.

• Try-catch Exception handling for remote calls

A try-catch on a direct invocation of a remote service will “swallow” the exception directly, causing Hystrix to fail to obtain network exceptions and other service unavailable exceptions. It is recommended that exceptions be thrown after catch logging processing.

2.5 fallback method

Hystrix supports graceful service degradation by adding a default value to the fallback method when relying on service invocation. However, there are also many points to pay attention to in the use of fallback, which can be summarized as follows:

1. The fallback method access level and parameters must be consistent with those of the corresponding dependent service

2. The logic performed in the fallback method should be as light as possible, such as using local cache or static default values, to avoid remote calls

3. If there are remote calls in the Fallback method, it is recommended to wrap them in Hystrix as well, and ensure that they are isolated from the main command thread pool

4. Fallback degradation is not recommended for remote calls to write operations

2.6 groupKey, commandKey, and threadPoolKey

We’ve all seen these three keys in Hystrix development, but many people don’t understand what they mean and what they do for Hystrix, especially threadPooKey, so here’s a summary:

groupKey

Group keys can be used to group command methods, facilitating Hystrix data statistics, alarms, and Dashboad display. Generally, remote services are differentiated according to service types. For example, an account service defines a group key, and an order service defines another group key.

The default is the name of the class in which the @hystrixCommand annotation annotates the method.

commandKey

Identifier of a command method, which is used to set dynamic parameters for the command.

The default is the name of the method annotated by the @hystrixCommand annotation.

threadPoolKey

Used to identify the thread pool to which the command belongs. Commands with the same threadPoolKey use the same thread pool.

If this key is not specified, the default value is groupKey, which is the class name of the method marked by the @hystrixCommand annotation.

In a real world project, we would recommend using threadPoolKey to specify thread pools rather than the groupKey default, because there will be scenarios where a command will need to be thread isolated from other commands in the group to avoid interaction.

2.7 Parameter Priority

Hystrix provides four levels of parameter values by default.

Global Default Value

Hystrix code default value, written in the source code of the value, the user does not configure any parameters.

Example: execution. The isolation. Thread. TimeoutInMilliseconds timeout global default value is 1000, unit of milliseconds

Dynamic global Default Property (Default Property)

This type of configuration parameter changes the global default value.

Example: through the property name hystrix.com mand. Default. Execution. The isolation. The thread. TimeoutInMilliseconds set the timeout value

Instant Value

Initial value of the fuse instance. After such parameters are configured, the default value is not used. Property values written in code annotations.

Example: @ HystrixProperty (name = “execution. The isolation. Thread. TimeoutInMilliseconds”, value = “5000”)

Dynamic Instance Parameters (Instant Property)

You can dynamically adjust the parameter values of a fuse instance

Example: through the property name hystrix.com mand. HystrixCommandKey. Execution. The isolation. The thread. TimeoutInMilliseconds set the timeout value

Priority relationship:

Dynamic Instance Property > Instance Initial Value > Dynamic Global Default Property > Default Value

2.8 Dynamic Parameter configuration based on the Configuration center

By default, Hystrix uses Archaius to implement dynamic Settings, and Archaius loads the config.properties file in the classpath by default. You can add the corresponding key-value property to the configuration file to dynamically control Hystrix behavior. It is standard to use the configuration center for unified configuration management in distributed projects. Therefore, dynamic configuration of Hystrix parameters is required based on the extension of the configuration center.

Tracing the creation of HystrixCommand, hystrix finally implements the HystrixDynamicProperties class to get values based on parameter attribute names, and Hystrix itself provides an extension mechanism for the HystrixDynamicProperties class. HystrixPlugins class 367 lines of code, Hystrix provides four extensions:

1. Through system parameters

2. Based on the Java SPI mechanism

3. Archaius dynamic property extension implementation class (default)

4. Hystrix built-in HystrixDynamicProperties implementation based on System. GetProperty;

2.8.1 Based on Java SPI mechanism

The implementation of the extension based on THE SPI mechanism depends on two classes: HystrixDynamicProperties and HystrixDynamicProperty. HystrixDynamicProperties is the spi interface of Hystrix dynamic property extension that needs to be implemented. Provides a number of methods to obtain dynamic properties, the interface definition is as follows:

public interface HystrixDynamicProperties {
    
    /**
     * Requests a property that may or may not actually exist.
     * @param name property name, never <code>null</code>
     * @param fallback default value, maybe <code>null</code>
     * @return never <code>null</code>
     */
    public HystrixDynamicProperty<String> getString(String name, String fallback);
    /**
     * Requests a property that may or may not actually exist.
     * @param name property name, never <code>null</code>
     * @param fallback default value, maybe <code>null</code>
     * @return never <code>null</code>
     */
    public HystrixDynamicProperty<Integer> getInteger(String name, Integer fallback);
    /**
     * Requests a property that may or may not actually exist.
     * @param name property name, never <code>null</code>
     * @param fallback default value, maybe <code>null</code>
     * @return never <code>null</code>
     */
    public HystrixDynamicProperty<Long> getLong(String name, Long fallback);
    /**
     * Requests a property that may or may not actually exist.
     * @param name property name
     * @param fallback default value
     * @return never <code>null</code>
     */
    public HystrixDynamicProperty<Boolean> getBoolean(String name, Boolean fallback);
}
Copy the code

The HystrixDynamicProperty class specifically represents a parameter attribute and has the ability to dynamically change. The interface definition is as follows:

public interface HystrixDynamicProperty<T> extends HystrixProperty<T>{
    
    public String getName();
    
    /**
     * Register a callback to be run if the property is updated.
     * @param callback callback.
     */
    public void addCallback(Runnable callback);
    
}
Copy the code

The addCallback method is the core of the dynamic property change. As the annotation explains, it registers the callback method to dynamically refresh the property when the property changes. This dynamic refresh logic is implemented internally in Hystrix, so we only need to save the callback when we customize the extension, and then trigger the callback method of the corresponding attribute object when the configuration center changes.

The implementation steps are as follows:

1. Define HystrixDynamicProperty implementation class

Complete the dynamic attribute class custom implementations, including String/Integer/Long/Boolean four types state dynamic properties.

As described in the HystrixDynamicProperty class description above, the callback needs to be saved and the callback methods for those properties are triggered when a configuration center property change is received to implement the dynamic change of the property. This logic can be designed with reference to the observer pattern.

The code is as follows:

private abstract static class CustomDynamicProperty<T> implements HystrixDynamicProperty<T>, PropertyObserver { protected final String name; protected final T defaultValue; protected List<Runnable> callbacks; protected CustomDynamicProperty(String propName, T defaultValue) { this.name = propName; this.defaultValue = defaultValue; PropertyObserverManager.add(this); } @Override public String getName() { return name; } @Override public void addCallback(Runnable callback) { if (callbacks == null) callbacks = new ArrayList<>(1); this.callbacks.add(callback); } @Override public String keyName() { return name; } @Override public void update(PropertyItem item) { if(getName().equals(item.getName())) { for(Runnable r : callbacks) { r.run(); } } } } private static class StringDynamicProperty extends CustomDynamicProperty<String> { protected StringDynamicProperty(String propName, String defaultValue) { super(propName, defaultValue); } @Override public String get() { return ConfigManager.getString(name, defaultValue); } } private static class IntegerDynamicProperty extends CustomDynamicProperty<Integer> { protected IntegerDynamicProperty(String propName, Integer defaultValue) { super(propName, defaultValue); } @Override public Integer get() { String configValue = ConfigManager.get(name); if(StringUtils.isNotEmpty(configValue)) { return Integer.valueOf(configValue); } return defaultValue; } } private static class LongDynamicProperty extends CustomDynamicProperty<Long> { protected LongDynamicProperty(String propName, Long defaultValue) { super(propName, defaultValue); } @Override public Long get() { String configValue = ConfigManager.get(name); if(StringUtils.isNotEmpty(configValue)) { return Long.valueOf(configValue); } return defaultValue; } } private static class BooleanDynamicProperty extends CustomDynamicProperty<Boolean> { protected BooleanDynamicProperty(String propName, Boolean defaultValue) { super(propName, defaultValue); } @Override public Boolean get() { String configValue = ConfigManager.get(name); if(StringUtils.isNotEmpty(configValue)) { return Boolean.valueOf(configValue); } return defaultValue; }}Copy the code

The ConfigManager class is the configuration center configuration management class by default and provides functions such as obtaining parameters and monitoring parameters. The PropertyObserver class (keyName/ Update method belongs to its definition) and PropertyObserverManager class are implemented by referring to the observer mode definition, responsible for observer registration and notification management, to complete the linkage between dynamic properties and configuration center change notification. These two classes are relatively simple to implement and will not be described.

2. Define HystrixDynamicProperties implementation class

Customize HystrixDynamicProperties based on the HystrixDynamicProperty extension class defined in Step 1. The code is as follows:

public class DemoHystrixDynamicProperties implements HystrixDynamicProperties {    @Override    public HystrixDynamicProperty<String> getString(String name, String fallback) {        return new StringDynamicProperty(name, fallback);    }
    @Override    public HystrixDynamicProperty<Integer> getInteger(String name, Integer fallback) {        return new IntegerDynamicProperty(name, fallback);    }
    @Override    public HystrixDynamicProperty<Long> getLong(String name, Long fallback) {        return new LongDynamicProperty(name, fallback);    }
    @Override    public HystrixDynamicProperty<Boolean> getBoolean(String name, Boolean fallback) {        return new BooleanDynamicProperty(name, fallback);    }}
Copy the code

3. Register the SPI implementation class

In the meta-inf/services/add com.net flix. Hystrix. Strategy. The properties. The HystrixDynamicProperties text files, The content is the full path name of step 2 HystrixDynamicProperties custom implementation class.

2.8.2 Extension based on default Archaius

Hystrix default parameters through Archaius dynamic access and Archaius itself also provide custom parameters acquisition modes, respectively is PolledConfigurationSource interface and AbstractPollingScheduler class, Which PolledConfigurationSource interface configuration access to the source, says AbstractPollingScheduler class represents the configuration time refresh mechanism.

The implementation steps are as follows:

1. Create a configuration source:

public class CustomCfgConfigurationSource implements PolledConfigurationSource { private final static String CONFIG_KEY_PREFIX = "hystrix"; @Override public PollResult poll(boolean initial, Object checkPoint) throws Exception { Map<String, Object> map = load(); return PollResult.createFull(map); } private Map<String, Object> load() throws Exception{ Map<String, Object> map = new HashMap<>(); Set<String> keys = ConfigManager.keys(); for(String key : keys) { if(key.startsWith(CONFIG_KEY_PREFIX)) { map.put(key, ConfigManager.get(key)); } } return map; }}Copy the code

The implementation is very simple, and the core implementation is the poll method, which iterates over all hystrix-starting configuration parameters in the configuration center and returns save.

2. Define the configuration refresh mode:

public class CustomCfgPollingScheduler extends AbstractPollingScheduler { private final static Logger logger = LoggerFactory.getLogger("CustomCfgPollingScheduler"); private final static String CONFIG_KEY_PREFIX = "hystrix"; @Override public void startPolling(PolledConfigurationSource source, final Configuration config) { super.startPolling(source, config); // ConfigManager.addListener(new ConfigListener() { @Override public void eventReceived(PropertyItem item, ChangeEventType type) { String name = item.getName(); if(name.startsWith(CONFIG_KEY_PREFIX)) { String newValue = item.getValue(); / / new & modify the if (ChangeEventType. ITEM_ADDED. Equals (type) | | ChangeEventType. ITEM_UPDATED. Equals (type)) { addOrChangeProperty(name, newValue, config); } else if(changeEventType.item_removed. Equals (type)) {deleteProperty(name, config); } else { logger.error("error config change event type {}.", type); }}}}); } private void addOrChangeProperty(String name, Object newValue, final Configuration config) { if (! config.containsKey(name)) { config.addProperty(name, newValue); } else { Object oldValue = config.getProperty(name); if (newValue ! = null) { if (! newValue.equals(oldValue)) { config.setProperty(name, newValue); } } else if (oldValue ! = null) { config.setProperty(name, null); } } } private void deleteProperty(String key, final Configuration config) { if (config.containsKey(key)) { config.clearProperty(key); } } @Override protected void schedule(Runnable pollingRunnable) { //IGNORE OPERATION } @Override public void stop() { //IGNORE OPERATION } }Copy the code

Define and initialize automatic configuration:

DynamicConfiguration dynamicConfiguration = new DynamicConfiguration(new CustomCfgConfigurationSource(), new CustomCfgPollingScheduler());
ConfigurationManager.install(dynamicConfiguration);
Copy the code

If you are careful, you may find that the DynamicConfiguration class is installed into the Hystrix configuration management class in step 3, and the timed refresh class in step 2 is also weak, so you may wonder if you can continue to simplify the above solution. You only need to implement a custom “DynamicConfiguration” to include configuration source access and monitoring configuration modification functions, as follows:

public class CustomCfgDynamicConfiguration extends ConcurrentMapConfiguration { private final static Logger logger = LoggerFactory.getLogger("CustomCfgDynamicConfiguration"); private final static String CONFIG_KEY_PREFIX = "hystrix"; public CustomCfgDynamicConfiguration() { super(); load(); initEvent(); } /** * Private void load() {Set<String> keys = configManager.keys (); for(String key : keys) { if(key.startsWith(CONFIG_KEY_PREFIX)) { map.put(key, ConfigManager.get(key)); }}} /** * listen for event callback processing through the configuration center, For Hystrix configuration parameter changes synchronized * / private void initEvent () {ConfigManager. AddListener (new ConfigListener () {@ Override public void  eventReceived(PropertyItem item, ChangeEventType type) { String name = item.getName(); if(name.startsWith(CONFIG_KEY_PREFIX)) { String newValue = item.getValue(); / / new & modify the if (ChangeEventType. ITEM_ADDED. Equals (type) | | ChangeEventType. ITEM_UPDATED. Equals (type)) { addOrChangeProperty(name, newValue); } else if(changeEventType.item_removed. Equals (type)) {deleteProperty(name); } else { logger.error("error config change event type {}.", type); }}}}); Private void addOrChangeProperty(String name, private void addOrChangeProperty, private void addOrChangeProperty, private void addOrChangeProperty) Object newValue) { if (! this.containsKey(name)) { this.addProperty(name, newValue); } else { Object oldValue = this.getProperty(name); if (newValue ! = null) { if (! newValue.equals(oldValue)) { this.setProperty(name, newValue); } } else if (oldValue ! = null) { this.setProperty(name, null); }} /** * private void deleteProperty(String key) {if (this.containsKey(key)) { this.clearProperty(key); }}}Copy the code

At last, by ConfigurationManager. Install (new CustomCfgDynamicConfiguration ()); “Install” the implementation can be.

Third, write at the end

The author summarized and shared the Hystrix usage based on the actual practice of the project, including the isolation policy, thread pool setting, parameter priority and other knowledge points, as well as the solution to specific problems such as annotation overlay, exception handling, and dynamic parameter configuration, hoping to be helpful to you.

Author: Development team of Vivo official website mall