The background,
Vivo Comment Center helps front desk businesses quickly build comment functions and provide comment operation capabilities by providing general capabilities such as comment publication, like, report and custom comment ordering, avoiding repetitive construction and data isolation problems of front desk businesses. At present, there are vivo short video, Vivo browser, negative one screen, Vivo mall and other 10+ business access. The traffic volume and fluctuation range of these services are different. How to ensure the high availability of foreground services and prevent the traffic surge of one service from causing the unavailability of other services? The comment data of all services are stored by the MEDIUM station. The data volume and DB pressure of the different services are different. As the medium station, how should the data of each service be isolated to ensure the high availability of the whole medium station system?
This article will share with you the vivo review center solution, mainly from the traffic isolation and data isolation of two parts.
2. Traffic isolation
2.1 Traffic Grouping
Vivo browser business hundreds of millions of daily activities, real-time hot news whole network push, for this kind of important business with a large number of users and large traffic, we provide a separate cluster to provide services for them, to avoid the impact of other businesses.
Vivo Review center provides services externally through Dubbo interface. We divide the whole service cluster logically by Dubbo tag routing. A Dubbo call can intelligently select the corresponding tag service provider according to the tag carried by the request. As shown below:
1) Provider labeling: At present, dynamic rule labeling and static rule labeling can be used to group instances. Dynamic rules have a higher priority than static rules. When two rules exist and conflict occurs, dynamic rules prevail. The internal operation and maintenance system of the company well supports dynamic marking, which can be done by marking machines with specified IP (non-Docker containers, machine IP is fixed).
2) The foreground consumer specifies the service label: set when initiating the request, as follows;
The foreground specifies the routing label for the middle platform
RpcContext.getContext().setAttachment(Constants.REQUEST_TAG_KEY,"browser");
Copy the codeThe scope of the request tag is every time. The tag is set before the invocation of the comment center service. The provider that the foreground service invoishes other services is not affected by the route tag.
2.2 Multi-tenant Traffic Limiting
Heavy traffic is isolated by a separate cluster. However, the cost of deploying an independent cluster is high. Therefore, a cluster cannot be deployed independently for each foreground service. In most cases, multiple services need to share a cluster. How to deal with the sudden traffic of services sharing a cluster? Yes, limit the flow! However, traffic limiting applies to the overall QPS of the interface in a one-size-suited manner. In this case, the traffic surge of a foreground service results in traffic limiting of all foreground service requests.
In this case, multi-tenant traffic limiting (one tenant can be regarded as a foreground service) comes into play. Traffic limiting of different tenants on the same interface is supported. The effect is shown in the following figure:
Implementation process:
We use the hotspot parameter flow limiting feature of Sentinel and use the service identity code as the hotspot parameter to configure different flow control sizes for each service.
So what is hotspot parameter limiting? First, let’s talk about hot spots. Hot spots are frequently accessed data. Most of the time, we want to collect the Top N data that is accessed most frequently in a certain hotspot data and restrict its access. Such as:
-
Item ID is used as parameter. The ID of the most commonly purchased item in a period of time is counted and restricted.
-
The user ID is a parameter that limits the user IDS that are frequently accessed within a period of time.
Hotspot parameter traffic limiting Collects statistics on hotspot parameters in the incoming parameters and implements traffic limiting for resource invocation containing hotspot parameters based on the configured traffic limiting threshold and mode. Hotspot parameter traffic limiting is a special kind of traffic control that only applies to resource calls containing hotspot parameters. Sentinel uses LRU strategy to count the most frequently accessed hotspot parameters and combines with token bucket algorithm to control the flow at parameter level. The following is a sample comment scenario:
Sentinel is used for resource protection, which is mainly divided into several steps: defining resources, defining rules, and implementing rules.
1) Defining resources:
Here it can be understood as the path of each mid-platform API interface.
2) Define rules:
Sentienl supports QPS flow control, adaptive traffic limiting, hotspot parameter traffic limiting, cluster traffic limiting, and more. Here we use standalone hotspot parameter traffic limiting.
Configure traffic limiting for hotspot parameters
{
"resource": "com.vivo.internet.comment.facade.comment.CommentFacade:comment(com.vivo.internet.comment.facade.comment.dto.CommentRequ estDto)".// The interface that needs to limit traffic
"grade": 1.// QPS limiting mode
"count": 3000.// The default traffic limit of the interface is 3000
"clusterMode": false.// Single-machine mode
"paramFieldName": "clientCode".// We have optimized the Sentinel component by adding this configuration property, which is used to specify the property name of the parameter object as the hotspot parameter key
"paramFlowItemList": [ // Traffic limiting rule for hotspot parameters
{
"object": "vivo-community".// If clientCode is set to this value, the traffic limiting rule is matched
"count": 1000.// The current limit is 1000
"classType": "java.lang.String"
},
{
"object": "vivo-shop".// If clientCode is set to this value, the traffic limiting rule is matched
"count": 2000.// The current limit is 2000
"classType": "java.lang.String"}}]Copy the code3) Effective treatment of rules:
When the flow limiting rule is triggered, sentinel throws a ParamFlowException exception. It is not elegant to throw the exception to the foreground service. Sentinel provides us with a unified exception callback handling portal, DubboAdapterGlobalConfig, which allows us to convert exceptions into business custom result returns.
Custom flow limiting return result;
DubboAdapterGlobalConfig.setProviderFallback((invoker, invocation, ex) ->
AsyncRpcResult.newDefaultAsyncResult(FacadeResultUtils.returnWithFail(FacadeResultEnum.USER_FLOW_LIMIT), invocation));
Copy the codeWhat additional optimizations have we made:
1) The internal traffic limiting console of the company does not support traffic limiting configuration of hotspot parameters, so we add a new traffic limiting configuration controller, which supports dynamic delivery of traffic limiting configuration through the configuration center. The overall process is as follows:
The traffic limiting configuration is delivered dynamically.
public class VivoCfgDataSourceConfig implements InitializingBean {
private static final String PARAM_FLOW_RULE_PREFIX = "sentinel.param.flow.rule";
@Override
public void afterPropertiesSet(a) {
// Customize the configuration parsing object
VivoCfgDataSource<List<ParamFlowRule>> paramFlowRuleVivoDataSource = new VivoCfgDataSource<>(PARAM_FLOW_RULE_PREFIX, sources -> sources.stream().map(source -> JSON.parseObject(source, ParamFlowRule.class)).collect(Collectors.toList()));
// Register the configuration validation listener
ParamFlowRuleManager.register2Property(paramFlowRuleVivoDataSource.getProperty());
// Initialize the traffic limiting configuration
paramFlowRuleVivoDataSource.init();
// Listen on the configuration center
VivoConfigManager.addListener(((item, type) -> {
if(item.getName().startsWith(PARAM_FLOW_RULE_PREFIX)) { paramFlowRuleVivoDataSource.updateValue(item, type); }})); }}Copy the code2) There are two ways to specify current limiting hotspot parameters in native Sentinel:
-
The first is to specify the NTH argument of the interface method.
-
The second method inherits the ParamFlowArgument and implements the ParamFlowKey method, which returns the value of the hotspot parameter value.
Neither of these methods is flexible. The first method does not support specifying object attributes. The second method requires us to change the code. If an interface parameter does not inherit the ParamFlowArgument and you want to configure the flow limiting of the hotspot parameter, you can only change the code and issue the version to solve the problem. Therefore, we have optimized the source code of Sentinel component for hotspot parameter limiting, adding “specify some attribute of parameter object” as hotspot parameter, and supporting object level nesting. Very small code changes, but greatly facilitates the configuration of hotspot parameters.
The modified hot spot parameter verification logic;
public static boolean passCheck(ResourceWrapper resourceWrapper, /*@Valid*/ ParamFlowRule rule, /*@Valid*/ int count,
Object... args) {
// Ignore some code
// Get parameter value. If value is null, then pass.
Object value = args[paramIdx];
if (value == null) {
return true;
}
// Assign value with the result of paramFlowKey method
if (value instanceof ParamFlowArgument) {
value = ((ParamFlowArgument) value).paramFlowKey();
}else{
// Get the hotspot parameter value based on the hotspot parameter specified by classFieldName
if (StringUtil.isNotBlank(rule.getClassFieldName())){
Reflection gets the value of the classFieldName property in the parameter objectvalue = getParamFieldValue(value, rule.getClassFieldName()); }}// Ignore some code
}
Copy the code3. MongoDB data isolation
Why data isolation? There are two reasons for this. First, the middle station stores the data of different services in the foreground, so the data of different services cannot affect each other during data query, and the data of service A cannot be queried from service B. The second point is that different services have different data levels and different pressures on db operations. For example, in traffic isolation, we provide a separate set of service clusters for the browser service, so that the DB used by the browser service can be completely isolated from the service pressures of other services.
Vivo Review Center uses MongoDB as storage medium (for details about database selection and MongoDB application, please refer to our previous introduction “MongoDB practice in Review Center”). In order to isolate data from different business parties, Review Center provides two data isolation schemes: Physical and logical isolation.
3.1 Physical Isolation
The data of different business parties are stored in different database clusters, which requires our system to support the multi-data source of MongoDB. The implementation process is as follows:
1) Find the right entry point
By analyzing the source code of the spring-data-mongodb process, it is found that before executing all statements, a getDB() action will be done to obtain the database connection instance, as shown below.
Spring-data-mongodb DB operation source code;
private <T> T executeFindOneInternal(CollectionCallback
collectionCallback, DbObjectCallback
objectCallback, String collectionName)
{
try {
// getDb()
T result = objectCallback
.doWith(collectionCallback.doInCollection(getAndPrepareCollection(getDb(), collectionName)));
return result;
} catch (RuntimeException e) {
throwpotentiallyConvertRuntimeException(e, exceptionTranslator); }}Copy the codeGetDB () executes org. Springframework. Data. The mongo. MongoDbFactory interface getDB () method, by default use MongoDbFactory SimpleMongoDbFactory implementation, It is natural to think of using the proxy pattern, replacing SimpleMongoDbFactory with the SimpleMongoDbFactory proxy object and creating a SimpleMongoDbFactory instance within the proxy object for each MongoDB set.
To perform the getDb() operation on the proxy object while performing the DB operation, it only needs to do two things;
-
Find the SimpleMongoDbFactory object for the corresponding cluster
-
Perform SimpleMongoDbFactory. Getdb () operation.
The class diagram is shown below.
The overall execution process is as follows:
3.1.2 Core code implementation
Dubbo filter gets the business identity and sets it to the context;
private boolean setCustomerCode(Object argument) {
// Get the business identity information from the string parameter
if (argument instanceof String) {
if(! Pattern.matches("client.*", (String) argument)) {
return false;
}
// Sets the business identity information to the context
CustomerThreadLocalUtil.setCustomerCode((String) argument);
return true;
} else {
// Get the parameter object from the list type
if (argument instanceofList) { List<? > listArg = (List<? >) argument;if (CollectionUtils.isEmpty(listArg)) {
return false; } argument = ((List<? >) argument).get(0);
}
// Get the business identity information from the object
try {
Method method = argument.getClass().getMethod(GET_CLIENT_CODE_METHOD);
Object object = method.invoke(argument);
// Verify that the business identity is valid
ClientParamCheckService clientParamCheckService = ApplicationUtil.getBean(ClientParamCheckService.class);
clientParamCheckService.checkClientValid(String.valueOf(object));
// Sets the business identity information to the context
CustomerThreadLocalUtil.setCustomerCode((String) object);
return true;
} catch (NoSuchMethodException | IllegalAccessException | InvocationTargetException e) {
log.debug("Reflection failed to get clientCode with entry parameter: {}", argument.getClass().getName(), e);
return false; }}}Copy the codeRouting proxy class of MongoDB cluster;
public class MultiMongoDbFactory extends SimpleMongoDbFactory {
// Database instance cache of different clusters: key indicates the configuration name of the MongoDB cluster, and value indicates the instance of the corresponding MongoDB cluster
private final Map<String, SimpleMongoDbFactory> mongoDbFactoryMap = new ConcurrentHashMap<>();
// Add the created MongoDB cluster instance
public void addDb(String dbKey, SimpleMongoDbFactory mongoDbFactory) {
mongoDbFactoryMap.put(dbKey, mongoDbFactory);
}
@Override
public DB getDb(a) throws DataAccessException {
Get the foreground business code from the context
String customerCode = CustomerThreadLocalUtil.getCustomerCode();
// Obtain the MongoDB configuration name of the service
String dbKey = VivoConfigManager.get(ConfigKeyConstants.USER_DB_KEY_PREFIX + customerCode);
// Get the corresponding SimpleMongoDbFactory instance from the connection cache
if(dbKey ! =null&& mongoDbFactoryMap.get(dbKey) ! =null) {
. / / execution SimpleMongoDbFactory getDb () operation
return mongoDbFactoryMap.get(dbKey).getDb();
}
return super.getDb(); }}Copy the codeCustomize MongoDB operation templates;
@Bean
public MongoTemplate createIgnoreClass(a) {
// Generate the MultiMongoDbFactory agent
MultiMongoDbFactory multiMongoDbFactory = multiMongoDbFactory();
if (multiMongoDbFactory == null) {
return null;
}
MappingMongoConverter converter = new MappingMongoConverter(new DefaultDbRefResolver(multiMongoDbFactory), new MongoMappingContext());
converter.setTypeMapper(new DefaultMongoTypeMapper(null));
// Use the multiMongoDbFactory agent to generate the MongoDB operation template
return new MongoTemplate(multiMongoDbFactory, converter);
}
Copy the code3.2 Logical Isolation
Physical isolation is the most thorough data isolation, but it is impossible to build a separate MongoDB cluster for every business. When multiple services share a database, logical data isolation is required.
Logical isolation generally falls into two categories:
-
One is table isolation: data of different businesses is stored in different tables in the same database, and different businesses operate different tables.
-
One is row isolation: the data of different service parties are stored in the same table. The data of different service parties is redundant in the table. When data is read, the data is isolated according to the service code filtering conditions.
For implementation costs and to comment on the business scenario, we chose table isolation. The implementation process is as follows:
1) Initialize the data table
Each time a new service is connected, we assign a unique identity code to the service. We directly use this identity code as the suffix of the name of the service table and initialize the tables, for example, comment_info_vshop and Comment_info_community.
2) Automatic table search
Direct use of the spring-data-mongodb @document annotation to support Spel, combined with our business identity context, to achieve automatic table search.
Homing table
@Document(collection = "comment_info_#{T(com.vivo.internet.comment.common.utils.CustomerThreadLocalUtil).getCustomerCode()}")
public class Comment {
// Table fields are ignored
}
Copy the codeThe overall effect of the combination of the two isolation methods:
Four, the last
Through the above practices, we have well supported different levels of foreground business, and achieved no intrusion to business code, and well decoupled the complexity between technology and business. In addition, the Redis cluster and ES cluster used in the project are also isolated for different businesses. The general idea is similar to the isolation of MongoDB, which is to make a layer of agents, which will not be introduced here.
Author: Vivo official website Mall development team -Sun Daoming