preface

Continuous Integration (CI) has always been one of the things that Internet software engineers have been exposed to most. Continuous integration has become an important part of the mainstream Internet software development process. At present, Youzan is practicing Continuous delivery (CD) internally, which can be regarded as the product of the post-continuous integration era. It is important to note that both CI and CD are focused more on practices as part of the software development delivery process than they are once delivered to a production environment. Good online dial measurement system is to make up for this deficiency. The existing online safeguard means can be divided into operation and maintenance level, product level, security level, service level and test level. This article focuses on our practice at the testing level.

Online monitoring generation based on test scripts

The original intention of our on-line dial-test system is as follows:

  1. Active early warning line problem. Youzan has many lines of business, each line of business has different development and test students docking, it is difficult to evaluate the impact surface accurately every release. The monitoring at the operation and maintenance level is more passive alarm, that is, we will receive an alarm only when the user traffic triggers the online bug, and the user experience is not good enough. We need to move from being passive to proactive in bug alert online and periodically know the health of each line of business.
  2. Find online problems quickly with low traffic. We usually launch our software in the early hours of the morning when traffic is very low. After the release, the regression time is long (manually), and the testing surface is limited (it is impossible to do the full regression for every release). At this time, it is necessary to construct a wave of traffic with full coverage, and under the background of small traffic, agile can find online problems.
  3. Be aware of the scope of business impact and subsequent convergence in an emergency. For example, if a non-software fault occurs in the production environment, such as a network exception, you need to know the impact on the service level. After the network is recovered, you need to know whether the service impact has been rectified. Prior to this, these scenarios required manual intervention by testers and had very poor agility. With this system, the test personnel can increase their attention to the scene, the scene can be activated by active trigger and timing trigger to execute, through the alarm system notification to the relevant personnel, do the first time troubleshooting, reduce the impact of the fault, reduce the fault duration.

Basic version

For version 1.0, we use the common SpringWeb build, which is internally called online robot inspection. The system structure is as follows

System architecture diagram version 1.0

The system is mainly composed of three modules:

  1. Task scheduling module. This module encapsulates use case execution as a system task, which is triggered periodically using Spring Quartz. Provide external API to connect with the favorable publishing platform, and actively trigger the execution of use cases when the system is released online.
  2. Test case module. Includes service access, assertions, and alarms. Test scenarios require the input of test students from all lines of business.
  3. Alarm module. Connect to the Youzan internal alarm platform.

Flow Chart version 1.0

The system divides use cases into basic use cases and scenario use cases, and supports concurrent or sequential execution of scenarios. The specific execution strategy is set by the use case designer during the use case development process.

Existing problems

The base version meets minimum availability, which has the advantage of being quick to use up front and inexpensive for those who write integration use cases frequently, but not for others (new testers, development, operations, etc.). Generally speaking, its disadvantages mainly focus on the following points:

  1. Once there are more lines of business, the cost of developing use case code increases;
  2. As the number of use cases increases, the maintenance cost of late use cases is very high.
  3. Use case on-line is not flexible, each use case change needs to be re-published;
  4. Unable to see the running status and service coverage;
  5. Each execution is performed in full regardless of services.
  6. The use case code is redundant and inefficient.

Configuration and visualization

Because of these unavoidable problems, we redesigned and released version 2.0. To solve the above problems:

  1. Test cases and test scenarios can be configured on the management platform.
  2. Use case configuration standardization, given standard use case structure and assertion strategy;
  3. Through the management platform to manage their own use cases, use case changes take effect in real time, no need to publish;
  4. The front-end display is added to visually display the operation status and business coverage through charts, which is convenient for different groups to consult;
  5. Docking publishing platform, according to the specified application name to distinguish which use cases to run;
  6. Design use case execution framework, realize core code reuse.

The new system architecture diagram is as follows

System architecture diagram version 2.0

The use-case model is as follows:

field If required instructions
Use case name is Suggested naming format: “Use Case Type: Service: Method”
Use case type is The options are HTTP or Dubbo
Use case description is Scene description
Subordinate to the business is The service threshold to which the use case belongs
The request url no Url of the HTTP call
Request header no http header
Request parameters no HTTP or Dubbo request entry parameter. Dynamic parameter injection is supported for inter-use case dependencies
The service name no Interface name for requesting dubbo protocol (package name + class name)
Request method is HTTP protocol :GET, POST, PUT, etc. Dubbo protocol: method name
assertions is Support multiple
Whether open no Control switch, no longer running after closing. The default open
Whether the login no After this function is enabled, you can use the default account to log in. Disabled by default
Whether to retry no After this function is enabled, the use case fails and retry once. The default no
Pre/post check no Before/after executing the use case, perform the before/after check. If the check fails, it is interrupted

In order to display the health of online business more directly, we added rich front-end report

The data show

The main differences between the new version and the old version are:

  1. The execution flow and data flow are separated. Test case design requires no coding and supports configuration. Use cases are stored in DB for repeated use as data.
  2. Common transactions are encapsulated, such as login, store switching, and so on, managed through a unified thread pool.
  3. Support for dynamic parameter injection, which enables interdependencies between use cases, which will be covered separately later.

The task execution flow is as follows:

Flow Chart version 2.0

The task execution engine is implemented through different worker threads. Different business use cases are executed concurrently, and internal business use cases are executed sequentially. The system is distributed to specific task flows based on the type of use case (HTTP/DUbbo).

Core class design

Implementation of dependencies between use cases

In terms of the complexity of use cases, our use cases can be divided into two categories: the basic use cases of a single scene and the combined use cases of complex scenes. Composite use cases are integrated on the basis of basic use cases, and the input and output of use cases depend on each other. We implement use case dependencies in two ways:

  1. By configuring the pre – and post-relationships of use cases.
  2. Through parameter injection.

The first way, when configuring the use case, is to give it a front case, which of course is also managed in the platform. This way, when it comes to the use case, the execution engine will execute the front use case first.

In the second way, we define the following format for parameter injection for jSON-formatted input parameters:

$#a,b,c#$

The meanings of each field are as follows:

A: ID of the dependent case B: key value of the response field of the dependent case, for example, name C: Optional field. If the dependent value is in array, use its index index as an example. {“code”:”$#8,data,0#$”,”type”:”$#10,type#$”}

The flow of parameter injection is as follows:

Parameter injection flow chart

Assertion module design

In the new version, we have designed four types of common assertions that can satisfy almost all of our own application scenarios. The four types are:

  1. Whether to include. The response content contains the specified content which is true, and otherwise false.

  2. Not empty/null. The response content is true if it is not null /null, and false if it is null /null.

  3. JSON “equality” determination of a value at a specific location. In this case, the response content is first converted to JSON, and the assertion is added by specifying the coordinates of the object to be compared in the JSON string. True if the value on that coordinate is equal to the specified value, false otherwise. So how do you set a unique coordinate for each value of a JSON string? Since json has nested relationships and keys may be duplicated, we use a composite key to represent the coordinates, such as json:

{” data “: {” list:”/” 1 “, “2”, “info” : {” name “:” zhang “, “age” : 18}}, “code” : 200}

{“data”:{“info”:{“name”:” zhang SAN “}}} true if the value of the position returned is “Zhang SAN”, false otherwise.

  1. Json-oriented pseudocode expression judgment

The first three types of assertions only satisfy partial scenarios, and are still not sufficient for some complex assertions, such as the list size assertion in JSON above. To do this, we introduce a fourth type of assertion, pseudo-code assertion. A list size assertion can be written like this:

The code will concatenate the expression behind a JSON object while processing. The assertion is true for the result of the entire code execution, false otherwise. Dynamic compilation, loading, and invocation of pseudocode is implemented using GroovyShell. This part of the code is implemented as follows:

public Result compare(String response) { Result result = new Result(); / / singleton get GroovyShell GroovyShell shell. = SingleGroovyUtil getGroovyShell (); Binding binding = null; JSONObject jsonObject = new JSONObject(); JSONArray jsonArray = new JSONArray(); Object value = null; try { if (response.startsWith("[")){ jsonArray = JSON.parseArray(response); binding = new Binding();  binding.setVariable("data", jsonArray);  value = InvokerHelper.createScript(shell.getClass(), binding).evaluate("data." + textStatement);  }else { jsonObject = JSON.parseObject(response); binding = new Binding(); binding.setVariable("data", jsonObject);  value = InvokerHelper.createScript(shell.getClass(), binding).evaluate("data." + textStatement);  } if((Boolean)value) { result.setSuccess(true); }else { result.setSuccess(false); String MSG = JsonUtil. FindErrMsgByJsonObject (jsonObject); result. SetMsg (the String. Format (" assertion failure. [%s]", this.textStatement, msg.length()>0? MSG :response));  } } catch (Exception e) { result.setSuccess(false); String msg = JsonUtil.findErrMsgByJsonObject(jsonObject); Result.setmsg (string.format (") exception occurred when asserting. (LLDB =[%s], actual=[%s]", LLDB =[%s], msg.length()>0? MSG :response));  } return result; }Copy the code

pluggable

The new system meets the requirements of configurability and visualization of use cases, while sacrificing some flexibility. For example, the pseudo code of some complex assertions will be very long and not readable, and it will make mistakes if you are not careful. Simple use case dependencies can be satisfied, but complex use case dependencies are difficult to satisfy. For example, use case A depends on use case B under some conditions and on use case C under other conditions. This kind of complex dependency is not suitable for configuration. Based on the above considerations, we added plug-in features on the basis of the existing system to support the access of complex use cases.

System architecture diagram 3.0

The design idea of plug-in is as follows:

  1. The platform provides a set of standards for use cases. Test students can develop use cases that meet the standards and add them to the platform to run.
  2. Use cases are completely decoupled from the platform, where they can be configured.
  3. Use cases support hot swap, and the platform does not need to restart.

Use case standards are provided externally in the form of interfaces, which are exposed as JAR packages. The use-case designer relies directly on the JAR package and implements the specified interface. The use-case interface is defined as follows:

public interface AbstractTestCase {

    CaseResult before();

    CaseResult run();

    void after();
}
Copy the code

After the use case is developed, it is packaged into a JAR package and uploaded to the platform. A JAR package can contain one use case or multiple use cases.

After the jar package is uploaded, the platform should do the following:

  1. Dynamically load jars into the JVM
  2. Parse classes that implement an AbstractTestCase interface
  3. Invoke a method in a class according to the specified policy
  4. Report and present the result data

Get the following code from the JAR that implements the AbstractTestCase interface:

Public static List<Class<? >> getAllImplClassesByInterface(Class c) { List<Class<? >> filteredList = new ArrayList<Class<? > > (); If (c.isInterface()) {try {List<Class> allClass = getClassesbypagename (); allClass.forEach(clazz -> { if (c.isAssignableFrom(clazz)) { if (! c.equals(clazz)) { filteredList.add(clazz); }}}); } catch (ClassNotFoundException e) { e.printStackTrace(); } catch (IOException e) { e.printStackTrace(); } } return filteredList; }Copy the code

In the future

In the future, Youzan online dial-test system will provide richer functions, such as more flexible use case execution strategy, higher frequency of core use case execution and lower frequency of edge business execution. More comprehensive alarm strategy, each business side can freely customize the use case of concern, online problems in the first time contact; Support for multiple machine rooms. At present, the system is only deployed in single room. The core business of Youzan has been deployed in multiple machine rooms, and the dialing and testing system will be adjusted accordingly. The system supports distributed deployment. To prevent a single point of failure, distributed deployment will be considered in the future.

At present, this system can ensure that the test students know the abnormal core business of Youzan in the first time. In the future, the breadth and depth of the business will be further improved, and it will become a crucial part of the online quality assurance of Youzan.