Interview Ali was asked, “How does your project handle repeated/concurrent requests?”

For some user requests, it may be repeated in some cases, if it is query operations, it does not matter, but some of them involve writing operations, once repeated, may lead to very serious consequences, such as the transaction interface if repeated requests may be repeated orders.

Repeated scenarios might be:

1. The hacker intercepted the request and replayed it
2. The front-end/client request was sent twice for some reason, or the user clicked twice in a very short period of time.
3. The gateway to resend
4. … .

This article discusses how to gracefully and uniformly handle this situation on the server side, and how to disallow client-side actions such as repeated user clicks is beyond the scope of this article.

Use unique request number to de-weigh

You might think that if a request has a unique request number, it can borrow Redis to do this de-duplication — as long as the unique request number exists in Redis to prove that it has been processed, it is considered duplicate

The code looks like this:

String KEY = "REQ12343456788"; Long expireTime = 1000; Long expireAt = System.currentTimemillis () + expireTime; String val = "expireAt@" + expireAt; / / redis key also exist to consider requests are repeated Boolean firstSet = stringRedisTemplate. Execute ((RedisCallback < Boolean >) connection - > connection.set(KEY.getBytes(), val.getBytes(), Expiration.milliseconds(expireTime), RedisStringCommands.SetOption.SET_IF_ABSENT)); final boolean isConsiderDup; if (firstSet ! = null &&firstset) {// First visit isadup = false; } else {workview = true; }Copy the code

Service parameters are deduplicated

The above scheme can solve the scenario with unique request number. For example, before each write request, the server returns a unique number to the client. The client makes a request with this request number, and the server can complete the de-interception.

However, in many cases, requests do not have such unique numbers! So can we identify a request with a request parameter?

Considering the simple scenario, where the request parameter has only one field, reqParam, we can use the following identifier to determine whether the request is repeated. User ID: Interface name: Request parameter

String KEY = "dedup:U="+userId + "M=" + method + "P=" + reqParam;
Copy the code

So when the same user accesses the same interface and comes in with the same reqParam, we can identify it as duplicate.

The problem is, our interfaces are often not that simple, and in the current mainstream, our argument is usually a JSON. So for this scenario, how do we go about replicating?

Compute a summary of the request parameters as parameter identifiers

Suppose we sort the request parameters (JSON) by KEY in ascending order, and then form a string as the KEY value? However, this can be very long, so we can consider taking an MD5 digest of the string and replacing reqParam with this digest.

String KEY = "dedup:U="+userId + "M=" + method + "P=" + reqParamMD5;
Copy the code

Now, the unique identifier for the request is marked!

Note: MD5 is theoretically possible to repeat, but deduplication is usually done within a short window of time (e.g., one second). It is almost impossible for the same user and the same interface to spell out different parameters in a short period of time leading to the same MD5.

Continue to optimize, consider eliminating part of the time factor

The above problem is actually a good solution, but in practice, you may find some problems: some request users repeatedly click in a short period of time (e.g., send three requests in 1000 milliseconds), but bypass the above de-rescheduling (different KEY values).

The reason is that there is a time field in the request parameter field. This field marks the time when the user requested it, and the server can use this field to discard old requests (e.g., 5 seconds ago). In the following example, all other parameters of the request are the same, except that the request time is off by one second:

// Both requests are the same, But the request time one second String the req = "{\ n" + "\" requestTime \ ": \" 20190101120001 \ ", \ n "+" \ "requestValue \" : \ "1000 \", \ n "+ "\"requestKey\" :\"key\"\n" + "}"; String req2 = "{\n" + "\"requestTime\" :\"20190101120002\",\n" + "\"requestValue\" :\"1000\",\n" + "\"requestKey\" :\"key\"\n" + "}";Copy the code

It is also likely that we need to block out repeated requests. Therefore, such time fields need to be eliminated before obtaining the summary of service parameters. A similar field could be a GPS latitude and longitude field (with minimal differences between repeated requests).

Request to re-tool class, Java implementation

Public class ReqDedupHelper {/** ** @param reqJSON request parameters, * @return MD5 digest */ public String dedupParamMD5(final String) reqJSON, String... excludeKeys) { String decreptParam = reqJSON; TreeMap paramTreeMap = JSON.parseObject(decreptParam, TreeMap.class); if (excludeKeys! =null) { List<String> dedupExcludeKeys = Arrays.asList(excludeKeys); if (! dedupExcludeKeys.isEmpty()) { for (String dedupExcludeKey : dedupExcludeKeys) { paramTreeMap.remove(dedupExcludeKey); } } } String paramTreeMapJSON = JSON.toJSONString(paramTreeMap); String md5deDupParam = jdkMD5(paramTreeMapJSON); log.debug("md5deDupParam = {}, excludeKeys = {} {}", md5deDupParam, Arrays.deepToString(excludeKeys), paramTreeMapJSON);  return md5deDupParam; } private static String jdkMD5(String src) { String res = null; try { MessageDigest messageDigest = MessageDigest.getInstance("MD5"); byte[] mdBytes = messageDigest.digest(src.getBytes()); res = DatatypeConverter.printHexBinary(mdBytes); } catch (Exception e) { log.error("",e); } return res; }}Copy the code

Here are some test logs:

Public static void main(String[] args) { But the request time one second String the req = "{\ n" + "\" requestTime \ ": \" 20190101120001 \ ", \ n "+" \ "requestValue \" : \ "1000 \", \ n "+ "\"requestKey\" :\"key\"\n" + "}"; String req2 = "{\n" + "\"requestTime\" :\"20190101120002\",\n" + "\"requestValue\" :\"1000\",\n" + "\"requestKey\" :\"key\"\n" + "}"; String dedupMD5 = new ReqDedupHelper().deDupParAMMd5 (req); String dedupMD52 = new ReqDedupHelper().dedupParamMD5(req2); System.out.println("req1MD5 = "+ dedupMD5+" , req2MD5="+dedupMD52); String dedupMD53 = new ReqDedupHelper(). DedupParamMD5 (req,"requestTime"); String dedupMD53 = new ReqDedupHelper(). String dedupMD54 = new ReqDedupHelper().dedupParamMD5(req2,"requestTime"); System.out.println("req1MD5 = "+ dedupMD53+" , req2MD5="+dedupMD54); }Copy the code

Log output:

req1MD5 = 9E054D36439EBDD0604C5E65EB5C8267 , req2MD5=A2D20BAC78551C4CA09BEF97FE468A3Freq1MD5 = C2A36FED15128E9E878583CAAAFEFDE9 , req2MD5=C2A36FED15128E9E878583CAAAFEFDE9
Copy the code

Log description:

• Since the two parameters are different in requestTime, we can find that the two values are not the same when extracting the recrement parameter summary
• The second call removes the requestTime and asks for the digest (” requestTime “is passed in the second argument), and the two digests are the same, as expected.

conclusion

At this point, we can get the complete solution for deduplication, as follows:

String userId= "12345678"; // user String method = "pay"; // Interface name String dedupMD5 = new ReqDedupHelper().deDupParAMMd5 (req,"requestTime"); String KEY =" dedup:U=" + userId + "M=" + method + "P=" + dedupMD5; long expireTime = 1000; Long expireAt = System.currentTimemillis () + expireTime; String val = "expireAt@" + expireAt; // NOTE: Direct SETNX does not support expiration time, so setting + expiration is not an atomic operation. In extreme cases, it may be set to expire, and the same request may be misread as need to delete, so use the underlying API here. Ensure SETNX + expiration time is atomic operation Boolean firstSet = stringRedisTemplate. Execute ((RedisCallback < Boolean >) connection - > connection.set(KEY.getBytes(), val.getBytes(), Expiration.milliseconds(expireTime), RedisStringCommands.SetOption.SET_IF_ABSENT)); final boolean isConsiderDup; if (firstSet ! = null && firstSet) { isConsiderDup = false; } else { isConsiderDup = true; }Copy the code