Description of log desensitization scenarios

In the log, our log usually prints the Json string of Model, such as the following Model class

Public class Request {/** * private String name; /** * id */ private String IDCard; /** * private String phone; Private String imgBase64; /** * imgBase64; }Copy the code

There are examples of the following classes

Request request = new Request(); Request. setName(" Aisin Gioro "); request.setIdcard("450111112222"); request.setPhone("18611111767"); request.setImgBase64("xxx");Copy the code

We normally use fastJson to print the Request’s JSON string:

log.info(JSON.toJSONString(request));
Copy the code

This will print all of the Request attributes as follows:

{" idcard ":" 450111112222 ", "imgBase64" : "XXX", "name" : "zhang", "phone" : "17120227942"}Copy the code

There are two problems with the log here

  1. Security: Personal information such as name, phone and IDcard is extremely sensitive and should not be printed out in plain text. We hope that these sensitive information can be output in desensitization form

  2. Field redundancy: ImgBase64 is the image base64, which is a very long string. In production, the image Base64 data is not helpful for troubleshooting, but increases the storage cost. Moreover, this field is the base64 of the front and back of the ID card, and it is also sensitive information, so it needs to be removed from the log. We’d like to desensitize and slim down (imgBase64 field removed) the following log:

    {” idcard “:” 450222 “, “name” : “love,”, “phone” : “1861767”, “imgBase64” : “”}

You can see that each field is desensitized, but it should be noted that the desensitization rules for these fields are different

  • Idcard, keep the first three, the last three, the rest of the number
  • Keep the first two digits of the name and type the rest
  • Keep the first three digits of the phone number and the last four digits of the phone number
  • The image base64 (imgBase64) displays the empty string directly

The json. toJSONString method specifies a parameter, ValueFilter, to customize the attributes to be converted. We can use this Filter to make the final JSON string not show or show the desensitized value. The logic is as follows

public class Util { public static String toJSONString(Object object) { try { return JSON.toJSONString(object, getValueFilter()); } catch (Exception e) { return ToStringBuilder.reflectionToString(object); } } private static ValueFilter getValueFilter() { return (obj, key, Value) -> {// obj- object key-field name value- return formatted value}; }Copy the code

As shown above, we only need to desensitize value in the getValueFilter method to display the desensitized log in the final log. Now the problem comes, how to deal with the field desensitization problem, we know that some fields need desensitization, some fields do not need desensitization, so some people may be according to the name of the key to determine whether desensitization, the code is as follows:

private static ValueFilter getValueFilter() { return (obj, key, If (objects.equal (key, "phone")) {return phone} if (objects.equal (key, "phone")) "Idcard ")) {return idcard} if (objects.equal (key, "name")) {return name}; }Copy the code

This does seem to fulfill the requirement, but is merely fulfilling the requirement enough? There is a serious problem with such an implementation:

The desensitization rule is closely coupled with the specific attribute name, so a lot of if and else judgment logic needs to be written in valueFilter, which is not scalable and universal. For example, due to business reasons, some fields of telephone are called phone and some are called TEL in our project. Some, called telephone, have the same rules for desensitization, but you have to write the following ugly code in the above method.

private static ValueFilter getValueFilter() { return (obj, key, Value) - > {/ / obj - key - value - field values if the field name (Objects. Equal (key, "phone") | | Objects. The equal (key, "Tel") | | Objects. Equal (key, "telephone") | |) {return the phone after desensitization} / / the rest do not need desensitization, according to the original value of the return to the return value}; }Copy the code

So can we use a general, extensible way to solve this problem? I’m sure you have a good idea from the title of the article. Yes, it is annotations

Definition and implementation of annotations

Annotations, also known as Java annotations, is a kind of Annotation mechanism introduced in JDK 5.0. If code annotations are for programmers, then annotations are for programs. After programs see annotations, they can get them at run time and enhance the ability of run time according to annotations. There are three common annotations that apply to your code

  • @Override checks to see if the method overrides the parent method, and if it does not exist in the parent class or the interface implemented, a compilation error is reported
  • Deprecated marks obsolete classes, methods, attributes, etc
  • @SuppressWarnings – Instructs the compiler to ignore warnings declared in annotations.

So how do these annotations work? Let’s go to @Override

@Documented
@Retention(RetentionPolicy.RUNTIME)
@Target(value={CONSTRUCTOR, FIELD, LOCAL_VARIABLE, METHOD, PACKAGE, PARAMETER, TYPE})
public @interface Deprecated {
}
Copy the code

And on these Deprecated annotations you have @documented, @Retention, and @Target. These are also called meta-annotations, which are Deprecated or any other Deprecated or custom annotation. The behavior of the other annotations is defined by these annotations. The types and functions of these meta-annotations are as follows

  • Documented indicates that it is processed by a tool such as Javadoc, so that the final annotation type information is included in the resulting document

  • There are three main Retention strategies for @Retention annotations

  • Retentionpolicy. SOURCE source-level annotations, indicating that the specified annotations are only visible at compile time and will not be written to the bytecode file. Override and SuppressWarnings are examples of these types. It doesn’t make much sense to save at run time, so it doesn’t end up being compiled into a bytecode file

  • Retentionpolicy.runtime means that annotations are compiled into the final code file and read by the JVM upon startup, so that we can use reflection to retrieve these annotations at RUNTIME and perform operations on them. This is the save strategy used by most custom annotations. One of the things you might wonder about here is why Deprecated is denoted as RUNTIME. For programmers, it’s theoretically enough to just care if the calling class, method, etc., is Deprecated. What’s the point of run-time fetching? Suppose you want to count how often outdated methods are called in production to assess the bad taste of your project or as a reference for refactoring.

  • Retentionpolicy.class annotations are compiled into the final character file, but are not loaded into the JVM (annotations are discarded when the CLASS is loaded). This saving strategy is not commonly used, and is mostly used in bytecode file processing.

  • @target indicates where the annotation can be used. By default, it can be used anywhere. The scope of the annotation is specified primarily by value.

  • FIELD applies to properties

  • METHOD acts on METHOD

  • Elementtype. TYPE: Applies to classes, interfaces (including annotation types), or enum declarations

  • Inherited – marks which annotation class this annotation inherits from (the default annotation does not inherit from any child class)

So at sign interface, what does this do, actually if you decompile it you’ll see that in the bytecode the compiler encodes it into something like this.

public interface Override extends Annotation {   
}
Copy the code

What’s the Annotation

We can see that the nature of annotations is actually an interface that inherits the Annotation interface, supplemented by meta-annotations that regulate annotations like Retention and Target, runtime behavior, scope, etc.

Deprecated annotations do not define attributes, but you can define attributes if necessary. For example, a Deprecated annotation can define a value attribute, which can be specified when the annotation is declared

@Documented
@Retention(RetentionPolicy.RUNTIME)
@Target(value={CONSTRUCTOR, FIELD, LOCAL_VARIABLE, METHOD, PACKAGE, PARAMETER, TYPE})
public @interface Deprecated {
    String value() default "";
}
Copy the code

So when I apply this annotation to properties and so on, I can specify the value, as shown below

public class Person {
    @Deprecated(value = "xxx")
    private String tail;
}
Copy the code

If the annotation’s save policy is retentionpolicy.runtime, we can get the annotation at RUNTIME in the following way to get the annotation’s property values and so on

field.getAnnotation(Deprecated.class);
Copy the code

Skillfully use notes to solve log desensitization problems

Having outlined the principles and writing of annotations, let’s take a look at how annotations can be used to desensitize our logs.

First we need to define a desensitized annotation. Since this annotation needs to be retrieved at RUNTIME, the save policy should be retentionPolicy.runtime, and the annotation should be applied to fields such as phone and IDcard. So the @target value is elementType. FIELD, and we also notice that fields like phone number and ID card are desensitized, but they have different desensitization strategies, so we need to define a property for this annotation, so that we can specify what kind of desensitization its property is. Our definition of desensitization is annotated as follows:

// Sensitive information Type public enum SensitiveType {ID_CARD, PHONE, NAME, IMG_BASE64 } @Target({ ElementType.FIELD }) @Retention(RetentionPolicy.RUNTIME) public @interface SensitiveInfo { SensitiveType type(); }Copy the code

With the annotations defined, we can now specify the annotations and their sensitive information types for our sensitive fields, as follows

public class Request {
    @SensitiveInfo(type = SensitiveType.NAME)
    private String name;
    @SensitiveInfo(type = SensitiveType.ID_CARD)
    private String idcard;
    @SensitiveInfo(type = SensitiveType.PHONE)
    private String phone;
    @SensitiveInfo(type = SensitiveType.IMG_BASE64)
    private String imgBase64;
}
Copy the code

For attribute specifies the annotations, how the implementation according to the annotation corresponding sensitive Field type of desensitization, you can use reflection, use reflection to obtain first class of each Field, to decide on whether there is a corresponding Field notes, if any, to determine the annotation for what kind of sensitive type annotation, again according to the corresponding Field accordingly desensitization operations, Directly on the code, notes written very clear, I believe you should be able to understand

private static ValueFilter getValueFilter() { return (obj, key, Field[] fields = obj.getClass().getDeclaredFields(); for (Field field : fields) { if (! field.getName().equals(key)) { continue; } // Determine if the attribute has a corresponding SensitiveInfo annotation. SensitiveInfo annotation = field.getannotation (sensitiveinfo.class); If (null! = annotation) {switch (annotation.type()) {case PHONE: return PHONE desensitization; Case ID_CARD: return ID card desensitization; Case NAME: return NAME desensitization; case IMG_BASE64: return ""; }}}}} catch (Exception e) {log.error("To JSON String fail", e); } return value; }; }Copy the code

Some people may say that the use of annotations to achieve desensitization code more than doubled, looks like it is not worth it, in fact, the previous way, desensitization rules and a field name strong coupling, maintainability is not good, and with annotations, just like the project appeared in the phone, TEL, ** @sensitiveinfo (type = sensitivetype.phone) ** Simply add a new SensitiveType for increased maintainability and extensibility. Therefore, using annotations is highly recommended in such scenarios.

Advanced use of annotations – Use annotations to eliminate duplicate code

In the process of docking with the bank, the bank provided some API interfaces, and the serialization of parameters was a little special. Instead of using JSON, we needed to put the parameters together to form a large string.

  • Form all the parameters into fixed-length data in the order of the API documentation provided by the bank, and then concatenate them together as the entire string.

  • Because each parameter has a fixed length, it needs to be filled if the length is not reached:

  • If the length of a string parameter is shorter than the length of a string parameter, fill it with an underscore to the right, that is, the string content is left.

  • The length of the parameter of the number type is 0 left, that is, the actual number is to the right;

  • The representation of the currency type requires the amount to be rounded down by 2 bits to the minute and left populated as a number type.

  • MD5 operation for all parameters as signature (for easy understanding, the Demo does not involve salt processing). Look briefly at the interface definitions for the two banks

1. Create a user

Insert a picture description here

2. Payment interface

The general practice is to fill in parameters, concatenate, and check the checkmark for each interface according to the previous rules. Take the above two interfaces as examples

Create user with payment request as follows:

Pojo@data public class CreateUserRequest {private String name; private String identity; private String mobile; private int age; } pojo@data public class PayRequest {private long userId; private BigDecimal amount; } public class BankService {// createUser method public static String createUser(CreateUserRequest request) throws IOException { StringBuilder stringBuilder = new StringBuilder(); Append (String.format("%-10s", request.getName()).replace(", '_')); Append (String.format("%-18s", request.getidEntity ()).replace(", '_'))); Append (string. format("%05d", age)); Append (string.format ("%-11s", mobile).replace(", '_')); String Builder.append(digestutils.md2hex (stringBuilder.toString())); return Request.Post("http://baseurl/createUser") .bodyString(stringBuilder.toString(), ContentType.APPLICATION_JSON) .execute().returnContent().asString(); } // Payment method public static String pay(PayRequest Request) throws IOException {StringBuilder StringBuilder = new StringBuilder(); Append (string.format ("%020d", request.getUserId()))); // The amount is rounded down 2 places to the minute, in minute units, as the number to the right, Padding stringBuilder.append(string.format ("%010d",request.getAmount().setScale(2, roundingmode.down).multiply(new) BigDecimal("100")).longValue())); String Builder.append(digestutils.md2hex (stringBuilder.toString())); return Request.Post("http://baseurl//pay") .bodyString(stringBuilder.toString(), ContentType.APPLICATION_JSON) .execute().returnContent().asString(); }}Copy the code

As you can see, there is a lot of overlap in the logic of just writing these two requests:

1. The formatting logic of string, currency, and number is very repetitive. Take strings as an example

As can be seen, the processing of formatting string is different except for the length of each field, and other formatting rules are completely the same. However, in the above article, we have integrated a set of the same processing logic for each string, and this set of stitching rules can be completely extracted (because only the length is different, the stitching rules are the same).

2. The logic of string concatenation, signing and request in the process is repeated in all methods.

3, due to the order of each field in joining together, these need our human flesh hard-coded guarantee the order of these fields, maintenance cost is great, and very easy to get wrong, imagine if tens of hundreds of parameters, these parameters need to be in a certain order to stitching, if needs meat to ensure that, and it is difficult to guarantee the correctness, and repeat too much work, do more harm than good

Let’s take a look at how annotations can greatly simplify our code.

1. First of all, for each call interface, they all need to request the network at the bottom, but the request method is not the same, for this point, we can make a note for the interface as follows

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE)
@Documented
@Inherited
public @interface BankAPI { 
    String url() default "";
    String desc() default ""; 
}
Copy the code

In this way, the method name of the corresponding interface can be uniformly annotated at the network request layer

2. For each POJO requested for the interface, we noticed that each attribute has three attributes: type (string/number/currency), length, and order, so we could define an annotation that contains these three attributes as follows

@Retention(RetentionPolicy.RUNTIME) @Target(ElementType.FIELD) @Documented @Inherited public @interface BankAPIField { int order() default -1; int length() default -1; String type() default ""; // M for currency, S for string, N for number}Copy the code

Next we apply the annotations defined above to the request POJO described above

For creating a user request

@BankAPI(url = "/createUser", @data public class CreateUserAPI extends AbstractAPI {@bankapiField (order = 1, type = "S", length = 10) private String name; @BankAPIField(order = 2, type = "S", length = 18) private String identity; @bankAPIfield (order = 4, type = "S", length = 11) @BankAPIField(order = 3, type = "N", length = 5) private int age; }Copy the code

For the payment interface

@bankapi (url = "/bank/pay", desc = "pay") @data public class extends AbstractAPI { type = "N", length = 20) private long userId; @BankAPIField(order = 2, type = "M", length = 10) private BigDecimal amount; }Copy the code

The next process invoked with annotations is as follows

  1. Gets the class’s array of fields based on reflection, and then sorts the fields based on the order value in the Field’s BankAPIField annotation
  2. The sorted fields are traversed one by one to determine their types and then format their values according to their types. If the value is determined to be “S”, the values are formatted according to the string format required by the interface, and the formatted Field values are sequentially spliced and signed
  3. After concatenation, the request is sent. At this time, get the annotation of POJO class, obtain the URL value of annotation BankAPI, combine it with baseUrl to form a complete URL, and add the concatenation string in step 2 to construct a complete request

The code is as follows:

Private static String remoteCall(AbstractAPI API) throws IOException {// Get the request address from BankAPI BankAPI = api.getClass().getAnnotation(BankAPI.class); bankAPI.url(); StringBuilder stringBuilder = new StringBuilder(); Array.stream (api.getClass().getdeclaredFields ()) // Get all fields. Filter (field -> Field. IsAnnotationPresent (BankAPIField. Class)) / / find markup annotation fields. The sorted (Comparator.com paringInt (a - > A.getannotation (bankapifield.class).order())) // Sort fields according to the order in the annotation. Peek (field -> field.setaccessible (true)) // Set access to private fields .foreach (field -> {// get the annotation BankAPIField BankAPIField = field.getannotation (bankapifield-class); Object value = ""; Try {// reflection get field value value = field.get(API); } catch (IllegalAccessException e) { e.printStackTrace(); } // Format the string switch (bankapifield.type ()) {case "S": { stringBuilder.append(String.format("%-" + bankAPIField.length() + "s", value.toString()).replace(' ', '_')); break; } case "N": { stringBuilder.append(String.format("%" + bankAPIField.length() + "s", value.toString()).replace(' ', '0')); break; } case "M": { if (! (value instanceof BigDecimal)) throw new RuntimeException(String.format("{} {must be BigDecimal", API, field)); stringBuilder.append(String.format("%0" + bankAPIField.length() + "d", ((BigDecimal) value).setScale(2, RoundingMode.DOWN).multiply(new BigDecimal("100")).longValue())); break; } default: break; }}); Append (digestutils.md2hex (stringBuilder.toString()))); String param = stringBuilder.toString(); long begin = System.currentTimeMillis(); / / send the Request String result = Request. Post (" http://localhost:45678/reflection "+ bankAPI. Url ()). BodyString (param, ContentType.APPLICATION_JSON) .execute().returnContent().asString(); Log.info (" call banking API {} URL :{} parameter :{} Time :{}ms", bankapi.desc (), bankapi.url (), param, System.currentTimemillis () -begin); return result; }Copy the code

Now look at the logic for creating users and making payments

Public static String createUser(CreateUserAPI Request) throws IOException {return remoteCall(request); } // Pay method public static String pay(PayAPI Request) throws IOException {return remoteCall(request); }Copy the code

RemoteCall removes a large amount of irrelevant code from the request logic and makes the code more maintainable. Using annotations and reflection allows us to generalize this kind of structural problem, which is Cool!

conclusion

If reflection gives us don’t know under the condition of class structure according to the fixed logic to handle the ability of the class members, annotation is the expansion of the metadata of the members to the ability, we used at the time of using the reflection to realize general logic, can get more from external data, we care about, in turn, general processing these data, by reflection, Can really let us achieve twice the result with half the effort, can greatly reduce the repeated code, effective decoupling, greatly improve the scalability.

The last

Thank you for reading here, the article is inadequate, welcome to point out; If you think it’s good, give me a thumbs up.

Also welcome to pay attention to my public number: programmer Maidong, Maidong will share Java related technical articles or industry information every day, welcome to pay attention to and forward the article!