One, foreword
Core privacy data is especially important for enterprises and users, so we should find ways to prevent the leakage of all kinds of privacy data. Chen takes you from the following three aspects to explain how to desensitize privacy data, but also need to pay attention to the daily development:
- Configuration file data desensitization
- The interface returned data desensitization
- Desensitization of log file data
The table of contents is as follows:
How to desensitize the configuration file?
It is often the case that a project has sensitive information in its configuration file, such as the URL of the data source, user name, and password…. If this information is exposed and the entire database is exposed, how do you hide this configuration?
In the past, it was manually written to the configuration file after encryption, and then manually decrypted when extracting. Of course, this is a kind of idea, but it can also solve the problem, but each time to manually encrypt and decrypt do not feel troublesome?
Public number: Code ape technology column
Today we introduce a scheme that enables you to encrypt and decrypt configuration files without awareness. Take advantage of an open source plug-in: Jasypt-spring-boot. The project address is as follows:
https://github.com/ulisesbocchio/jasypt-spring-boot
Copy the codeIt’s easy to use. All you need to do to integrate Spring Boot is add a starter.
1. Add dependencies
<dependency>
<groupId>com.github.ulisesbocchio</groupId>
<artifactId>jasypt-spring-boot-starter</artifactId>
<version>3.0.3</version>
</dependency>
Copy the code2. Configure the key
Add an encrypted secret key (of any kind) to the configuration file as follows:
jasypt:
encryptor:
password: Y6M9fAJQdU7jNp5MW
Copy the codeOf course, it is not safe to put the secret key directly in the configuration file. We can configure the secret key at project startup by using the following command:
java -jar xxx.jar -Djasypt.encryptor.password=Y6M9fAJQdU7jNp5MW
Copy the code3. Generate encrypted data
This step is to encrypt the configuration plaintext with the following code:
@SpringBootTest
@RunWith(SpringRunner.class)
public class SpringbootJasyptApplicationTests {
/** * inject the encryption method */
@Autowired
private StringEncryptor encryptor;
/** * generate ciphertext manually. Here we show url, user, password */
@Test
public void encrypt(a) {
String url = encryptor.encrypt(\ \ "JDBC: mysql \ \ : / / 127.0.0.1 \ \ : 3306 / test? useUnicode\\=true&characterEncoding\\=UTF-8&zeroDateTimeBehavior\\=convertToNull&useSSL\\=false&allowMultiQueries\\=true &serverTimezone=Asia/Shanghai");
String name = encryptor.encrypt("root");
String password = encryptor.encrypt("123456");
System.out.println("database url: " + url);
System.out.println("database name: " + name);
System.out.println("database password: " + password);
Assert.assertTrue(url.length() > 0);
Assert.assertTrue(name.length() > 0);
Assert.assertTrue(password.length() > 0); }}Copy the codeThe above code encrypts the URL, user, and password of the data source in plain text, and the output result is as follows:
database url: szkFDG56WcAOzG2utv0m2aoAvNFH5g3DXz0o6joZjT26Y5WNA+1Z+pQFpyhFBokqOp2jsFtB+P9b3gB601rfas3dSfvS8Bgo3MyP1nojJgVp6gCVi+B/XUs0 keXPn+pbX/19HrlUN1LeEweHS/LCRZslhWJCsIXTwZo1PlpXRv3Vyhf2OEzzKLm3mIAYj51CrEaN3w5cMiCESlwvKUhpAJVz/uXQJ1spLUAMuXCKKrXM/6dS RnWyTtdFRost5cChEU9uRjw5M+8HU3BLemtcK0vM8iYDjEi5zDbZtwxD3hA=
database name: L8I2RqYPptEtQNL4x8VhRVakSUdlsTGzEND/3TOnVTYPWe0ZnWsW0/5JdUsw9ulm
database password: EJYCSbBL8Pmf2HubIH7dHhpfDZcLyJCEGMR9jAV3apJtvFtx9TVdhUPsAxjQ2pnJ
Copy the code4. Write the encrypted ciphertext to the configuration
Jasypt uses ENC() by default, and the data source configuration is as follows:
spring:
datasource:
Basic data source configuration
username: ENC(L8I2RqYPptEtQNL4x8VhRVakSUdlsTGzEND/3TOnVTYPWe0ZnWsW0/5JdUsw9ulm)
password: ENC(EJYCSbBL8Pmf2HubIH7dHhpfDZcLyJCEGMR9jAV3apJtvFtx9TVdhUPsAxjQ2pnJ)
driver-class-name: com.mysql.jdbc.Driver
url: ENC(szkFDG56WcAOzG2utv0m2aoAvNFH5g3DXz0o6joZjT26Y5WNA+1Z+pQFpyhFBokqOp2jsFtB+P9b3gB601rfas3dSfvS8Bgo3MyP1nojJgVp6gCVi+B/ XUs0keXPn+pbX/19HrlUN1LeEweHS/LCRZslhWJCsIXTwZo1PlpXRv3Vyhf2OEzzKLm3mIAYj51CrEaN3w5cMiCESlwvKUhpAJVz/uXQJ1spLUAMuXCKKrXM /6dSRnWyTtdFRost5cChEU9uRjw5M+8HU3BLemtcK0vM8iYDjEi5zDbZtwxD3hA=)
type: com.alibaba.druid.pool.DruidDataSource
Copy the codeThe above configuration uses the default prefix=ENC(, suffix=), of course, we can change according to our own requirements, just need to change in the configuration file, as follows:
jasypt:
encryptor:
## Specifies the prefix and suffix
property:
prefix: 'PASS('
suffix: ') '
Copy the codeThe configuration must be decrypted using the PASS() package, as follows:
spring:
datasource:
Basic data source configuration
username: PASS(L8I2RqYPptEtQNL4x8VhRVakSUdlsTGzEND/3TOnVTYPWe0ZnWsW0/5JdUsw9ulm)
password: PASS(EJYCSbBL8Pmf2HubIH7dHhpfDZcLyJCEGMR9jAV3apJtvFtx9TVdhUPsAxjQ2pnJ)
driver-class-name: com.mysql.jdbc.Driver
url: PASS(szkFDG56WcAOzG2utv0m2aoAvNFH5g3DXz0o6joZjT26Y5WNA+1Z+pQFpyhFBokqOp2jsFtB+P9b3gB601rfas3dSfvS8Bgo3MyP1nojJgVp6gCVi+B /XUs0keXPn+pbX/19HrlUN1LeEweHS/LCRZslhWJCsIXTwZo1PlpXRv3Vyhf2OEzzKLm3mIAYj51CrEaN3w5cMiCESlwvKUhpAJVz/uXQJ1spLUAMuXCKKrX M/6dSRnWyTtdFRost5cChEU9uRjw5M+8HU3BLemtcK0vM8iYDjEi5zDbZtwxD3hA=)
type: com.alibaba.druid.pool.DruidDataSource
Copy the code5. To summarize
Jasypt also has many advanced uses. For example, you can configure your own encryption algorithm. For details, see the Github documentation.
3. How to desensitize the data returned by the interface?
Usually, some sensitive data in the interface return value should also be desensitized, such as id number, mobile phone number, address….. The usual method is to hide some data with *, of course, can also be customized according to your needs.
Anyway, how do you do that gracefully? There are two implementations, as follows:
-
Integrate Mybatis plug-in to desensitize specific fields when querying
-
Integrate Jackson to desensitize specific fields during the serialization phase
-
Data desensitization based on Sharding Sphere, check the previous article: One-click desensitization based on Sharding Sphere
There are many ways to implement the first solution on the web, and the second one, integrating Jackson, is shown below.
1. Customize a Jackson annotation
A custom desensitization annotation is required. Once the attributes are marked, desensitization should be performed as follows:
/** * Custom Jackson annotation, annotated on properties */
@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.FIELD)
@JacksonAnnotationsInside
@JsonSerialize(using = SensitiveJsonSerializer.class)
public @interface Sensitive {
// Desensitization strategy
SensitiveStrategy strategy(a);
}
Copy the code2. Customize desensitization strategies
According to the requirements of the project, desensitization rules of different fields are customized, for example, the middle digits of the mobile phone number are replaced by *, as follows:
/** * desensitization policies, enumerating classes, customized specific policies for different data */
public enum SensitiveStrategy {
/** * User name */
USERNAME(s -> s.replaceAll("(\\S)\\S(\\S*)"."$1 * $2")),
/** * id */
ID_CARD(s -> s.replaceAll("(\\d{4})\\d{10}(\\w{4})"."$1 * * * * $2")),
/** * Mobile phone number */
PHONE(s -> s.replaceAll("(\\d{3})\\d{4}(\\d{4})"."$1 * * * * $2")),
/** * address */
ADDRESS(s -> s.replaceAll("(\\S{3})\\S{2}(\\S*)\\S{2}"."$1 $2 * * * * * * * *"));
private final Function<String, String> desensitizer;
SensitiveStrategy(Function<String, String> desensitizer) {
this.desensitizer = desensitizer;
}
public Function<String, String> desensitizer(a) {
returndesensitizer; }}Copy the codeThe above only provides part of the configuration according to your project requirements.
3. Customize JSON serialization implementation
The important implementation is to desensitize @sensitive.
JsonSerializer
: Specifies the String type. The serialize() method is used to load modified data into */
public class SensitiveJsonSerializer extends JsonSerializer<String> implements ContextualSerializer {
private SensitiveStrategy strategy;
@Override
public void serialize(String value, JsonGenerator gen, SerializerProvider serializers) throws IOException {
gen.writeString(strategy.desensitizer().apply(value));
}
/** * gets the annotation attribute */ on the attribute
@Override
publicJsonSerializer<? > createContextual(SerializerProvider prov, BeanProperty property)throws JsonMappingException {
Sensitive annotation = property.getAnnotation(Sensitive.class);
if (Objects.nonNull(annotation)&&Objects.equals(String.class, property.getType().getRawClass())) {
this.strategy = annotation.strategy();
return this;
}
returnprov.findValueSerializer(property.getType(), property); }}Copy the code4. Define the Person class to desensitize its data
Data desensitization is performed using the @sensitive annotation:
@Data
public class Person {
/** * Real name */
@Sensitive(strategy = SensitiveStrategy.USERNAME)
private String realName;
/** * address */
@Sensitive(strategy = SensitiveStrategy.ADDRESS)
private String address;
/** * Phone number */
@Sensitive(strategy = SensitiveStrategy.PHONE)
private String phoneNumber;
/** * ID card number */
@Sensitive(strategy = SensitiveStrategy.ID_CARD)
private String idCard;
}
Copy the code5. Simulate the interface test
The above four steps have completed the Jackson annotation of data desensitization. Write a controller to test it, and the code is as follows:
@RestController
public class TestController {
@GetMapping("/test")
public Person test(a){
Person user = new Person();
user.setRealName("No, Mr. Chen.");
user.setPhoneNumber("19796328206");
user.setAddress("Wenzhou, Hangzhou city, Zhejiang Province....");
user.setIdCard("4333333333334334333");
returnuser; }}Copy the codeCall the interface to check whether the data is desensitized normally, and the results are as follows:
{
"realName": "No * Mr. Chen"."address": "Wenzhou city, ****, Zhejiang Province.. * * * *"."phoneNumber": "197 * * * * 8206"."idCard": "4333 * * * * 34333"
}
Copy the code6. Summary
There are many ways to achieve data desensitization, the key is which is more suitable, which is more elegant…..
How to desensitize log files?
Desensitization of configuration files and interface return values is mentioned above. Now it is always the turn of log desensitization. It is inevitable to print logs in a project. Sensitive data (ID card, number, user name…..) must be filtered out. .
Spring Boot log configuration: Spring Boot log configuration: Spring Boot log configuration Spring Boot is the second bullet, configuration file how to make? .
The following uses log4j2 as an example to explain how to desensitize logs. The general idea of other log frameworks is the same.
1. Add log4j2 log dependency
The Spring Boot default logging framework is logback, but we can switch to log4j2, depending on the following:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
<! -- Remove springBoot default configuration -->
<exclusions>
<exclusion>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-logging</artifactId>
</exclusion>
</exclusions>
</dependency>
<! Replace LogBack with log4j2 -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-log4j2</artifactId>
</dependency>
Copy the code2. Create the log4j2. XML configuration in the /resource directory
Log configuration for log4j2 is simple. You only need to create a new log4j2. XML configuration file in the /resource folder, as shown in the following figure:
How each node is configured and what it means is explained in detail in my previous two articles.
The configuration above does not implement data desensitization, which is a normal configuration using PatternLayout
3. Customize PatternLayout to achieve data desensitization
The configuration in Step 2 uses PatternLayout to implement the log format, so we can also customize a PatternLayout to implement filtering desensitization of the log.
The class diagram for PatternLayout inherits as follows:
As you can clearly see from the diagram above, PatternLayout inherits an abstract class AbstractStringLayout, so you just need to inherit this abstract class to customize.
Create a CustomPatternLayout that inherits the abstract class AbstractStringLayout
The code is as follows:
/** * log4j2 desensitization plugin extends AbstractStringLayout **/
@Plugin(name = "CustomPatternLayout", category = Node.CATEGORY, elementType = Layout.ELEMENT_TYPE, printObject = true)
public class CustomPatternLayout extends AbstractStringLayout {
public final static Logger logger = LoggerFactory.getLogger(CustomPatternLayout.class);
private PatternLayout patternLayout;
protected CustomPatternLayout(Charset charset, String pattern) {
super(charset);
patternLayout = PatternLayout.newBuilder().withPattern(pattern).build();
initRule();
}
/** * The regular expression map */ to match
private static Map<String, Pattern> REG_PATTERN_MAP = new HashMap<>();
private static Map<String, String> KEY_REG_MAP = new HashMap<>();
private void initRule(a) {
try {
if (MapUtils.isEmpty(Log4j2Rule.regularMap)) {
return;
}
Log4j2Rule.regularMap.forEach((a, b) -> {
if (StringUtils.isNotBlank(a)) {
Map<String, String> collect = Arrays.stream(a.split(",")).collect(Collectors.toMap(c -> c, w -> b, (key1, key2) -> key1));
KEY_REG_MAP.putAll(collect);
}
Pattern compile = Pattern.compile(b);
REG_PATTERN_MAP.put(b, compile);
});
} catch (Exception e) {
logger.info(">>>>>> failed to initialize log desensitization rule ERROR: {}", e); }}/** * Process the log information and desensitize * 1. Check whether the desensitization field * 2 is configured in the configuration file. Determine whether there is sensitive information that needs desensitization * 2.1 Return directly if there is no sensitive information that needs desensitization * 2.2 Processing: id card, name, mobile phone number sensitive information */
public String hideMarkLog(String logStr) {
try {
//1. Check whether the desensitization field is configured in the configuration file
if (StringUtils.isBlank(logStr) || MapUtils.isEmpty(KEY_REG_MAP) || MapUtils.isEmpty(REG_PATTERN_MAP)) {
return logStr;
}
//2. Determine whether the content contains sensitive information that needs desensitization
Set<String> charKeys = KEY_REG_MAP.keySet();
for (String key : charKeys) {
if(logStr.contains(key)) { String regExp = KEY_REG_MAP.get(key); logStr = matchingAndEncrypt(logStr, regExp, key); }}return logStr;
} catch (Exception e) {
logger.info(">>>>>>>>> desensitization processing ERROR:{}", e);
// If an exception is thrown, the original information is returned in order not to affect the process
returnlogStr; }}/** * matches the corresponding object. * *@param msg
* @param regExp
* @return* /
private static String matchingAndEncrypt(String msg, String regExp, String key) {
Pattern pattern = REG_PATTERN_MAP.get(regExp);
if (pattern == null) {
logger.info(">>> Logger did not match the corresponding regular expression");
return msg;
}
Matcher matcher = pattern.matcher(msg);
int length = key.length() + 5;
boolean contains = Log4j2Rule.USER_NAME_STR.contains(key);
String hiddenStr = "";
while (matcher.find()) {
String originStr = matcher.group();
if (contains) {
// The distance between the calculated keyword and the desensitized word is less than 5.
int i = msg.indexOf(originStr);
if (i < 0) {
continue;
}
int span = i - length;
int startIndex = span >= 0 ? span : 0;
String substring = msg.substring(startIndex, i);
if(StringUtils.isBlank(substring) || ! substring.contains(key)) {continue;
}
hiddenStr = hideMarkStr(originStr);
msg = msg.replace(originStr, hiddenStr);
} else{ hiddenStr = hideMarkStr(originStr); msg = msg.replace(originStr, hiddenStr); }}return msg;
}
/** * Mark sensitive text rule **@param needHideMark
* @return* /
private static String hideMarkStr(String needHideMark) {
if (StringUtils.isBlank(needHideMark)) {
return "";
}
int startSize = 0, endSize = 0, mark = 0, length = needHideMark.length();
StringBuffer hideRegBuffer = new StringBuffer("(\\S{");
StringBuffer replaceSb = new StringBuffer("$1");
if (length > 4) {
int i = length / 3;
startSize = i;
endSize = i;
} else {
startSize = 1;
endSize = 0;
}
mark = length - startSize - endSize;
for (int i = 0; i < mark; i++) {
replaceSb.append("*");
}
hideRegBuffer.append(startSize).append("})\\S*(\\S{").append(endSize).append("})");
replaceSb.append("$2");
needHideMark = needHideMark.replaceAll(hideRegBuffer.toString(), replaceSb.toString());
return needHideMark;
}
/**
* 创建插件
*/
@PluginFactory
public static Layout createLayout(@PluginAttribute(value = "pattern") final String pattern,
@PluginAttribute(value = "charset") final Charset charset) {
return new CustomPatternLayout(charset, pattern);
}
@Override
public String toSerializable(LogEvent event) {
returnhideMarkLog(patternLayout.toSerializable(event)); }}Copy the codeWhat about some of the details, such as @plugin and @pluginFactory? How to customize a plugin for log4j2 is not described in detail here, but is not the focus of this article. If you are interested, check out the official documentation for log4j2.
2. Customize your own desensitization rules
Log4j2Rule in the above code is the static class of desensitization rule, I am directly placed in the static class configuration, the actual project can be set to the configuration file, the code is as follows:
/** * Now there are three types of encrypted logs: * 1, ID card * 2, name * 3, ID card number * Encryption rules can be optimized in the configuration file **/
public class Log4j2Rule {
/** * rematches keyword categories */
public static Map<String, String> regularMap = new HashMap<>();
/** * TODO configurable * this item can be placed later in the configuration item */
public static final String USER_NAME_STR = "Name, Name, contact, Name";
public static final String USER_IDCARD_STR = "EmpCard,idCard, idCard, id number";
public static final String USER_PHONE_STR = "Mobile,Phone, Phone, Phone, Phone.";
/** * Regular matching, according to your own service requirements */
private static String IDCARD_REGEXP = "(\\d{17}[0-9Xx]|\\d{14}[0-9Xx])";
private static String USERNAME_REGEXP = "[\ \ u4e00 - \ \ u9fa5] {2, 4}";
private static String PHONE_REGEXP = "(?
;
static{ regularMap.put(USER_NAME_STR, USERNAME_REGEXP); regularMap.put(USER_IDCARD_STR, IDCARD_REGEXP); regularMap.put(USER_PHONE_STR, PHONE_REGEXP); }}Copy the codeAfter the two steps above, the custom PatternLayout is complete, and it’s time to rewrite the log4j2.xml configuration file.
4. Modify the log4j2. XML configuration file
This is a simple change. The original configuration file was formatted directly with the PatternLayout log, so just replace the default
The configuration file can be replaced globally, and the modification is complete.
5. Demo effect
In Step 3, the static class Log4j2Rule for desensitization rules is customized, which defines the name, ID card, and number of the three desensitization rules as follows:
Here is a demonstration of the three rules can be correct desensitization, direct use of log printing, the code is as follows:
@Test
public void test3(a){
log.debug("Id card: {}, name: {}, Tel: {}"."320829112334566767"."No, Mr. Chen."."19896327106");
}
Copy the codeThe console logs are as follows:
Id:320829* * * * * *566767, name: No. ***, Tel:198* * * * *106
Copy the codeOh huo, it works, so easy!!
6. Summary
Log desensitization scheme many, Chen also just introduced a common, interested can be studied.
Five, the summary
This article introduces the privacy data desensitization implementation scheme from three dimensions, code word is not easy, quickly click like collection bar!!
Source code has been uploaded to GitHub, need public number ape technical column, reply keyword data desensitization access.