preface

Hello, everyone, and double 叒 yi to meet, I put everyone every day pretty three knife.

Before being shut down by everyone, I made a “big dream”, must update the article this week. Now it seems that flag is useful…

This article is a summary of more than a month of work I’ve done to help deprecate the FastJSON framework in my group by migrating most of the Java repository from FastJSON to Gson.

The main reason for this was that the company was fed up with the frequent fastjson security breaches, and every time a bug appeared it had to force a company-wide fastjson upgrade, which was a headache.

In the first half of this article, I will briefly analyze the pros and cons of various JSON parsing frameworks, and present several solutions for migrating JSON frameworks for enterprise projects.

In the second half of this article, I will summarize the usage issues of Gson based on the experience of this month and the migration of FastJSON to the pit that Gson has trod.

Article Contents:

  • Why give up FastJSON?
  • Fastjson alternative
    • Features of three JSON frameworks
    • The performance comparison
    • Final choice
  • Considerations when replacing dependencies
    • Caution, caution, caution
    • Good communication between development team and test team
    • Do regression/interface testing
    • Consider performance differences before and after migration
  • Use Gson instead of Fastjson
    • Json deserialization
    • Paradigm to deal with
    • The List/Map to write
    • Hump and underline conversion
  • Migration faQS trampling pits
    • Date is serialized in different ways
    • Abnormal SpringBoot
    • Abnormal Swagger
    • @Mapping JsonObject is abnormal as an input parameter

Note: Whether to use Fastjson has been a controversial topic in recent years. This article is not intended to discuss the right and wrong framework selection, but to reflect on the problems encountered in the migration process. If you have any comments, please feel free to discuss them in the comments section.

About 5 minutes to read this article

The code word is not easy, welcome to pay attention to my personal public account: Back-end technology talk (QR code see the bottom of the article)

Why give up FastJSON?

The reason is that fastjson vulnerabilities occur frequently, leading to frequent internal urging of all lines of business to upgrade Fastjson version to prevent security problems.

Fastjson frequently exposed a security vulnerability in 2020 that bypassed the autoType switch to deserialize remote code execution and gain server access.

From V1.2.59 released in July 2019 to V1.2.71 released in June 2020, AutoType has been updated in each of the 13 official versions.

Fastjson version history related to AutoType:

1.2.59 release, enhance AutoType open security fastjson 1.2.60 release, increased the AutoType blacklist, repair the denial of service security fastjson 1.2.61 release, Add AutoType security blacklist fastjson 1.2.62 release, add AutoType blacklist, enhance date deserialization and JSONPath Fastjson 1.2.66 release, Bug fixes security hardening, and do security hardening, AutoType blacklist fastjson 1.2.67 release, Bug fix security hardening, add AutoType blacklist fastjson 1.2.68 release, support GEOJSON, add AutoType blacklist 1.2.69 release, Fix newly discovered high-risk AutoType switch bypass security vulnerabilities, added AutoType blacklist 1.2.70 release, improve compatibility, added AutoType blacklist 1.2.71 release, added security blacklist, no new use, preventive supplementCopy the code

In contrast, other JSON frameworks, such as Gson and Jackson, have far fewer bugs and fewer high-risk vulnerabilities, which is the main reason the company wanted to replace the framework.

Fastjson alternative

This paper mainly discusses Gson to replace the fastJSON framework of the actual combat problems, so here do not discuss the advantages and disadvantages of various JSON framework in detail, only give a conclusion.

After evaluation, two JSON frameworks, Jackson and Gson, are mainly taken into consideration for comparison with FastJSON.

Features of three JSON frameworks

FastJson

Speed is fast

Fastjson is fast compared to other JSON libraries, and since fastjson’s 1.1.x release in 2011, its performance has never been surpassed by other JSON libraries implemented in Java.

Widely used

Fastjson is widely used in Alibaba, deployed on tens of thousands of servers, and is widely accepted in the industry. In 2012, it was selected as one of the most popular domestic open source software by Open Source China.

The test is complete

Fastjson has a large number of Testcases, with over 3321 testcases in 1.2.11. Regression testing is performed for each release to ensure quality stability.

Using a simple

The Fastjson API is very concise.

Jackson

Easy to use – The Jackson API provides a high-level look and feel to simplify common use cases.

No need to create mappings – THE API provides default mappings for most object serialization.

High performance – fast, low memory footprint, suitable for large object charts or systems.

Clean JSON-Jackson creates a clean and compact JSON result that is easy to read.

Independent – The library does not require any other libraries except the JDK.

Gson

Provide a mechanism to make converting Java objects to JSON or vice versa as simple as using toString() and constructors (factory methods).

Allows pre-existing immutable objects to be converted to JSON or vice versa.

Allows you to customize the representation of an object

Arbitrarily complex objects are supported

Output lightweight, easy-to-read JSON

The performance comparison

Performance comparison source code written by colleagues:

Github.com/zysrxx/json…

This paper does not discuss performance differences in detail, because it involves a lot of implementation ideas and optimization of various frameworks, so it only gives the conclusion:

1. The serialization performance of a single object is Fastjson > Jackson > Gson

2. Performance of serializing large objects Jackson> Fastjson > Gson. When serializing large Json objects Jackson> Gson > Fastjson, performance of serializing large data Jackson has obvious advantages

3. Deserialize single object performance Fastjson > Jackson > Gson, the performance gap is small

4. Performance of deserialization of large objects Fastjson > Jackson > Gson, the performance difference is very small

Final choice

  • Jackson applies to high performance scenarios, and Gson applies to high security scenarios
  • For new project repositories, fastJSON is no longer used. For the stock system, considering the Json replacement cost, the following options are available:
    • The project does not use the autoType function, so it is recommended to switch to non-Fastjson. If the switching cost is high, you can continue to use Fastjson and disable Safemode.
    • Services use the autoType function. It is recommended to scrap FastJSON.

Replace dependency considerations

Characteristics of enterprise projects or large projects:

  • The code structure is complex and maintained by multiple teams.
  • Undertake important online business, once serious bug may lead to serious accident.
  • If it is an old project, it may lack documentation and cannot be modified at will.
  • The project has many development branches that are constantly coming online in iterations.

So for large projects, migrating the underlying FastJSON to Gson can be a complex and painful task, as can any other dependency replacement.

I’ve summarized the following issues to consider when replacing project dependencies.

Caution, caution, caution

You can’t be too cautious, and if the project you’re trying to change is a critical business, making mistakes can be costly. And, for both the business and product teams, it’s “unbearable” to have the system crash without new features coming online. Although you may feel aggrieved that only you or your team know that while the business may seem the same, the underlying code has changed dramatically.

So, be careful!

Good communication between development team and test team

In the process of dependency replacement, it is necessary to make good project planning, such as module replacement and strict subdivision schedule.

Do a good job of pre-planning, development and testing can be orderly work.

Developers need to communicate development considerations in advance, such as version-dependent issues, so as to prevent multiple developers from changing the code at the same time and finding that the interface usage is different in different versions, which can be embarrassing and take extra time to deal with.

For testing, communicate well in advance. In general, testing doesn’t care too much about technical items that have no change to the business, because it’s neither optimization speed nor new functionality. But in fact, migration involves the bottom layer, it is easy to appear bugs. It takes a lot of testing time for the test team to understand that changing project dependencies can cost as much as new features, so make it as important as possible.

Do regression/interface testing

As mentioned above, the testing team needs to invest a lot of time, which is mainly used in the overall regression of the project function, namely regression testing.

Of course, not just business regression testing, but interface regression testing if possible.

If the company has an interface management platform, it can greatly improve the efficiency of such project testing.

For example, after a module has been modified, in a test environment (or sandbox environment), deploy an online version, deploy a modified version, and directly compare the data returned by the interface. Json comparison is generally a Json comparison. There are many Json comparison tools available on the web:

www.sojson.com/

Consider performance differences before and after migration

As described above in the performance comparison between Gson and Fastjson, replacement frameworks need to be aware of the performance differences between frameworks, especially for traffic traffic, i.e. high concurrency projects, where a large change in response time can cause additional consequences upstream and downstream.

Use Gson instead of Fastjson

Here is a summary of two common json framework methods, posted a detailed code example, to help you quickly get started with Gson, seamless switch!

Json deserialization

String jsonCase = "[{\ \" id ": 10001, \" the date \ ": 1609316794600, \" name \ ": \" xiao Ming \ "}, {\ \ "id" : 10002, \ "the date \" : 1609316794600, \ "name \" : \ "xiao li \}"] ". // fastjson JSONArray jsonArray = JSON.parseArray(jsonCase); System.out.println(jsonArray); System.out.println(jsonArray.getJSONObject(0).getString("name")); System.out.println(jsonArray.getJSONObject(1).getString("name")); / / output: / / / {" date: "1609316794600," name ":" xiao Ming ", "id" : 10001}, {" date: "1609316794600," name ":" xiao li ", "id" : 10002}] xiaoming / / / / / / Gson xiao li JsonArray jsonArrayGson = gson.fromJson(jsonCase, JsonArray.class); System.out.println(jsonArrayGson); System.out.println(jsonArrayGson.get(0).getAsJsonObject().get("name").getAsString()); System.out.println(jsonArrayGson.get(1).getAsJsonObject().get("name").getAsString()); Output: / / / / / {" id ": 10001," date ": 1609316794600," name ":" xiao Ming "}, {" id ": 10002," date ": 1609316794600," name ":" xiao li "}] / / / / the man xiao liCopy the code

It can be seen that the difference between the two mainly lies in the various types of GET. Gson call method has changed, but the change is not big.

So, let’s see if empty object deserialization is an exception:

String jsonObjectEmptyCase = "{}"; // fastjson JSONObject jsonObjectEmpty = JSON.parseObject(jsonObjectEmptyCase); System.out.println(jsonObjectEmpty); System.out.println(jsonObjectEmpty.size()); Jsonobjectempty = gson.fromjson (jsonObjectEmptyCase, jsonObject.class); System.out.println(jsonObjectGsonEmpty); System.out.println(jsonObjectGsonEmpty.size()); // output: // {} // 0Copy the code

No abnormality, happy.

Look at the empty array, after all [] feels more error-prone than {}.

String jsonArrayEmptyCase = "[]"; // fastjson JSONArray jsonArrayEmpty = JSON.parseArray(jsonArrayEmptyCase); System.out.println(jsonArrayEmpty); System.out.println(jsonArrayEmpty.size()); // [] // 0 // Gson JsonArray jsonArrayGsonEmpty = gson.fromjson (jsonArrayEmptyCase, jsonarray.class); System.out.println(jsonArrayGsonEmpty); System.out.println(jsonArrayGsonEmpty.size()); // output: // [] // 0Copy the code

There are no problems with both frameworks, perfect parsing.

Paradigm to deal with

Parsing generics is a very common feature, and most of the FastJSON code in our project is parsing JSON and Java beans.

// entity class User User = new User(); user.setId(1L); User. SetUserName (" ma "); // fastjson List<User> userListResultFastjson = JSONArray.parseArray(JSON.toJSONString(userList), User.class); List<User> userListResultFastjson2 = JSON.parseObject(JSON.toJSONString(userList), new TypeReference<List<User>>(){}); System.out.println(userListResultFastjson); System.out.println("userListResultFastjson2" + userListResultFastjson2); / / output: UserListResultFastjson [User [Hash = 483422889, ID =1, userName= Ma], Null] // userListResultFastjson2[User [Hash = 488970385, ID =1, userName= Ma], null] // Gson List<User> userListResultTrue = gson.fromJson(gson.toJson(userList), new TypeToken<List<User>>(){}.getType()); System.out.println("userListResultGson" + userListResultGson); UserListResultGson [User [Hash = 1435804085, ID =1, userName= Ma], null]Copy the code

As you can see, Gson can also support generics.

The List/Map to write

Fastjson differs from Gson in that Gson does not support directly writing a List to a value, whereas Fastjson does.

Therefore, Gson can only parse the List and write it into the value, as shown in the following code:

// entity class User User = new User(); user.setId(1L); User. SetUserName (" ma "); // fastjson JSONObject jsonObject1 = new JSONObject(); jsonObject1.put("user", user); jsonObject1.put("userList", userList); System.out.println(jsonObject1); / / output: / / {" userList ": [{" id" : 1, "userName" : "ma"}, null], "user" : {" id ": 1," userName ":" ma "}} / / Gson JsonObject JsonObject = new JsonObject(); jsonObject.add("user", gson.toJsonTree(user)); System.out.println(jsonObject); Output: / / / / {" user ": {" id" : 1, "userName" : "ma"}, "userList" : [{" id ": 1," userName ":" ma "}, null]}Copy the code

This way, Gson doesn’t look as convenient as Fastjson because the List is put in the form of gson.tojsonTree (user). This prevents you from entering the object first and modifying it later. (Some of you are more used to putting objects in first and then modifying them, so you have to change the code.)

Hump and underline conversion

The hump conversion underscores depend on changing Gson’s serialization mode to LOWER_CASE_WITH_UNDERSCORES

GsonBuilder gsonBuilder = new GsonBuilder(); gsonBuilder.setFieldNamingPolicy(FieldNamingPolicy.LOWER_CASE_WITH_UNDERSCORES); Gson gsonUnderScore = gsonBuilder.create(); System.out.println(gsonUnderScore.toJson(user)); {"id":1,"user_name":" ma "}Copy the code

Faqs demining

Below are the pits we stepped on in the process of Gson migration in the company’s project. These pits feel less technical when written now. But that’s why I wrote this article, to help you avoid these hard-to-find pits.

Some of these problems are discovered during regression testing, some are discovered during self-testing, and some are discovered after going live, such as when Swagger fails.

Date is serialized in different ways

I don’t know if you’ve thought about this, but if you have a caching system in your project, and you write to the cache using FastJSON, after you switch to Gson, you need to parse it out using Gson. So it is important to ensure that the parsing logic of both frameworks is the same, but this is obviously a good wish.

During testing, the Date type was discovered and resolved differently in the two frameworks.

  • Fastjson: Date directly parses to Unix
  • Gson: Serializes directly to the standard format Date

As a result, Gson reported an error directly when deserializing the JSON and could not convert it to Date.

Solution:

Create a new class that resolves the Date type:

import com.google.gson.TypeAdapter; import com.google.gson.stream.JsonReader; import com.google.gson.stream.JsonWriter; import java.io.IOException; import java.util.Date; public class MyDateTypeAdapter extends TypeAdapter<Date> { @Override public void write(JsonWriter out, Date value) throws IOException { if (value == null) { out.nullValue(); } else { out.value(value.getTime()); } } @Override public Date read(JsonReader in) throws IOException { if (in ! = null) { return new Date(in.nextLong()); } else { return null; }}}Copy the code

Next, when creating Gson, put it into a special processing class as Date:

Gson gson = new GsonBuilder().registerTypeAdapter(Date.class,new MyDateTypeAdapter()).create();
Copy the code

This allows Gson to process Date as Unix.

Of course, this is just for compatibility with older caches, so if you don’t think your repository has this concern, you can ignore it.

Abnormal SpringBoot

After switching to Gson, the interface of the Web project built with SpringBoot cannot be requested directly. Error similar to:

org.springframework.http.converter.HttpMessageNotWritableException
Copy the code

Since SpringBoot’s default Mapper is Jackson parse, we switched to Gson as the return object, and Jackson could not parse.

Solution:

Application. Properties add:

#Preferred JSON mapper to use for HTTP message conversion
spring.mvc.converters.preferred-json-mapper=gson
Copy the code

Abnormal Swagger

This problem is similar to the SpringBoot exception above, because Gson is introduced in SpringBoot, causing Swagger to fail to parse JSON.

Use a solution similar to the following (add Gson adapter) :

Yuyublog. Top / 2018/09/03 /…

  1. GsonSwaggerConfig.java
@ Configuration public class GsonSwaggerConfig {/ / set the swagger support gson @ Bean public IGsonHttpMessageConverter IGsonHttpMessageConverter() { return new IGsonHttpMessageConverter(); }}Copy the code
  1. IGsonHttpMessageConverter.java
public class IGsonHttpMessageConverter extends GsonHttpMessageConverter { public IGsonHttpMessageConverter() { Super.setgson (new GsonBuilder().registerTypeAdapter(json.class, New SpringfoxJsonToGsonAdapter ()). SerializeNulls () / / null value was also involved in serialized. The create ()); }}Copy the code
  1. SpringfoxJsonToGsonAdapter.java
public class SpringfoxJsonToGsonAdapter implements JsonSerializer<Json> { @Override public JsonElement serialize(Json json, Type type, JsonSerializationContext jsonSerializationContext) { return new JsonParser().parse(json.value()); }}Copy the code

@Mapping JsonObject is abnormal as an input parameter

Sometimes, we use things like:

public ResponseResult<String> submitAudit(@RequestBody JsonObject jsonObject) {}
Copy the code

If you use this code, you’re using Gson to parse json strings. However, this method is very risky, so please avoid using JsonObject to accept parameters directly.

In Gson, a JsonObject with a numeric field will be serialized to double, i.e. count = 0 will be serialized to count = 0.0.

Why does this happen? To put it simply, When Gson parses JSON to Object, it converts numeric types to double by default.

If Json corresponds to Object, it will be resolved to Map<String, Object>. The Object type is related to the specific value in Json, for example, the “” value of the double quotation mark is translated into STRING. We can see that the numeric type (NUMBER) is all converted to Double, so we have the problem we had before, where the integer data is translated to Double, for example, 30 becomes 30.0.

Look at Gson’s ObjectTypeAdaptor class, which inherits Gson’s TypeAdaptor abstract class:

For detailed source code analysis and principle elaboration, you can see this extended reading:

www.jianshu.com/p/eafce9689…

Solution:

The first option: accept the input as an entity class, not a JsonObject

The second solution: similar to the above solution to the Date type problem, define your own Adaptor, to accept numbers, and processing. This idea, which I think is feasible but difficult, can affect other types of parsing and needs to be taken into account when designing adapters.

conclusion

This article is intended for students who need to migrate their projects to the Gson framework.

Generally speaking, small personal projects do not need to spend so much energy on migration, so this article may target a relatively narrow group of people.

However, the article also offers some ideas for solving common problems, such as how to evaluate the need for a migration framework. Many issues such as framework compatibility, performance difference between the two, and time spent on migration need to be considered.

Hope this article is helpful to you.

reference

How to Migrate from Fastjson to Gson

Juejin. Cn/post / 684490…

The author of “FastJson Migration to Jackson” wrapped his own utility class to do the migration

Mxcall. Making. IO/posts / % E5% B…

Do you Really Know how to use Gson? How to Use Gson

www.jianshu.com/p/e74019622…

Json Performance Comparison

Github.com/zysrxx/json…

Fastjson official document

Github.com/alibaba/fas…

Easy hundred tutorial

www.yiibai.com/jackson

Pay attention to my

I am an Internet backend development engineer struggling in the front line.

Focus on back-end development, data security, edge computing and other directions, welcome to exchange.

I can be found on every platform

  • Wechat official account: A ramble on back-end technology
  • Making:@qqxx6661
  • CSDN: @Pretty three knives
  • Zhihu: @ Ramble on back-end technology
  • Nuggets: @ pretty three knife knife
  • Tencent Cloud + community: @ Back-end technology ramble
  • Blog Garden: @ Back-end Technology Ramblings
  • BiliBili: @Pretty three knives

Original article main content

  • Back-end development practice (Java based)
  • Technical interview
  • Algorithm problem solving/data structure/design pattern
  • Interesting things about my life

Personal public account: Back-end technology ramble

If this article is helpful to you, please like it and bookmark it