Original: Coding diary (wechat official ID: Codelogs), welcome to share, reprint please reserve the source.

Introduction to the

Recently, in order to reduce the burden of a core system, the group decided to migrate some query interfaces with heavy call volume to another system. Due to the complexity of the interface logic, in order to ensure the consistency of the interface logic, we decided to replay the request parameters within a week on the two interfaces and verify the response results of the interfaces on both sides with scripts. The data returned by the interface is in json structure, and I wanted to use ICDIFF and JQ to achieve it, but I didn’t come up with it due to time constraints. My colleague used Python script to achieve this comparison, but later, after careful thinking, I found a way to use ICDIFF and JQ to achieve it.

Icdiff compares text

For normal text comparisons, the diff command can be used, but ICdiff is better and the results are more intuitive.

$ text1='{"name":"apple","item_no":2}'
$ text2='{"name":"a apple","item_no":2}'
$ icdiff <(echo "$text1") < (echo "$text2")
Copy the code

But if you format it with JQ and compare it, it looks a little clearer.

$ icdiff <(echo "$text1"|jq .) < (echo "$text2"|jq .)
Copy the code

However, sometimes json data may have the same data, but the key order is inconsistent, for example:

$ text1='{"name":"apple","item_no":2}'
$ text2='{"item_no":2,"name":"apple"}'
$ icdiff <(echo "$text1"|jq .) < (echo "$text2"|jq .)
Copy the code



For operations like object to JSON, it is possible to return JSON keys in different order on different machines.

Jq provides the -s option to sort the key of the output JSON, as follows:

$ icdiff <(echo "$text1"|jq -S .) < (echo "$text2"|jq -S .)
Copy the code

We thought this would solve all the problems, but when we run the script, we find that the order of the items in the JSON array may not be consistent, as follows:

$ text1='{ "order_id": 121345435624, "waybills": [ { "waybill_id": 1, "name": "package1", "items": [ { "name": "orange", "item_no": 1 }, { "name": "apple", "item_no": 2 } ] }, { "waybill_id": 2, "name": "package2", "items": [ { "name": "pear", "item_no": 3 }, { "name": "banana", "item_no": 4 } ] } ] }'
$ text2='{ "order_id": 121345435624, "waybills": [ { "waybill_id": 2, "name": "package2", "items": [ { "name": "banana", "item_no": 4 }, { "name": "pear", "item_no": 3 } ] }, { "waybill_id": 1, "name": "package1", "items": [ { "name": "orange", "item_no": 1 }, { "name": "apple", "item_no": 2 } ] } ] }'
$ icdiff <(echo "$text1"|jq -S .) < (echo "$text2"|jq -S .)  
Copy the code



Because ICdiff doesn’t recognize this sequential difference, it just compares the two JSON files line by line, and even though the two json files are logically the same, it can’t tell the difference.

This order difference can occur in a program when:

  1. When querying data from a database, if SQL does not have an ORDER BY statement, the order in which the results are returned is uncertain.
  2. If the data is in a Map, when values() is converted to List to return the data, the order of return structures is also uncertain.

It’s a bit of a hassle to do this, because you need to sort the array in JSON and then compare it, which I couldn’t figure out how to do before. Later, HOWEVER, I took a closer look at the JQ MAN documentation and found a useful function called Walk, so I tried it out and realized it could be implemented.

$ json_sort='walk(if type == "array" and length >0 then (if .[0].item_no then sort_by(.item_no) elif .[0].waybill_id then sort_by(.waybill_id) else. end) else . end)'
$ icdiff <(echo "$text1"|jq -S "$json_sort") < (echo "$text2"|jq -S "$json_sort")
Copy the code

conclusion

Linux commands are always more powerful when combined, but it takes some time.

Content of the past

Linux text command tips (2) Linux text command tips (1) AWK is really a magic ah easy to use parallel command common network command summary