This is the 24th day of my participation in Gwen Challenge

GO in thegjsonApplication and sharing

Last time we shared the data of using GO to crawl static web pages, let’s review it

• Share a brief description of static and dynamic web pages
• GO crawls simple data from static web pages
• GO crawls pictures on web pages
• And crawl resources on the web page

If you’re still interested in GO crawling static data, check out this article to share a wave of GO crawlers

jsonWhat is?

JSON(JavaScript Object Notation) is a lightweight data interchange format

It is based on a subset of ECMAScript (the JS specification of the European Computer Association) and uses a text format that is completely independent of the programming language to store and represent data

Json has the following advantages:

• The hierarchy is simple and clear
• Easy to read and write
• Easy machine parsing and generation
• Improves network transmission efficiency

Let’s briefly list some of the ways we serialize data

• json
• xml

Extensible Markup Language (EXTENSIBLE Markup Language) is a simple data storage language

• protobuf

Is a platform – independent, language – independent, extensible, lightweight and efficient protocol for serializing data structures for network communication and data storage

gjsonWhat is?

It’s a library in GO

Primarily, it provides a very fast and easy way to retrieve values from JSON documents

This gJSON library, in fact, is the abbreviation of get + JSON, the same sJSON library, friends know he represents the meaning of it, is set + json meaning

gjsonHow to use it?

How to use GJSON, XDM, HERE I put the basic use of this library, involving knowledge points, and matters needing attention, to comb through

If you want to see gJSON source code is how to achieve efficient and fast operation json, interested friends, can be referenced on the basis of the first to take a detailed look at the source code

This article shares the following four aspects to practice and comb out the use of GJSON:

• gjsonSimple use of
• gjson 的 json 行
• gjsonAnd custom modifiers
• gjsonMatching rules for key paths

gjsonSimple use of

Let’s simply use a GJSON, and the following encoding involves the following points:

• Set specificjsondata
• * * checkjsonData ** is valid
• Get a single value at a time
• Get multiple values at once

package main

import (
   "log"
   "github.com/tidwall/gjson"
)

func main(a) {

   // Set the parameters to print the number of lines
   log.SetFlags(log.Lshortfile | log.LstdFlags)
   
  // Set the JSON data
   json := ` { "author": { "name": "xiaomotong", "age": 18, "hobby": "writing" }, "extra": "hello wolrd" "picList":[{"name":"xiaozhu1"},{"name":"xiaozhu2"}] } `
   // Verify that the JSON string is valid
   // If it is illegal, gjson will not panic and may get a strange value
   if gjson.Valid(json){
      log.Println("json valid ...")}else{
      log.Fatal("json invalid ... ")}// Get the value of author.name
   aName := gjson.Get(json, "author.name")
   log.Println("aName :", aName.String())

   // Get the value of extra
   extra := gjson.Get(json, "extra")
   log.Println("extra:", extra)

   // Get the value of a nonexistent key
   non := gjson.Get(json, "non")
   log.Println("non:", non)


   // Get multiple key values of JSON at once
	res := gjson.GetMany(json, "author.age"."author.hobby"."picList")
	for i, v := range res{
		if i == 0{
			log.Println(v.Int())
		}else if i == 2{
			for _,vv := range v.Array(){
				log.Println("picList.name :",vv.Get("name"))}}else{
			log.Println(v)
		}
	}
}
Copy the code

After running the above code, you should see the following:

2021/06/xx xx:32:04 main.go:28: json valid ...
2021/06/xx xx:32:04 main.go:35: aName : xiaomotong
2021/06/xx xx:32:04 main.go:39: extra: hello wolrd
2021/06/xx xx:32:04 main.go:43: non:
2021/06/xx xx:32:04 main.go:50: 18
2021/06/xx xx:32:04 main.go:57: writing
2021/06/xx xx:32:04 main.go:53: picList.name : xiaozhu1
2021/06/xx xx:32:04 main.go:53: picList.name : xiaozhu2
Copy the code

We need to be careful to get our data source right, that is, our JSON data must be valid, otherwise, using the GJSON library data will not be the value we expect

• usegjson.Get(), gets a single value
• usegjson.GetMany(), gets multiple values
• usegjson.Valid()To determinejsonWhether the data is valid

gjson 的 json 行

Now look at the JSON line

Gjson provides the following syntax to parse JSON row data:

• ..#

Output the length of the JSON row array

• ** #.author **

Output the value of the author in each row of json as an array

• ..#(author=”xiaomotong”).hobby

Output The hobby value of author = xiaomotong in the JSON line

• ..1

Output the second line of the JSON line array if.. 2 prints line 3

• traverseJson line

Use gjson.ForEachLine to traverse each row of the JSON row, and the details of each row can also be traversed

Let’s write a DEMO to cover the syntax used above:

package main

import (
   "github.com/tidwall/gjson"
   "log"
)

const json = ` {"author": "xiaomotong", "age": 18, "hobby":"play"} {"author": "xiaozhu", "age": 19 , "hobby":"eat"} {"author": "zhangsan", "age": 20, "hobby":"drink"} {"author": "lisi", "age": 21, "hobby":"sleep"}`

func main(a) {

   // Set the parameters to print the number of lines
   log.SetFlags(log.Lshortfile | log.LstdFlags)

   // Outputs the length of the JSON array of rows
   log.Println(gjson.Get(json, ".. #"))
    
   // Outputs the third line of the JSON row array
   log.Println(gjson.Get(json, ".. 2. ""))
    
   // Outputs the value of the author in each row of json to form an array
   log.Println(gjson.Get(json, ".. #.author"))
    
   // Output The hobby value corresponding to author = xiaomotong in the JSON line
   log.Println(gjson.Get(json, `.. #(author="xiaomotong").hobby`))

   // Iterate over the JSON line
   gjson.ForEachLine(json, func(jLine gjson.Result) bool {
      log.Println("author:", gjson.Get(jLine.String(), "hobby"))
      return true})}Copy the code

After the above code is run, the result is as follows:

2021/06/xx xx:17:52 main2.go:20: 4
2021/06/xx xx:17:52 main2.go:22: {"author": "zhangsan"."age": 20."hobby":"drink"}
2021/06/xx xx:17:52 main2.go:24: ["xiaomotong"."xiaozhu"."zhangsan"."lisi"]
2021/06/xx xx:17:52 main2.go:26: play
2021/06/xx xx:17:52 main2.go:30: author: play
2021/06/xx xx:17:52 main2.go:30: author: eat
2021/06/xx xx:17:52 main2.go:30: author: drink
2021/06/xx xx:17:52 main2.go:30: author: sleep
Copy the code

Let’s look at the implementation of gjson.ForEachLine:

// ForEachLine iterates through lines of JSON as specified by the JSON Lines
// format (http://jsonlines.org/).
// Each line is returned as a GJSON Result.
func ForEachLine(json string, iterator func(line Result) bool) {
   var res Result
   var i int
   for {
      i, res, _ = parseAny(json, i, true)
      if! res.Exists() {break
      }
      if! iterator(res) {return}}}Copy the code

Each row returns a JSON Result, Result

ParseAny parses the specific JSON data for each row. The parseAny function goes into a lot of detail about how to determine how to process each character

// parseAny parses the next value from a json string.
// A Result is returned when the hit param is set.
// The return values are (i int, res Result, ok bool)
func parseAny(json string, i int, hit bool) (int, Result, bool) {
   var res Result
   var val string
   // do it character by character
   // Different characters have corresponding logic, interested XDM can be detailed
   for ; i < len(json); i++ {
      if json[i] == '{' || json[i] == '[' {
         i, val = parseSquash(json, i)
         if hit {
             // Assign the corresponding character
            res.Raw = val
            res.Type = JSON
         }
         return i, res, true
      }
      if json[i] <= ' ' {
         continue
      }
      // After the above special cases are excluded, proceed with the character processing as follows
      switch json[i] {
      case '"':
         i++
         var vesc bool
         var ok bool
         i, val, vesc, ok = parseString(json, i)
         if! ok {return i, res, false
         }
         if hit {
            res.Type = String
            res.Raw = val
            if vesc {
               res.Str = unescape(val[1 : len(val)- 1])}else {
               res.Str = val[1 : len(val)- 1]}}return i, res, true
      case The '-'.'0'.'1'.'2'.'3'.'4'.'5'.'6'.'7'.'8'.'9':
         i, val = parseNumber(json, i)
         if hit {
            res.Raw = val
            res.Type = Number
            res.Num, _ = strconv.ParseFloat(val, 64)}return i, res, true
      case 't'.'f'.'n':
         vc := json[i]
         i, val = parseLiteral(json, i)
         if hit {
            res.Raw = val
            switch vc {
            case 't':
               res.Type = True
            case 'f':
               res.Type = False
            }
            return i, res, true}}}return i, res, false
}
Copy the code

Let’s look at the concrete data structure of Result

// Result represents a json value that is returned from Get().
type Result struct {
   // Type is the json type
   Type Type
   // Raw is the raw json
   Raw string
   // Str is the json string
   Str string
   // Num is the json number
   Num float64
   // Index of raw value in original json, zero means index unknown
   Index int
}
Copy the code

Str String, Num float64, Index int

Let’s look at the types of these

const (
   // Null is a null json value
   Null Type = iota
   // False is a json false boolean
   False
   // Number is json number
   Number
   // String is a json string
   String
   // True is a json true boolean
   True
   // JSON is a raw block of JSON
   JSON
)
Copy the code

So it looks at a glance, or for the gJSON library parsing corresponding characters to further research, you can download gJSON, see gjson.go source file inside the specific implementation

gjsonMatching rules for key paths

What is the key path?

Let’s create a table to see what matching rules gJSON supports

tag	instructions
?	Matches a single character, for examplehell? I can match hello, but I can’t match Helloo
*	Matches any number of characters, for examplehell* It can match Hello, helloooo, whatever
xx.xx	Used to match arrays, such as Hello is an array, thenhello.0It matches the first element of the array hello.1That’s the two elements that match
xx.#	Gets the length of the array, for example, Hello.#
If it appears in the key name., then need to use\escape	It’s easy to understand, for example, the key namehello.world ， When you want to use this key, you need to escape it like thishello\.world
= =,! =, <, <=, >, >=	For example,helloIs a group number, and there are elements in the array that are fieldsname 和 age When we match, we can add#To have the flexibility to match the data we want For example: hello. # (name = “xiaozhu”). The age
%	Pattern matching, for example hello.#(name%”n*”).age
! %	Pattern matching, for example hello.#(name! %”n*”).age

Let’s use the matching rules above:

package main

import (
   "github.com/tidwall/gjson"
   "log"
)
// json source data
const json = ` { "author":{"name":"xiaomotong", "nick": "xiaozhu"}, "age": 18, "hobby": ["play", "eat", "drink"], "love.music": "one day", "location": [ {"province": "gd", "city":"gz", "area": "huangpu"}, {"province": "gd", "city":"sz", "area": "nanshan"}, ] } `

func main(a) {

   // Set the parameters to print the number of lines
   log.SetFlags(log.Lshortfile | log.LstdFlags)

   // Get the name
   log.Println("author:", gjson.Get(json, "author.name"))
   // Get the age
   log.Println("age:", gjson.Get(json, "age"))
	
   / / use? # * Operation Hobby
   log.Println("hobby:", gjson.Get(json, "hobb?"))
   log.Println("hobby count:", gjson.Get(json, "hobby.#"))
   log.Println("second hobby:", gjson.Get(json, "ho? by.1"))
   log.Println("third hobby:", gjson.Get(json, "ho*.2"))
	
   // key with. We use \. To escape
   log.Println("love.music", gjson.Get(json, `love\.music`))
	
   Gjson. Get(json, "location.0. City ");
   log.Println("location first city :", gjson.Get(json, "location.0"))
   log.Println("location second city :", gjson.Get(json, "location.1"))}Copy the code

The results of the above code are as follows:

2021/06/xx xx:03:26 main3.go:27: author: xiaomotong
2021/06/xx xx:03:26 main3.go:29: age: 18
2021/06/xx xx:03:26 main3.go:31: hobby: ["play"."eat"."drink"]
2021/06/xx xx:03:26 main3.go:32: hobby count: 3
2021/06/xx xx:03:26 main3.go:34: second hobby: eat
2021/06/xx xx:03:26 main3.go:35: third hobby: drink
2021/06/xx xx:03:26 main3.go:37: love.music one day
2021/06/xx xx:03:26 main3.go:39: location first city : {"province": "gd"."city":"gz"."area": "huangpu"}
2021/06/xx xx:03:26 main3.go:40: location second city : {"province": "gd"."city":"sz"."area": "nanshan"}
Copy the code

Gjson library of all kinds of rules, is not difficult, we can look at the use of them, their actual operation to find, memory will be more profound, when the time really used, to check again, is not unfamiliar

gjsonAnd custom modifiers

Finally, let’s talk about modifiers in the GJSON library. Modifiers are also very powerful, and are generally played with key addresses

Let’s take a look at the built-in modifiers:

tag	instructions
@reverse	Flipping an array
@ugly	Remove all whitespace from JSON
@valid	Verify the validity of JSON
@pretty	Make JSON easier to read
@flatten	Array is flat, the [” 1 “the pig, [” pig 2″, “three pigs”]] to [” pig 1 “, “pig 2”, “three pigs”]
@this	Returns the current element, which can be used to return the root element
@join	To combine multiple objects into one object

Go to DEMO

package main

import (
	"github.com/tidwall/gjson"
	"log"
)

const json = ` { "author":{"name":"xiaomotong", "nick": "xiaozhu"}, "age": 18, "hobby": ["play", "eat", "drink"], "love.music": "one day", "location": [ {"province": "gd", "city":"gz", "area": "huangpu"}, {"province": "gd", "city":"sz", "area": "nanshan"}, ] } `
func main(a) {

	// Set the parameters to print the number of lines
	log.SetFlags(log.Lshortfile | log.LstdFlags)

	// Flip the Hobby array
	log.Println("Reverse Array Hobby Reverse :", gjson.Get(json, "hobby|@reverse"))
    
	// Remove whitespace
	log.Println("Remove whitespace location.0:", gjson.Get(json, "location.0|@ugly"))

	// Make JSON easier to read pretty
	log.Println("Make JSON easier to read pretty Location:", gjson.Get(json, "location.1|@pretty"))
    
	// Outputs the entire JSON
	log.Println("Output entire JSON this:", gjson.Get(json, "@this"))

	test := '[" piggy 1", [" piggy 2", "piggy 3"]]'
	/ / flat
	log.Println("Flat this:", gjson.Get(test, "@flatten"))}Copy the code

Running the code above, we can see something like this:

2021/06/xx xx:30:24 main4.go:27Array Hobby Reverse: ["drink"."eat"."play"]
2021/06/xx xx:30:24 main4.go:29: Removes the whitespace location. 0: {"province":"gd"."city":"gz"."area":"huangpu"}
2021/06/xx xx:30:24 main4.go:32Pretty Location: {make json easier to read"province": "gd"."city": "sz"."area": "nanshan"
}

2021/06/xx xx:30:24 main4.go:34Output the entire json this: {"author": {"name":"xiaomotong"."nick": "xiaozhu"},
  "age": 18."hobby": ["play"."eat"."drink"]."love.music": "one day"."location": [{"province": "gd"."city":"gz"."area": "huangpu"},
    {"province": "gd"."city":"sz"."area": "nanshan"]}},2021/06/xx xx:30:24 main4.go:39: flatten this: ["The pig 1"."The pig 2"."Three pigs"]
Copy the code

Ha-ha, did you think it was easy,

We can also customize modifiers. Let’s take a look

Use the gjson.AddModifier function to add our custom modifier implementation, as shown below

func main(a) {
   gjson.AddModifier("myTo".func(json, arg string) string {
       // Change the string to uppercase
      if arg == "title" {
         return strings.ToTitle(json)
      }

      return json
   })

   const json = `{"children": ["hello", "world", "xiaomotong"]}`
   fmt.Println(gjson.Get(json, "children|@myTo:title"))}Copy the code

The results are as follows:

["HELLO"."WORLD"."XIAOMOTONG"]
Copy the code

AddModifier binds custom modifier commands to GJSON syntax. This action is not thread-safe and should be performed before all other GJSON functions are used.

// AddModifier binds a custom modifier command to the GJSON syntax.
// This operation is not thread safe and should be executed prior to
// using all other gjson function.
func AddModifier(name string, fn func(json, arg string) string) {
	modifiers[name] = fn
}
Copy the code

conclusion

• To share thejsonwithgjsonWhat do they represent
• gjsonSimple use of
• gjsonCheck, get the value
• gjsontheJson line
• gjsonKey path matching rules for
• gjsonAnd custom modifiers

Welcome to like, follow and favorites

Friends, your support and encouragement, I insist on sharing, improve the quality of the power

Ok, this time here, next GO permission management Casbin

Technology is open, our mentality, should be more open. Embrace change, live in the sun, and strive to move forward.

I am Nezha, welcome to like, see you next time ~