Go World Cup backstage management system

Hello everyone, my name is Xie Wei, I am a programmer.

Take advantage of the weekend for an update. Last time I talked about how to get familiar with a project quickly, and at the end of this article I said that the best way is to re-implement a project using the same technology stack.

This paper is to use the same technology stack to achieve the 2018 World Cup background management system.

The main technologies used are:

• Gin quickly build web Server
• Gin-swagger automates build API documentation
• Gorm operates the database
• Fresh implements Web Server listening
• Viper implements reading user configuration
• The database uses PostGRE
• Goquery implements web page parsing

The main ideas are as follows:

The first step:

Since it is the 2018 World Cup backstage management system, then certainly need the data of this World Cup, then where does the data come from?

Target website Russia 2018 World Cup

Now that you know the target site, what’s the next step?

Web crawler.

• matches
• teams
• groups
• players
• statistics
• awards
• classic

The main information needed is this.

The second step:

Analyze the web source code. Web crawler. One of the better libraries for web page parsing in GO is GoQuery

Analyze the desired target data one by one.

Step 3:

Where is the data stored?

You certainly have to do whatever you want, save the text, or save the database. General enterprise applications, will be stored locally?

So I’ll just keep it in the database. Database selection, press yourself, I choose PostGRE.

Since the use of the database, it is necessary to operate the database, if you want to code filled with SQL statements, then you can choose to write SQL statements, of course, I think the better maintenance is to use ORM, GO use ORM technology, a better library is GORM.

Gorm allows you to easily add, delete, change and check your database.

Step 4:

Since the data has, then how to achieve background management system?

Use restful apis to add, delete, modify, and query resources.

Gin is recommended. Of course it’s OK if you like other frameworks, or even if you like native ones, that’s OK.

However, I think GIN is fast, lightweight and cheap to learn. You can easily implement a Web Server.

Gin can be extended using middleware.

Step 5:

If you don’t want your data to be freely accessible to anyone, how do you limit it? The effect of the front end is that you need to log in to access resources, so how to achieve the back end?

JWT: JSON Web Token Uses JSON to transfer data, which is used to determine whether a user is logged in.

Specific practices:

• Log in and get JWT
• When accessing, JWT is mounted in the Header when requesting it

This is just the core code:

1. Project structure

├ ─ ─ configs ├ ─ ─ docs │ └ ─ ─ swagger ├ ─ ─ domain ├ ─ ─ infra │ ├ ─ ─ adapter │ ├ ─ ─ the config │ ├ ─ ─ crypt │ ├ ─ ─ the download │ ├ ─ ─ init │ │ └ ─ ─ model ├ ─ ─ UI └ ─ ─ API - server │ ├ ─ ─ admins │ ├ ─ ─ allow │ ├ ─ ─ classic │ ├ ─ ─ coaches │ ├ ─ ─ controller │ ├ ─ ─ groups │ ├── matches │ ├─ players │ ├─ statistics │ ├─ soccer ├─Copy the code

• Configs Indicates data configuration information, such as host address, port, user name, and password
• Docs API documents, gin-Swagger built automatically, not created manually
• Domain domain layer, mainly the analysis and crawling of web page information and storage
• Infra infrastructure layer, mainly string processing, encryption algorithms, access to web source code, database model definition
• The UI user visualization layer, primarily the operations of the API built by GIN, includes routing, responses, and Swagger document annotations
• Vendor Third-party library

2. Obtain the source code of the web page

Use the built-in NET/HTTP

func Downloader(url string) (*goquery.Document, error) {
	request, err := http.NewRequest("GET", url, nil)
	iferr ! = nil {return nil, ErrDownloader
	}

	request.Header.Add("User-Agent"."Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36")
	client := http.DefaultClient

	response, err := client.Do(request)
	iferr ! = nil {return nil, ErrDownloader
	}

	defer response.Body.Close()
	return goquery.NewDocumentFromReader(response.Body)
}


Copy the code

If you encounter dynamic loading data and don’t want to bother analyzing web pages, you can use Selenium

func DownloaderBySelenium(url string) (string, error) {
	caps := selenium.Capabilities{
		"browserName": "chrome",
	}

	imageCaps := map[string]interface{}{
		"profile.managed_default_content_settings.images": 2,
	}
	chromeCaps := chrome.Capabilities{
		Prefs: imageCaps,
		Path:  "",
		Args: []string{
			"--headless"."--no-sandbox"."- the user-agent = Mozilla / 5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/604.4.7 (KHTML, like Gecko) Version/11.0.2 Safari/604.4.7",
		},
	}
	caps.AddChrome(chromeCaps)

	service, err := selenium.NewChromeDriverService(
		config.ChromeDriverPath, 9515,
	)
	defer service.Stop()

	iferr ! = nil { fmt.Println(ErrSeleniumService)return "", ErrSeleniumService
	}
	webDriver, err := selenium.NewRemote(caps, fmt.Sprintf("http://localhost:%d/wd/hub", 9515))

	iferr ! = nil { fmt.Println(ErrWebDriver)return "", ErrWebDriver
	}

	err = webDriver.Get(url)

	iferr ! = nil { fmt.Println(ErrWebDriverGet)return "", ErrWebDriverGet
	}
	return webDriver.PageSource()

}
Copy the code

3. Database table definition and response information definition

Database table definitions control gorM Model definitions, types, non-null, default values, and so on using tags

// Awards table definitiontype Award struct {
	ID        uint   `gorm:"primary_key; column:id"`
	AwardName string `gorm:"type:varchar(64); not null; column:award_name"`
	URL       string `gorm:"type:varchar(128); not null; column:url"`
	Info      string `gorm:"type:varchar(128); not null; column:info"'} // API response information definitiontype AwardSerializer struct {
	ID        uint   `json:"id"`
	AwardName string `json:"award_name"`
	Info      string `json:"info"`
	URL       string `json:"url"`
}

func (a *Award) Serializer() AwardSerializer {
	return AwardSerializer{
		ID:        a.ID,
		AwardName: a.AwardName,
		Info:      a.Info,
		URL:       a.URL,
	}
}


Copy the code

4. Crawl information into the database

func Awards(doc *goquery.Document) error {
	var err error
	count := 0
	urlList := make([]string, 0, 0)
	urlList = append(urlList, "/worldcup/awards/golden-boot/")
	urlList = append(urlList, "/worldcup/awards/golden-glove/")
	urlList = append(urlList, "/worldcup/awards/golden-ball/")
	for _, url := range urlList {
		completeAwardURl := config.RootURL + url
		doc, err := download.Downloader(completeAwardURl)
		iferr ! = nil { err = ErrorAwardDownloaderbreak
		}
		// db save
		awards := callBack(completeAwardURl, doc)
		fmt.Println(completeAwardURl)
		for _, award := range awards {
			fmt.Println(award)
			count++
			// push data into db
			initiator.POSTGRES.Save(&award)

		}
	}
	fmt.Println(count)

	return err
}

func callBack(url string, doc *goquery.Document) []model.Award {

	allAwardInfo := make([]model.Award, 0, 0)

	awardName := doc.Find("h1").Eq(2).Text()

	doc.Find("div p").Each(func(i int, selection *goquery.Selection) {

		if i > 6 {

			awardInfo := selection.Text()
			if strings.HasPrefix(awardInfo, "*") {
				return
			}
			oneAward := model.Award{}
			oneAward.URL = url
			oneAward.AwardName = awardName
			oneAward.Info = awardInfo
			allAwardInfo = append(allAwardInfo, oneAward)
		}

	})
	return allAwardInfo
}

Copy the code

5. Build restful apis

func awardsRegistry(r *gin.RouterGroup) {
	r.GET("/awards", awards.ShowAllAwardHandler)
	r.GET("/awards/:awardID", awards.ShowAwardHandler)
}
Copy the code

package awards

import (
	"FIFA-World-Cup/infra/init"
	"FIFA-World-Cup/infra/model"
	"fmt"
	"github.com/gin-gonic/gin"
	"github.com/pkg/errors"
	"net/http"
)

var (
	ErrorAwardParam = errors.New("award param is not correct")
)

// ShowAwardHandler will list Awards
// @Summary List Awards
// @Accept json
// @Tags Awards
// @Security Bearer
// @Produce  json
// @Param awardID path string true "award id"// @Resource Awards // @Router /awards/{id} [get] // @Success 200 {object} model.AwardSerializer func ShowAwardHandler(c  *gin.Context) { id := c.Param("awardID")

	var award model.Award
	if dbError := initiator.POSTGRES.Where("info LIKE ?", fmt.Sprintf("%%%s%%", id)).First(&award).Error; dbError ! = nil { c.AbortWithError(400, dbError)return
	}
	c.JSON(http.StatusOK, award.Serializer())

}

type ListAwardParam struct {
	Search string `form:"search"`
	Return string `form:"return"`
}

// ShowAllAwardHandler will list Awards
// @Summary List Awards
// @Accept json
// @Tags Awards
// @Security Bearer
// @Produce  json
// @Param search path string false "award_name"
// @param return path string false "return = all_list"
// @Resource Awards
// @Router /awards [get]
// @Success 200 {array} model.AwardSerializer
func ShowAllAwardHandler(c *gin.Context) {

	var param ListAwardParam

	iferr := c.ShouldBindQuery(&param); err ! = nil { c.AbortWithError(400, ErrorAwardParam)return
	}

	var awards []model.Award

	ifparam.Search ! ="" {
		if dbError := initiator.POSTGRES.Where("award_name LIKE ?", fmt.Sprintf("%%%s%%", param.Search)).Find(&awards).Error; dbError ! = nil { c.AbortWithError(400, dbError)return}}if param.Return == "all_list" {
		ifdbError := initiator.POSTGRES.Find(&awards).Error; dbError ! = nil { c.AbortWithError(400, dbError)return
		}
	}

	var result = make([]model.AwardSerializer, len(awards))

	for index, award := range awards {
		result[index] = award.Serializer()
	}
	c.JSON(http.StatusOK, result)
}

Copy the code

The comments above the concrete response function are needed to build the automated document.

6. JWT certification

package controller

import (
	"FIFA-World-Cup/infra/init"
	"FIFA-World-Cup/infra/model"
	"errors"
	"fmt"
	"github.com/gin-gonic/gin"
	"strings"
)

var (
	ErrorAuth      = errors.New("please add token: 'Authorization: Bearer xxxx'")
	ErrorAuthWrong = errors.New("Token is not right, example: Bearer XXXX")
)

func AuthRequired() gin.HandlerFunc {
	return func(c *gin.Context) {
		if vendor := c.Request.Header.Get("X-Requested-With"); vendor ! ="" {
			c.Set("X-Requested-With", vendor)
		}

		header := c.Request.Header.Get("Authorization")
		if header == "" {
			c.AbortWithError(400, ErrorAuth)
			return
		}

		authHeader := strings.Split(header, "")

		iflen(authHeader) ! = 2 { c.AbortWithError(400, ErrorAuthWrong)return
		}

		token := authHeader[1]

		var admin model.Admin
		fmt.Println(token)
		if dbError := initiator.POSTGRES.Where("auth_token = ?", token).First(&admin).Error; dbError ! = nil { c.AbortWithError(400, dbError) }else {
			c.Set("current_admin", admin)
			c.Next()
		}
	}
}
Copy the code

What does that mean?

1. Users need to register or log in, and the background generates the corresponding auth_token

select * from admins;

 id |          created_at           |          updated_at           | deleted_at |      name      |                auth_token                |                     encrypted_password' | phone | state ----+-------------------------------+-------------------------------+------------+----------------+--------------------- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- 2, 2018-07-20 16:10:11. 099085 2018-07-20 16:10:11. 099085 FIFA World Cup - c6d81d35bc598ddedf3e0b798cd5d463139ab6c9 $2a$04$wKHmdGixgrISJM7wV3rKn.6HX5Bjg8.JbelGYl/443ber3aXI/K8K 110120119 adminCopy the code

Each user generates a corresponding Auth_token

This token is required to access the resource HEADER for authentication purposes.

Effect of 7.

Swagger – API documentation

The API list

Video version explanation

BiliBili

8. The source code

FIFA-World-Cup-2018

This is the end, I am Xie Wei, goodbye, thank you.