I. Database

1. Batch operation is best for data

Whether you add, delete, check or change, do it in batches, not one by one.

2. Set the maximum number of connections

Generally, the database has a maximum number of connections. Therefore, you need to set the maximum number of connections on the server to prevent errors.

3. Does the Go-Redis package have connection number connection pool? Will there be bugs when multiple coroutines compete for redis clients? To prevent this, open a separate Goroutine to manipulate the Redis client to store data (option 2: lock dao layer functions).

Note: this bag can set up the connection pool, github.com/gomodule/redigo/redis currently use github.com/go-redis/redis our project

type Data struct {
    ...
}

var saveData = make(chan Data)

func main(){
    go SaveHistory()
    for. { gofunc() {... data := Data{ } saveData <- data }() } ... } // the SaveHistory loop reads the channel and saves the data to redis funcSaveHistory() {
	for {
		select {
		case data := <-saveData:
			err := dao.RedisSaveHistory(data.SaveHistory, data.Unit)
			redisKey := dao.HistoryRedisKey(data.SaveHistory[0].Meta.Platform, data.SaveHistory[0].Meta.Pairid, data.Unit)
			iferr ! = nil { log.Error("persistence err, ", err.Error(), ", redisKey: "+redisKey)
			}
		}
	}
}
Copy the code

4. Redis should use less keys method, which will cause great performance loss. It can be replaced by hkeys, and corresponding data structure should also be changed to hash

5. Operate the database concurrently

We will need to learn more about this later, but we now know that optimistic locking and pessimistic locking are the methods to maintain database data consistency, but these two locks are not suitable for high concurrency scenarios.

Second, the concurrent

1. Coroutines do not need coroutines pool, but the number of coroutines should be limited, otherwise a large number of waiting G will block the memory, resulting in a heavier burden on the scheduler, which may eventually break down.

Third, program writing

1. It’s best not to make assumptions about data sets

Example: Do not assume that the data obtained from the database is ordered, even if it was ordered before, what if the data in the database is wrong? It’s best to sort it out yourself.

Request validation

1. Set an IP address restriction whitelist on the firewall

Scenario: Confirm the IP address list of the requester

2. Use AES to generate a key pair for digital signature

Steps:

  1. Generate a key pair,
  2. Server saves key
  3. Public Key to user
  4. The user encrypts the account, password, and data using the public key and sends the data to the server
  5. The server decrypts the account and password with the private key to verify the account and password

Object oriented programming

In the process of development, I only used procedural programming, the code is messy, too many functions, not easy to organize. Try object-oriented programming later in your refactoring or secondary development.