Hello, everyone. I’m a fisherman from Go School. Today I will show you how to implement a local memory caches based approach to improve system performance in the Go project in the scenario of small data volume and frequent reads.
We are all familiar with caching. The definition of Baidu Encyclopedia is as follows:
The cache is the memory that can exchange data at high speed. It exchanges data with the CPU before the memory, so the speed is very fast.
Thus, caching is used to speed up data exchange. The cache we are going to talk about today is not the cache in the CPU, but the cache in the application for the database. The application reads data from the cache before the database to reduce the strain on the database and improve the read performance of the application.
In real projects, I believe you have encountered similar situations: small amount of data, but frequently accessed (such as national standard administrative region data), and want to store it completely in local memory. This avoids direct access to mysql or Redis, reduces network traffic, and improves access speed. So how do you do that?
This article describes a method often used in Go projects: load data from the database into a local file, and then load the data from the file into memory, where the data is directly available to the application. As shown below:
This article will ignore the database-to-local file process because it is a process of uploading and downloading files locally. So we’ll focus on loading data from a local file into memory.
01 target
In a Go language project, data from a local file is loaded into the application’s memory for direct use by the application.
Let’s break the goals down into two goals:
1. When the program is started, the data of the local file is initialized to the memory, that is, cold start
2. During the running of the program, when the local file is updated, the data will be updated to the memory.
02 Code implementation
The main purpose of this article is to explain the implementation of the goal, so it will not take you step by step analysis, but through explaining the implementation of the code to provide you a reference implementation.
Therefore, we first give the class diagram we designed:
As you can see from the class diagram, there are two main structures: FileDoubleBuffer and LocalFileLoader. Let’s go through the properties and method implementations of the two structures one by one.
2.1 Scenario Assumptions
We take the weather situation of the city as an example, and store the real-time temperature and wind force of each city in JSON format in a file. When the temperature or wind force of the city changes, the file will be updated. As follows:
{
"beijing": {
"temperature": 23."wind": 3
},
"tianjin": {
"temperature": 20."wind": 2
},
"shanghai": {
"temperature": 20."wind": 20
},
"chongqing": {
"temperature": 30."wind": 10}}Copy the code2.2 Call to Main
Here, first give the call example of main function, according to the implementation of main function, we see the realization of the two main structures in the figure step by step, the code is as follows:
// The first step is to define the structure to load the data in the file
type WeatherContainer struct {
Weathers map[string]*Weather // Real weather for each city
}
// The weather condition of each city in the file data
type Weather struct {
Temperature int // Current temperature 'json:"temperature"'
Wind int // Current wind 'json:"wind"'
}
func main(a) {
pwd, _ := os.Getwd()
// The path to the loaded file
filename := pwd + "/cache/cache.json"
// Initialize the local file loader
localFileLoader := NewLocalFileLoader(filename)
// Initialize the file buffer instance with localFileLoader as the underlying file buffer
fileDoubleBuffer := NewFileDoubleBuffer(localFileLoader)
// Start loading the contents of the file into buffer variables, essentially loading the file data using load and reload
fileDoubleBuffer.StartFileBuffer()
// Get data
weathersConfig := fileDoubleBuffer.Data().(*WeatherContainer)
fmt.Println("weathers:", weathersConfig.Weathers["beijing"])
blockCh := make(chan int)
// This channel is used to block the process so that the Reload coroutine can execute
<-blockCh
}
Copy the code2.3 FileDoubleBuffer structure and its implementation
This structure is primarily application-oriented (in our case, the main function), allowing applications to fetch data directly from memory, i.e., bufferData. The structure is defined as follows:
// The main application mainly obtains data from this structure
type FileDoubleBuffer struct {
Loader *LocalFileLoader
bufferData []interface{}
curIndex int32
mutex sync.Mutex
}
Copy the codeFirst look at the properties of the structure:
**Loader: ** is a LocalFileLoader type (defined later) used to load data from a specific file into bufferData.
**bufferData slice: ** A variable that receives data from a file. On the one hand, the data from the file is loaded into the variable. On the other hand, the application gets the desired data information directly from this variable, rather than from a file or database. The data type of this variable is interface{}, indicating that any type of data structure can be loaded. Also, note that this variable is a slice that has only 2 elements, both of which have the same data structure and are used in conjunction with the curIndex attribute.
**curIndex: ** This property specifies which index is being used by bufferData. The value of this property cycles between 0 and 1 for switching between old and new data. For example, the current data of the index element curIndex=1 is used externally. When new data is stored in the file, the file is loaded to index 0 first. After the file is fully loaded, the value of curIndex is pointed to 0. In this way, when there is new data in the file to refresh the data in memory, the application does not affect the use of old data.
Look again at the functions in FileDoubleBuffer:
Data () function
The application uses this function to get the dataBuffer data in the FileDoubleBuffer. The concrete implementation is as follows:
func (buffer *FileDoubleBuffer) Data(a) interface{} {
// bufferData actually stores two elements of the same structure for switching between old and new data
index := atomic.LoadInt32(&buffer.curIndex)
return buffer.bufferData[index]
}
Copy the codeThe load function
This function is used to load data from a file into bufferData. The code implementation is as follows:
func (buffer *FileDoubleBuffer) load(a) {
buffer.mutex.Lock()
defer buffer.mutex.Unlock()
// Determine which element of the bufferData array is currently in use
// Since bufferData has only two elements, it is either 0 or 1
curIndex := 1 - atomic.LoadInt32(&buffer.curIndex)
err := buffer.Loader.Load(buffer.bufferData[curIndex])
if err == nil {
atomic.StoreInt32(&buffer.curIndex, curIndex)
}
}
Copy the codeReload function
Used to load new data from a file into bufferData. This is actually a for loop that executes the load function at regular intervals as follows:
func (buffer *FileDoubleBuffer) reload(a) {
for {
time.Sleep(time.Duration(5) * time.Second)
fmt.Println("Start loading...")
buffer.load()
}
}
Copy the codeStartFileBuffer function
This function starts loading and updating data as follows:
func (buffer *FileDoubleBuffer) StartFileBuffer(a) {
buffer.load()
go buffer.reload()
}
Copy the code**NewFileDoubleBuffer(loader LocalFileLoader) FileDoubleBuffer function
This function initializes the FileDoubleBuffer instance as follows:
func NewFileDoubleBuffer(loader *LocalFileLoader) *FileDoubleBuffer {
buffer := &FileDoubleBuffer{
Loader: loader,
curIndex: 0,}// This allocates memory space so that the value from the file can be loaded into the variable for use by the program
buffer.bufferData = append(buffer.bufferData, loader.Alloc(), loader.Alloc())
return buffer
}
Copy the codeThe LocalFileLoader structure is created because we first load data from the database into a local file, and then load the file’s data into the memory buffer. The purpose of this structure is to perform specific file data loading and file update detection tasks. LocalFileLoader is defined as follows:
type LocalFileLoader struct {
filename string // The file to load, the full path
lastModifyTime int64 // Time when the file was last modified
}
Copy the codeLet’s start with the properties of the structure:
**filename: ** specifies a filename to load data from
**modifyTime: ** Time when the file was last loaded. If the file update time is longer than this time, the file is updated
Look again at the function in LocalFileLoader:
Load (filename string, I interface) function
This function is used to load data from the filename file into variable I. The variable I is actually an element of bufferData passed in from FileDoubleBuffer as follows:
// Where the I variable is actually an element in dataBuffer passed in from the load method of the FileDoubleBuffer structure
func (loader *LocalFileLoader) Load(i interface{}) error {
// The WeatherContainer structure is defined according to the data stored in the file, which will be covered later
weatherContainer := i.(*WeatherContainer)
fileHandler, _ := os.Open(loader.filename)
defer fileHandler.Close()
body, _ := ioutil.ReadAll(fileHandler)
_ := json.Unmarshal(body, &weatherContainer.Weathers)
// We omit those err judgments here
return nil
}
Copy the codeDetectNewFile () function
This function is used to detect whether the filename file has been updated. If the file has been modified for a time greater than modifyTime, FileDoubleBuffer loads the new data into dataBuffer. The code is as follows:
// This function checks if the file is updated and returns true if it is, false otherwise
func (loader *LocalFileLoader) DetectNewFile(a) bool {
fileInfo, _ := os.Stat(loader.filename)
// If the file modification time is longer than the last modification time, the file is updated
if fileInfo.ModTime().Unix() > loader.lastModifyTime {
loader.lastModifyTime = fileInfo.ModTime().Unix()
return true
}
return false
}
Copy the code**Alloc() interface{} **
Used to assign specific variables for loading data in a file. The variables allocated here are eventually stored in dataBuffer data in FileDoubleBuffer. The code is as follows:
// Assign specific variables to carry the specific contents of the file. The variable structure needs to be consistent with the structure in the file
func (loader *LocalFileLoader) Alloc(a) interface{} {
return &WeatherContainer{
Weathers: make(map[string]*Weather),
}
}
Copy the codeWe also need a function to initialize the LocalFileLoader instance:
// Specify the file path to be loaded
func NewLocalFileLoader(path string) *LocalFileLoader {
return &LocalFileLoader{
filename: path,
}
}
Copy the codeconclusion
This mode is applicable to scenarios where the data volume is small and the data is frequently read. As you can see in the figure at the beginning of this article, because servers tend to be clusters, the file contents on each machine may vary briefly, so this implementation is also not suitable for scenarios where data is strongly consistent.