In this article, you and Quinn will get a quick start on flask development and the use of mongodb to build a stable and robust distributed multi-machine server resource monitoring platform
In the previous article, The Python System Resource Information Retrieval Tool, Have you used it? In Python, you learned how to use the third-party library psutil in Python to get information about your system’s resources, such as CPU usage, memory usage, hard disk, and process status. And At the end of his article, Quinn made a big deal about how psutil could be used to build a distributed server monitoring platform
Dreus: Hum! What a big talk!!
Let’s take a look at what Quinn said at the end of his last post about distributed server monitoring platforms:
Delevin: Why did the people at Demacia build such a surveillance platform? Could there be some kind of conspiracy?
The story before the project began
Quinn now has dozens of Linux servers that host many crawlers and services that crawlers rely on, and the De Marcia camp wants Quinn to save as much money as possible, given the military’s tight budget since the last war against Norknoxus. Quinn was brought in to optimize the crawler project.
However, before optimization, the resource peak value and the corresponding number of crawler processes on the server must be recorded to estimate the correlation between the number of crawlers and resource consumption. There are plenty of powerful server monitoring platforms out there, so why would Quinn build one instead of building one?
The requirements of this project is not big, also not complex, and customization degree is higher, if you use a mature server monitoring platform is also too waste, and requires frequent recording server peak, I’m afraid I have to change the code – quinn considering these problems, in the heart feel better rewrite to true, after all, only the wheels, can progress faster.
What kind of wheel
The project figure above is the hypothesis that the requirements under the condition of production, but now have real demand, it should be to adjust the figure above and elaboration, thought of here, quinn frowned: if HuaLuo here, there is at least a negotiable, but HuaLuo was sent to perform a task, quinn had to own fencing:
First of all, this module does the misnamed task of getting server resource information, including the current CPU, memory, and disk usage, as well as a list of processes related to the current Python (since crawlers are written in Python, monitoring Python process information). Should be changed to system resource information capture module.
Then in the storage center, Redis doesn’t seem to be a good fit, and if noSQL is going to be used, mongodb might be a better fit. There are also distributed issues to consider. Quinn was well aware that distribution was simply the use of a central storage where data from different ends was stored together in a data center, and this led to three scenarios:
- A). Peer-to-peer distribution — the code on each server is the same and the functionality is the same
- B). Master-slave distributed push mode — multiple slave machines push data to the host, and the host is responsible for storage and other operations.
- C). Master-slave distributed pull mode — the host takes the initiative to request data from the slave machine, which is vividly understood as pull.
A vivid image flashed through Quinn’s mind:
Summoner’s Canyon
Three wolves
Select push mode and pull mode
Push and pull, seemingly in the same direction, require different operations (can you say red Buff and blue Buff are the same?). .
The first thing to consider is the environment where the host resides. If the host is not in the public network, but the slave servers are in the public network (for example, the Ari cloud server), the host can access the slave server, but the slave server cannot access the host. The push mode is not applicable, and vice versa.
Quinn consulted with Canyon Vanguard and learned that the servers in demacia’s camp currently have hosts that are not on the Internet, but slaves that are on the Internet, so pull mode is a good fit. As a rule of thumb, the pull pattern in code is usually represented by the host request.get.
Therefore, the storage center only needs to be provided to the host, and the slave machine does not need to access the storage center, so the figure can be changed to:
Warning center
Quinn is a good fighter, and even the importance of an alarm, how could a project this big not have an alarm feature?
Tyrone: It doesn’t matter how careful you are!
Based on our experience against Norknoxus, their people are realistic and aggressive and stupid, so the alarm center needs to be both prompt and accurate, so the alarm center is divided into two parts: detection and notification. That is, threshold detection and mail delivery
Data processing and visualization
In order to convince people, they have to show real evidence, so the data processing and visualization are essential, but since Warlow is out on a mission, Quinn has no clue about the presentation of the data, so let’s leave it as it is.
Therefore, the structure of the server information monitoring platform is as follows:
guns
Now that it’s all planned out, it’s almost ready to start. Quinn is good with tools, he has the most powerful crossbow in the league and knows the importance of tool choice.
Since it’s a platform, it needs WEB services to support it. Looking at Python, the most popular ones are Django and Flask. Since the project is small, lightweight and fast, Quinn chose Flask restful click to documentation, which he has never used before.
For the data storage center, select Mongodb, but for quick development, select an intermediate tool that will connect Flask and Mongodb together. There are several options available:
- A).pymongo — Python connection library for mongodb
- B).Flask-pymonggo — A layer is encapsulated on the basis of A
- C).MongoEngine — a mature Mongodb ORM framework
All three will do the job, and Warlow told Quinn that ORM was the best choice for speed, code structural integrity, ease of use, and maintainability, so go to MongoEngine and click on documentation
Flask restful and MongoEngine were tools quinn had rarely used before, but he trusted Warlow’s recommendations and his ability to learn quickly
The most important ability
Quinn, who had fought hundreds of battles, big and small, against the mighty Nokthas and the monsters that haunt summoner’s Canyon, knew that you can never learn everything, but if you learn how to learn fast, you can be strong.
start
After the above analysis and planning, “The project seems to be clear in the chest and ethereal in the canyon”
Quinn: Do it!
Tool installation
If you really start at zero, then all learning can start with official documentation. Official documents are certainly a more reliable alternative to the rumors that circulate in the canyons and mists (BY which I mean the dead-end articles plagiarized online), just as Quinn chose to fight alongside Warlow.
According to Flask restful documentation
MongoEngine
Mongodb
Mongodb visualization tool Robt 3T
Installation introduction
Quinn had them installed quickly because of official documentation
Run the Flask
Flask restful documentation in the Quick Start section gives a simple sample code that you can use to start Flask. Quinn never did, of course, because you can’t learn quickly by simply copying and pasting a document, you have to make the document your own immediately:
from flask import Flask
from flask_restful import Resource, Api
app = Flask(__name__)
api = Api(app)
class Home(Resource):
def get(self):
return {'status': 'success'.'message': 'Flask restful is running'}
api.add_resource(Home, '/')
if __name__ == '__main__':
Cui Qingcai | Jingmi, Wei Shidong | Quinn invites you to follow the wechat official number
app.run(debug=True)
Copy the codeThen we run the code
Flask has started, and you can click on the link to go to your browser and see what the output is:
As written in the code, the output is correct, indicating that appropriate changes are possible, not afraid.
The Flask and MongoEngine
MongoEngine document Connecting to MongoDB
from flask import Flask
from flask_restful import Resource, Api
from mongoengine import *
connect(
host='mongodb://localhost/server_resource'
)
app = Flask(__name__)
api = Api(app)
class Home(Resource):
def get(self):
return {'status': 'success'.'message': 'Flask restful is running'}
api.add_resource(Home, '/')
if __name__ == '__main__':
Cui Qingcai | Jingmi, Wei Shidong | Quinn invites you to follow the wechat official number
app.run(debug=True)
Copy the codeThe data model
Data models make it easy to manipulate databases and keep code clean and aesthetically pleasing — just as soldiers like to polish their weapons, as Quinn often does.
The document clearly describes how to build the data model, and uses some examples to show how field Settings and definitions should be selected.
Now, what data models do we need to define?
First of all, going back to the push-pull mode, we’ve decided to use pull mode (not Dragon Turtle Ramos), so that means the host pulls data from the slave machine, how does the host determine in the data record which slave machine the data comes from?
Therefore, we need a server queue, which records the name, IP address, and remarks of the slave machine, so that the data model can be defined.
Add a new data model class to the code:
class ResourceModel(Document):
"""Server queue model"""
name = StringField(max_length=255, required=True)
address = StringField(max_length=160, required=True)
description = StringField(max_length=255)
Copy the codeAfter running the code, you can open the Mongodb visualization tool and view the newly generated Collection.
summary
In addition to writing code, requirements analysis, planning, and technical validation are also very important during the development of a project. That’s all for today, because Quinn is out on a mission
Next time we will perfect this distributed multi – machine monitoring platform