This is the 21st day of my participation in the First Challenge 2022
Learn about cloud server development with my boss (if you are reading this series for the first time, it is highly recommended to learn the following articles) :
Pilot: Have a server, I unexpectedly so cool?
Alternative project: write a resume page in 10 lines of code!
How to apply for a free SSL certificate for a domain name
A pagoda in Linux, a real pagoda! Detailed tutorial
Finally, a website that everyone can visit
How to send alarm notifications to spikes using Python?
How to send tweets automatically in Python?
Finally, I “built” a job that could program anywhere, anytime
How to build a nail robot that can automatically reply
One, say something
Really want to have a fund monitoring robot, convenient view themselves on fund various index or, in time to stop or check, starting today, we first built LouJi, take you achieve a fund holding query robot, the main can query the date specified fund data and check the fund net value charts, slowly behind new features.
Two, start using your head
2.1 Environment Preparation
- Linux, Mac, Windows
- Python 3.7 and above
- Software packages: pandas, requests, Re, AkShare, Matplotlib, datafamee-image
2.2 Obtaining fund data of the specified date segment
Fund data can be obtained from some financial related websites, such as Titian Fund network, Sina fund network, etc., you can write a crawler to obtain website data, or you can use a ready-made tool kit to obtain data, such as a line of code to obtain stock and fund data, and draw akshare used in the K-line graph.
Here we introduce two methods simultaneously:
2.2.1 Review akshare’s acquisition of fund data
At present, Akshare does not support the acquisition of the fund net value data within the specified date range, but it can obtain the historical fund net value data at one time, call the function fund_em_open_fund_info to obtain the historical fund data, and then choose the time to analyze it according to the date.
import akshare as ak
fund_data = ak.fund_em_open_fund_info(fund='005827', indicator='Net Unit Value Trend')
print(fund_data)
Copy the code
Call the ready-made data interface yourself
In essence, Akshare also obtains data from some financial related websites. We can also write codes to obtain data by ourselves. Through the browser, we can quickly search the fund data interface, which is from The Daily Fund website of Oriental Wealth.
f'http://fund.eastmoney.com/f10/F10DataApi.aspx?type=lsjz&code={code}&page={page}&sdate={start_date}&edate={end_date}&per={per}'Code - Fund code Page - Fund data page number STARt_date - Data start date END_date - Data end date per - Maximum amount of data displayed per page40
Copy the code
According to the specified parameter, the browser will return the specified parameter, a JS assignment code, including fund data (Content), Total records (Records), total pages (Pages), current page (curpage).
The format is very neat, and we can extract the data directly through the re,
Get single page fund data
def get_html(code, start_date, end_date, page=1, per=40) :
url = f'http://fund.eastmoney.com/f10/F10DataApi.aspx?type=lsjz&code={code}&page={page}&sdate={start_date}&edate={end_date}&per={per}'
# print(url)
rsp = requests.get(url)
html = rsp.text
return html
Copy the code
Pandas’ read_html is used to parse the data in the table tag.
Parse out the data table part from the HTML and parse it into DF
def parses_table(html) :
Get fund data table
pattern = 'content:"<table(.*)</table>",'
table = re.search(pattern, html).group(1)
table = '<table' + table + '</table>'
fund_data = pd.read_html(table)[0]
return fund_data
Copy the codeAs mentioned above, the fund data interface returns a maximum of 40 entries per page, so to get all the data, we may need to traverse each page, so we also need to get the total number of pages through the re, and then traverse call get_html and parses_table to parse all the data.
Parse out the data table part from the HTML and parse it into DF
def parses_table(html) :
Get fund data table
pattern = 'content:"<table(.*)</table>",'
table = re.search(pattern, html).group(1)
table = '<table' + table + '</table>'
fund_data = pd.read_html(table)[0]
return fund_data
# Obtain the accumulated net value and other data within the specified date
def get_fund_data(code, start_date, end_date) :
first_page = get_html(code, start_date, end_date)
Get the total number of pages
pattern = 'pages:(.*),'
pages = re.search(pattern, first_page).group(1)
Convert to int data
try:
pages = int(pages)
except Exception as e:
r = F '{e}'
# print(r)
return r
# Store the fund data obtained on each page in dataframe format for later merging
fund_df_list = []
# Loop to facilitate all pages
for i in range(pages):
if i == 0:
fund_data = parses_table(first_page)
else:
page_html = get_html(code, start_date, end_date, page=i+1)
fund_data = parses_table(page_html)
fund_df_list.append(fund_data)
# Merge the data for each page
fund_df = pd.concat(fund_df_list)
return fund_df
Copy the code
The above two methods can obtain the net value data of the fund. Finally, I choose akshare to obtain the data. I set a scheduled task to update all the data of the fund I care about at 3 o ‘clock every day and store it locally.
- Regular task: every morning at 3 o ‘clock to obtain all concerned fund history data, stored locally
# Timed task: Get all concerned fund history data at 3am every day and store it locally
def get_all() :
try:
# Read the list of fund codes you care about from the file
with open('./FD/funds.txt') as f:
funds = [i.strip() for i in f.readlines()]
Update data one by one
for fund in funds:
fund_df = ak.fund_em_open_fund_info(fund, indicator='Net Unit Value Trend')
fund_df = fund_df.sort_values(by=['Net date'], ascending=False)
fund_df.to_csv(f"./FD/DATA/F{fund}_data.csv", index=False)
# print(f"./FD/DATA/F{fund}_data.csv")
time.sleep(random.randint(1.5))
return 'Fund data update completed'
except Exception as e:
r = F "[error message]{e}"
return r
Copy the code- Obtain the net value data of the specified date segment of the specified fund
Get the net value data of the specified date segment of the specified fund
def get_fund_data(fund, start_d, end_d) :
fund_df = pd.read_csv(f'./FD/DATA/{fund}_data.csv')
result_df = fund_df.query(f"'{start_d}<= net date <='{end_d}'")
return result_df
Copy the code2.3 Returned data presentation mode
For now, let’s keep it simple and set the rules as follows:
- 1) If the number of data is less than or equal to 30, the original data graph is returned
In this case, we use the third-party package Datafame-image. It is very simple to use. After PIP is installed, you can directly call export function to quickly convert datafrmae data into images.
# Transform the dTAFrame table into an image
def df_to_img(fund_df, fund, start_d, end_d) :
if fund_df.shape[0] < =1:
dfi.export(fund_df, f'./FD/IMG/{fund}_{start_d}_{end_d}_data.png')
return
Format the table to highlight the maximum and minimum values
fund_df = fund_df.style.highlight_max(subset=['Net unit value'], color='red')\
.highlight_min(subset=['Net unit value'], color='green')\
.format({'Daily growth rate': '{...} %'})
dfi.export(fund_df, f'./FD/IMG/{fund}_{start_d}_{end_d}_data.png')
Copy the codeTo make the picture data look better, we also used df.style to set the data table style (maximum net value per unit, minimum value highlighted, and percentage sign added daily growth).
- 2) If the number of data is greater than 30, the original data trend graph is returned
The original data trend chart is to visualize the data and then return it to the user. Here, we choose to draw the trend chart of the data and use Matplotlib to draw it.
# Draw the net value of the fund unit chart
def draw_fund_line(fund_df, fund, start_d, end_d) :
plt.rcParams['figure.figsize'] = (8.0.4.0) # set the figure_size size
plt.rcParams['savefig.dpi'] = 300 # Save image resolution
# Do not display the right and top borders
ax=plt.gca()
ax.spines['right'].set_color('none')
ax.spines['top'].set_color('none')
Set the grid
plt.grid(axis="y", color='gray')
# Calculate the maximum and minimum coordinates and label them in the graph
fund_max = fund_df.loc[fund_df['Net unit value'].idxmax()]
fund_min = fund_df.loc[fund_df['Net unit value'].idxmin()]
ax.annotate(f'({fund_max[0]}.{fund_max[1]}) ', xy=(fund_max[0], fund_max[1]), color='red')
ax.annotate(f'({fund_min[0]}.{fund_min[1]}) ', xy=(fund_min[0], fund_min[1]), color='green')
# drawing
plt.plot(fund_df['Net date'],fund_df['Net unit value'], color="c")
plt.title('Net Fund Unit Chart')
plt.xticks(rotation=30)
plt.xlabel('Net date')
plt.ylabel('Net unit value')
plt.savefig(f'./FD/IMG/{fund}_{start_d}_{end_d}_data.png')
Copy the codeThe line chart is used here, there are some Settings on the image style, such as: size, border, Max/min, but it is still not very beautiful, continue to optimize later.
- Complete call
# return data
def response_data(fund, start_d, end_d) :
Check whether the query result already exists
imgs = os.listdir('./FD/IMG/')
if f'{fund}_{start_d}_{end_d}_data.png' in imgs:
return f'./FD/IMG/{fund}_{start_d}_{end_d}_data.png'
# Fetch data
fund_df = get_fund_data(fund, start_d, end_d)
If the number of data is less than or equal to 30, return the original data graph
if fund_df.shape[0] < =30:
df_to_img(fund_df, fund, start_d, end_d)
else:
Otherwise, the data trend graph is returned
fund_df = fund_df.sort_values(by=['Net date'])
draw_fund_line(fund_df, fund, start_d, end_d)
return f'./FD/IMG/{fund}_{start_d}_{end_d}_data.png'
Copy the code2.4 Set daemon program for docking nail robot
Currently, the project uses two kinds of robots introduced before: Spike swarm robot and enterprise robot. The configuration method and code can be seen in the previous article: How to send alarm notification to Spike using Python? And how to build a nail robot that can respond automatically, very detailed.
Nail group robot is mainly used to report the daily automatic report fund data update, the back can also add the fund up and down detection situation.
Enterprise robot is mainly used for automatic reply function of fund data query, and can also expand the initiative to send messages to users, which will be studied in the future.
2.5 Problems encountered and solutions
2.5.1 Error Converting datafrmae-image on Linux
At first I was told there was no Chrom, and then I followed the online tutorial to install Google Chrom.
Reference: segmentfault.com/a/119000002…
SyntaxError: Not a PNG file.
The error message is “Pillow” and “Matplotlib”, the same version as the local version will not work.
Finally looked at the source, found that can convert methods in addition to using Chrom, can also use matplotlib, after modification, can indeed normally generate pictures, but no format!!
Mysql > select table_conversion from table_conversion to table_conversion. Mysql > select table_conversion from table_conversion to table_conversion.
[0209/162730.350381: ERROR: zygote_host_impl_linux. Cc (90)] Running as root without - no - the sandbox is not supported. See https://crbug.com/638180Copy the codeUnder root, create a new user od, grant the /root directory permission to it, and then run su to switch to the new user.
useradd od
chown -R od /root
su od
Copy the codeRunning it again does solve image generation and data format problems, but there is a new problem: table header Chinese does not display…
Finally, I suddenly thought that I could do it locally, and the versions of the two packages were the same. It should not be a code problem. Is it because there is no Chinese font installed in Linux?
As the root user, create a directory to store Chinese fonts. After creating the directory, you can upload the local SimHei fonts to the corresponding directory using the pagoda.
mkdir -p /usr/share/fonts/my_fonts
Copy the codeThe following command can be used to check whether the Chinese font is successfully installed.
fc-list :lang=zh
Copy the code
Run the code again, the generated image is normal ~ happy!
2.5.2 Chinese display problem of Matplotlib pictures
Permanent solution to Matplotlib
2.5.3 The nail robot cannot directly transmit pictures
The Nail bot currently only supports transports: normal text, Markdown text, connections, actionCard messages, and feedCard message types.
If I want to send the generated fund data graph to the user, the best way is to convert the image into a link as before, and then transfer it through markdown.
If the system is only for personal use and the amount of data is not large, we don’t need to choose existing graph bed tools on the network (so we need to write interface docking code), we can directly open an HTTP port to share our pictures. Enterprise robots themselves use flask, so we can implement this function more easily.
app = Flask(__name__, static_folder='xxx/FD/IMG', static_url_path='/static')
Copy the codeFor example, when you initialize the flask app, specify the directory where the static files are stored and the route suffix of the static files. In this way, you can access the corresponding image through http:// server IP address: port number /static/ image file name.
The image link is then embedded in Markdown and returned to the user as normal.
2.6 Final rendering
- Specify the query
View the net value of a fund within a period of time. (Within 30 pieces of data, table display; > 30, trend chart)
Query format: F Fund Code Start Date End Date, for example, F005827 2021-12-03 2022-02-10
- Common query
View the net value and daily growth rate of a fund in the past 10 days + trend chart
Third, the last words
This project is big or small, with 100 lines of code. The native testing is still smooth, mainly after the migration to Linux, there are some problems, from the initial Python version problems (a 3.7.9 installed) to datafrmae-image problems. Extensions for Linux include installing Google, setting up new users, assigning permissions, and learning source code testing.
It took me a long time to meet problems and solve them, which once made me very upset. However, this process also made me feel very beneficial. It was a process of continuous accumulation, continuous practice and continuous consolidation.
At present, the fund monitoring robot is still relatively simple, and there is even no monitoring function (currently only support data query and update), but this building is very stable and deep, and it will be simple and convenient to add other functions later. Welcome to leave a message in the comment section, and say what functions you want to add for this robot.
The code will be shared when the next version of the article is released, and you are welcome to comment on your feedback and requirements.
Persistence and hard work: results.
Like to see the message forwarding, four support, the original is not easy. Ok, see you next time, I love the cat love technology, more love si si’s old cousin Da Mian ଘ(˙꒳˙)ଓ Di Di