1. The scene

Some friends often ask me in the communication group, how do I get the morning news every day?

In fact, the scheme used in the early stage was to use crawler to get the headlines of some news websites, then do some simple data cleaning, and finally send them to the designated community by Itchat.

Due to the instability of crawler facing website revision and the insecurity of ItChat, this scheme was abandoned

In the later stage, I changed a scheme, by modifying crawler scheme, creating API service, writing App to obtain data, and then manually sending it to wechat group

This article will talk about the implementation process in detail

2. Data crawl

Step 1: Get the data through the Python crawler

There are two ways to obtain it, namely, the existing news website and the headlines of the news website

Select a crawler, you can quickly crawl to the target site data

Step 2: Data cleaning and sorting

The data will be firstly filtered by keywords, and then sorted by hot spots according to the number of likes or reads

Step 3: Parameterize and de-weight

Remove duplicate news stories from the data, and then parameterize the number of final displays

# Based on the number of likes, Sorted (news, key=itemgetter('news_approve_num'), reverse=True) result = [] for news_sorted_item in news_sorted_pro: Result. Append (news_sorted_item.get('title')) # delete result = sorted(set(result), Result =result [:self.news_num] if len(result) >= self.news_num else resultCopy the code

It should be noted that, because news websites often change version, it is recommended to climb multiple news websites, do exception handling, do a priority, if a website fails to climb data, switch to the next level of website to climb data

3. As a service

The purpose of servitization of data, that is, writing apis, is to facilitate terminal invocation

If you write apis in Python, it is recommended to use the FastAPI or Flask framework because these two frameworks are quick and easy to develop apis. Take FastAPI as an example:

Step 1, install dependencies

Includes the FastAPI framework and hyperCorn dependencies, which are standalone ASGI servers that facilitate FastAPI project deployment

Pip3 Install HypercornCopy the code

Step 2, write the API

Using FastAPI is convenient, and you can write an interface service in less than 10 lines of code

Simply instantiate the FastAPI object, specify the request method and path using the decorator, and call the crawler method above.

@app. Get ("/last_news") def get_last_news(): """ news = get_news() data = {'code': 0, 'news': news} # Encapsulate return dataCopy the code

To debug the API locally, run the project through the uvicorn command

# 运行项目
uvicorn news:app --reload
Copy the code

Then visit the address below to see the news data returned

http://127.0.0.1:8000/last_news

Step 3, generate the dependency migration file

Using the PIP freeze command will generate the dependency migration file locally

PIP freeze > requirements.txtCopy the code

Step 4: Upload the code

Upload code to code hosting platforms such as Code Cloud, GitLab, etc

Step 5, server pull code

Git pulls the code from the server and installs all dependencies with one click through the dependencies file

Pip3 install -r requirements.txtCopy the code

Step 6, run the service

Run the FastAPI project with HyperCorn to keep the process running in the background and save run log information

# nohup Hypercorn news:app --bind 0.0.0.0:8000 > /news.log 2> &1& # nohup Hypercorn news:app --bind 0.0.0.0:8000 > /news.log 2> &1&Copy the code

Note that the port number bound to the project must be enabled in the firewall and cloud server security groups

Of course, if you write apis in Java, it is recommended to use Spring Boot, which can quickly develop a Restful API service

4. Write the App

After completing the API service, the next step is to write an App in the terminal to access the API, get the data and display it

Take writing an Android app for example

First, we place a text display box and a button control on the interface

Then, the button control is set to listen for click events

get_news_btn = findViewById(R.id.get_news_btn); // Get_news_btn.setonClickListener (this); @overridePublic void onClick(View v) {switch (v.getid ()) {case r.id.get_news_bTN: news_et.settext ("... ); getNewsMet(); break; }}Copy the code

Next, the API is called to retrieve the data using Android’s OkHttp framework for web requests

/*** * get news */ private void getNewsMet() {OkHttpClient OkHttpClient = new OkHttpClient(); Request = new request.builder ().url(url).get().build(); // Call call = okHttpClient.newCall(request); okHttpClient.newCall(request).enqueue(new Callback() { @Override public void onFailure(Call call, IOException e) {log.d ("xag", "failed "); showResult(false, ""); } @Override public void onResponse(Call call, final Response response) throws IOException { Log.d("xag", "Get success ") parseJsonWithJsonObject(response); }}); }Copy the code

Finally, the news data is displayed in a text control and copied to the system clipboard

Private void copyToClip(String Content) {// Get clipboard manager:  ClipboardManager cm = (ClipboardManager) getSystemService(CLIPBOARD_SERVICE); MClipData = ClipData. NewPlainText ("Label", content); // Put the ClipData contents into the system clipboard. if (null ! = cm) { cm.setPrimaryClip(mClipData); }}Copy the code

If you use Xcode to write iOS applications on only iOS devices, you are advised to upgrade Xcode to 11.0 or higher

When creating a project, it is recommended to use SwiftUI to build the UI and then use CocoaPods to add the Alamofire Network request dependency library. Other steps are similar to those on Android

# Uncomment the next line to define a global platform for your project
source 'https://github.com/CocoaPods/Specs.git'

platform :ios, '9.0'

use_frameworks!

target 'news_eve' do  
# Comment the next line if you don't want to use dynamic frameworks  
pod 'Alamofire'  pod 'SwiftyJSON' 
pod 'HandyJSON'  
# Pods for news_eve
end
Copy the code

5. The last

You only need to click the button in the App to copy the morning news to the system clipboard, and then it can be forwarded to multiple wechat groups

Of course, SoloPi or barrier-free can also be used as the last step to forward the content with one click through automated operation

If you think the article is good, please like it and share it. Your affirmation is my biggest encouragement and support.

The article reprinted