directory
introduce
rendering
Here’s the idea
Utility class code
1. All dependencies of utility classes
2. Check whether the folder exists and the file is not created
2.1 This method can be used separately. See file usage for aspects handling IO aspects
3. A private parameter
4. The constructor loads FileWriter
5. Write text
6. Close the IO
7. Call the main method
8. Full code
9. External crawler class is used to call, which is used to record the data parsed out in real-time update. This is the self-adjustment of crawler HTML parsing of crawler interface
9.1 Code Highlights
WriteFileTextUitl wfTU = null; Define an outer band to make it easier for other methods below to write
9.1.2 Close the IO at the end
9.1.3 Write content method use, this item can be added or not, in the tool class has been added to the newline \ here is to add two lines, you can decide
10. Log effect drawing
11. File write refresh as file data effect
11.1 This stuff sometimes shows up as 0KB, but it’s actually real data
introduce
Recently, WHEN I wrote a crawler, I found that sometimes, when there were too many things to climb, the crawler would die. The crawler would die after it had been resolved for a long time
For convenience, all data is saved. Write this custom log to the local directory
File service
rendering
Here’s the idea
1. Create a FileWriter file
2. Instantiate a utility class with FileWriter
3. Enter the address and name of the file to be created
Constructor, heap FileWriter new an object
5. Write write methods
6. Write the shutdown IO method
7. Record data To record the value of the monitored data strip, through the strip value record
8. If the service hangs or crawls, it has been monitored as crawls and normal data is not returned,
9. Change the IP address, shut down the service, change the initial location of the crawl value, and continue the crawl
Utility class code
1. All dependencies of utility classes
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import com.alibaba.druid.support.logging.Log;
import com.alibaba.druid.support.logging.LogFactory;
Copy the code
2. Check whether the folder exists and the file is not created
/** * If the file is empty create a file ** @author jianghy ** @param logAddress * @param logFileName File name ** / public static void createFile(String logaddress,String logFileName) { File file0 = new File(logaddress); // Check whether the address exists if (! file0.exists()) { file0.mkdir(); } File file1 = new File(logaddress+logFileName); // Check whether the file exists without adding if (! file1.exists()) { try { file1.createNewFile(); } catch (IOException e) {log.error(" create file error: "+ e.tostring ()); return; }}}Copy the code
2.1 This method can be used separately. See file usage for aspects handling IO aspects
public static void main(String[] args) { String logFileName = "zjwsj"+System.currentTimeMillis()+".log"; String logaddress = "D:/test/wexxxbrary/author/log/"; log.info("file address:"+logaddress+logFileName); / / 1 alone create file use WriteFileTextUitl. CreateFile (logaddress logFileName); }Copy the code
3. A private parameter
private FileWriter writer = null;
Copy the code
4. The constructor loads FileWriter
/** * public WriteFileTextUitl(String fileAddress,String FileName){ try { createFile(fileAddress,FileName); writer = new FileWriter(fileAddress+FileName, true); } catch (IOException e) {log.error(" object FileWriter created open file exception "+ e.tostring ()); return; }}Copy the code
5. Write text
/** * @param text */ public void writeText(String text){try {writer.write("\r"+text); writer.flush(); } catch (IOException e) {log.error(" object FileWriter write exception "+text+" "+ e.tostring ()); }}Copy the code
6. Close the IO
/** */ public void close(){try {writer.close(); } catch (IOException e) {log.error(" object FileWriter closed exception "+ e.tostring ()); }}Copy the code
7. Call the main method
public static void main(String[] args) { String logFileName = "zjwsj"+System.currentTimeMillis()+".log"; String logaddress = "D:/test/wenzhuxieCreateLibrary/author/log/"; log.info("file address:"+logaddress+logFileName); //2 Use WriteFileTextUitl wfTU = new WriteFileTextUitl(logAddress,logFileName); Wftu.writetext (" ha ha ha "); wftu.close(); // You must manually close IO}Copy the code
8. Full code
package com.superman.uitl;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import com.alibaba.druid.support.logging.Log;
import com.alibaba.druid.support.logging.LogFactory;
/**
* 写入文件文本 工具类<br/>
*
* 为方便爬虫工具进行写入日志<br/>
*
*
* @author jianghy
*
*/
public class WriteFileTextUitl {
private final static Log log = LogFactory.getLog(WriteFileTextUitl.class);
public static void main(String[] args) {
String logFileName = "zjwsj"+System.currentTimeMillis()+".log";
String logaddress = "D:/test/wenxxsexibrary/author/log/";
log.info("file address:"+logaddress+logFileName);
//1单独创建文件使用
//WriteFileTextUitl.createFile(logaddress,logFileName);
//2编辑日志使用
WriteFileTextUitl wftu = new WriteFileTextUitl(logaddress,logFileName);
wftu.writeText("哈哈哈");
wftu.close();//最后必须手动关闭io
}
private FileWriter writer = null;
/**
* 写入文件名和文件地址
*
* @param fileAddress
* @param FileName
*
* @author jianghy
*
*/
public WriteFileTextUitl(String fileAddress,String FileName){
try {
createFile(fileAddress,FileName);
writer = new FileWriter(fileAddress+FileName, true);
} catch (IOException e) {
log.error("对象 FileWriter 创建 打开文件异常 "+e.toString());
return;
}
}
/**
* 写入文本
*
* @param text
*/
public void writeText(String text){
try {
writer.write("\r"+text);
writer.flush();
} catch (IOException e) {
log.error("对象 FileWriter 写入异常 "+text+" "+e.toString());
}
}
/**
* 关闭io
*
*/
public void close(){
try {
writer.close();
} catch (IOException e) {
log.error("对象 FileWriter 关闭异常 "+e.toString());
}
}
/**
* 如果文件为空创建文件
*
* @author jianghy
*
* @param logaddress 地址
* @param logFileName 文件名称
*
*/
public static void createFile(String logaddress,String logFileName) {
File file0 = new File(logaddress);//判断地址是否存在
if (!file0.exists()) {
file0.mkdirs();
}
File file1 = new File(logaddress+logFileName);//判断文件是否存在，没有添加
if (!file1.exists()) {
try {
file1.createNewFile();
} catch (IOException e) {
log.error("创建文件异常："+e.toString());
return;
}
}
}
}
Copy the codeAn update has been made here
- Before is file0 mkdir ();
- To file0. The file.mkdirs (); This allows you to create multiple folders. The top one can only create one folder
9. External crawler class is used to call, which is used to record the data parsed out in real-time update. This is the self-adjustment of crawler HTML parsing of crawler interface
package com.superman.test.wenzhuxie; import java.io.File; import java.io.FileWriter; import java.io.IOException; import java.util.ArrayList; import java.util.Date; import java.util.List; import com.alibaba.druid.support.logging.Log; import com.alibaba.druid.support.logging.LogFactory; import com.alibaba.fastjson.JSONArray; import com.alibaba.fastjson.JSONObject; import com.superman.uitl.ExcelUitl; import com.superman.uitl.ExeclToJson; import com.superman.uitl.NetworkUitl; import com.superman.uitl.WriteFileTextUitl; /** @author Jianghy ** / public class authorAddInfo implements Runnable {private final static Log = LogFactory.getLog(wzxWebsiteAuthor.class); /** * climb XXX data ** first name ** a to z data ** h */ WriteFileTextUitl wfTU = null; /** * Perform a task * @throws IOException ** / public void task() {String logFileName = "zjwsj"+System.currentTimeMillis()+".log"; String logaddress = "D:/test/wenzhxxxary/author/log/"; log.info("file address:"+logaddress+logFileName); wftu = new WriteFileTextUitl(logaddress,logFileName); Long startTime = system.currentTimemillis (); / / / / to get start time, xx open file to the specific site query String fileUrl = "D: / test/wenzxxy/author/XXXX 1611888141102. XLSX"; JSONObject jo = ExeclToJson.excelToJson(fileUrl); // log.info(jo.toJSONString()); JSONArray ja0 = jo.getJSONArray("shell1"); log.info(ja0.size() + ""); List<String> restja = new ArrayList<String>(); for (int i = 138; i <1000; I ++) {getPageData(ja0.getJsonObject (I).getString(" for xx "), I, restja, 1); } // End of service long endTime1 = system.currentTimemillis (); / / System. Get over time out. Println (" -- -- -- -- -- -- -- -- -- programs _ _ persistence running time: "+ (endTime1 - startTime) +" ms "); } catch (Exception e1) {e1.printStackTrace(); return; }finally{ wftu.close(); Public void getPageData(String author,int authorId, List<String> restJa, int pageNum) { for (int i = 0; i < pageNum; i++) { String url = "http://ss.xx.com/" + "?key=" + NetworkUitl.getURLEncoderString(author) + "&status=0&ajaxdata=1&type=1&pagenum=" + (i + 1); System.out.println("request: "+ url); execute(author, authorId,restJa, url); Public void execute(String author,int authorId, List<String> restJa, String url) { try { String context = NetworkUitl.getHTMLContentV2(url); System.out.println("rest Context :" + context); if (context.equals("")) return; JSONObject jo = (JSONObject) JSONObject.parse(context); JSONArray list = jo.getJSONObject("data").getJSONArray("itemList"); for (int i = 0; i < list.size(); i++) { JSONObject joit = list.getJSONObject(i); String worksauthor = joit.getString("author"); // author xx String worksName = joit.getString(" itemName "); / / work xx String restStr = authorId +, __, "" + +", __, "the author + worksauthor +, __," "+ worksName; restJa.add(restStr); wftu.writeText("\r"+restStr); } } catch (Exception e) { //e.printStackTrace(); log.error(e.toString()); return; } finally { } } @Override public void run() { task(); } public static void main(String[] args) { new Thread(new authorAddInfo()).start(); }}Copy the code
9.1 Code Highlights
WriteFileTextUitl wfTU = null; Define an outer band to make it easier for other methods below to write
WriteFileTextUitl wftu = null;
Copy the code
9.1.2 Close the IO at the end
finally{
wftu.close();
}
Copy the code
9.1.3 Write content method use, this item can be added or not, in the tool class has been added to the newline \ here is to add two lines, you can decide
wftu.writeText("\r"+restStr);
Copy the code
10. Log effect drawing
It means to add \r it means to change lines and change lines, and that’s it
11. File write refresh as file data effect
11.1 This stuff sometimes shows up as 0KB, but it’s actually real data
The reason for this is that, after testing, the Windows refresh did not refresh
You can right click on the file and it will indicate the update progress and how many KB files have been inserted
ok
Continuously updated