preface

Some time ago in the group discussion, want to achieve a file timing upload server to how to achieve. I remember a project I worked on before: writing FOTA function for Iot module of Qualcomm: Implementation module can download upgrade image package remotely to realize version upgrade function. And then used a super powerful tool called cURL. Whim, decided to write a special article, to the need of friends.

The article is divided into two parts

To cURL:
To achieve a regular upload log file a simple procedure.

concept

CURL is a common command-line tool used to request Web servers. Its name stands for client’s URL utility.

CURL was originally written by Daniel Stenberg (currently the core developer of cURL) and is a senior member of the IETF HTTPbis Working Group. Daniel founded the Curl project in 1998. He wrote the original version of Curl and created the LibCurl library. More than half of the 24,000 commits included in the code repository so far have been made by Daniel himself, who remains the core developer of the project. Daniel says he considers Curl his own child.

As a powerful tool, The supported protocols include DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, TELNET and TFTP), also supports POST, cookies, authentication, download some files from the specified offset and other functions, with user agent string, speed limit, file size, progress bar, cookie support, user authentication, breakpoint continuation and other features.

1. Command installation

sudo apt-get install curl
Copy the code

CURL cURL

curl [options] [URL...]
Copy the code

3. URL format

Refer to RFC 1808 for the URL format definition.

Address: http://www.w3.org/Addressing/rfc1808.txt the Relative Uniform Resource Locators"Copy the code

The URL consists of the resource type, host domain name for storing the resource, and resource file name. It can also be regarded as consisting of four parts: protocol, host, port, and path The general syntax format of URL is as follows:

protocol :// hostname[:port] / path / [;parameters][?query]#fragment
Copy the code

(Square brackets [] are optional).

Protocol

Specify the transport protocol to use. The following table lists the valid scheme names for the protocol attribute. The most commonly used protocol is HTTP, which is also the most widely used protocol on the WWW.

File resources are files on the local computer. The format is file:///, followed by three slashes.
FTP Access resources through FTP. Format: FTP://
Gopher accesses the resource through gopher.
HTTP Access to the resource through HTTP. Format: HTTP://
HTTPS Access the resource using secure HTTPS. Format: HTTPS://
Mailto resources are E-mail addresses accessed through SMTP. Format mailto:
MMS plays this resource by supporting the MMS (streaming media) protocol. (Software: Windows Media Player) format: MMS://
Ed2k accesses this resource through P2P software that supports the ED2K (Dedicated Download Link) protocol. (software: ed2k://) format
Flashget This resource is accessed through P2P software that supports the Flashget (Dedicated Download Link) protocol. (represents software: express) format Flashget://
Thunder Access this resource through P2P software that supports the Thunder (Dedicated Download Link) protocol. (stands for software: thunderbolt) format thunder://
News Accesses the resource through NNTP.

Hostname (hostname)

It is the domain name System (DNS) host name or IP address of the server where resources are stored. Sometimes, the hostname can also be preceded by the username and password needed to connect to the server (format: username:password@hostname).

Port (Port number)

This parameter is optional. If omitted, use the default port number of the solution. All transport protocols have default port numbers. If omitted, the default port number is used. Sometimes, for security or other reasons, a port can be redefined on the server to use a non-standard port number. In this case, the port number cannot be omitted from the URL.

Path (path)

A string separated by zero or more slashes (/) to indicate the address of a directory or file on a host.

Parameters:

This is optional for specifying special parameters.

The query (query)

This parameter is optional. It is used to pass parameters to dynamic web pages (such as web pages made using CGI, ISAPI, PHP/JSP/ASP/ASP.NET, etc.). Multiple parameters can be separated by ampersand (&), and the name and value of each parameter are separated by =.

Fragment (information)

A string that specifies a fragment in a network resource. For example, if there are multiple definitions in a web page, you can use fragment to locate a specific definition.

Curl command parameter description

The Linux curl command has a large number of parameters. The following table lists only some common parameters. For more parameters, run the “man curl” command.

parameter	function
-a/–append	When uploading a file, attach it to the destination file
-A/–user-agent	Set up the user agent to send to the server
-anyauth	You can use “any” authentication method
-b/–cookie <name=string/file>	Cookie string or file read location
–basic	Use HTTP basic authentication
-B/–use-ascii	Use ASCII/text transfer
-c/–cookie-jar	The cookie is written to the file when the operation is complete
-C/–continue-at	Breakpoint continued to turn
-d/–data	Data is transmitted in HTTP POST mode
–data-ascii	Post data in ASCII mode
–data-binary	Post data in binary mode
–negotiate	Use HTTP authentication
–digest	Use digital authentication
–disable-eprt	Do not use EPRT or LPRT
–disable-epsv	Do not use the EPSV
-D/–dump-header	Write the header information to the file
–egd-file	Set EGD socket path for random data (SSL)
–tcp-nodelay	Use the TCP_NODELAY option
-e/–referer	Source url
-E/–cert <cert[:passwd]>	Client certificate file and password (SSL)
–cert-type	Certificate File Type (DER/PEM/ENG) (SSL)
–key	Private key File name (SSL)
–key-type	Private key File Type (DER/PEM/ENG) (SSL)
–pass	Private Key Password (SSL)
–engine	The encryption engine uses SSL. “–engine list” for list
–cacert	The CA certificate (SSL)
–capath	CA directory (made using C_rehash) to verify peer against (SSL)
–ciphers	SSL password
–compressed	Using deflate or gzip to return a compressed situation
–connect-timeout	Set the maximum request time
–create-dirs	Create a directory hierarchy for local directories
–crlf	Uploading is the transformation of LF into CRLF
-f/–fail	Does not display HTTP errors when connection fails
–ftp-create-dirs	If the remote directory does not exist, create a remote directory
–ftp-method [multicwd/nocwd/singlecwd]	Control the use of CWD
–ftp-pasv	Use PASV/EPSV instead of ports
–ftp-skip-pasv-ip	When using PASV, ignore the IP address
–ftp-ssl	Try using SSL/TLS for FTP data transfer
–ftp-ssl-reqd	SSL/TLS is required for FTP data transfer
-F/–form <name=content>	Simulate HTTP form submission data
–form-string <name=string>	Simulate HTTP form submission data
-g/–globoff	Disable url sequences and ranges using {} and []
-G/–get	Send data as get
-H/–header	Custom header information is passed to the server
–ignore-content-length	Length of HTTP header ignored
-i/–include	The output includes the protocol header
-I/–head	Only request header information is displayed
-j/–junk-session-cookies	Ignore session cookies when reading files
–interface	Use the specified network interface/address
–krb4	Use KRB4 at the specified security level
-k/–insecure	Allows you to use no certificates to SSL sites
-K/–config	The specified configuration file is read
-l/–list-only	List the file names in the FTP directory
–limit-rate	Set the transmission speed
–local-port	Enforce the use of local port numbers
-m/–max-time	Set the maximum transmission time
–max-redirs	Set the maximum number of directories that can be read
–max-filesize	Set the maximum number of files that can be downloaded
-M/–manual	Display full manual
-n/–netrc	Read the username and password from the NEtrC file
–netrc-optional	Override -n with.netrc or URL
–ntlm	Use HTTP NTLM authentication
-N/–no-buffer	Disabling buffered output
-o/–output	Write the output to the file
-O/–remote-name	Write the output to the file, keeping the name of the remote file
-p/–proxytunnel	Using HTTP proxies
–proxy-anyauth	Select any proxy authentication method
–proxy-basic	Use basic authentication on the proxy
–proxy-digest	Use digital authentication on the proxy
–proxy-ntlm	Use NTLM authentication on the proxy
-P/–ftp-port	Use port addresses instead of PASV
-q	As the first argument, close.curlrc
-Q/–quote	Before file transfer, send commands to the server
-r/–range	Retrieves byte ranges from HTTP/1.1 or FTP servers
–range-file	Read (SSL) random files
-R/–remote-time	Retain the remote file time when a file is generated locally
–retry	Number of retries when a transmission problem occurs
–retry-delay	Set the retry interval when a transmission problem occurs
–retry-max-time	Set the maximum retry time when transmission problems occur
-s/–silent	Silent mode. It doesn’t output anything
-S/–show-error	Display an error
–socks4 <host[:port]>	Socks4 proxy for the given host and port
–socks5 <host[:port]>	Socks5 proxy for the given host and port
–stderr
-t/–telnet-option <OPT=val>	Telnet Option Settings
–trace	Debug the specified file
–trace-ascii	Like — trace but no hex output
–trace-time	Add a timestamp when tracing/detailing output
-T/–upload-file	Upload a file
–url	Spet URL to work with
-u/–user <user[:password]>	Set the user name and password for the server
-U/–proxy-user <user[:password]>	Set the proxy user name and password
-w/–write-out [format]	What output is finished
-x/–proxy <host[:port]>	Use an HTTP proxy on a given port
-X/–request	What command to specify
-y/–speed-time	The time required to abandon the speed limit. Default is 30
-Y/–speed-limit	Stop transmission speed limit, speed time

Linux curl command exit code

The following is the error code of the Linux curl command and its corresponding error message. When the command is executed incorrectly, you can use the error code to check the cause of the error, which is convenient for development and debugging.

Exit code	Error description
1	Unsupported protocol. This build of curl has no support for this protocol.
2	Failed to initialize.
3	URL malformed. The syntax was not correct.
5	Couldn’t resolve proxy. The given proxy host could not be resolved.
6	Couldn’t resolve host. The given remote host was not resolved.
7	Failed to connect to host.
8	FTP weird server reply. The server sent data curl couldn’t parse.
9	FTP access denied. The server denied login or denied access to the particular resource or directory you wanted to reach. Most often you tried to change to a directory that doesn’t exist on the server.
11	FTP weird PASS reply. Curl couldn’t parse the reply sent to the PASS request.
13	FTP weird PASV reply, Curl couldn’t parse the reply sent to the PASV request.
14	FTP weird 227 format. Curl couldn’t parse the 227-line the server sent.
15	FTP can’t get host. Couldn’t resolve the host IP we got in the 227-line.
17	FTP couldn’t set binary. Couldn’t change transfer method to binary.
18	Partial file. Only a part of the file was transferred.
19	FTP couldn’t download/access the given file, the RETR (or similar) command failed.
21	FTP quote error. A quote command returned error from the server.
22	HTTP page not retrieved. The requested url was not found or returned another error with the HTTP error code being 400 or above. This return code only appears if -f/–fail is used.
23	Write error. Curl couldn’t write data to a local filesystem or similar.
25	FTP couldn’t STOR file. The server denied the STOR operation, used for FTP uploading.
26	Read error. Various reading problems.
27	Out of memory. A memory allocation request failed.
28	Operation timeout. The specified time-out period was reached according to the conditions.
30	FTP PORT failed. The PORT command failed. Not all FTP servers support the PORT command, try doing a transfer using PASV instead!
31	FTP couldn’t use REST. The REST command failed. This command is used for resumed FTP transfers.
33	HTTP range error. The range “command” didn’t work.
34	HTTP post error. Internal post-request generation error.
35	SSL connect error. The SSL handshaking failed.
36	FTP bad download resume. Couldn’t continue an earlier aborted download.
37	FILE couldn’t read file. Failed to open the file. Permissions?
38	LDAP cannot bind. LDAP bind operation failed.
39	LDAP search failed.
41	Function not found. A required LDAP function was not found.
42	Aborted by callback. An application told curl to abort the operation.
43	Internal error. A function was called with a bad parameter.
45	Interface error. A specified outgoing interface could not be used.
47	Too many redirects. When following redirects, curl hit the maximum amount.
48	Unknown TELNET option specified.
49	Malformed telnet option.
51	The peer’s SSL certificate or SSH MD5 fingerprint was not ok.
52	The server didn’t reply anything, which here is considered an error.
53	SSL crypto engine not found.
54	Cannot set SSL crypto engine as default.
55	Failed sending network data.
56	Failure in receiving network data.
58	Problem with the local certificate.
59	Couldn’t use specified SSL cipher.
60	Peer certificate cannot be authenticated with known CA certificates.
61	Unrecognized transfer encoding.
62	Invalid LDAP URL.
63	Maximum file size exceeded.
64	Requested FTP SSL level failed.
65	Sending the data requires a rewind that failed.
66	Failed to initialize SSL Engine.
67	The user name, password, or similar was not accepted and curl failed to log in.
68	File not found on TFTP server.
69	Permission problem on TFTP server.
70	Out of disk space on TFTP server.
71	Illegal TFTP operation.
72	Unknown TFTP transfer ID.
73	File already exists (TFTP).
74	No such user (TFTP).
75	Character conversion failed.
76	Character conversion functions required.
77	Problem with reading the SSL CA cert (path? access rights?) .
78	The resource referenced in the URL does not exist.
79	An unspecified error occurred during the SSH session.
80	Failed to shut down the SSL connection.
82	Could not load CRL file, missing or wrong format (added in 7.19.0).
83	Issuer check failed (added in 7.19.0).
XX	More error codes will appear here in future releases. The existing ones are meant to never change.

Vi. Usage Demonstration:

To save space, some operations are no longer labeled with execution results.

1, view the source of the web page

Curl curl curl curl curl curl curl curl curl curl curl curl curl curl curl curl curl Take the url www.sina.com (chosen primarily because of its short web code) :

root@ubuntu:/home/peng# curl www.sohu.com
<html>
<head><title>307 Temporary Redirect</title></head>
<body bgcolor="white">
<center><h1>307 Temporary Redirect</h1></center>
<hr><center>nginx</center>
</body>
</html>
Copy the code

307 Temporary Redirect is displayed, indicating that the URL needs to be redirected.

To save the page, use the -o argument, which is equivalent to using the wget command.

Curl -o [file name] www.sohu.comCopy the code

2, automatic jump

Some urls jump automatically. Using the -l parameter to curl will redirect to the new url.

curl -L www.sohu.com
Copy the code

Type the above command, and the result automatically jumps to www.sohu.com.cn.

3. Display header information

The -i argument displays the HTTP response header, along with the web page code.

root@ubuntu:/home/peng/driver/test# curl -i www.sohu.comHTTP/1.1 307 Temporary Redirect Content-Type: text/ HTML Content-Length: 180 Connection: keep-alive Server: nginx Date: Tue, 25 Aug 2020 10:10:54 GMT Location: https://www.sohu.com/ FSS - Cache: the from 9790436.18244590.10468709 FSS - Proxy: < HTML >< head><title>307 Temporary Redirect</title></head> <body bgColor ="white">
<center><h1>307 Temporary Redirect</h1></center>
<hr><center>nginx</center>
</body>
</html>

Copy the code

The -i parameter displays only HTTP response headers.

4. Display the communication process

The -v parameter displays the entire HTTP communication process, including port connections and HTTP request headers.

root@ubuntu:/home/peng/driver/test# curl -v www.sohu.com
* About to connect() to www.sohu.com port 80 (# 0)* Trying 240e:83:201:3700::5... Connected > GET/HTTP/1.1 > user-agent: Curl /7.22.0 (i686-pc-linux-gnu) libcurl/7.22.0 OpenSSL/ 1.0.1zlib /1.2.3.4 libidn/1.23 librtmp/2.3 > Host: www.sohu.com > Accept: */* > < HTTP/1.1 307 Temporary Redirect < content-type: text/ HTML < content-length: 180 < Connection: keep-alive < Server: nginx < Date: Tue, 25 Aug 2020 10:11:49 GMT < Location: https://www.sohu.com/ < FSS - Cache: the from 9855973.18375663.10534247 < FSS - Proxy: < < HTML >< head><title>307 Temporary Redirect</title></head> <body bgColor ="white">
<center><h1>307 Temporary Redirect</h1></center>
<hr><center>nginx</center>
</body>
</html>
* Connection #0 to host www.sohu.com left intact
* Closing connection # 0

Copy the code

If you don’t think the above information is enough, the following command can look at the communication in more detail.

curl --trace output.txt www.sohu.com
Copy the code

curl --trace-ascii output.txt www.sohu.com
Copy the code

After running, please open output. TXT file to view.

5. Send form information

There are two methods for sending form information: GET and POST. The GET method is relatively simple, simply attaching the data to the url.

curl example.com/form.cgi?data=xxx
Copy the code

The POST method must separate data from the url. Curl uses the –data parameter.

curl -X POST --data "data=xxx" example.com/form.cgi
Copy the code

Curl curl curl curl curl curl curl curl curl curl curl curl curl curl curl curl curl curl curl

curl -X POST--data-urlencode "date=April 1" example.com/form.cgi
Copy the code

6. HTTP verbs

The default HTTP verb for curl is GET, and other verbs can be supported with the -x argument.

curl -X POST www.example.com
curl -X DELETE www.example.com
Copy the code

7. File upload

Suppose the file upload form looks like this:

<form method="POST" enctype='multipart/form-data' action="upload.cgi">
　<input type= the file name = the upload > < inputtype=submit name=press value="OK">
</form>
Copy the code

You can use curl to upload files like this:

curl --form upload=@localfilename --form press=OK [URL]
Copy the code

8. Referer field

Sometimes you need to provide a referer field in the HTTP request header to indicate where you jumped from.

curl --referer http://www.example.com http://www.example.com
Copy the code

9. User Agent field

This field is used to represent client device information. Servers sometimes use this field to return web pages in different formats for different devices, such as mobile and desktop.

The User Agent for iPhone4 is

Mozilla / 5.0 (iPhone; U; CPU iPhone OS 4_0 like Mac OS X; En-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8A293 Safari/6531.22.7Copy the code

Curl can be modeled like this:

curl --user-agent "[User Agent]" [URL]
Copy the code

10, cookie,

Curl sends a cookie using the –cookie argument.

curl --cookie "name=xxx" www.example.com
Copy the code

The value of the specific cookie can be obtained from the SET-cookie field in the HTTP response header.

-C cookie-file saves the cookie returned by the server to a file. -b cookie-file can use the file as cookie information for subsequent requests.

curl -c cookies http://example.com
curl -b cookies http://example.com
Copy the code

Add header information

Sometimes you need to add a header to your HTTP request. The –header argument will do the trick.

$ curl --header "Content-Type:application/json" http://example.com
Copy the code

12, certification

Using the curl option -u, you can perform HTTP or FTP authentication. You can specify a password or no password. Enter the password in the following operations:

curl -u user:pwd http://man.linuxde.net
curl -u user http://man.linuxde.net

Copy the code

13, FTP

1) List the directories on the FTP server

Curl ftp://www.xxx.com/ --user name:passwd curl ftp://www.xxx.com/ --u name:passwd# Brevity
curl ftp://name:[email protected] # 2
Copy the code

For example, set up an FTP server at 192.168.43.117 and set the user name to user and password to 123456. Now we want to display information about all files in the root directory of the server.

The curl -u user: 123456 ftp://192.168.43.117Copy the code

The result is as follows: Succinctly written:

The curl ftp://user:[email protected]Copy the code

The result is as follows:

2) List only the directories without displaying the progress bar

Curl ftp://www.xxx.com -u name:passwd -sCopy the code

3) Download a file:

format

The curl ftp://www.xxx.com/size.zip - u name: passwd - o size. ZipCopy the code

An example is as follows: Download the test.c file from the root directory of the server and save it to a local directory. The local file name is also test.c. [Note] Without the -o option, the program directs the data stream to stdout, which displays the contents of the file directly to the terminal.

Curl ftp://user:[email protected]/test.c -o test. CCopy the code

The result is as follows: Concise model

Curl the -u user: 123456 ftp://192.168.43.117/list.h - o list. HCopy the code

The result is as follows:

4) Upload a file:

Curl -- u name:passwd -t size.mp3 ftp://www.xxx.com/mp3/Copy the code

Examples are as follows:

Curl -u user:123456 ftp://192.168.43.117/ -t list.hCopy the code

You can see that the file did not upload successfully, and the error code is 25. Refer to Chapter 5

25	FTP couldn't STOR file. The server denied the STOR operation, used for FTP uploading.
Copy the code

The server does not have the storage permission. Therefore, set the write permission for the server.

Delete files from the FTP server using curl

Curl -u name:passwd ftp://www.xxx.com/ -x'DELE mp3/size.mp3'
Copy the code

Curl does not support recursive downloading, but you can use an array to download a file. For example, you can download a file named 1-10.gif

Curl -- u name:passwd ftp://www.xxx.com/img/[1-10].gif -- O#O uppercase
Copy the code

7) Download multiple files consecutively:

Curl -- u name:passwd ftp://www.xxx.com/img/[one,two,three].jpg -- O#O uppercase
Copy the code

6. Upload log files periodically

function

The last modification time of the current log file should be recorded when the program runs.
Every 10 seconds to check whether the log file is modified, if not, sleep for 10 seconds;
If the log file is modified, copy the current log file to a backup file with the name of the backup file and the current time.
Send it to the FTP server using curl.
To delete backup files, repeat Step 2.

The program flow chart is as follows:

Function Description

init()

The current log file time is first recorded in the global variable last_mtime.

check_file_change()

Select last_mtime and compare it with last_mtime. If the value is the same, return 0. If the value is different, return 1.

file_name_add_time()

Copy the current log file to a backup file with the name of the backup file and the current time.

stat()

Get the attribute information of the corresponding file and store it in the struct stat structure variable.

system()

Execute the command corresponding to the string in the argument

The code is as follows:

/* Copyright (C) yikouLinux */
#include <sys/stat.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <time.h>
 
typedef struct stat ST;
unsigned long last_mtime;

/* The username and password should be saved in the configuration file */
char name[32] ="user";
char pass[32] ="123456";
char ip[32]     ="192.168.43.117";
char filename[32] ="t.log";
char dstfile[256] = {0};

int init(void)
{
	// Prepare the structure
	ST status;
	
	// Call the stat function
	int res = stat(filename,&status);
	if(- 1 == res)
	{
		perror("error:open file fail\n");
		return 0;
	}
	last_mtime = status.st_mtime;
	printf("init time:%s \n",ctime(&last_mtime));
	return 1;
}
	
int  check_file_change(void)
{
	// Prepare the structure
	ST status;
	
	// Call the stat function
	int res = stat(filename,&status);
	if(- 1 == res)
	{
		perror("error:open file fail\n");
		return 0;
	}
//	printf("old:%s new:%s",ctime(&last_mtime),ctime(&status.st_mtime));
	if(last_mtime == status.st_mtime)
	{
		printf("file not change\n");
		return 0;
	}else{
		printf("file updated\n");	
		last_mtime = status.st_mtime;
		return 1; }}void file_name_add_time(void)
{
	ST status;
	time_t t; 	
	struct tm *tblock;	
	char cmd[1024] = {0};
		
	t = time(NULL);
	tblock = localtime(&t);
	
	sprintf(dstfile,"t-%d-%d-%d-%d-%d-%d.log",
		tblock->tm_year+1900,
		tblock->tm_mon,
		tblock->tm_mday,
		tblock->tm_hour,
		tblock->tm_min,
		tblock->tm_sec);
	sprintf(cmd,"cp %s %s",filename,dstfile);
//	printf("cdm=%s\n",cmd);
	system(cmd);
}
int main(void)
{

	char cmd[1024] = {0};

	init();
	while(1)
	{	
		if(check_file_change() == 1)
		{
			file_name_add_time();
			sprintf(cmd,"curl -u %s:%s ftp://%s/ -T %s",name,pass,ip,dstfile);
	// printf("cdm=%s\n",cmd);
			system(cmd);
			unlink(dstfile);
		}
		sleep(10); }}Copy the code

Run screenshot:

The first step: Because the log file has not been modified, the program will not be uploaded.

The second step:Manually enter the string yikoulinux into the log file t.log.Curl updated the updated file to see the log where curl uploaded the file.The following is the root directory of the FTP server, and you can see the uploaded log files:t-2020-7-26-1-19-45.log.

[supplementary]

Configuration information, written directly in the code, usually should be read from the configuration file, for the convenience of readers, this code does not add the function;
The FTP server was set up, which was not explained in this paper. There are many related files, so you can search by yourself.

3. Usually, such programs that need to run for a long time need to be set as daemons. This article does not add corresponding functions, so readers can search by themselves. If strongly requested, you can write a detailed introduction. 4. The management function of time in the code, please search related articles by yourself. Curl: curl: curl: curl: curl: curl: curl: curl: curl: curl: curl: curl: curl: curl. 6. The reason why you copy the log file as a backup file is that other modules may modify the log file at any time to protect it.

To communicate with bloggers, please follow the public account “A Mouthful of Linux”.

Linux command cURL detailed explanation, and the implementation of file timing upload to FTP server procedures