The background,

  • Demand: The current customer demand is to automatically upload the BACKUP files of Tencent Cloud CDB to Tencent Cloud COS, which is a good way to start a project. There are many similar demands that can be solved by using this method. Offline IDC data files can be backed up to COS in the cloud, or files can be uploaded to COS according to the file download address URL.
  • First get CDB backup download URL, upload file through COS API, if you have a better way welcome to discuss.

Two, technical details

  • COS: COS has an API and an SDK, which makes it very convenient for us to use Python to perform various operations on COS, COS SDK for Python

  • CDB: CDB has API, but there is no CORRESPONDING SDK for CDB query backup download. At this time, it can only be obtained through API. The signature of Tencent cloud API is very complicated. Construct parameter dictionary -> sort dict -> concatenate sign-> encode sign-> concatenate final URL -> complete call, sign method, query backup API

  • requirements:

Cos python - SDK - v5 = = 1.5.2 requests = = 2.19.1 tencentcloud - SDK - python = = 3.0.15 urllib3 = = 1.23Copy the code
  • File directory structure

Three, code,

Making the address

3.1 Configuration File

# auth:kaliarch
# func: Upload Tencent Cloud CDB backup files to the bucket specified by COS
# python version:python3+
# cos version:v5
# https://console.cloud.tencent.com/cos5/bucket

# Tencent cloud public information configuration section
[common]
# Tencent Cloud secretid
secret_id = AKIDMdjegcmoGxxxxxxxxxxxxxxxxxxxx
# Tencent Cloud SecretKey
secret_key = d5MRL4VoxyvlQvxxxxxxxxxxxxxx

# Tencent cloud COS information configuration section
[cosinfo]
# where the cosine is
cos_region = ap-chengdu

# Tencent cloud bucket name (COS V5 bucket name composition :bucket+ appID)
bucket_name = xuel-test-bucket-125396xxxx

# Tencent cloud CDB information configuration section
[cdbinfo]
# CDB instance id
cdb_instanceid = cdb-rqaxxxxx

# CDB location
cdb_region = ap-shanghai

# CDB log backup types, coldbackup, binlog, and slowlog_day
cdb_bak_type = coldbackup

Log file information configuration section
[loginfo]
Log file directory name
logdir_name = rds_to_cos
# Log file name
logfile_name = rdsbak_to_cos.log
Copy the code

3.2 CDB API core operation code

# build a dictionary
keydict = {
		'Action': self.cdb_action,
		'Timestamp': str(int(time.time())),
		'Nonce': str(int(random.random() * 1000)),
		'Region': self.cdb_region,
		'SecretId': self.secret_id,
		# 'SignatureMethod': SignatureMethod,
		'cdbInstanceId': self.cdb_instanceid,
		'type': self.cdb_bak_type
}
# dictionary sort
sorted(zip(keydict.keys(), keydict.values()))
String concatenation
sign_str_init = ' '
for value in sortlist:
		sign_str_init += value[0] + '=' + value[1] + '&'
sign_str = 'GET' + self.cdb_api_url + sign_str_init[:-1]
return sign_str, sign_str_init
Get the signature string and encode it
secretkey = self.secret_key
signature = bytes(sign_str, encoding='utf-8')
secretkey = bytes(secretkey, encoding='utf-8')
my_sign = hmac.new(secretkey, signature, hashlib.sha1).digest()
my_sign = base64.b64encode(my_sign)
parse.quote(my_sign)
Get the final URL
result_url = 'https://' + self.cdb_api_url + sign_str + '&Signature=' + result_sign
Copy the code

Running this module alone yields the following information:

3.3 COS SDK core operation code

Select simple upload or block upload automatically according to file size. Block upload has breakpoint continuation function
with open(filename, 'wb') as localfile:
		localfile.write(requests.request('get', url).content)
# upload
response = cos_client.upload_file(
		Bucket=self.bucket_name,
		LocalFilePath=filename,
		Key=filename,
		PartSize=partsize,
		MAXThread=maxthread
)
# Delete local files
if os.path.exists(filename):
		os.remove(filename)
Copy the code

3.4 Logging core code

Create a directory
def create_dir(self):
		_LOGDIR = os.path.join(os.path.dirname(__file__), self.logdir_name)
		_TIME = time.strftime('%Y-%m-%d', time.gmtime()) + The '-'
		_LOGNAME = _TIME + self.logfile_name
		LOGFILENAME = os.path.join(_LOGDIR, _LOGNAME)
		if not os.path.exists(_LOGDIR):
				os.mkdir(_LOGDIR)
		return LOGFILENAME
# define log files
def create_logger(self, logfilename):
		logger = logging.getLogger()
		logger.setLevel(logging.INFO)
		handler = logging.FileHandler(logfilename)
		handler.setLevel(logging.INFO)
		formater = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
		handler.setFormatter(formater)
		logger.addHandler(handler)
		return logger
Copy the code

4. Test results

Get the CDB download link

Five, the summary

  • Optimization: Automated tasks can be completed later by cooperating with scheduled tasks
  • Extension: Source: not only limited to CDB backup files, for any download URL, can be uploaded to COS. Terminal: Terminal is not limited to Tencent CLOUD COS, this idea can also be used in other cloud platforms such as Ali OSS, Amazon S3, Baidu cloud BOS, etc.