preface
First of all, I have to admit that I am a bit of a claptrap. The title of this article, more accurately, should be: How to implement the file upload function for projects deployed on Tencent Cloud via Serveerless Framework?
When it comes to file uploading, there may not seem to be much to share for traditional development, but in Serverless architecture, it should be mentioned that in fact, both SCF of Tencent Cloud and Lambda of AWS have a packet size limit when triggering functions through the API gateway. In the case of Tencent Cloud, the limit is 6M.
What do you think of this limit of 6M? Can be considered if we are to the client (may be a Web, could be a small program or other) upload a file through the API gateway to SCF, then the whole packet is no more than 6 m, and the function of cloud to accept binary content, seemingly innate lack of soul, so is often the case, is recommended by Base64 before uploading. It is well known that packet sizes can vary after Base64, so in practice we might only upload about 4M packets to cloud functions this way.
What is 4M?
As shown in the figure above, I took out my phone and took an image with a size of 6.21M, which means that if I want to upload the image to SCF for some processing, it is “impossible”. Or, I make an album function, I upload the image directly to SCF, and SCF stores it in the object store. This operation is limited by packet size.
So at this point, there is a second solution (don’t call my timing diagram ugly, because this is not a timing diagram, just understand it, hahahahaha).
In this method, the client has launched three requests, acquire temporary upload address, upload files to COS, obtain the results (of course, if you don’t need to get the results of what, for example, the user simply upload a file to your own account, the third time that this kind of situation will not need to request).
This approach may not be easy to think of and is a bit more complicated to implement than the SCF method of uploading files directly and then persisting them.
Recommendations for several different scenarios:
-
Scenario 1: User uploading profile picture Function For this scenario, plan 1 is used. Because under normal circumstances, the head is not big, can completely on the client side for image compression and cropping, completed, direct users with some parameters, such as the user’s token, uploaded to the SCF, archived in SCF to picture the COS and the image associated with user information, and some results back to the client. The whole process only needs one function, which is convenient and quick.
-
Scenario 2: For such a scene, scheme 2 is actually better, because most of the time, when uploading pictures to albums, they want to keep the original picture rather than compress it, so the size of the original picture is likely to exceed 6M. Scheme 1 is not very reasonable, and the combination of APIGW+SCF, It is not very suitable for file transfer, so it is reasonable to upload COS first at this time. Users can take the image to upload photo album and photo name, the user’s token by acquire temporary keys into the function 1, 1 () function converts a user, photo album, pictures and status (for example be uploaded, processed, handled, etc.) information such as association, and storage, then temporary address returned to the client, the client upload pictures to COS, COS trigger is used to trigger function 2, which compresses the image (under normal circumstances, compressed images will be displayed in the album list, and complete lossless images will be available only after clicking on the album details), and associate with the previous information to modify the data state. After users to upload pictures, if necessary, the client can be launched the third request for image storage/processing as a result, the function of 3 will query the database state, within a certain time threshold, if the data is complete, has said the data has been uploaded and finished part processing, otherwise you will return the corresponding exception information.
Show CCCCCCCCCode
Function 1 implements the first scheme, in which files are passed to SCF through Base64 and then transferred to COS by SCF:
def uploadToScf(event, context):
print('event', event)
print('context', context)
body = json.loads(event['body'])
You can use the token sent from the client for authentication. Only after the authentication is passed can you obtain the temporary upload address
# This part can be modified as needed, for example, the user's token can be obtained in Redis, can be obtained through some encryption methods, etc
Username = token; username = token; token = token
# match it and so on, this will improve security as much as possible
if "key" not in body or "token" not in body or body['token'] != 'mytoken' or "key" not in body:
return {"url": None}
pictureBase64 = body["picture"].split("base64,") [1]
with open('/tmp/%s' % body['key'].'wb') as f:
f.write(base64.b64decode(pictureBase64))
region = os.environ.get("region")
secret_id = os.environ.get("TENCENTCLOUD_SECRETID")
secret_key = os.environ.get("TENCENTCLOUD_SECRETKEY")
token = os.environ.get("TENCENTCLOUD_SESSIONTOKEN")
config = CosConfig(Region=region, SecretId=secret_id, SecretKey=secret_key, Token=token)
client = CosS3Client(config)
response = client.upload_file(
Bucket=os.environ.get("bucket_name"),
LocalFilePath='/tmp/%s' % body['key'],
Key=body['key'],)return {
"uploaded": 1."url": 'https://%s.cos.%s.myqcloud.com' % (
os.environ.get("bucket_name"), os.environ.get("region")) + body['key']}Copy the codeFunction 1, to achieve the second scheme, to obtain temporary signature URL:
def getPresignedUrl(event, context):
print('event', event)
print('context', context)
body = json.loads(event['body'])
You can use the token sent from the client for authentication. Only after the authentication is passed can you obtain the temporary upload address
# This part can be modified as needed, for example, the user's token can be obtained in Redis, can be obtained through some encryption methods, etc
Username = token; username = token; token = token
# match it and so on, this will improve security as much as possible
if "key" not in body or "token" not in body or body['token'] != 'mytoken' or "key" not in body:
return {"url": None}
Initialize the COS object
region = os.environ.get("region")
secret_id = os.environ.get("TENCENTCLOUD_SECRETID")
secret_key = os.environ.get("TENCENTCLOUD_SECRETKEY")
token = os.environ.get("TENCENTCLOUD_SESSIONTOKEN")
config = CosConfig(Region=region, SecretId=secret_id, SecretKey=secret_key, Token=token)
client = CosS3Client(config)
response = client.get_presigned_url(
Method='PUT',
Bucket=os.environ.get('bucket_name'),
Key=body['key'],
Expired=30.)return {"url": response.split("? sign=") [0]."sign": urllib.parse.unquote(response.split("? sign=") [1]),
"token": os.environ.get("TENCENTCLOUD_SESSIONTOKEN")}
Copy the codeHTML page basic implementation:
HTML part:
<div style="width: 70%">
<div style="text-align: center">
<h3>Uploading files on the Web</h3>
</div>
<hr>
<div>
<p>Scheme 1: Upload SCF, process it, and then save it to COS. This method is more intuitive, but the problem is that SCF can only receive data less than 6M from APIGW, and it is not good for binary file processing.</p>
<input type="file" name="file" id="fileScf"/>
<input type="button" onclick="UpladFileSCF()" value="Upload"/>
</div>
<hr>
<div>
<p>Scheme 2: Directly uploaded to the COS, the process is to first get temporary address from SCF, data storage, such as to save the file information as redis, etc.), and then from the client to upload the COS, upload end by cosine trigger function, from the storage system (for example, have been stored in the redis) read more for information, in the image processing.</p>
<input type="file" name="file" id="fileCos"/>
<input type="button" onclick="UpladFileCOS()" value="Upload"/>
</div>
</div>
Copy the codeOption 1 upload part of JS:
function UpladFileSCF() {
var oFReader = new FileReader();
oFReader.readAsDataURL(document.getElementById("fileScf").files[0]);
oFReader.onload = function (oFREvent) {
const key = Math.random().toString(36).substr(2);
var xmlhttp = window.XMLHttpRequest ? (new XMLHttpRequest()) : (new ActiveXObject("Microsoft.XMLHTTP"))
xmlhttp.onreadystatechange = function () {
if (xmlhttp.readyState == 4 && xmlhttp.status == 200) {
if (JSON.parse(xmlhttp.responseText)['uploaded'] = =1) {
alert("Upload successful")}}}var url = " https://service-f1zk07f3-1256773370.bj.apigw.tencentcs.com/release/upload/cos"
xmlhttp.open("POST", url, true);
xmlhttp.setRequestHeader("Content-type"."application/json");
var postData = {
picture: oFREvent.target.result,
token: 'mytoken'.key: key,
}
xmlhttp.send(JSON.stringify(postData)); }}Copy the codeOption 2: Upload some JS:
function doUpload(key, bodyUrl, bodySign, bodyToken) {
var fileObj = document.getElementById("fileCos").files[0];
xmlhttp = window.XMLHttpRequest ? (new XMLHttpRequest()) : (new ActiveXObject("Microsoft.XMLHTTP"));
xmlhttp.open("PUT", bodyUrl, true);
xmlhttp.onload = function () {
console.log(xmlhttp.responseText)
if(! xmlhttp.responseText) { alert("Upload successful")}}; xmlhttp.setRequestHeader("Authorization", bodySign);
xmlhttp.setRequestHeader("x-cos-security-token", bodyToken);
xmlhttp.send(fileObj);
}
function UpladFileCOS() {
const key = Math.random().toString(36).substr(2);
var xmlhttp = window.XMLHttpRequest ? (new XMLHttpRequest()) : (new ActiveXObject("Microsoft.XMLHTTP"))
xmlhttp.onreadystatechange = function () {
if (xmlhttp.readyState == 4 && xmlhttp.status == 200) {
var body = JSON.parse(xmlhttp.responseText)
if (body['url']) {
doUpload(key, body['url'], body['sign'], body['token'])}}}var getUploadUrl = 'https://service-f1zk07f3-1256773370.bj.apigw.tencentcs.com/release/upload/presigned'
xmlhttp.open("POST", getUploadUrl, true);
xmlhttp.setRequestHeader("Content-type"."application/json");
xmlhttp.send(JSON.stringify({
token: 'mytoken'.key: key,
}));
}
Copy the codeOs.environ. get(“TENCENTCLOUD_SECRETID”) is used to obtain the user’s key information. Using the Serverless Framework as an example, you can use Tencent -cam-role, for example to create a global component:
Conf:
component: "serverless-global"
inputs:
region: ap-beijing
runtime: Python3.6
role: SCF_UploadToCOSRole
bucket_name: scf-upload-1256773370
Copy the codeThen create a component that adds Role:
UploadToCOSRole:
component: "@gosls/tencent-cam-role"
inputs:
roleName: ${Conf.role}
service:
- scf.qcloud.com
policy:
policyName:
- QcloudCOSFullAccess
Copy the codeNext, create a function that will bind to the role you just created:
getUploadPresignedUrl:
component: "@gosls/tencent-scf"
inputs:
name: Upload_getUploadPresignedUrl
role: ${Conf.role}
codeUri: ./fileUploadToCos
handler: index.getPresignedUrl
runtime: ${Conf.runtime}
region: ${Conf.region}
description: Obtain cos temporary upload address
memorySize: 64
timeout: 3
environment:
variables:
region: ${Conf.region}
bucket_name: ${Conf.bucket_name}
Copy the codeAlso bind this function to APIGW:
UploadService:
component: "@gosls/tencent-apigateway"
inputs:
region: ${Conf.region}
protocols:
- http
- https
serviceName: UploadAPI
environment: release
endpoints:
- path: /upload/cos
description: Upload cos via SCF
method: POST
enableCORS: TRUE
function:
functionName: Upload_uploadToSCFToCOS
- path: /upload/presigned
description: Obtain temporary address
method: POST
enableCORS: TRUE
function:
functionName: Upload_getUploadPresignedUrl
Copy the codeIn addition, this example also needs a COS bucket to be used as a test, because the Web service may have cross-domain problems, so we need to set the COS cross-domain:
SCFUploadBucket:
component: '@gosls/tencent-cos'
inputs:
bucket: ${Conf.bucket_name}
region: ${Conf.region}
cors:
- id: abc
maxAgeSeconds: '10'
allowedMethods:
- POST
- PUT
allowedOrigins:
- The '*'
allowedHeaders:
- The '*'
Copy the codeOnce done, you can quickly deploy:
(venv) DFOUNDERLIU-MB0:testDfounderliu $SLS -- Resolving the Template'static variables. DEBUG ─ Collecting components from the template. DEBUG ─ Downloading any NPM components found in the template. ... . apis: - path: /upload/cos method: POST apiId: api-0lkhke0c - path: /upload/presigned method: POST apiId: API - B7J5IKOC 15S › uploadToSCFToCOS › doneCopy the codeFinally, for your convenience: github.com/anycodes/Se… I put it on Git:
