The guidelines are divided into three steps.

  • Upload compressed packages from local to platform personal data
  • Download platform personal data into the instance
  • Perform training with automatic upload results and shutdown

The operation process is as follows:

Upload personal data locally

Locally, the trained data is compressed into a zip, tar.gz file. Then download and install the OSS command line tool locally and use the tool to upload the compressed package to personal Data on the platform.

The Windows system is used as the local environment here. For other oss, see the OSS command line tool.

Download the [OSS (Windows)] executable file. After downloading, change the file name of oss_windows_x86_64.exe to oss.exe. Open CMD (command prompt) or PowerShell in Windows.

Use the CD command to switch to the directory of oss.exe, where oss.exe is saved under D:\Download. In this case, it is executed using a PowerShell terminal.

PS C:\Users\windows> cd D:\Download 
PS D:\Download>
Copy the code

This command can be executed by executing.\oss in this directory. You need to log in before uploading. The login account is the account name and password of Hengyuan Cloud platform, not the instance. The account name is mobile phone number. If it is not a Mobile phone number in mainland China, the country number with + should be added.

# execution. \ oss login PS D: \ Download >. \ oss login Username: 139 * * * * * * * * Password: * * * * * * * * * * * 139 * * * * * * * * login successfully!Copy the code

After logging in successfully, the operation of transferring files will be carried out, and the local compressed package will be uploaded to [personal data]. Notice The file name extension supports only common compressed package formats. The zip path in this case is D:\Datasets\ mnist.zip.

\oss cp D:\Datasets\ mnist. zip oss:// PS D:\Download>.\oss cp D:\Datasets\ mnist. zip oss:// Start at 2021-10-22 06:21:48. 2061924 + 0000 UTC [-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --] 100.00% 52.34 MB/s (76.55 76.55 MB 1.53 MB/s the Upload Successfully, 76.55MB, n/a, D:\Datasets\ mnist. zip --> oss:// mnist. zip, cost [7020], status [200], request id [0000017CA6A9BE01901395D26CE9A228]Copy the code

After the file is uploaded, you can view it on the personal data page of the platform.

Example Download personal data

After the instance is created or started on the platform, you can access the terminal through the page of [JupyterLab] or [SSH client] to connect the terminal.

On the terminal, use oss to log in first. The login account is the account name and password of Hengyuan Cloud platform, not the instance. The account name is mobile phone number. If it is not a Mobile phone number in mainland China, the country number with + should be added.

# implement oss login ~ # oss login Username: 139 Password: * * * * * * * * * * * * * * * * * * * 139 * * * * * * * * login successfully!Copy the code

After the login is successful, you can download the uploaded file to/HY-tmp (/ HY-NAS can be used for models with shared storage).

Oss cp oss:// mnist. zip /hy-tmp ~# oss cp oss:// mnist. zip /hy-tmp Start at 2021-10-22 06:37:17.227649376 +0000 UTC [-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -] 100.00% 62.85 MB/s (76.55 to 76.55 MB 1.42 MB/s Waiting to rename temporary file... Download successfully, 76.55MB, n/a, OSS :// mnist. zip --> / HY-tmp/mnist. zip, cost [2422], status [206], request id [0000017CA6B7D6419012DB7767D11959]Copy the code

After the package is downloaded, go to the download path and decompress the package.

# unzip -q mnist. zip unzip -q mnist. zip /hy-tmp# unzip -q mnist. zipCopy the code

Power off after performing training and automatically uploading results

Now that the data has been stored in the example, the next step is training. You can write code using tools such as [JupyterLab] or [VSCode]. Note that the data path referenced in the code needs to be changed to the data path within the instance, which was /hy-tmp in the example above. Long training tasks need to be run in the background through [Tmux], which can shut down the local computer without interrupting training.

Pay by volume instances can be achieved through the following operations after the completion of training to upload the results to personal data, after the shutdown to terminate billing. This operation can save the usage cost of pay-per-view and avoid the failure to view and download the data of/HY-tmp caused by the machine being occupied.

The operations in the example script must meet the following requirements. Some operations must be adjusted based on site requirements.

  • The results of the training are in/hy-tmp/result 中
  • In this example, you need to use the platform account mobile phone number and password in advanceoss loginSuccess of landing

Create an upload.sh script using vim in the /root directory.

cd
vim upload.sh
Copy the code

The script content is as follows. Paste the following content using the Vim editor and save it.

#! File ="result-$(date "+%Y%m%d-%H% m% S").zip Oss cp "${file}" oss://backup/ rm -f "${file}" #Copy the code

Add execute permission to the script.

chmod u+x upload.sh
Copy the code

This script implements /hy-tmp/result as a zip package and uplows this file to personal data. After the command is successfully executed, power off the device.

Run the /root/upload.sh script at the end of the training task code. It is recommended to test it for the first time. As referenced in Python code:

import os
os.system('/root/upload.sh')
Copy the code

When the training execution is shut down, the packaged result file can be downloaded locally using the OSS command line tool. If the instance machine is occupied and cannot be started, you can also create a new instance on another machine and download data to the new instance through OSS. For details about OSS operations, see the OSS command Tool.

# Log in the Hengyuan Cloud account and use the account name and password of Hengyuan Cloud. The account name is mobile number. Need the area code of coupled with + ~ # oss login Username: 139 Password: * * * * * * * * * * * * * * * * * * * 139 * * * * * * * * login successfully! ~# oss ls -s oss:// Folder list: oss:// oss://backup/ Object list: Oss ://backup/result-20211018-164323.zip # download personal data result-20211018-164323.zip to the current directory ~# oss cp oss://backup/result-20211018-164323.zip . Download successfully, 310B, n/a, oss://backup/result-20211018-164323.zip --> /root/result-20211018-164323.zip, cost [57], status [200], request id [0000017C92929EF49014BE16738685B7]Copy the code