Big Data Computing Services (MaxCompute, formerly ODPS) is a fast, fully managed TB/PB level data warehouse solution. MaxCompute provides users with a comprehensive data import solution and a variety of classical distributed computing models, which can quickly solve the problem of computing massive data for users, effectively reduce enterprise costs, and ensure data security. MaxCompute mainly serves to store and compute batch structured data, and provides massive data warehouse solutions and analysis and modeling services for big data. With the continuous enrichment and improvement of social data collection methods, more and more industry data have been accumulated. Data scale has grown to the traditional software industry cannot carry massive data (100 GB, TB, or even PB) level. In the scenario of massive data analysis, due to the processing capacity limitation of a single server, data analysts usually adopt distributed computing mode. However, the distributed computing model puts forward higher requirements for data analysts and is difficult to maintain. With a distributed model, data analysts need to be familiar with both the business requirements and the underlying computing model. MaxCompute is designed to provide users with a convenient way to analyze and process massive amounts of data. Users can analyze big data without worrying about the details of distributed computing. MaxCompute has been widely applied in Alibaba Group, such as data warehouse and BI analysis of large Internet enterprises, log analysis of websites, transaction analysis of e-commerce websites, user characteristics and interest mining, etc. The MaxCompute development tool has three methods, this article describes the MaxCompute client installation, configuration, testing methods, and possible problems. Odpscmd is the MaxCompute base client that accesses the MaxCompute project space and uses various functions.
Step 1: Download and install JDK (1.7 or later) Because ODPSCMD is developed based on JAVA, you need to configure the JAVA runtime environment. JDK you can find on the Internet to download, I provide a download link, just for reference. www.oracle.com/technetwork… Windows and Mac OS versions can be downloaded. After downloading, you can follow the installation process.
Step 2: Configure the Windows JDK version: 1. Right-click computer → Properties → Advanced System Settings → Advanced → Environment Variables 2. System variable → New variable Name: JAVA_HOME Variable value: Variable value Enter the JDK installation directory, for example, I am
3. Find the Path variable in the system variable, select Edit, then select New, and enter
The final confirmation is OK.
4. Check whether the installation is complete. Run the Java -version command in the CMD tool of the system. If the following information is displayed, the configuration is complete.
This 1.8.0_171 is the installed version. Mac OS version: Mac OS configuration environment variables this article is very clear, you can refer to: blog.csdn.net/vvv_110/art…
Step 3: Download the ODPSCMD client installation package help.aliyun.com/document_de… Click on the link above and find the content below to download it.
Step 4: decompress the installation package we download the installation package to the appropriate hard disk, download down are ZIP files, their own computer compression tools should be able to decompress, if you can not decompress, we download a zip decompression tool. Once unzipped, there are four folders: bin, conf, lib, and plugins.
Step 5: Configure the client Edit the odps_config.ini file in the conf folder and configure the client as follows: Access_id =* Access_key =* Accesss ID and Access Key are cloud account information of users. You can log in to the official website of Aliyun and Access the AccessKeys page of the management console to view the information. Project_name =my_project # Specifies the project space that the user wants to enter. End_point=service.odps.aliyun.com/api # MaxCompute service access links. Tunnel_endpoint=dt.odps.aliyun.com # MaxCompute specifies the access link of the Tunnel service. Log_view_host=logview.odps.aliyun.com after a user executes a job, the client returns the LogView address of the job. Opening this address will show details of job execution. Https_check =true # Https_check =true # determine whether to enable HTTPS access because. Ini file computer default open TXT application, computer built-in text files open after a long sliver, is not good for viewing, you can download some TXT viewing tools, such as notepad, which is good for viewing. Because the service is enabled in different regions, you need to configure end_point and TUNel_endpoint based on the Regoin you have enabled and where data resides. For details, see the official document help.aliyun.com/document_de…
Save the configuration.
After modifying the configuration file, run the MaxCompute client in the bin directory (run in Linux./bin/odpscmd, run in Windows./bin/odpscmd.bat).
Step 7: Check whether the installation is complete. Run the test statement on the client, as shown in the following example: Create table tBL1 (id bigint); insert overwrite table tbl1 select count(*) from tbl1; select ‘welcome to MaxCompute! ‘ from tbl1; If the following information is displayed, the installation is correct.
Possible problems: 1, there is no JDK installed, no environment variable configuration, direct run odpscMD error. 2. Check the port 80 of the system, which is usually open. Closing the port will also cause an error. 3. The config configuration file is not correctly configured, especially the space name and endpoint. As a result, the client cannot be used normally. 4. If you encounter other problems, consult the MaxCompute developer community.
The original link
This article is the original content of the cloud habitat community, shall not be reproduced without permission.