Abstract: Shanxi Kun Peng Development Training Camp is mainly divided into kunpeng ecological introduction, Kunpeng application code migration guidance, C/C++ and Java code migration sandbox practice and micro authentication.
On August 27, Sharon, the developer of Huawei DevRun, came to Taiyuan and held the “Shanxi Kunpeng Development Training Camp” in the West Hall conference room, first floor, Management Committee of Shanxi Transformation and Comprehensive Reform Demonstration Zone. Kunpeng Development Training Camp is a technology empowerment activity held by Shanxi Kunpeng Ecological Innovation Center under the guidance of Shanxi Department of Industry and Information Technology and the Innovation and Development Department of Shanxi Comprehensive Reform Demonstration Zone Management Committee. The training camp consists of kunpeng ecological introduction, Kunpeng application code migration guidance, C/C++ and Java code migration sandbox practice and micro authentication.
Introduction of kunpeng ecology
COO Yu Chao of Shanxi Kunpeng Ecological Innovation Center introduced the development status of Kunpeng ecology to the participants. Yu chao mentioned that today’s human society is ushering in the fourth industrial revolution, whose representative industry is intelligent technology. In this field, Huawei Kunpeng chips mainly follow the ARM technology route and continue to evolve in the aspects of high-performance computing, security and reliability and open ecology. Kunpeng architecture has had large-scale commercial applications in scenarios such as big data, database, cloud services, distributed storage and native applications. Adhering to the concept of open hardware, open source software and enabling partners, Huawei kunpeng provides various industries with the support of “Kunpeng chip + Ceng AI” in computing industry and builds the ecology of Kunpeng computing industry.
At present, more than 800 manufacturers and more than 2,000 schemes have passed kunpeng ecological certification. Huawei has built 18 Kunpeng eco-innovation centers in cooperation with provincial and municipal government departments across the country. Shanxi Kunpeng Ecological Innovation Center, which was just completed in July this year, is an important part of the “No.1 Project” of Shanxi Credit and innovation Industry. It aims to provide various professional services for the enterprises and personnel participating in Kunpeng Ecology and jointly promote the enterprise application innovation and software project incubation activities based on kunpeng ecological research and development. Through the technical support of Shanxi Kunpeng Ecological Innovation Center, the market of Kunpeng computing industry in Shanxi Province can be comprehensively developed, the competitiveness of local enterprises in Shanxi can be improved, and the ecological and innovation highland of credit and innovation industry in Shanxi Province can be built together. In the next 3-5 years, Huawei also plans to invest $1.5 billion in kunpeng ecological construction in universities, cooperate with colleges and universities to compile kunpeng technology textbooks, jointly build teacher teams, laboratories and training bases, and carry out vocational certification training examinations, so as to cultivate excellent talents in Kunpeng industry from universities and colleges.
Kunpeng code migration guide
Software migration is an important part of Kunpeng ecological application. In this training camp, Liu Kun, a senior expert of Kunpeng computing industry, brought an in-depth course of Software migration of Kunpeng. Once the code of computer application is written in high-level programming language, it will be compiled into assembly language through compiler and regenerated into binary machine code. The CPU instruction set architecture is the bridge between the bottom layer of the chip and the upper layer of the program, the latter is translated into instruction sequence, through the instruction set architecture instructions execution. The KUNpeng processor uses an ARMv8 instruction set with a completely different architecture from x86, so programs must be recompiled when migrating from x86 to Kunpeng. Typically, the software migration process consists of five steps: migration preparation, migration analysis, compilation migration, performance tuning, and test certification. One of the most important steps is compile migration. Preparation for migration: mainly collect information of underlying hardware and upper software to prepare compilation environment (kunpeng server or OpenLab online environment). Migration analysis: Analyzes the application software stack, including its language, underlying OS, middleware, dependencies, compilers, runtime environment, etc. Compile migration: This step falls into two categories. The first category is code migration, such as code migration from compiled languages such as C/C++ or interpreted languages such as Java/Python. The second type is software packages, mainly the migration of RPM packages and JAR packages. Performance tuning: Optimize efforts to achieve desired performance goals after the migration is complete. Testing and certification: Ensure migration quality through testing and certification, and find reasons for non-compliance before tuning. In order to help developers reduce the migration threshold of applications, Huawei also developed Kunpeng software migration suite, which mainly includes analysis scanning tool, migration tool and performance tuning tool.
C/C++ code migration
C/C++ is a compiled language. The source code is compiled into an executable file through several steps. A precompiled file is generated and then compiled into assembly code. Then the target binary file is generated through assembly process, and finally the executable file is generated by adding SO and other links through linking process. Since x86 and Kunpeng are different in instruction set architecture, registers, extended instruction set (SIMD) and other underlying attributes, their assembly codes are different, so cross-platform application transplantation requires compilation and migration. C/C++ code engineering generally falls into two categories: compile build scripts (such as Makefiles, Configure, etc.) and source code. The former migration mainly involves the change of compilation options, while the latter involves compilation macro migration, builtin function, inline assembly, SIMD and so on. Not all layers need to be touched in the actual application migration. Some applications simply need to compile directly from the source code, or change a compilation option or two. In application migration, the source code should be obtained first, then the environment should be compiled, Makefile should be generated, SO library should be migrated, and finally the installation and deployment should be done.
Typical problems in C/C++ code migration
1. Compile option migration. For example, when the application is compiled to 64-bit, m64 option is used on x86 platform and MABI = LP64 option is used on Kunpeng. It is enough to replace the compilation options with architectural differences according to the compilation document. 2. Compile macro migration. Compilation macros are tools used to identify architectural platforms and select corresponding branches of code. There are only two compiler macros in x86, which can be replaced directly on kunpeng platform. User – defined compilation macros also need to be substituted according to their own writing. However, after compiling macro replacement, if there is no ARM corresponding source branch, you need to implement the corresponding function. 3. Builtin function transplantation. The builtin function is a function implemented internally by the compiler. There are few functions that need to be migrated, mainly focusing on the CRC32 checksum calculation. 4. Inline assembly migration. There are two ways to transfer in-line assembly. One is to replace one instruction by assembly instruction, the key is the idea of instruction to instruction replacement. The other is to replace it by builtin function, which selects the corresponding builtin function according to the function of the instruction. 5.SIMD command migration. SSE, AVX are x86 platform SIMD instruction, SIMD is short for single instruction multiple data flow, can be used to speed up computing. The Kunpeng platform also has its own SIMD directive called Neon. When SIMD instructions from x86 platform are transferred to Kunpeng platform, it only needs to find the corresponding function replacement implementation. In practice, it is recommended to adopt the open source project instead of migrating one by one. Huawei, for example, has open-source a project that implements most of the SIMD instructions. In practice, the open source file can be referenced in the application directory, which is very convenient.
Special commands that cannot be covered by the open-source project need to be manually replaced. In this case, you need to search for the implementation principles of the code to replace them. You can contact Huawei engineers for help.
To simplify application migration, Huawei has also developed the Porting Advisor. It first scans the source code of the build script to identify problems that may be involved in the migration. Then according to the existing migration problem database to give guidance to the detected problems, avoid duplication of work. Most of the migration issues mentioned above can be addressed in the migration report provided by the tool. Just follow the instructions.
Java/Python code migration
Java/Python is an interpreted language, and migration issues are rare. Java application development, after installing JDK, use JDK to fill in the code, compile bytecode files through the Java compiler, package some resource libraries to generate JAR packages as required, and upload the environment to deploy and run. There are two main steps in Java application migration: reinstall JDK. For example, Oracle JDK on x86 is not supported on The Kun Peng platform, so it is recommended to use OpenJDK instead. The classic version can be installed directly from the software source, or you can compile the JDK source code yourself. The SO library needs to be recompiled. The SO library of x86 platform was identified through the Porting Advisor to screen the JAR package. Then THE SO library of KUNpeng platform was generated from the Porting Advisor and repackaged. Python code migration is similar, including migration of the runtime environment and SO library. The migration of the runtime environment is not difficult, SO library migration is the same as Java steps.
Maven software warehouse building
Maven is a common open source software building tool. Maven repositories mainly store JAR packages, including local, remote, and central repositories. Remote repositories are servers set up inside the enterprise or by third-party organizations. Central repositories are those set up by the Maven team itself. Maven’s default jar search sequence is local, remote, and central. If you find a jar you need, pull it down and build it. However, if the jar package containing x86 platform dependencies is pulled down, the construction and operation on Kunpeng platform will make mistakes, and it needs to analyze scanning, packaging and compilation again, which is a cumbersome process.
To simplify operations, Huawei developed a Maven repository exclusively owned by Kunpeng, which contains common jar package porting versions. Developers only need to change the XMR configuration file of Maven warehouse to preferently search the Kunpeng Maven warehouse to quickly find the corresponding JAR package of Kunpeng platform, simplify the migration process and reduce the workload of Java migration.
Taking Hive compilation instance as an example, the migration tool was used to scan and found that there were four JAR packages with x86 dependencies in the application. Then the corresponding JAR packages of Kunpeng platform could be pulled after the Maven storehouse was configured, and the migration could be completed after the compilation.
RPM Software Package Migration
RPM is a common software package type in Linux. The essence of RPM package transplantation is to transplant some binary files and replace x86 dependent files with those of Kunpeng platform. During the transplantation, firstly, find the RPM package corresponding to Kunpeng platform, and try to obtain it from the following channels: local source of operating system; The remote source of the operating system; Huawei Yunkunpeng images (including many RPM packages on kunpeng platform); RPM packages only available under x86 can be disassembled and reconstructed, replacing all SO and binary files and then packaging; If all else fails, download the source code and recompile. RPM package reconstruction involves scanning, compiling, packaging, and verification. Scan for x86 dependencies, replace them if they have jar implementations, or compile them from source code. Then replace the packaging and verify. The Porting Advisor Porting Tool provides an RPM package reconstruction module, which can take full advantage of Kunpeng Maven barn to automatically scan and find existing JAR package replacement implementations. However, if it is the difference of SO file, it needs to be handled manually. When migrating, developers can refer to the resources of kunpeng community, Kunpeng Xiaozhi Assistant and developer’s official account. There are 24 hours q&A in Kunpeng community, as well as abundant migration and performance tuning documents and q&A materials for many questions. Kunpeng xiaozhi assistant can easily obtain the specific migration guidance under Kun Peng platform; The developer public account will regularly publish some technical information.
Migrating sandbox practices and microauthentication
After the theory transfer course of sandbox training camp, sandbox practice session was arranged for the on-site partners. The sandbox environment is open to the public online, and interested developers can try it out on their own. Sandbox operation is mainly divided into three steps:
To register a Huawei cloud account, real-name authentication is required. Huawei Cloud Platform has OpenLab virtual Lab, which has four sandbox practices: C/C++, Java, software package and Maven warehouse. Select the corresponding sandbox to start the experiment, the cloud will be assigned a VIRTUAL machine, can complete the compilation operation. Developers can complete experiments according to the manual and instructions on the web. The microauthentication Cloud Laboratory has a microauthentication module, which is divided into two courses of Software migration and performance tuning of Kunpeng. After purchasing certification, you can first take courses, complete self-tests and LABS, and finally take exams. Real-name authentication is required, and cameras are used to capture the exam to prevent cheating. The content includes judgment, single choice and multiple choice forms. After passing the exam, a micro certificate will be issued online.
Links to relevant community resources involved in this bootcamp are summarized below: Inline assembly rules reference: gcc.gnu.org/onlinedocs/… Different GCC compiler options query (gCC7.3 as an example) : gcc.gnu.org/onlinedocs/… Kunpeng development suite – Porting Advisor:www.huaweicloud.com/kunpeng/sof… Kunpeng Maven warehouse link: mirrors.huaweicloud.com/kunpeng/mav… Kunpeng community: bbs.huaweicloud.com/forum/forum… Kunpeng little wisdom assistant: ic-openlabs.huawei.com/chat/#/ huawei cloud lab: lab.huaweicloud.com/ micro authentication: edu.huaweicloud.com/certificati…
This article is shared by Sylvia CC in “[Shanxi Kunpeng Training Camp] Migrating from application to Platform Micro-authentication: In-depth Interpretation of Technology Dry Goods”, published by Huawei Cloud community.
Click to follow, the first time to learn about Huawei cloud fresh technology ~