Abstract: Kunpeng DevKit, a development kit of Kunpeng, can achieve quick code migration within two days. It provides a full-stack development tool, which integrates code migration, compilation and debugging, performance tuning, exception diagnosis and other tools and functions.
This article is shared from the compilation code of “[Cloud Resident Co-create] Kunpeng DevKit to Achieve Efficient Migration for two people in a day” by Huawei cloud community. The original author is Victue.
First, the ambition of Kunpeng
1.1 Kun Peng Platform
The Kunpeng platform offers developers around the world a new computing power option. From the opening of hardware to the open source of system software, it covers the whole industrial chain from the bottom hardware and basic software to the upper industry application, and has achieved significant results or progress in various industries of national economy and people’s livelihood. KunpengDevKit is a dragon slaving blade created by Huawei for kunpeng developers to help them develop efficiently and migrate with one key.
1.2 Kunpeng DevKit
KunpengDevKit provides full-stack development tools, integrating code migration, compilation and debugging, performance tuning, exception diagnosis and other tools and functions. It provides a convenient, fast and professional tool kit for developers to work in series. Through Kun Peng DevKit, users can develop efficiently and show their plans.
The real martial arts master, internal strength is very strong, external strength is also very strong! DevKit product design and user experience are equally good!
Time is limited, this article focuses on sharing assembly translation techniques and KunpengDevKit product experience.
Second, assembly automatic translation
2.1 Assembly Code
As we all know, assembly language has a long history and is difficult to get started. It is the low-level language of any computer or other programmable devices. Assembly language corresponds to different machine language instruction sets on different devices.
An assembly language dedicated to a computer system structure, such as X86_64 or AARCH64 assembly. This means that to the extent that assembly language can truly control hardware, it can do things that higher-level languages cannot. Similarly, assembly language is obscure and difficult to learn because of its proximity to the hardware platform.
2.2 Assembly Style
Each computer platform corresponds to its own computer platform, memory, registers and other syntax complex format.
The use of assembly language is also diverse, single instruction simple embedded assembly, multi-instruction simple embedded assembly, full assembly file. This is equivalent to speaking ancient prose and poetry in our modern society, using classical Chinese or quoting two ancient poems.
2.3 Significance and Challenges
Moving code from one platform to another can be extremely difficult if assembly language translation is involved.
The goal is to migrate the application in two days, but if the code contains assembly language, it may not be possible for 12 people to do the migration. No, no! Huawei must develop assembly migration tools!
“Tens of thousands of lines of code, just a few lines of assembly, no tools for migration, two lines of development.”
2.4 Application Scenarios
When using assembly language, the number of instructions embedded in assembly is mostly less than three instructions, which are relatively simple to use and relatively concentrated.
SIMD instructions account for the vast majority of the whole assembly files, and the instructions are relatively concentrated.
It can be found that the common assembly instruction and SIMD instruction account for the majority of assembly language. As long as the common instructions and SIMD instructions can be solved, the project is more than half successful.
2.5 General Ideas
The goal is to recognize all X86 assembles and automatically translate (most scenarios with minimal performance degradation).
In the beginning, the commonly used instructions are translated, and then, with the expansion of user scale and the increase of usage, the instructions required by users are gradually added according to their needs.
First technical point: automation
Second technical point: technology reuse
The overall technical architecture is future-oriented and has vitality. This leads to the following design principles
Design principles
-
Ease of use – one-click migration
-
Extensibility – Translation range
-
Maintainability – source code translation
-
Architectural reuse – Different scenarios
-
Technology reuse – do not build wheels
2.6 Overall Design
W + 2
“Two”, direct translation, implementation from x86-64 to AArch64
Although the operation is relatively complex, the “W” takes advantage of the existing capability of compilation technology and transforms it into translation, thus realizing the optimization of translation efficiency. Based on the complex nature of assembly as described above and the goal of platform convenience, the following design principles and solutions were generated
Design principles
-
Complex assembly style
-
There are big differences between platforms
-
The goal is high
Design scheme
-
Can be used with
-
Need to change all
-
When the fast is fast
-
Twists and turns on
Part of assembly automatic translation technology
3.1 Recognition of embedded assembly
For the whole assembly, the recognition method is relatively simple, the assembly file can be recognized by the suffix of the file, the problem lies in the recognition of embedded assembly. We use the Clang-Tidy compilation technique to achieve this goal.
Clang-tidy: LlVM-based static syntax checking framework that provides syntax (AST) parsing capabilities.
The clang-Tidy toolset extends assembly recognition and replacement rules, exposing AST key nodes. After locating assembly code, you can do whatever you want and translate it.
The identification flow chart is as follows:
Here is a small example:
3.2 Automatic translation of simple embedded assembly
Rule-based automatic translation refers to the identification and transformation of corresponding code by some feature matching rules.
1. Instruction operand mapping
2. Assembly statement feature Analysis (Binding Step 1)
3. Rule base rule matching (binding Step 2)
4. Translate code according to rules
First of all, the assembly statement is syntactically parsed, after which each feature can be obtained. Then feature rules are matched in the rule base, that is, they are matched with the grammatical structure. When the rules are converted, the corresponding grammatical structure is also converted.
3.3 Automatic translation of complex assembly
For complex assembly, the number of instructions is large, the syntax structure is complex, ever-changing, there may be external references and other problems.
1. The combination of instructions cannot be exhausted
2. Assembly module references external variables
3. Eliminate call differences between platforms
4. Peer-to-peer translation of SIMD instructions
The “W + ii” design structure above plays an important role in complex assembly translation. The next “I” carries out SIMD direct translation.
3.4 Compilation and translation examples
Translate x86-64 to AArch64 with one click and provide two alternatives, the old path can still be executed on X86 without any problems, and the new branch created can run on the Kunpeng platform. Once the code is generated, you can replace the old code with one click. Provide a main function, which can be thought of as a driver, and the test driver can be run on both X86 and Kunpeng platforms. The results are consistent. Keeping X86 compatibility, code running on different platforms can remain 100% compatible and correct.
Simple compilation translation:
Complex assembly translation:
Oh, my God, this is so fast. Friends, download and use it as soon as possible, one-click translation, and transfer your application to kunpeng platform. Washed up!
Experience design
Kunpeng DevKit is powerful and elegant to use. Super convenient one-stop solution to help developers speed up software development, migration and power upgrades. One-stop development suite, whole process efficiency improvement. One-stop migration, compilation, testing, analysis.
4.1 Application Migration Experience Design
Developers can be broadly divided into three categories based on personal experience and habits:
Less experienced users, who rely heavily on automation, expect fully automated migration
Have some experience, reasonably rely on automation, want to be the main migration process
Experienced, shallow degree of automation, their own purely manual migration
Kunpeng DevKit designed a large number of product experience layers for the above three different types of users.
The first group of developers, through the full automation capabilities of the system, achieves imperceptibly migration. Provides dynamic binary translation tools that automatically translate X86 instruction sets into Kunpeng instruction sets. The process is transparent, fully automated, and requires no additional work to be applied to the Kunpeng platform.
The second group of developers want to participate in the entire application migration process and understand the translation process. Kunpeng DevKit automatically identifies problems, provides suggestions to assist decision making, and provides replacement code — key modification for a minimalist experience.
Third, there are developers who believe that automated refactoring is a black box and cannot be trusted. In this regard, DevKit provides transparency and other things to convince users that the tool is safe.
4.2 Performance Tuning Experience Design
When designing the architecture, people’s thoughts and behaviors are integrated into it, so that the use of tools is more natural and more consistent with human cognitive thinking, from the whole to the local, from the macro to the micro.
4.3 small example
A few small examples can illustrate the above experience design. One is hardware-tuned 3D visualization, based on cognitive model, performance model minimalism, mapping from the physical world to the virtual world.
The second example is a visual visualization of process/thread congestion during performance analysis.
In the system design, run a large amount of information visualization, at a glance, clearly visible, elegant experience.
Five, the summary
No need to say more, a diagram flow summary!
Kunpeng DevKit products are excellent both in terms of design and functions. It can be found that For the development and growth of Kunpeng, Huawei has put a lot of effort into this product. I hope all the developers can try it more, maybe Kunpeng is more powerful, friendly and convenient than you and I imagined.
“There is a fish in the Northern Sea named Kun. Kun big, I do not know its thousands of miles also. Into a bird, its name is Peng. The back of peng, I do not know its thousands of miles; When he flies in anger, his wings hang like clouds of the sky.”
Click to follow, the first time to learn about Huawei cloud fresh technology ~