Abstract: This article will share with readers how huawei cloud is to achieve the speed of “one cloud a day” (soft installation) to copy the full stack cloud platform to the customer’s data center in large quantities at low cost, for customers to build a digital transformation of the highway.
This article is shared from huawei cloud community “[Huawei Cloud Stack] [Big Shelf Presence] Issue 9: Efficient Cloud Construction, one Cloud a Day behind the story”, author: Huawei Cloud Stack engineering architecture field architect Wei Shijiang.
Abstract
The strategic goal of “accelerating digital development and building a digital China” has been clearly stated in the 14th Five-Year Plan and the outline of the long-term goals for 2035. Cloud computing with “all services” concept, to provide users with the like water and electricity can be on-demand access to computing, storage, network resources and a large number of mature, big data, the application of AI control such services, has become a digital economy, the digital society, the government’s important infrastructure, digital transformation, government and enterprises is a vital part of the process of business innovation. As the base of the digital economy, cloud computing was clearly included in the “fourteenth Five-year plan” program “Key industries of the digital economy” list.
Cloud computing while greatly speed up the efficiency of the business innovation, but because of the inherent complexity of the cloud, the initial construction process generally involves the computer room, IT equipment procurement, equipment configuration, cloud platform software deployment, and other links, construction cycle is long, has the potential to be the important factors that affect digital transition floor rhythm. If “digital economy” is a large shopping mall, then the cloud platform is the shopping mall building and its ancillary facilities, when the building is built, directly affects the shopping mall when it opens.
This article will share with readers how Huawei cloud achieves the speed of “one cloud a day” (soft installation) to replicate the full-stack cloud platform to customers’ data centers in large quantities and at low cost, and build a digital transformation highway for customers.
Cloud platforms are inherently complex
Building a cloud is a huge undertaking, mainly because of the inherent complexity of the cloud platform itself.
Before we discuss cloud platform installation and deployment, let’s take a look at some simple software installation costs and implementation skill requirements:
As can be seen from the above table, when the delivery of an application software with the same complexity as the enterprise ERP system, the skill requirements of delivery personnel are already relatively high, and its delivery cycle has seriously affected the business development of customers.
Let’s take a look at the overall architecture of Huawei Cloud Stack to get a sense of its complexity:
Figure 1: Overall architecture of Huawei Cloud Stack
Huawei cloud Stack is logically divided into the resource pool layer, public components, cloud service layer, and o&M operation system. Each layer is composed of dozens or even hundreds of microservices and their dependent DB, middleware, and LB components. In addition, there are complex interactions among microservices and services. The overall complexity is several orders of magnitude higher than the ERP systems mentioned earlier.
In the face of such a complex distributed system, if the traditional manual delivery method is adopted, the delivery cycle will be unacceptable, and the technical ability of the delivery personnel will be very high requirements.
To meet the challenges of batch and fast delivery of Huawei Cloud Stack, we mainly take the following measures:
1. In product development stage, building block architecture is adopted to ensure high cohesion and low coupling of a logical layer and each service unit.
2. Drive the bottom-up automatic deployment of the whole cloud through the self-developed one-click automated delivery platform.
3. Ensure the effectiveness of delivery and reduce the cost of on-site debugging through the automatic acceptance tool of cloud platform.
4. Standardized data center-level modeling is used to provide data support for subsequent upgrades to ensure long-term sustainable upgrade evolution of delivered sites.
Each of these measures is described in detail below.
Building blocks with high cohesion and low coupling
A building block software architecture is a prerequisite for efficient delivery. Imagine that you want to build a building, but all you have to deliver to the site are piles of sand, cement, steel and wood, which need to be measured and poured on site. Then the construction efficiency and quality will be extremely uncontrollable. Therefore, in the field of modern architecture, standard prefabricated parts are generally mass-produced in factories according to the drawings designed by architects, and then assembled on the construction site to quickly complete construction by building blocks.
During the development of Huawei Cloud Stack, software deliverability has been fully considered. The following measures are taken:
• Use the software modeling tool developed by Huawei to design and review the delivery model and deployment model of each service unit in detail, and determine the deployment and operation dependencies of each service unit.
• The interface between each layer is clearly defined through the layered architecture method, and the API control tool is used to ensure the stability of the API provided by each layer, thus reducing the complexity of interconnection between services.
• Extracting common components that most services rely on through domain modeling and managing them collectively in a service-oriented manner, thus improving their standardization and delivery efficiency.
• Developed a unified installation and deployment development framework and specifications to ensure uniform deployment of various cloud services from different teams.
Take the cloud host service ECS as an example. In the architectural design stage, the following deliverables should be completed:
Figure 2: Cloud service design deliverability related design
During the development phase, in addition to the development of the business function code, each service also develops the automatic deployment plug-in package according to the development specification of automatic deployment, which contains the description of each sub-step of the installation and deployment, parameter declaration, dependency declaration and related deployment scripts. Deployment plug-ins mask the intricacies of cloud services and provide a unified delivery interface externally. For huawei Cloud Stack delivery personnel, they do not need to know whether to deploy the IaaS service or ManageOne platform, which greatly reduces their learning costs and skill requirements.
Through the above measures, each service unit in huawei cloud Stack becomes a Lego block, providing a standard “card slot” and laying a solid foundation for efficient delivery
Easy to use automated delivery platform
With good architectural design, we have a collection of building blocks that can be flexibly assembled according to customer needs. However, those who have played Lego may know that for large lego models, although each piece has a standard interface, it is still a huge task to assemble it in the face of hundreds of blocks.
In order to realize the building block delivery of hundreds of microservices in HUAWEI cloud Stack, we have developed a highly flexible visual and automated delivery Platform (HUAWEI CLOU Stack Deployment Platform (HCSD)). The main functions of the platform are shown below:
Figure 3: Automated Deployment Platform (HCSD) architecture
The platform is equivalent to an automated robot building blocks. Front-line delivery personnel import software packages and common atomic capability libraries into the delivery platform, and then select services or components to be deployed as required through the visual UI. Import parameter tables according to the prompts on the page and start the deployment project. Then the task execution engine will load all the deployment plug-ins in the cloud service package, generate daGs according to the subtask dependency description defined in the plug-in, and sort topologies. Then, the installation task will be executed concurrently to complete the deployment and configuration of the cloud platform software. After deployment, E2E acceptance cases defined in each cloud service plug-in package are automatically executed and acceptance reports are automatically generated.
Based on the platform, the overall software delivery process and working hours of Huawei Cloud Stack are as follows (200 physical servers are used as an example) :
Figure 4: Huawei Cloud Stack end-to-end delivery process and man-hours
Foolproof automated end-to-end acceptance
In the traditional software delivery process, after the software is deployed, professionals who are proficient in business processes verify the software delivery. This method is feasible for simple software. However, in a complicated distributed system like Huawei Cloud Stack, it is time-consuming and labor-intensive to execute all acceptance cases manually, and it is inevitable that the validity of the acceptance results will be affected by various errors.
In order to improve the efficiency and effectiveness of acceptance, testability design was introduced in the software design stage. Testability was fully considered at the beginning of functional design and E2E use cases were designed in detail. During the development phase, these E2E test cases are codified according to uniform specifications and released with the deliverables.
The major automation E2E acceptance use cases covered so far include:
• End-to-end invocation of major apis of cloud services, such as creating a VPC, uploading VM images, creating VMS, verifying VM network connectivity, and deleting VMS.
• Automatically click the link and button of the Cloud service Web Console to capture a screenshot.
Based on the automated acceptance and commissioning tool, we shorten the acceptance and commissioning period of cloud platform from more than 3 days to hours, and automatically generate acceptance reports of Web version and Word version. More importantly, it reduces the skill requirements for delivery personnel and provides important support for mass replication of Huawei Cloud Stack.
Standardized model precipitation provides a strong guarantee for the continuous upgrade and evolution of cloud platform
Digital construction is a long-term and continuous process. As the base of digital platform, cloud platform must have long-term sustainable upgrading and evolution ability. Therefore, the scalable capability of the cloud platform should be fully considered in the development and delivery stages to provide necessary data support for subsequent maintenance and upgrade.
At present, the Stack of Huawei cloud uses ManageOne CMDB as the carrier to precipitates core metadata related to o&M changes of the entire cloud.
Figure 5: CMDB operation data model
These data are maintained by each cloud service team during the development phase, and summarized by HCSD and injected into CMDB during the deployment phase.
In order to reduce the maintenance cost of operation and maintenance data, we defined a data model verification baseline based on JSON Schema and configured access control for all cloud service code bases. Every code submission would trigger automatic model verification to ensure the accuracy of data.
In the stage of cloud service software release, each cloud service will package the data along with the deliverables in accordance with the specifications, and then HCSD will automatically write the data into CMDB.
In this way, Huawei Cloud Stack achieves one-click end-to-end automatic upgrade of the entire cloud, laying a solid foundation for the sustainable operation of government and enterprise customers’ digital assets.
conclusion
At present, huawei Cloud Stack has formed a highly standardized pipeline from architecture design to code development, and from release to delivery. In the future, we will continue to make more innovative explorations in new delivery modes, and copy the new technologies and features of Huawei public cloud to customer data centers in the first time, so as to create a highway leading to digital transformation for customers.
Click to follow, the first time to learn about Huawei cloud fresh technology ~