Ali Mei
DataWorks is independently developed by Alibaba, which supports 99% data business construction and governance in Alibaba economy and is used by tens of thousands of data development and algorithm development engineers every day. From 2010 to the current version, it has undergone many technological changes and architecture upgrades, and also left a lot of historical baggage. Technological innovation and business development complement and constrain each other. There are slow demand access, code affecting the whole body, complex environment and other problems, chronic. The previous iterations did not fundamentally upgrade DataWorks, but merely improved performance, optimized engineering structures, and reduced repetitive code, rather than leading to a fundamental technological revolution.
This paper will discuss how to change the practical problems of DataWorks platform through the current hot micro-service architecture, and explore a feasible way to change the technical architecture from the complex engineering
Let’s start by talking about the pain points that DataWorks is currently experiencing as the driving force for technological change.
1.1 Heavy historical baggage
First want to bring all sorts of problems is why historical legacy, DataWorks multiple versions synchronous development in history, technology stack change many times before and after the end, once launched, it is difficult to scrap, an exposed API, is likely to be developed five years ago, but still have business in dependence, even these ancient, head of the business are normally I can’t find it. When our service is running normally, no one pays attention to it. Once offline, several users may come out of nowhere to complain. The same is true for the functions on the page. Sometimes it is just a bug introduced in the development of some students in the past, but it also becomes the truth because of our large user base. Had been hidden deep in the history and are function point has its own fan, once ignored by our development accidentally lost, and do will have complaints and work order, not the kui is a dental laboratories honed so DataWorks platform big data platform of benchmarking, simple interface hides under the function point of countless nuanced. If you want to recreate a data development platform that has been proven (tortured) by countless data development engineers in alibaba’s economy, you need to think about what has happened to our platform in the last decade.
1.2 Complex software and hardware environments
DataWorks facing environment across alibaba economies are complex, and its rare for a hybrid cloud (i.e., proprietary cloud) this private independent and closed environment, triad version after we must give up depend on play in mature middleware system, only to find the same in all three conditions of technology to support. Therefore, many of the dependencies that are missing in a given environment, if we cannot solve them by switching them on or off, or if we judge their complexity is not high, will be solved by self-development or relying on some open source systems. However, all kinds of network environment problems on the public cloud almost rely on human flesh to check, from the daily high number of questions we can see how much manpower we are affected by environmental problems. In addition, the influence brought by the sino-US trade war was also transmitted to the DataWorks platform recently. The operating environment needs to match domestic chips. Our process should not only run on X86 instruction set, but also run on domestic chips based on ARM instruction set. At the same time, in order to meet the purchase requirements of some small and medium-sized enterprise users, agility is also what we need to tailor the design.
DataWorks platform although large and complex, but in a variety of more complex software and hardware environment, it also needs to be flexible and light, easy to disassemble and assemble at any time and anywhere, in order to meet the expectations of users in different business scenarios. Because picky users have always wanted the best solution for the least amount of money, DataWorks’ competitiveness clearly lies in its flexibility and ability to evolve.
1.3 The whole body is affected by a single action
The complex relationship between projects has always been the traditional Service Oriented Architecture (SOA) : Service Oriented architecture (SOA) has developed to a certain scale, no matter how reasonable the initial design, how clear the field, once experienced the accumulation of needs, scale expansion, personnel change, will gradually face the problem of blurred boundaries. The most typical example is RESTful apis between individual services. Often, the Schema of these apis can only add elements, not subtract them, because the dependent does not know how many other services are relying on the interface. Maybe one day a service will wake up from its long sleep in the server. It turns out that your API’s Schema has changed, and you’ll be called back to fix it. This kind of problem is more prominent in the framework of the separation of the front and back ends. Therefore, in order to reduce the impact of schema changes, some front-end students simply let the back end pass through a large string. Even if there are private goods inside, the page will not crash inexplicably. DataWorks platform is also going through this process, after the development to a certain size, each function changes should carefully, for fear of unknown impact to other modules, or sometimes we need to put a lot of time into the effects of changes in research clear, even if such obsession, are likely to have the net, the final close, A single failure order can wipe out all the work you’ve put into it.
1.4 Requirements change and are frequently released
In addition to the problems of engineering architecture, there are also problems caused by the cooperation of many developers. Gitlab helped us solve the problem of code conflicts between versions, but not in the product release cycle. When multiple requirements are released to the public, especially in hybrid cloud, large releases need to be produced on a monthly basis, we need to quickly launch new features while packaging these features into the proprietary cloud version in a similar way to the waterfall model. The release rhythm of missile, public cloud and hybrid cloud is completely different, and many features will appear in different version iterations according to different rhythms. The circuit breaker mechanism in the past has intensified the risks caused by concentrated release in the window period. In a single service in SOA, M features developed by N developers are released online according to what combination of intervals, so that the release pace is neither too frequent nor the version span is too large due to the lack of release frequency. If these M features are dependent on each other, the contradiction between the increase and decrease of release frequency will be further amplified.
1.5 Problems caused by internationalization
Time zones, daylight saving time, languages, habits, localised ICONS, and so on. For DataWorks, a platform with 20 regions around the world, it’s a big deal. Our team has accumulated a lot of experience in internationalization, and we open source these excellent experiences: github.com/alibaba/rea… .
1.6 Dependent Coupling
Based on the Starter of SpringBoot, we improve the reuse rate of the code, and the careful design of the Starter, to ensure that students do not need to repeatedly step on the pit by different students. But this is after all the common dependence of different projects, when the Starter hidden bug, all the use of these dependencies will be affected by the project, or even a student accidentally modify a widely dependent Starter when the bug is written, it is possible to trigger the system avalanche.
1.7 Clunky grayscale mechanism
We know that the search engine in different algorithms to find the best solution, is to fill these algorithms into different buckets, and then drainage through these algorithm buckets, through the comparison of various indicators, the best algorithm will be selected. This is a gray scale mechanism based on the architecture design, and does not need human intervention to guide the flow. Different searches from the entrance naturally flow through different algorithm buckets. With massive access, the optimal solution is bound to emerge. But now under the SOA architecture of gray often rely on early designed switch, DataWorks was so, when we need to verify some function if there is a problem, the traditional way is to find the front classmate, let front end design a switch mechanism, the function of the filter part of the user to enter a new design, after a period of trial run and adjustment, Gradually solve the problem, while the impact on users can also be controlled in a relatively small range. But this is clearly not a repeatable, arbitrary, natural grayscale mechanism. The artificial intervention and the design and development for gray scale make the cost of gray scale remain high. Sometimes our students even ignore gray scale verification because they want to avoid trouble. When the function we want to verify through gray scale is very local, or the user is irrelevant, the workspace is irrelevant, or we do not know which users will use certain functions, gray scale mechanism will fail in the traditional architecture, even if we want to design, but also have no way to start.
For single service under SOA, gray scale cannot rely on architecture design, but the Gateway cluster of Alisa, the underlying scheduling service under DataWorks platform, can also realize the gray scale mechanism based on architecture design due to the large number of machines. Among hundreds of gateways, we can take out a small part and deploy the new version to be verified. After a task is delivered, compare different versions to find potential problems. But this is not the norm at the back end of the DataWorks platform, where almost all individual services are not deployed on so many machines, and therefore it is almost the state of affairs for most SOAs, facing a grayscale mechanism based on human intervention rather than architectural design.
1.8 Uncertainty of external associated services
External associated services are complex and changeable, unreliable and unstable, and may break down or network interruption at any time. Even external service upgrades forget to inform us, which leads to frequent failures. This is especially true for data integration, an application that moves data across dozens of engines and thousands of database instances. In order to deal with the uncertainty of external service, we will have many rely on their own application design robustness is particularly outstanding, but it will add our own code logic complexity, occasional issues will also be the robustness of the code mask, problems if not handled in time, will gradually accumulate, when all the conditions of the malfunction, ginny, One major P1 fault occurs at a time.
1.9 Shortage front end
The DataWorks development platform also faced a shortage of front-end staff, which is a common problem faced by the development teams of rich interactive products. The front-end is subject to different interactions, different styles, different businesses, as well as the differences in the understanding of business among the r&d students, so that the reusable front-end components are extremely limited. Out of the vast array of front-end libraries, components, and styles, there are very few that can be cost shifted. In the design mode of front and rear end separation, based on the mainstream front-end framework, compared with the historical front and rear end integrated design system, the user experience is improved. However, this is based on the front-end development students’ hard work, a little bit of style adjustment and interaction results, there is no shortcut. And we can only hope to reuse the front-end development of students’ RESEARCH and development results, so that each design does not become a one-time use of labor.
DataWorks R & D platform many functions cover the daily work of data development engineers, users in our platform for years ata desk work, the design of some function points have their own sense of body, and these body sense is our platform r & D feel. PD and our UED collect requirements, use them, and feel for themselves. However, after all, they are not specialized in data development itself, so it is hard to feel the subtle frustration of data development engineers after long-term use. When it comes to the segmented vertical business market, how to use our platform in different industries varies greatly. For finance, banks, governments, large state-owned enterprises, Internet companies, traditional enterprises, private enterprises, education, etc., their usage is completely different, and some industries even do not know what to do after getting DataWorks platform. Users’ needs vary widely and their minds are in different stages.
Therefore, in the fields that we have no time to take into account, the delivery team or company in front applies the DataWorks platform to specific industries, and then brings the exclusive needs of the industry back to our PD for analysis. The platform itself will also open up some apis for the front team to package into products for customers in specific fields to help customers solve practical problems.
New product planning is still under constant formulation, the engine team to design their own products also need to reduce the difficulty of users to get started on the DataWorks platform, if only DataWorks platform development students in accordance with the schedule to gradually complete these access and customized requirements, the platform is doomed to sustainable development and growth. Therefore, from the front and back end architecture level, from the countless cooperation and competition scenarios, we urgently need to carry out a technology revolution for ourselves, completely free the production workforce from SOA, introduce more user side RESEARCH and development force, and help the platform to develop in a more sound direction.
Change in any technology is gradual, in line with the philosophy of Rod Johnson, the founder of Spring. He introduced the “Wheel theory,” or “Don’t Reinvent the Wheel,” which is a Western proverb that says, Don’t Reinvent the Wheel. In addition, after long-term practice, he summarized and clarified the idea of evidence-based architecture in his book, that is, “there is no best architecture, only the most appropriate architecture”. This is the beginning of the evolution theory of the architecture world, which means that our architecture is constantly evolving to meet changing business needs.
In traditional SOA, services tend to be stable and centralized, individual services are equal to each other, or at least similar in common structure, and services depend on each other through network communication. Each single service in SOA may be developed by multiple developers working together, so replacing any single service is a big break. When there are big stack changes, such as upgrading from WebX 3.0 to SpringMVC, upgrading from SpringMVC to SpringBoot, The human cost is huge, and the upgrade cycle is often in years, not to mention the near-impossible task of replacing a web service already designed in J2EE with Go or Python’s Django framework. It also means that once we get into the technology system, it’s hard to get out of it, let alone talk about the evolution and evolution of the architecture.
In traditional SOA, the way to improve engineering efficiency is extreme code reuse. All reusable code is extracted into a binary package and designed into a class library for different individual services to rely on. Configuration is getting more complex, but it does reduce the need to reinvent the wheel.
Another approach, like the one I took when trying out new architectural designs in 2015, is abstractions that can be abstracted into a binary package and not automatically generated by an automated code generation tool. A set of drills mature SOA architecture engineering, from the directory structure to configure assembly is almost stable, even if have change, also is in accordance with the MVC three-tier architecture for fine-tuning, so we can put some unable to abstract, or contains a complex logic, need to be flexible to adjust the code generated by automatic code generation tool, and the function of redundant code as much as possible, Let developers subtract as much as possible from the generated engineering code or modify as little code as possible according to the business logic, thus improving the cleanliness of the project and development efficiency.
In addition to code reuse, and service reuse, we designed a few services, centralized monomer is used to complex logic or start a slower or need cache, etc. So some functionality encapsulated in these core services, further reduce the service around the center of the size and complexity of the other application, of course, it also may introduce a single point reliability issues, However, to ensure the stability and reliability of core services, it is not difficult under a certain scale of service volume.
Even so, the efficiency gains of SOA are limited, and rapid and efficient initial project establishment does not mean that such efficiency will be sustained over the long term. The pain points described above gradually show up and there is no effective solution. Because the developers uniformly use the same set of technology stack or even the same set of engineering directory tree structure, although they have a better understanding when they cooperate, they also eliminate the racing mechanism of cutting-edge technology in the team. Research and development students study how to squeeze out the final efficiency and performance under a set of gradually ancient and backward framework, but often ignore that maybe a change of framework, a change of technology stack or even a change of language will bring about qualitative changes. Therefore, following the idea of “there is no best architecture, only the most suitable architecture”, we began to think about how to shift DataWorks technology direction to flexible microservices architecture.
3.1 Microservice architecture
When it comes to micro services, it is naturally associated with Cloud Native, which includes the following three points:
1) the conversation
Development and operations are no longer separate teams, but you are part of me, and you are part of me.
2) Continuous delivery
Continuous delivery means frequently releasing new features to users without affecting their use of the service.
3) Containerization
The advantage of containerization is that there is no need to care about the technology stack used by each service during operation and maintenance. Each service is encapsulated indiscriminately in a container and can be managed and maintained indiscriminately.
A cloud native environment that meets the above three points is a natural fit for microservices. When a product grows to a certain scale and has enough volume and a large number of sub-products that the interactions and dependencies become increasingly complex, the microservice architecture will bring obvious benefits to the engineering community under such a product. Right in this paper, the basic concept of micro service do in detail and the general practice in general, these articles online books, here only to recommend two good book “micro service design (concept explanation can also previous chapters, the back part) and the micro service architecture design pattern, which is suitable for preliminary understanding, which is suitable for advanced. This article only discusses the approach of DataWorks under the new architecture.
A change in the direction of microservices architecture may not be the answer to all of the problems that DataWorks has created for many applications, but it certainly is the antidote to the problems that afflict current development models.
★ 3.1.1 Know who you are
The DataWorks development platform is a typical PaaS application, but there are also data services that make it to the SaaS layer. The pain points of traditional SOA, and the customization capabilities that we will open up to customers in the future, need to be addressed with microservices architecture, gradually shifting the focus of r&d from PaaS to SaaS.
When we adopt K8S containerized micro-service architecture, developers can integrate the basic OpenAPI opened by DataWorks platform into our independently developed micro-service platform, and also integrate API of external applications to conduct data integration and business logic writing in micro-service. The result is a set of apis accessible at the front end of the platform for use by front-end functional modules. In the process, we can address some of the pain points mentioned above.
★ 3.1.2 Resolve pain points
Taking the historical baggage as an example, we can gradually replace the functionality contained in the aging, obsolete SOA monolithic services, cutting the rapidly deteriorating pie into small, low-coupling slices and achieving gradual replacement rather than a long cycle of one-time whole replacement. Old projects do not need to develop and update, only to maintain the basic operation, to avoid the overall replacement of long cycle, many failures, difficult to roll back problems. Moreover, we can easily verify whether the new “small cake” has successfully replaced the functions of a small part of the “bad big cake” through canary release, remove the small cake with problems quickly and timely through blue and green deployment, and verify which small cake has better performance and more reasonable design through ABTest.
Taking personalized demand as an example, when we open the micro-service platform coupled with DataWorks business, business teams with self-research ability (such as data/report development team) can quickly design their own needs to the back end of our DataWorks platform with the help of micro-service design. At the same time, we also set aside a “private site” (plug-in as described below) for the business team to design and develop. Engine access can also follow this mode, the next part of DataWorks module access can be more stupid, such as checker, such as powerful custom nodes, users can quickly access and use after simple development according to the document, but for more customized functions, For example, the visual table building part of ADB engine is currently being designed on DataWorks platform. Due to its high complexity, it must be developed through the front-end slot of micro-service docking (described below), so as to realize the independent self-access of complex business logic.
To describe the grayscale mechanism based on the architecture, blue-green deployment, Canary publishing and ABTest can be easily implemented under the microservice architecture. Our microservice design should be as domain-oriented as possible (of course, it is unlikely to be 100% domain-oriented). High cohesion and low coupling is still the design tenet of individual microservices. We can release multiple versions of microservices to test whether a problem in a particular domain has been improved in some way, or we can release multiple microservices designed entirely in different frameworks or even different languages to verify who is the best solution in a particular domain. With the help of architecture-based grayscale, cloud-native, this is all very efficient and reliable, and even if problems occur, problematic microservices can be quickly taken down without further impact.
There are other pain points, too, not to describe microservices-based solutions, such as the all-encompassing problem, which will disappear after the DataWorks platform is fully built on microservices. Each student can be in charge of multiple micro-services that are small enough for the field. When an interface needs to be redesigned, it is not necessary to replace the old interface immediately, but to redesign the corresponding micro-service under the interface at low cost. After the traffic is switched to the new micro-service, the old version of micro-service can be gradually offline. For example, the coupling introduced by Starter under SpringBoot will be decoupled through service discovery under the microservices framework, instead of coupling associations through code-level dependencies.
★ 3.1.3 Evidence-based framework
Going back to one of the things we haven’t talked about in this article, what exactly are our microservices? Microservices of course don’t have to be small; it’s still an SOA, but much lighter and domain-oriented. Take the robot factory as an example, the product has a function that allows users to configure some intents to jump to the response logic designed by the users themselves. This part is embracing the micro-service architecture. After the completion of online, users can design their own micro-services according to the input, regardless of language. This also prevents the user’s microservices from crashing in the event of a bad design, which will not affect other running microservices. The robot factory scenario can also be designed as FaaS, where users can write functions to complete their own response logic and measure usage by visits.
The micro-service platform designed by DataWorks team fully embraces the popular Service Mesh, namely Service grid, through which part of the work is encapsulated in the front micro-services. These system-level micro-services and the micro-services designed by developers run in the same POD, making the micro-services designed by developers simpler. A Mesh is more like a HandlerInterceptor or Filter in the Spring framework. AOP oriented developers are good at developing interceptors and filters in projects. In the microservice framework that integrates Service Mesh, It is easy to use system-level microservices to do some of the work of traditional interceptors. Such as login jump, permission control, service discovery, such as traffic limiting, monitoring, log aggregation, and so on.
We also designed DMF (DataWorks MicroService FrameWork) FrameWork groups in different languages to help developers quickly get started with the development of microservices. “There is no best architecture, only the most suitable architecture.” In the future, we will also open up the development design of DMF, allowing more business parties to contribute their own “most suitable microservices framework.” In order to better support DevOps, which is very relevant to our business, we developed DMSP: DataWorks MicroService Platform to manage the deployment and release of microservices, as well as service governance and other operations and maintenance work.
3.2 Front-end system coordination
The plug-in mentioned above refers to the XStudio plug-in scheme designed by our front-end team. Plug-ins combine with back-end microservices to form an overall solution. Based on this, the front-end team of DataWorks platform hopes to explore a methodology to improve the efficiency of front-end research and development. The XStudio plug-in is implemented based on single-SPA and the Qiankun framework. The framework provides important features such as multi-instance pattern, slot mechanism and visual plug-in choreography, which further improves the efficiency of plug-in development. The schematic diagram of the whole plug-in is as follows:
Front-end students based on XStudio plug-in design page, leave a good slot, slot can be inside a button, or any other type of component, the component binding behind a service, the contents of a slot on the inside, we can replace, micro back-end services along with the rapid assembly of page functions, so as to realize a development in use. At the same time, the content in the slot can also be provided by the business development team, so the business development team only needs to design such a plug-in integrated with the front and back end to place in the front slot, to achieve customized development of personalized requirements.
In traditional plug-in design, developers either provide a two-party package that follows an interface protocol, or provide a set of protocol-compliant apis that are exported forward by the SOA architecture. This creates problems that either intrude on SOA services, affect the overall security and stability of SOA services, are limited by the programming language, or have no flexibility, and microservices & plug-ins perfectly solve these problems.
To take up more of the page space design, we can set a vast area into a replaceable component, such as the editor is part of the above, let the user to replace the area of the page content, with the back end of one or more micro service associate embedded into the front page of DataWorks, convenient business team to realize more complex custom business logic. The current ADB visual table building part is designed to follow this approach, developed by the ADB engine team and integrated into the DataWorks platform.
And at the same time, the front-end system is important to the study of data monitoring and alarm, we design the various dimensions of reporting and monitoring indicators, whether your business or write in the components of the external business team, don’t write one line of code, can be through the “automated complete quantity buried point technology”, to observe and understand the usage of components.
3.3 Running of plug-ins
At present, we have combined some front-end components based on XStudio architecture with back-end microservices to achieve plug-in encapsulation, such as Terminal, DWEditor, directory tree, inspector and other plug-ins. Take Terminal as an example. After the plug-in design is completed, it can be inserted into different studios to connect with different engines, and can automatically pull up or destroy container instances according to the user’s usage, thus saving running resources.
The Terminal plug-in is connected to multiple engines, and no additional development is needed for the engine for both front-end and back-end micro-services, thus improving the development efficiency. Plug-in packaging design can not only save development resources, but also realize the purpose of using a set of microservices for multiple applications. Elastic orchestration and automatic scaling mechanism ensure the performance of services, while avoiding wasting machine resources.
In addition, we can build some implementations of SaaS based on microservices architecture, such as FaaS, BaaS (Backend as a Service), and BFF (Backend For Frontend). Taking BFF as an example, the application of BFF in mobile terminal DataWorks can reduce the network consumption of H5 page on mobile terminal, and provide interfaces provided by multiple back-end microservices to the mobile terminal through Gateway assembly to realize the aggregation of microservices. If SSR is made through BFF (Server Side Render: page ishomogeneity, equivalent to directly rendering HTML to output to the browser on the Server Side), it can further reduce the consumption of rendering performance on the mobile end.
★ 3.3.1 DataWorks microservice platform
The integration of the front and back end based on the DataWorks business plug-in, is also an important reason for us to adhere to the research and development of DataWorks MicroService Platform (DMSP: DataWorks MicroService Platform). DMSP gets through the binding association between the release of front-end components and the back-end microservices. Through technical means like Swagger, the front and back end can quickly become a business plug-in after deployment. Allow both the front and back ends of the team to implement DevOps in DMSP, releasing new features to customers on a continuous delivery basis.
It is especially worth noting that DMSP is also targeted at three environments, namely, in-missile, public cloud and hybrid cloud. After the plug-in is developed, we need to continuously deliver it to the environment of up to 20 regions of public cloud through DMSP, and realize the unified packaging and deployment of microservices in private cloud. In addition, DMSP should also let students develop plug-ins as far as possible to the complex external deployment environment without feeling.
In the future, we expect that most of the page content of the whole DataWorks platform will be based on plug-in design, so as to solve the problem mentioned in the previous pain point: “flexible and light, easy to disassemble and assemble at any time and anywhere”. Architecture drives not only development patterns, but also the blue ocean of the entire product.
The important premise of tectonic ecology is to have competition and survival of the fittest. At the same time, tectonic ecology is to construct a technological system that can evolve and apply the theory of evolution. As the sublimation of “evidence based architecture”, microservice architecture is obviously superior in evolution. Evidence based architecture is a top-down technology evolution path, while microservice architecture is a bottom-up technology evolution path. Containerization is language independent and framework independent, and each microservice is encapsulated indiscriminately in a container, so that multiple microservices can be developed for a function, similar to the optimization mechanism of algorithm bucket, and the optimal solution can be selected from these microservices that complete the same function. Ideally, applications can self-assemble best practices over a period of evolution without the active intervention of the superarchitect. Situation, of course, this is just a theory, we are in the real world by many external factors of interference, the actual situation, is limited by the developers of technological literacy, external dependence is uneven, even is restricted to the KPI of the guidance, will make this ideal of best practices can not be reached, but the service architecture gives we can through team collaboration and hard work, To get infinitely close to the final solution in theory.
Under the micro-service architecture, the vertical business customized development mentioned above will also become a possibility. Industry-oriented delivery teams can make use of the plug-in capability provided by the DataWorks platform to customize an intelligent R&D platform completely adapted to the characteristics of the industry for customers. Then create a dynamic innovation ecosystem on the DataWorks research and development platform to provide customers with more colorful choices. Architecture will drive the survival of the fittest throughout the ecosystem, thus evolving in a more competitive direction.
Our DataWorks RESEARCH and development team is also looking forward to realizing the win-win cooperation mode within and outside the bomb on this framework, and promoting the product formation of cloud intelligent business group.
The so-called architecture is not only a technical thing, but also to provide a complete set of guidelines for the organization of manpower allocation. After the application of the microservices architecture, it became clear that the responsibilities of the DataWorks r&d team could not be limited to the daily requirements development. As an advocate of microservices architecture, I also thought about the responsibilities of the front and back of the team while promoting the architecture. First of all, a framework group should be formed at the front and back end to influence the evolution of the whole architecture by guiding the design of front-end components and back-end microservices. There should be discussions within the team focusing on domain-oriented design, analyzing whether each plug-in is domain-specific or not, and at the same time, there should be students with checks. Review the plugin design submitted by the second party (developers outside the team) and the third party (delivery team outside the team or industry developers from the customer) to prevent bad applications from destroying the system construction.
At the same time, due to the language independence of micro-service architecture, it also smoothen some technological gaps. Many front-end students are good at NodeJS and can also show their skills in micro-service design, which makes the technical communication between the front and back end more tacit. We also have a special product in our product line: AppStudio, specializing in the research and development of WebIDE, can be combined with AppStudio in the future, whether it is the data export provided by data services, or the function in FaaS, or the development of micro-services themselves, independently developed by users, can completely not separate from DataWorks global big data platform, From data development to report design, and then writing business logic through micro-services, to achieve the goal of data output, one-stop to complete the customized needs of users.
The figure above is an idea of the division of responsibilities of the team in the future. Under this organizational structure, the front and back r&d students will make a set of combination fist, directly attack the pain points, help users overcome technical difficulties, and achieve ecological prosperity.
What is the future of technology and architecture? In my ideal, software engineering research and development technology should be a never-ending and borderless, and increasingly intelligent field. Many of DataWorks’ products are already moving in the direction of intelligence, such as an intelligent programming plug-in based on VSCode that uses Markov algorithms. The future of r&d teams largely depends on the architecture of the system. We should encourage innovation and exploration of technological frontiers and boundaries, and should not create too many artificial rules that limit the imagination of thinking. If one day, intelligent programming finally starts to replace human development, then by changing the design of the architecture, developers will surely find new responsibilities in the new architecture.
Our technological vision should be to build mature and constantly evolving engineering systems for a changing world. Look to the future, embrace change, for incalculable value! Welcome to identify the qr code below, learn about the team information, join the flying big data communication group and DataWorks products to communicate!