Alibaba Cloud DataWorks – The sequel
An overview of
Because I have been doing front-end work related to Alibaba Cloud Flying Big data platform for a long time, I have been thinking about a question: “What is the difference between the front end of big data and the front end of other business teams?” Specifically, in the wave of big data and artificial intelligence, what is the impact on the development of front-end technology?
For example, the team is in charge of ali Cloud Flying Big data platform. Since the first line of code was written in 2009, it has been the 11th year of Ali big data development. I was joined in 2011, ali, after I have been responsible for large data related to the front end of the work, basically participation in ali, the history of development of the vast majority of big data process, looking back now, I’m glad myself in a period of historical change into the line, more lucky to witness the birth of some of the landmark data products, and its change of front end technology.
If we think of 2010 as the first year of big data Web product adoption, it is an interesting year. Why?
Looking back at the history of the front end, there was a big wave of technology change around 2005, which was the transition from Web1.0 to Web2.0.
Before that, the front end was more of a static display of pure content, like the apple and Yahoo websites of that time.
After the front end began to gradually make the dynamic web pages; One of the most important historical signs of this is Gmail’s adoption of new technologies like Ajax.
Around 2010, a variety of big data applications entered an explosion period, and many well-known applications of Ali basically emerged during that period. Now, looking back at the history of that period, one of the major reasons is that with the development of the Internet, especially after Web2.0, there was a huge explosion of data growth.
The chart below illustrates this trend. If Web applications were more about “generating” data in the past, how to better “present” data after 2010 has taken on a new dimension, and many front-end technologies have opened up a new chapter.
The influence of data intelligence wave on the development of front-end technology will be discussed in three main lines based on my own practice, namely data visualization, software webization and interaction diversification.
Data visualization
Under the wave of big data, one of the most obvious features is the exponential growth of data, which can be seen from the figure above. The subsequent challenge is how to present data in a more vivid and interactive way, which is commonly referred to as “data visualization”.
Going back to the technology itself, the biggest impact of data visualization on the front end is probably the growth of SVG, Canvas and WebGL.
The above picture quotes Yuying teacher in “Visualization with Yuying” about the relationship and difference between web front-end and visualization
Among them, in addition to the upgrading of the underlying technology of the browser, a large number of excellent works have emerged in the upper visualization library and visualization application, among which the outstanding ones include:
Open source technology component level:
-
AntV
-
Echarts
-
HighLights
-
.
Heavy data visualization products
- Aliyun large screen visualization product DataV
- Quick BI of Aliyun
- BI analysis tool Tebleau
- Feature domain analysis products, such as Plantir
There are lots of opportunities in different scenarios in specialized segments such as geography, security, new retail, etc
For example, DataWorks products of ali Cloud, a one-stop big data development and governance platform, have interesting visualizations such as DAG for process choreography, graph analysis, and blood analysis of data.
Software, Web,
You should notice a phenomenon recently that is: Web systems are becoming more and more complex, many of the original desktop interoperability applications are gradually “pan-web”, even many applications are the first version of Web technology.
The pan-Web here can be divided into two kinds of manifestations:
One is to directly use front-end technology to make desktop software, of which the landmark event is the rapid development of NW. JS and Electron since they started in 2013. The familiar IDE VSCode is a typical representative of this; Ali’s desktop version uses Web technology heavily in the UI layer
The other is to do it directly on the Web, for example you’ve been seeing various Web’X’ systems lately (Google Docs)
The driving force behind this is, first, with the gradual unification of browser correlation, cross-terminal can be realized more easily with its technology. The other is the rise of cloud computing big data, especially the storage and computing power of cloud has gradually broken through the original performance boundary of local PC, thus reshaping the original entrance of human-computer interaction.
I want to focus on the second point. To explain this logic, I have to talk briefly about the development of computers, from the world’s first universal computer “ENIAC” with an area of 170 square meters, to the personal PC of the Era of Apple and Microsoft, the iPhone and Android in the mobile era, To large computing clusters in the age of cloud computing.
For developer tools, before many software, many are local, because it often use local PC computing power is enough, but the big data calculation under the scenario of local work force is certainly not enough, it is dependent on the cloud computing cluster (in terms of our ali flying big data platform, we have 100000 sets of computing the size of the cluster). How to use these computing power in a more convenient and flexible way on the user side is the focus of our front-end, and this is the original software architecture can not allow you to customize, or the cost of customization is very high (sometimes even more than the cost of making a new set), so many systems will choose to start a new version
WebIDE and WebExcel are typical examples of this. We are responsible for alibaba Cloud Dataworks.
Dataworks is designed and developed according to the idea of cloud native from the beginning. The back end needs to provide powerful computing power through cloud computing to replace the original local computing power, and the front end needs to achieve a more sophisticated architecture design to correspond to the increasingly complex interaction ability; Specifically for our application, it includes but is not limited to
-
Architecture level:
-
State management
-
pluggable
-
.
-
Overlapping components
-
Editor
-
Form/Excel
-
Tree
-
Logivew
-
.
Interactive diversification
In recent years, under the vigorous development of data-driven artificial intelligence, especially in image recognition, speech recognition, natural language processing has achieved great breakthroughs, so that the front end of the new interaction has also achieved great progress.
UX
A Graphical User Interface (GUI) for UX is transformed into a Graphical User Interface (XUI). Users can not only operate the Graphical User Interface (GUI) with a mouse and keyboard, but also provide new forms of interaction through facial expressions, body movements, and voice interaction.
The figure below is our practice in some man-machine dialogues and interactive games in Ali Cloud ET around 2016. For details, please see here
This round of technological change has two big, macro backgrounds
1. The third wave of AI technology
Around 2010, with the maturity of deep learning technology, the improvement of computing power, and the wealth of big data accumulated in the Internet era, artificial intelligence technology began a Renaissance that was quite different from the past. Respectively in speech recognition, image recognition, natural language processing and other related technologies to obtain fundamental breakthroughs.
For example, from 2012 to 2015, in the ImageNet competition (ILSVRC), which represents the cutting edge of computer-intelligent image recognition, the ai algorithms competing in the competition have made leaps in recognition accuracy. In 2014, deep learning-based computer programs outperformed the average human eye when it came to recognizing people, animals, vehicles or other common objects in pictures.
The picture below, taken from Artificial Intelligence by Kaifu Lee, illustrates this trend
- For the front-end, another prerequisite is the maturity of WebRTC (Web Real-time Communication) technology, which was open source on June 1, 2011 and included in the W3C recommendation standard with the support of Google, Mozilla and Opera. With it, the front end can easily handle images, video, voice and other content. You’re seeing a lot of interesting interactions right now that rely on him at the bottom
DX
In front end developer (DX) : Intelligent means can improve our r&d efficiency and experience. From our own implementation of Ali and Ant, Imgcook (D2C: Desgin to Code), intelligent Code prompt, intelligent visualization AVA and front-end machine learning Pipcook are all interesting attempts.
conclusion
Above is my about data in practice under the wave front some thinking on the development of technology, of course, the front-end technology can get so great progress in recent years, in addition to data intelligence, other trends (such as mobile interconnection, 5 g, IoT) also profoundly affected the front-end technology, but these would not be in the scope of this article, have a chance to talk with you.
I have always liked an idea mentioned by Wu Jun in his book The Intelligent Age: “2% of the people will control the future, become them or be eliminated”. It is expected that all front end students can find their own positioning in this wave of intelligent data.
Write in the last
If you are interested in this field and want to work with Alibaba in big data and artificial intelligence, please feel free to send me a private letter or resume at [email protected]. We work together, we do something meaningful, we hire people for a long time