Brief introduction: This article will briefly summarize the FFA Keynote topics. If you are interested, you can find relevant theme videos on the FFA official website [2] to watch live replays.
The author | MeiYuan (Yuan Mei) source | ali technology to the public
As the sun rises and everything begins to change, this poem is apt to describe the big data field in 2021, and Flink will start a new chapter in 2021.
On January 8-9, 2022, Flink Forward Asia (FFA) Online Summit was successfully held. Flink Forward Asia is a conference hosted by Apache Flink Chinese Community and authorized by Apache. Flink Forward Asia is now one of the largest Apache Top-level project conferences in The country and an annual event for Flink developers and users. In conjunction with the online summit, FFA also held its first Flink Hackathon dedicated to real-time computing, with 267 teams participating, 27 of which were selected for the offline finals. In the future, Flink Hackathon will also be held on a regular basis.
The FFA conference summarized Flink’s development over the past year in terms of community development, industry influence and eco-technology evolution. On the community side, Flink has ranked as one of the most active projects in the Apache community for three consecutive years, according to core metrics published in the Apache Software Foundation’s fiscal year 2021 report. As the smallest atom in the community, Flink has more than 1,400 contributors, with an annual growth rate of more than 20 percent. In particular, it is worth mentioning that Flink’s Chinese community is thriving: the official official account of Flink has more than 50,000 subscribers, and sends more than 140 updates on Flink technology, ecology and industry practices every year. Recently, Flink community opened the official video number of Flink, hoping to let everyone have a more comprehensive understanding of Flink through more abundant and innovative forms from more latitudes. In addition, Flink community reconstructed and revised Flink Learning[1], the official Learning website of Flink that was opened last year, hoping to collect Learning materials, scenarios, cases and activity information related to precipitation and Flink through this Learning website. To make Flink Learning truly become a good helper for everyone to study and explore Flink.
In terms of industry impact, Flink has become the de facto standard for real-time computing in the industry. More and more companies not only use Flink, but also actively participate in the development and construction of Flink to improve Flink together. Currently, Flink’s code developers are from over 100+ companies around the world. Alibaba, Bytedance, Ctrip and 360 all provided strong support for the four offline meet Up held last year. This year’s FFA conference will feature 83 keynote speeches from more than 40 well-known companies from various industries such as Internet, finance, energy, manufacturing and telecommunications. From the perspective of ecological technology evolution, Flink has achieved good results in four main directions of cloud native, high availability, streaming and batch integration and AI. It is particularly worth mentioning that Flink has launched the advanced version of Streaming batch integration, the concept of Streaming Warehouse, to achieve real-time analysis of Streaming batch integration, in a real sense to complete the integration of Streaming batch integration computing and Streaming batch integration storage, so that the data flow of the entire data Warehouse. Streaming data warehouse will be one of the most important directions of Flink in the future, and will be promoted simultaneously in the Flink community.
This article will briefly summarize the FFA Keynote topics. If you are interested, you can find relevant theme videos on the FFA official website [2] to watch live replays.
Agenda of the main meeting
Before the main topic, Mr. Jia Yangqing, vice President of Alibaba Group, head of Alibaba Open Source Technical Committee and head of Ali Cloud intelligent Computing Platform, as the opening guest, shared his thoughts on open source in the context of cloud computing: Open source, whether in terms of technical contribution or ecological development, has gradually developed from the original replacement and complement to the role of innovation and leadership. Alibaba has opened more than 2,700 projects so far, making it a pioneer among Chinese Internet technology companies. As one of the most influential open source projects of Alibaba, Flink is indisputable in terms of technological advancement and ecological richness. Moreover, In the past few years, Alibaba has actively expanded the applicable scenarios of Flink, polished and iterated open source technologies through its own large-scale business, and then returned these technologies to Flink community, and joined hands with other open source projects to form a more comprehensive joint solution, truly realizing open source, continuous feedback, and accelerated popularization.
Let’s focus on some of the main topics.
1 Flink Next –– Beyond Stream Processing
As usual, the main topic was started by Mr. Wang Feng (aka Mo Wen), the founder of Apache Flink Chinese community and the head of Alibaba open Source Big data platform. He mainly introduced the achievements of Flink community in 2021 and the future development direction, including four parts: cloud native, Flink fault tolerance, stream batch integration and machine learning.
Cloud native — Deployment architecture evolution
Speaking of the development of open source big data, there is no way around the cloud native, the two depend on each other and complement each other. As the engine class of open source big data, how Flink’s deployment mode evolves in the context of cloud native is an interesting topic. The earliest deployment mode of Flink is the classic Static Standalone mode. The Static mode means that the user must reserve resources according to the service estimate. Therefore, the resource cannot run without the job, so in most cases, the resource must be reserved according to the maximum amount. It is obvious that this mode is both complex and inefficient for users. The second mode is called Active mode. The Active mode means that Flink applies for and releases resources to the underlying Kubernetes or Yarn based on the usage of business resources. This mode requires deep integration between Flink and underlying Kubernetes or Yarn. It is suitable for users who need to control resources in depth. It is too complicated for small and medium users. This leads to a third model which we call Adaptive/Reactive. In this mode, Flink operates like any other application on the cloud, dynamically adjusting its topology based on the resources given to it (adding or subtraction resource Pods). From the user’s point of view, he does not need to know how resources are allocated, so the third mode has a lower threshold for users.
There is a problem worth thinking about what is cloud native brought Flink, in addition to the elasticity of resource management, data backup, adaptive operations management, standardized tools and operation, the author think that the use of the more important is to reduce the user threshold, with smaller cost provide users with more simple, stable and rich experience.
Flink fault tolerance — Stable and fast Checkpoint
Discussions related to Checkpointing have run through almost the entire development of Flink, which is the core of the entire Flink fault-tolerant architecture. Flink periodically Checkpoint all operator states. If the Flink job fails, the job is resumed from the previous Checkpoint. In practice, we find that a large part of Oncall problems in the engine layer are related to Checkpoint, so how to perform Checkpoint at high frequency and stably is the key to improve Flink’s high availability (fault tolerance). The reasons for Checkpoint failure (timeout) are as follows: Slow flow of intermediate data causes slow flow of Checkpoint Barrier; excessive status of the operator causes timeout of status data upload. Flink has major projects on both fronts: Buffer Debloating and Generalized Log-based Checkpoint.
Buffer Debloating is designed to reduce upstream and downstream data that needs to be cached to just the operator without affecting throughput and latency. Currently, by default, upstream Buffer Debloating dynamically caches downstream data that can be processed for 1 second (this time can be configured). Buffer Debloating has been released in Flink-1.14. Generalized log-based Checkpoint is a kind of log-based Checkpoint method, which is similar to the traditional DB write ahead Log. The advantage is that the Checkpoint can be performed quickly, frequently, and stably. The trade-off is that you need to write/save an extra log. We know that Flink Checkpoint consists of synchronous and asynchronous processes. The synchronous process is usually fast, and the main time is spent in the asynchronous uploading of status files. The principle of a Generalized log-based Checkpoint is to separate Checkpointing from the time-consuming asynchronous file upload process and decouple the physical and chemical process of the underlying state store. A Generalized log-based Checkpoint is expected in flink-1.15.
This section is covered in more detail in “Flink Fault Tolerance 2.0”.
Flow and Batch integration — architecture evolution and implementation
In recent years, Flink has been pushing for the innovative idea of stream batch integration. Since the idea was first proposed, it has been widely accepted now. Teacher Mo Wen shared the process of the evolution of stream batch integration in various levels of Flink system architecture and its implementation scenarios, as shown in the figure below.
1) Architecture evolution
Last year, the unified SQL/Table API was implemented for the first time in the large-screen analysis of Tmall marketing activities, the core of Alibaba’s Singles’ Day [3]. This year, the Imperative API has been integrated further. A unified DataStream API for streaming batch will be formed, while the old DataSet API will be phased out. At the architecture level, the same job can process both finite and infinite data sets at the same time. The Connector framework can also interconnect streaming and batch storage, so that one set of code can handle two sets of data sources. At the operational level, a set of scheduling framework can be applied to both stream and batch jobs; Shuffle is pluggable and uses a set of Shuffle interfaces. This year, Alibaba real-time computing team opened source the Remote Shuffle Service[4], which separates storage and computation, and put it into the sub-project group Flink-Extended of Flink open source project. Flink- Extended [5] contains a number of other Flink ecological projects that interested students can check out.
After launching the core big-screen business on Tmall’s Singles’ Day last year, Libidi has been gradually promoted in more core businesses of Alibaba this year. Besides Alibaba, more and more companies are embracing the idea of streaming and batch integration. This year FFA has a dedicated streaming and batch integration sub-forum where bytedance, Meituan, JD.com and Xiaomi share the practices of streaming and batch Integration in their businesses. In addition, there is a special talk “New Progress of Flink Runtime for Stream Batch Architecture” in the special session of core technology. Students who are interested in this topic can learn about it. Students who are interested in the principle of the new connector framework can refer to the “New Trend of Flink Connector community and Hybrid Source Principle Practice” in the core technology special session.
2) Landing the scene
Teacher Mo Wen pointed out that the implementation of the technical concept of stream batch integration needs the support of specific scenes to reflect its real value. Based on this, he shared two typical application scenarios of stream batch integration.
Scenario 1 Flink CDC: Fully incremental integrated data integration
In traditional data integration, offline data integration and real-time data integration are two sets of different stacks, which require full and incremental timing merging and have poor timeliness. Flink’s integrated streaming and batch capability combined with Flink CDC capability can realize integrated data integration: full synchronization of historical data is completed, and then the breakpoint is automatically connected, and real-time incremental data is continuously transmitted to achieve one-stop data synchronization (automatic switchover after reading full database data, through binlog incremental synchronization). The implementation of automatic switching here is based on the new stream batch Source framework.
Flink CDC currently supports most major databases including MySQL, Postgres, Oracle, MongoDB, MariaDB, and others such as TiDB, DB2, and SQL Server are also under active development. Those who are interested in how Flink CDC can achieve one-stop data integration can refer to the talk “How Flink CDC Can simplify real-time data entry into the lake and warehouse” in the real-time Data Lake session of the sub-forum.
Scenario 2 Streaming Warehouse: Streaming Warehouse
As mentioned above, one of the highlights of this year is the concept of Streaming Warehouse proposed by Teacher Mo Wen. The background of this concept is to solve the problem of real-time offline data Warehouse integration.
Real-time offline data warehouse integration this problem is currently more commonly used solution is to use real-time and offline two links to achieve: 1) real-time stream processing link (Flink + Kafka) layered data ODS, DWD, DWS, and real-time write online service layer, provide online services (real-time OLAP); 2) At the same time, there will be an offline link to periodically supplement and revise the real-time data. In addition to the common problems of development efficiency, maintenance cost and inconsistent flow batch caliber caused by inconsistent flow batch, there is a more hidden and more difficult problem to solve: To ensure real-time performance, layered data such as ODS, DWD and DWS in real-time links are stored in message queues (such as Kafka). However, data in message queues cannot be effectively analyzed in real time. If other OLAP systems are introduced, system complexity will be increased and data consistency cannot be guaranteed.
To solve the problem that message queues cannot perform real-time analysis efficiently, Flink introduces Dynamic Table to store layered data generated by real-time links, as shown in the figure above. In this way, Flink can be through the flow of Flink SQL batch integrated capability of real-time series of the entire hierarchical data warehouse; Flink SQL provides real-time analysis capability for OLAP queries of Dynamic tables. We can understand this as the advanced version of StreamHouse = Streaming + Warehouse, which is the concept of StreamHouse = Streaming + Warehouse proposed by Teacher Mo Wen. It really achieves a set of API, a set of calculation under the framework of a set of methodology. Full link integration of a set of intermediate storage.
Dynamic Table is a built-in Table of Flink, different from Source and Sink in general sense. It is called a dynamic table because of the flow table duality of the table. Stream table duality is supported by columns LSM Tree and Log in two different forms of storage, corresponding to the batch (full analysis) and stream (incremental processing) modes of Flink SQL respectively. Dynamic Table through Flink’s own Checkpointing consistency semantic mechanism to ensure the flow Table duality in two storage forms of consistency semantics. It should be noted that the data consistency problem of the two image storage of the flow Table is a problem that the mashup system (introducing other OLaps and message queues) cannot easily avoid and solve (because it involves the consistent read and write synchronization between multiple systems), which is also one of the core competitiveness of Flink Dynamic Table that distinguishes it from other similar systems. If you are interested in the implementation of Dynamic tables, you can check out the talk “Building a data warehouse based on Flink Dynamic Table” in the stream batch sub-forum for a more detailed introduction to Dynamic tables.
At the end of this part, there is a stream data warehouse demo, which uses the above integration methodology to show a typical scenario of stream batch real-time analysis integration after real-time OLAP analysis found that the business logic has errors, and how to do batch correction and real-time support OLAP query correction. Students who want to have a more detailed understanding of the streaming data warehouse can refer to the interview of Mr. Mo Wen on the streaming data warehouse [6].
Machine learning — Apache Flink ML 2.0 architecture
Machine learning is another important scenario of Apache Flink. On the basis of the further improvement of Flink streaming batch API and architecture this year, it has been fully upgraded to Flink ML 2.0 based on the streaming batch API reconstruction. The biggest feature of Flink ML is real-time offline integration, as well as its matching real-time offline integration management scheduling (Flink AI Flow) and execution. There are several new highlights in Flink ML 2.0 that are worth taking a look at: 1) Flink’s new iterative computing framework, which is based on DataStream’s native support in the engine section, supports more flexible distributed synchronous and asynchronous iterations; 2) Released a new version of Flink ML Pipeline API, following the sciKit-learn style (Transformer, Estimator, Model) that machine learning users are more familiar with; 3) Support integrated deep learning integration, Flink ML Estimator can pull Up Pytorch and Tensorflow; 4) Stream batch integration capability enables Flink ML 2.0 to interconnect with stream and batch datasets simultaneously.
Flink ML 2.0 has been jointly completed by Alibaba Real-time Computing team and Machine learning team, and contributed to the Flink community as a sub-project of Flink, Flink-ML[7]. It is worth mentioning that in addition to Alibaba, there are also many other companies working together to build the ecology of Flink ML, such as 360 contributing to Clink[8]. The talk “Algorithmic Interfaces and Iterative Engines for Real-time Machine Learning” in the core Technology session details the architectural evolution of Flink ML 2.0, and there is also a machine learning session this year for those interested.
For PyFlink, Flink has more complete support for Python, the mainstream AI development language: PyFlink completely equals the capabilities of the Table and Data Stream apis in terms of functionality, and is innovative in terms of performance by calling C from JNI and then calling the Python parser from C, eliminating Python UDF and Java cross-process communication. The Performance of Python UDFs is similar to that of Java UDFs, with both development and operational efficiency. This section is explained in more detail in the sub-forum core Technology special “Introduction to PyFlink’s Next Generation Python runtime based on FFI”.
Development and prospect of real-time computing in Bytedance
The second session of the main topic was brought by Mr. Shi Rui, head of Bytedance computing infrastructure. Bytedance’s product business scenarios are mainly based on real-time information flow recommendation, so real-time computing supported by Flink is widely used in bytedance’s various products. The total MAU of all bytedance products is now over 1.9 billion. Due to its business characteristics, its data volume (EB level, 1EB = 2^60 Bytes) and real-time recommended requests (million QPS) are huge. We can see that in the comparison chart of resource usage of Bytedance engine shared by Teacher Shi Rui, Flink and Spark are basically the same, which is not common in ordinary companies. From this aspect, it can be seen that bytedance’s entire business line is dependent on flink-based stream computing.
Bytedance started research and gradually used Flink streaming computing in 2017. By early 2019, all streaming operations had been migrated from JStorm to Flink. Since 2019, with the maturity of Flink SQL and Flink Batch computing, Flink Batch has also been implemented in scenes such as bytedance data synchronization. Now there are about 10W + Flink Batch jobs running every day. Teacher Shi Rui especially mentioned that since last year, Bytedance has gradually promoted its application in bytedance. Interested partners can refer to the talk “Practice of Bytedance in Bytedance Feature Platform” in the special forum of Bytedance. At present, byteDance has 4W Flink streaming jobs worldwide, of which SQL jobs account for 30%. The number of CPU cores used exceeds 4 million. The QPS of Flink jobs processing messages in the evening peak reaches 9 billion, and Checkpoint peak throughput reaches 600GB/s.
In the sharing of Bytedance, BMQ, a stream-batch integrated message queue based on memory separation architecture, is worth mentioning (BMQ currently receives 90% of message queue traffic in bytes). Before BMQ, bytes used Kafka as message queues, and cluster scaling required a lot of copying, so it took about a week to complete a cluster upgrade. To solve this problem, the Byte team redesigned the message queue, BMQ, based on the architecture of memory separation. In BMQ architecture, data is stored in HDFS and Meta is stored in K-V storage. Due to stateless Proxy at the computing layer of BMQ, it is easy to scale and scale, and the migration time can be completed at the minute level. On the other hand, BMQ can provide both Stream API and Batch API, so it supports both Stream and Batch consumption, realizing the integration of Stream and Batch at the storage layer. Some of you may wonder, is this the same as Dynamic tables mentioned above? I think it is very different, because to solve the problem is not the same: dynamic table to solve the integration of real-time analysis of stream batch, so its stream batch storage format is completely different (in order to speed up stream processing and batch query respectively); All BMQ data is stored in the HDFS to support efficient large-scale message transmission and read/write services.
Teacher Shi Rui said that their next plan is to push Flink OLAP to the ground. He noted that Flink has a rich connector ecosystem that enables cross-data source queries, that Flink OLAP capabilities have been tested internally to be comparable to Presto, and in some cases better, and that improvements and optimizations related to Flink OLAP are now being actively promoted in the Flink community. This FFA bytedance has seven sessions of talk, covering all aspects from core technology to industry practice. Students who are interested in how Flink has evolved inside Bytedance can go to see it.
Iii. Construction process and prospect of REAL-TIME big data platform of ICBC
The third session of the main topic was brought by Mr. Yuan Yi, head of the big data platform of INDUSTRIAL and Commercial Bank of China, who shared the process and ideas of the construction of icbc’s real-time big data platform from the perspective of the financial industry.
First let’s take a look at a diagram of icbc data flow, as shown in the figure above. The data generated by the application is written to relational databases such as MySQL or Oracle, and then the logs generated by the database are copied to Kafka message queues as data sources for the real-time processing platform. The real-time processing platform has three data outlets. First, real-time data can be entered into the lake through Flink real-time ETL. Second, Flink results are output to HBase or ES online databases to provide application-oriented data center services. Third, analysis-oriented BI analysis capabilities are provided through Presto or CK analysis engines. The highly time-sensitive business scenarios within ICBC can basically be included in this link system.
As the savvy might have noticed, this complex data link is almost identical to the Flink Streaming Warehouse scenario. However, through Flink’s flow data warehouse, we can simplify the link of ICBC through many systems and components into Flink single link, and complete the real-time entry into the lake, real-time data service and real-time analysis through Flink’s Dynamic Table which provides real-time flow batch real-time analysis integration capability!
Another interesting point is that the data center in the financial industry will take data privacy and security into special consideration when it is designed. They use the following methods: 1) use the full life cycle data monitoring audit, for the audit and traceability of data access; 2) Watermarking the data itself when the data moves can facilitate traceability; 3) Dynamic data access permission control at natural person level is realized through SQL; 4) Automatic identification of sensitive data in mass data through expert rules and Machine Learning. These ideas and methods are of great reference significance in today’s data security and data privacy are more and more valued. Teacher Yuan Yi also shared a lot of business scenes related to the financial industry in detail, which I believe will inspire students who are interested in business scenes.
Four Deconstructing Stream Storage
The last session of the main topic by The founder of Pravega China community, Dell Technology Group OSA software development director Teng Yu teacher finale: deconstruction stream storage.
Pravega is an open source distributed streaming storage that provides unified streaming batch capability. It has the following characteristics: 1) Data can be ordered under the same key value; 2) Dynamically expand and shrink storage units according to data flow; 3) Support transactional write; 4) Support Checkpointing and consistent reading and writing; 5) Hierarchical storage design. All of these features are encapsulated in a Stream abstraction that shields streaming computing from much of the complexity on the Stream storage side. In this sharing, Ms. Teng highlighted Pravega’s architecture, Tiered Storage: Tiered Storage, with a persistent primary Storage based on distributed file/object Storage at the bottom, a global memory-based Cache layer in the middle, and a distributed Log abstraction layer at the top. Professor Teng also shared the difference between Pravega’s layered storage architecture and Kafka and Pulsar, as well as the impact on the performance of the two messaging systems. Interested students can go to the detailed understanding.
There are a few more interesting points in Pravega’s sharing:
The first is Pravega’s customized optimization of edge computing in the Internet of Things, which is now quite popular. For example, Pravega implements two-stage data aggregation for multiple clients by performing the first-stage aggregation in Writer and the second-stage aggregation in Segment Store, which greatly improves throughput. This kind of data aggregation optimization is ideal for situations where there are a large number of clients but each client generates a relatively small amount of data, which is typical of the Internet of Things.
The other is the end-to-end auto-scaling of Pravega and Flink. Elastic scale-up is a very important issue in the background of cloud primary. As mentioned above, one of the main features of Pravega is that it can automatically expand and scale up and adjust the number of segments, which can be a good indicator of Flink Reactive Scaling. The two can be combined to achieve end-to-end auto-scaling from computing to storage, and this work is currently being planned in collaboration between communities on both sides. Another Demo shared by Teacher Teng Yu showed the scaling effect of Pravega and Flink.
Teacher Teng Yu said that in the future, the boundary between stream and table is gradually blurred between storage and computation. Pravega’s streaming and batch integrated storage design also reflects a very important development direction of Flink in the future. The Pravega community is actively working with the open source community, including Flink, to build solutions. This year, Pravega and the Flink community jointly published a white paper, and we expect to work more with the Flink community in the future to push Flink computing to the data generation side, enabling data flow from end to cloud through Pravega.
Fifth Round Table
This year, FFA added a new round table (two sessions in Beijing and Shanghai), which invited many big data experts from Alibaba, Bytedance, Meituan, Kuaishou, Xiaomi, ICBC, Dell Technology Group and Xiaohongshu to discuss Flink and the future of real-time computing. You are friendly, sincere and down-to-earth to discuss a lot of issues that we are concerned about. Due to space constraints, only part of the relevant topics are listed here. You can find videos to feel them:
- What do you think of Flink’s maturity in real-time computing? What do people do with real-time computing today?
- What is the future of real-time computing (both technical and business)? Based on this, what new areas should Flink explore and what key issues should it address?
- Some people think that the threshold and cost of real-time computing is relatively high, relatively small; There are also many people who believe that real-time computing is the direction of the future, and big data and AI will evolve towards real-time. What do you think about that?
- How should Flink position itself and differentiate itself in the whole open source big data ecosystem?
- What is the relationship between internal technology practices, technology innovation and the open source community, and what are the strategies for people to use and give back to the community?
- What are the advantages of using and contributing to open source projects? What aspects of Flink are being explored internally? What challenges did you encounter along the way?
- What are Flink’s future plans for internal use, and what innovative technologies are planned to contribute to the community next?
- How do you see the relationship between Flink and ecological projects?
- What kind of open source community is helpful? And a sustainable community at the same time?
Summary and impressions
2021 is the year of big data, and it will be crucial for Apache Flink, the leader in real-time computing, to seize it. When Flink SQL tends to mature and the integration of stream and batch is gradually accepted and settled in the industry, we need to think about the future of Flink, which is what we are doing. On this basis, Flink launched the advanced version of Streaming batch integration, the concept of Streaming Warehouse, hoping to achieve real-time Streaming batch analysis integration, complete the integration of Streaming batch integration computing and Streaming batch integration storage in a real sense, so as to achieve a set of API, a set of calculation under the framework of a set of methodology, Full link integration of a set of intermediate storage. Flow data warehouse will be the most important direction of Flink in the future, road resistance and long, line is coming, line and continuous, the future can be!
[1] Flink Learning (Flink -learning.org.cn/)
[2] flink-forward.org.cn/
[3] Four billion a second! Flink is behind the first landing of Alibaba Double 11
[4]Remote Shuffle Service (github.com/flink-exten…
[5] Flink – extended (github.com/flink-exten…
[6] Apache Flink goes beyond computing, data warehouse architecture or a new revolution (c.tb.cn/F3.0OfNLU)
[7] Flink – ML (github.com/apache/flin…
[8] Clink (github.com/flink-exten…
The original link
This article is the original content of Aliyun and shall not be reproduced without permission.