Heron: Next generation streaming engine application from Twitter

Inside Twitter, Heron replaces Storm and is the standard for stream processing.

Data model differences

Heron is compatible with Storm’s data model, or Heron is compatible with Storm’s API, but the implementation behind it is completely different. So the scenarios are the same, Heron is used where Storm is used. But Heron offers better efficiency, more features, more stability and easier maintenance than Storm.

Storm Trident is a project based on Storm and provides a high-level API like Heron’s functional API. Trident realizes exactly once by checkpoint and rollback; Heron implements Effectively Once with the distributed snapshot algorithm invented by Chandy and Lamport.

Application architecture differences

Storm’s worker runs multiple threads per JVM process, performing multiple tasks per thread. The logs of these tasks are mixed together, making it difficult to debug the performance of different tasks. Storm’s Nimbus cannot isolate worker resources, so resources in multiple topologies affect each other. ZooKeeper is also used to manage heartbeat, which makes it easy to become a bottleneck.

Each Heron task is a separate JVM process, facilitating debugging and resource isolation management while saving resources for the entire topology. ZooKeeper stores only a small amount of data in Heron. Heartbeat is managed by the Tmaster process and has no pressure on ZooKeeper.

Spark Streaming processes tuples in a micro-batch granularity. Tuples in a time window of half a second to several seconds are submitted to Spark as a micro-batch. Heron uses a processing granularity of tuple. The average response time of Spark Streaming can be considered as the length of half a time window due to time window limitations, which Heron does not have. So Heron is low latency and Spark Streaming is high latency.

Spark Streaming recently announced a proposal to include a new mode in the next version 2.3 that does not use micro-batch for calculations.

Data model differences

Semantically, Spark Streaming and Heron both implement exactly once/effectively once. Spark Streaming and Heron both implemented stateful processing at the state level. Spark Streaming supports SQL, while Heron does not. Spark Streaming and Heron both support Java and Python interfaces. It should be noted that Heron’s API is pluggable and supports many programming languages besides Java and Python, such as C++.

Application architecture differences

For task assignment, Spark Streaming uses a single thread for each task. A JVM process may have multiple threads running at the same time. Heron is a separate Heron-instance process for each task, which is designed to be easy to debug because when a task fails, the task process can only be taken out for inspection, avoiding the interaction of the various task threads in the process.

In terms of resource pools, Spark Streaming and Heron can run on YARN and Mesos. It should be noted that Heron’s resource pool design is modeled on pluggable Interface, which can connect to many resource managers such as Aurora, etc. You can see [11] for the resource pools supported by Heron.

Based on the comparison of the above tables, we can get the following key points of selection:

• Storm is suitable for scenarios that require rapid response and moderate traffic. Storm and Heron are API compatible and basically interchangeable in functionality; Twitter moved from Storm to Heron, indicating that if Storm and Heron had to choose between them, they would usually choose Heron.

• Kafka Streams is bound to Kafka. If your existing system is built on Kafka, consider using Kafka Streams to reduce overhead.

• Spark Streaming is thought to have the highest traffic of these projects, but it also has the highest response latency. Spark Streaming can be used for systems that require low response speed but high flux. If you take this situation to the extreme you can use the Spark system directly.

• Flink uses a stream processing kernel and provides both stream and batch interfaces. Flink is suitable for projects that require both stream and batch processing. At the same time, due to the need to take into account the trade-offs of both sides, it is not easy to carry out targeted optimization and processing in a single aspect.

Spark Streaming, Kafka Streams, and Flink all have specific application scenarios, and Heron can be used for other general stream processing scenarios.

Find Heron’s publishing page at github.com/twitter/her… Find the latest version, 0.16.5. You can see that Heron provides several versions of the installation files, which are divided into several categories: client, toolkit Tools, development kit API, and so on.

Installing the Client

Download the client installation file heron-client-install-0.16.5-ubuntu.sh:

Wget HTTP: / / https://github.com/twitter/heron/releases/download/0.16.5/heron-client-install-0.16.5-darwin.sh

Then execute this file:

Chmod +x heron-*. Sh./heron-client-install-0.16.5-- platform. sh --userCopy the code

The –user argument installs the Heron client into the current user directory ~/.hedon and creates a link under ~/bin to the executable under ~/.heorn/bin.

The Heron client is a command line program named Heron. The heron command can be accessed directly by exporting PATH=~/bin:$PATH. Run the following command to check whether the heron command was successfully installed:

heron version

Run the example Topology

Add localhost to /etc/hosts. Heron uses /etc/hosts to resolve local domain names in single-machine mode.

The Heron client was installed with a jar for the example Topology package in the ~/.heron/example directory. We can run one of the examples Topology as an example:

heron submit local ~/.heron/examples/heron-examples.jar \com.twitter.heron.examples.ExclamationTopology ExclamationTopology \--deploy-deactivatedCopy the code

The Heron submit command submits a topology to heron to run. The heron submit command format can be viewed with the Heron help Submit command.

When Heron runs in standalone local mode, it stores information such as running status and logs in the ~/. Herondata directory. We can look at the example Topology directory we just ran:

ls -al ~/.herondata/topologies/local/${USER_NAME}/ExclamationTopology

Topology lifecycle

The lifecycle of a topology consists of the following phases:

• Submit: Submits the topology to the Heron-Scheduler. The topology has not yet handled the tuples, but it is ready to be activated;
• Activate /deactivate: Tells the topology to start/stop processing the tuples;
• Restart: Restarts a topology so that the resource manager can re-allocate the container.
• Kill: Cancels the topology to release the resource.

These phases are managed through the Heron command line client. You can view the command format in heron help.

Since Twitter made Heron open source in the summer of 2016, the Heron community has developed many new features, In particular, in 2017, Heron added “online dynamic capacity expansion/reduction”, “Effectively Once Transmission Semantics”, “functional API”, “Multiple programming language support”, and “self-regulating”.

In addition to the original tuple transmission mode at most once and at least Once, Heron newly adds effectively Once. At most once and at least once have some disadvantages. At least once repeats some tuples. Well Once aims to be accurate and reliable when calculations are deterministic.

Effectively once can be summarized into two aspects:

• Distributed state checkpoint;
• The topology state rolled back.

The tmaster periodically sends a marker tuple to spout. When a node in the topology collects all upstream marker tuples, it writes its status to a state storage. This process is checkpoint. When all nodes in the topology checkpoint, the state storage stores a snapshot of the entire topology. If the topology encounters an exception, the snapshot can be read from the State storage to recover and start processing the data again.

Whereas topology writers used to write topology using storm-compatible Java apis, Heron now provides Python and C++ apis so that programmers familiar with Python and C++ can write topology. Python and C++ API design is similar to the Java API, they contain the underlying API for constructing dags, and in the future they will provide functional apis for topology developers to focus on the business logic.

In implementation, both Python and C++ apis have heron-instance implementations of Python and C++. They do not overlap with the Java implementation of Heron-Instance, so they reduce the overhead of interlanguage translation and increase efficiency.

In this article, we compare Heron with common stream processing projects, including Storm, Flink, Spark Streaming and Kafka Streams, and summarize the key points of system selection. In addition, we practice a case study of Heron. Finally, we discussed the new features Heron developed during the year.

Finally, the author hopes that this article can provide you with some experience related to Heron application, and we welcome your suggestions and help. If you are interested in developing and improving Heron, you can check out the Heron website (Heronstreaming.io) and the code (github.com/twitter/her…). .

reference

[1] Maosong Fu, Ashvin Agrawal, Avrilia Floratou, Bill Graham, Andrew Jorgensen, Mark Li, Neng Lu, Karthik Ramasamy, Sriram Rao, And Cong Wang.” Streaming Engines. “In 2017 IEEE 33RD International Conference on Data Engineering (ICDE), pp. 1165-1172. IEEE, 2017. [2] Sanjeev Kulkarni, Nikunj Bhagat, Maosong Fu, Vikas Kedigehalli, Christopher Kellogg, Sailesh Mittal, Jignesh M. Patel, Karthik Ramasamy, and Siddarth Taneja. “Twitter Heron: Stream Processing at scale. “In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, pp. 239-250. ACM, 2015. [3] Maosong Fu, Sailesh Mittal, Vikas Kedigehalli, Karthik Ramasamy, Michael Barry, Andrew Jorgensen, Christopher Kellogg, Neng Lu, Bill Graham, “Streaming@ Twitter.” IEEE Data Eng. Bull. 38, No. 4 (2015): 15 to 27. [4] storm.apache.org/ [5] storm.apache.org/releases/cu… [6] flink.apache.org/ [7] spark.apache.org/streaming/ [8] kafka.apache.org/documentati… [9] twitter. Making. IO/heron/API/p… [10] github.com/twitter/her… [11] github.com/twitter/her…

Huijun Wu is a software engineer at Twitter, working on the research and development of Heron, a real-time streaming engine. He graduated from Arizona State University, specializing in big data processing and mobile cloud computing. He has published several academic papers in international top journals and conferences, and has a number of patents. Lu Neng is a member of Twitter’s real-time computing platform team. Focused on distributed systems, participated in the development of Twitter’s Manhattan key-value storage system and Obs monitoring alarm system, and is currently responsible for the development research of Heron. He has published many academic papers in top international journals and conferences. Maosong Fu is head of Twitter’s real-time computing platform team, responsible for Heron, Presto and other services. One of the original authors of Heron. He has published several papers in SIGMOD, ICDE and other conference journals, focusing on distributed systems. Graduated from Huazhong University of Science and Technology with a bachelor’s degree. She is a graduate of Carnegie Mellon University. This article is the programmer’s original article, shall not be reproduced without permission.

On January 13th,SDCC 2017 Database Online SummitWill be strong attack, adhering to the dry goods solid material (case) content principle, invited fromalibaba,tencent,weibo,Netease etc.Database experts from a number of enterprises and university researchers will focus on hot database technologies such as Oracle, MySQL, PostgreSQL, Redis, etc., from the deep digging of core technologies to the analysis of high availability practices, to create the essence of compressed sharing, draw a conclusion by reference, think and fight, registration and more details can be foundClick here toLook at it.