The Apache Flink community announces the release of Flink 1.10.0!

This Release fixes 1.2k issues, makes significant improvements to the overall performance and stability of Flink jobs, and adds support for K8S, Python.

This release marks the completion of the integration with Blink and enhances the integration of streaming SQL with Hive. This article details the new features and major improvements.

First, memory management optimization

The original TaskExecutor has some disadvantages:

Stream processing and batch processing use different configuration models;

The off-heap configuration of flow processing RocksDB is complex and requires user configuration;

To make memory management more intuitive, Flink 1.10 makes significant improvements to the TaskExecutor memory model and configuration. This change makes Flink more suitable for a variety of deployment environments: K8S, Yarn, Mesos.

This change unifies entry points and makes programming for downstream frameworks such as Zeppelin easier.

2. Integrate Kubernetes

This is great news for users who want to use Flink in containers.

Active Kubernetes integration was launched in Flare 1.10

Flink’s ResourceManager (K8sResMngr) communicates locally with Kubernetes to allocate new pods on demand, similar to Flink’s Yarn and Mesos integration. Users can also use namespaces to start Flink clusters for multi-tenant environments with limited aggregate resource consumption. Configure the RBAC roles and service accounts with sufficient permissions in advance.

Users can simply refer to Kubernetes configuration options and then submit jobs on the CLI to an existing Flink session on Kubernetes using the following command:

./bin/flink run -d -e kubernetes-session -Dkubernetes.cluster-id=<ClusterId> examples/streaming/WindowJoin.jarCopy the code

3. Integrate Hive

Flink 1.10 is developed to integrate Hive into Flink for use in production environments.

Flink supports a list of Hive versions:

  • 1.0
    • 1.0.0
    • 1.0.1
  • 1.1
    • 1.1.0
    • 1.1.1
  • 1.2
    • 1.2.0
    • 1.2.1
    • 1.2.2
  • 2.0
    • 2.0.0
    • 2.0.1
  • 2.1
    • 2.1.0
    • 2.1.1
  • 2.2
    • 2.2.0
  • 2.3
    • 2.3.0
    • 2.3.1
    • 2.3.2
    • 2.3.3
    • 2.3.4
    • 2.3.5
    • 2.3.6
  • 3.1
    • 3.1.0
    • 3.1.1
    • 3.1.2

Need to introduce dependencies

<! -- Flink Dependency --> <dependency> <groupId>org.apache.flink</groupId> < artifactId > flink - connector - hive_2. 11 < / artifactId > < version > 1.10.0 < / version > < scope > provided < / scope > < / dependency > < the dependency > < groupId > org. Apache. Flink < / groupId > < artifactId > flink - table - API - Java - bridge_2. 11 < / artifactId > The < version > 1.10.0 < / version > < scope > provided < / scope > < / dependency > <! -- Hive Dependency --> <dependency> <groupId>org.apache.hive</groupId> <artifactId>hive-exec</artifactId> <version>${hive.version}</version> <scope>provided</scope> </dependency>Copy the code

Connecting to Hive code

val settings = EnvironmentSettings.newInstance().useBlinkPlanner().inBatchMode().build()
val tableEnv = TableEnvironment.create(settings)

val name            = "myhive"
val defaultDatabase = "mydatabase"
val hiveConfDir     = "/opt/hive-conf" // a local path
val version         = "2.3.4"

val hive = new HiveCatalog(name, defaultDatabase, hiveConfDir, version)
tableEnv.registerCatalog("myhive", hive)

// set the HiveCatalog as the current catalog of the session
tableEnv.useCatalog("myhive")Copy the code

PyFlink: Supports UDF

Since Flink 1.10, PyFlink has supported UDF functions.

Users can also PIP easily install PyFlink using the following methods:

pip install apache-flinkCopy the code

Other important changes

  • Flink can now compile and run on Java 11.
  • A new Elasticsearch sink that fully supports Elasticsearch version 7.x.
  • Kafka 0.8 and 0.9 versions have been deprecated and are no longer supported.
  • Removed the authentication network traffic configuration options taskmanager.net work. Credit. The model.
  • Removed the old Web UI.

6. List of contributors

Finally, when we look at the list of contributors, there are many national gods

Achyuth Samudrala, Aitozi, Alberto Romero, Alec.Ch, Aleksey Pak, Alexander Fedulov, Alice Yan, Aljoscha Krettek, Aloys, Andrey Zagrebin, Arvid Heise, Benchao Li, Benoit Hanotte, Benoit Paris, Bhagavan Das, Biao Liu, Chesnay Schepler, Congxian Qiu, Cyrille Chepelov, Cesar Soto Valero, David Anderson, David Hrbacek, David Moravek, Dawid Wysakowicz, Dezhi Cai, Dian Fu, Dyana Rose, Eamon Taaffe, Fabian Hueske, Fawad Halim, Fokko Driesprong, Frey Gao, Gabor Gevay, Gao Yun, Gary Yao, GatsbyNewton, GitHub, Grebennikov Roman, GuoWei Ma, Gyula Fora, Haibo Sun, Hao Dang, Henvealf, Hongtao Zhang, HuangXingBo, Hwanju Kim, Igal Shilman, Jacob Sevart, Jark Wu, Jeff Martin, Jeff Yang, Jeff Zhang, Jiangjie (Becket) Qin, Jiayi, Jiayi Liao, Jincheng Sun, Jing Zhang, Jingsong Lee, JingsongLi, Joao Boto, John Lonergan, Kaibo Zhou, Konstantin Knauf, Kostas Kloudas, Kurt Young, Leonard Xu, Ling Wang, Lining Jing, Liupengcheng, LouisXu, Mads Chr. Olesen, Marco Zuhlke, Marcos Klein, Matyas Orhidi, Maximilian Bode, Maximilian Michels, Nick Pavlakis, Nico Kruber, Nicolas Deslandes, Pablo Valtuille, Paul Lam, Paul Lin, PengFei Li, Piotr Nowojski, Piotr Przybylski, Piyush Narang, Ricco Chen, Richard Deurwaarder, Robert Metzger, Roman, Roman Grebennikov, Roman Khachatryan, Rong Rong, Rui Li, Ryan Tao, Scott Kidder, Seth Wiesman, Shannon Carey, Shaobin.Ou, Shuo Cheng, Stefan Richter, Stephan Ewen, Steve OU, Steven Wu, Terry Wang, Thesharing, Thomas Weise, Till Rohrmann, Timo Walther, Tony Wei, TsReaper, Tzu-Li (Gordon) Tai, Victor Wong, WangHengwei, Wei Zhong, WeiZhong94, Wind (Jiayi Liao), Xintong Song, XuQianJin-Stars, Xuefu Zhang, Xupingyong, Yadong Xie, Yang Wang, Yangze Guo, Yikun Jiang, Ying, YngwieWang, Yu Li, Yuan Mei, Yun Gao, Yun Tang, Zhanchun Zhang, Zhenghua Gao, Zhijiang, Zhu Zhu, a-suiniaev, azagrebin, beyond1920, biao.liub, blueszheng, bowen.li, caoyingjie, catkint, chendonglin, chenqi, chunpinghe, cyq89051127, danrtsey.wy, dengziming, dianfu, eskabetxe, fanrui, forideal, gentlewang, godfrey he, godfreyhe, haodang, hehuiyuan, hequn8128, hpeter, huangxingbo, huzheng, ifndef-SleePy, jiemotongxue, joe, jrthe42, kevin.cyj, klion26, lamber-ken, libenchao, liketic, lincoln-lil, lining, liuyongvs, liyafan82, lz, mans2singh, mojo, openinx, ouyangwulin, shining-huang, shuai-xu, shuo.cs, stayhsfLee, sunhaibotb, sunjincheng121, tianboxiu, tianchen, tianchen92, tison, tszkitlo40, unknown, vinoyang, vthinkxie, wangpeibin, wangxiaowei, wangxiyuan, wangxlong, wangyang0918, whlwanghailong, xuchao0903, xuyang1706, yanghua, yangjf2019, yongqiang chai, yuzhao.cyz, zentol, zhangzhanchum, zhengcanbin, zhijiang, zhongyong jin, zhuzhu.zz, Zjuwangg, Zoudaokoulife, Yantian, Xie Lei, Zhang Zhihao, Cao Jianhua

Flink series:

Flink introduction to Apache Flink Introduction to Apache Flink Architecture

Getting started with Flink (3) — Environment and deployment

Introduction to Flink (iv) — Programming model

For more blog posts on real-time computing,Flink,Kafka and other related technologies, welcome to real-time streaming computing