Based on the environment

Prepare three VMS

Configure no password login

Configuration method :ipooli.com/2020/04/lin…

And do host mapping.

Download the Flink

www.apache.org/dyn/closer….

And unzip

The deployment of

Standalone Cluster Standalone mode

Start the

Enter the flink-1.10.1 folder

Direct execution:

./bin/start-cluster.sh
Copy the code

Cluster pattern

Modifying a Configuration File

Enter the flink-1.10.1 folder

  • Modify the./conf/flink-conf.yaml

Modify the following parameters:

jobmanager.rpc.address: bigdata1
jobmanager.rpc.port: 6123
jobmanager.heap.size: 1024m
taskmanager.memory.process.size: 1568m
taskmanager.numberOfTaskSlots: 3
parallelism.default: 3
Copy the code
  • Modify the./conf/mastersConfiguring the Master Node

Is amended as:

bigdata1:8081
Copy the code
  • Modify the./conf/slavesConfigure slaves

Is amended as:

bigdata1
bigdata2
bigdata3
Copy the code

Copy to the other two machines


scp -r /home/admin/flink/ root@bigdata2:/home/admin/

scp -r /home/admin/flink/ root@bigdata3:/home/admin/

Copy the code

Start the cluster

Execute on bigData1:

./bin/start-cluster.sh
Copy the code

Then visit http://bigdata1:8081/

You can see that there are three Task Managers and one Job Manager is BigdatA1

HA high availability mode

Two JobManagers. When the active JobManager is down, the standby JobManager is used. After the interrupted JobManager recovers, the standby JobManager becomes the standby JobManager

Download the Hadoop dependency package

  • Download the hadoop version

The hadoop version I use is: Hadoop-2.6.5 Dependency package: Download

Other versions: Download

  • Place dependency packages in the Lib directory of Flink
  • Configuring environment Variables
Vi /etc/profile # Add environment variables export HADOOP_CONF_DIR=/home/admin/hadoop-2.6.5/etc/hadoop # Environment variables take effect source /etc/profileCopy the code

Modify the./conf/flink-conf.yamlThe configuration file

Modify the following parameters

high-availability: zookeeper
high-availability.storageDir: hdfs://bigdata1/flinkha/
high-availability.zookeeper.quorum: bigdata1:2181
high-availability.zookeeper.path.root: /flink
state.checkpoints.dir: hdfs:///flink/checkpoints
state.savepoints.dir: hdfs:///flink/savepoints

Copy the code

Start the cluster.

Hadoop configuration files and startup methods are not covered here.

Start ZooKeeper in Flink

Go to the Flink folder

./bin/start-zookeeper-quorum.sh
Copy the code

JPS checks whether it is started

Start the Flink

Execute in bigData1

./bin/start-cluster.sh
Copy the code

To view

Open access separately:

http://bigdata1:8081/

http://bigdata2:8081/

You can view cluster information on both pages

test

  • We can kill the Job Manager on the bigdatA1 machine, and then the standby Job Manager can be used.

  • Start Job Manager of BigDatA1
./bin/jobmanager.sh start
Copy the code

Summary: This paper introduces three deployment modes of Flink: single machine, cluster and HA.

More articles :www.ipooli.com