1. Basic cluster information

  1. Number of clusters: 3

  2. MyNode01

    • System: Centos7.6

    • Memory: 8 gb

    • Hard disk: 1 t

  3. MyNode02

    • System: Centos7.6

    • Memory: 12 gb

    • Hard disk: 500 gb

  4. MyNode03

    • System: Centos7.6

    • Memory: 12 gb

    • Hard disk: 500 gb

Ii. Software package version

  1. system

    • CentOS-7-x86_64-DVD-1810.iso

  2. CDH framework version: 5.14.2

  3. Hadoop

    • Hadoop – server – cdh5.14.2. Tar. Gz

    • jdk-8u141-linux-x64.tar.gz

    • Zookeeper 3.4.5 – cdh5.14.2. Tar. Gz

  4. Hive

    • Hive – 1.1.0 – cdh5.14.2. Tar. Gz

    • Mysql connector – Java – 5.1.38. Jar

  5. Hbase

    • Hbase – 1.2.0 – cdh5.14.2. Tar. Gz

    • Apache – phoenix – 4.14.0 – cdh5.14.2 – bin. Tar. Gz

  6. Flume

    • The flume – ng – 1.6.0 – cdh5.14.2. Tar. Gz

  7. Sqoop

    • Sqoop – 1.4.6 – cdh5.14.2. Tar. Gz

    • java-json.jar

    • Mysql connector – Java – 5.1.38. Jar

  8. Impala

  9. Spark

    • Scala – 2.11.8. Zip

    • The spark – 2.3.3 – bin – hadoop2.7 TGZ

  10. Kafka

    • Kafka_2. 11-1.0.1. TGZ

    • Kafka – eagle – bin – 1.2.3. Tar. Gz

  11. Flink

    • Flink, version 1.8.1 – SRC. TGZ

Development environment

  1. Operating system: MacOS

  2. Development tool: IDEA 2019

  3. Auxiliary tools: FinalShell 3.9

4. Cluster Description

  • Blitz got three used laptops, and then slapped them all with Centos7.6 (reason: virtual machines don’t work very well). The total memory capacity of the cluster is 32GB and the total hard disk capacity is 2 TB. Later period is not enough, white brother to make a few ~ ~

  • PS: If there are more than 50 people, Brother Bai will write an article to introduce the process in detail. (If you want to know more about the cluster building process, you can also leave a message ~)

  • Big data cases will be based on the above cluster environment for practical operation.