This is the 22nd day of my participation in the August Wen Challenge.More challenges in August

WangScaler: A writer with heart.

Declaration: uneducated, if there is a mistake, kindly correct. \

Now artificial intelligence and big data are hot industries, how can we as programmers not to research, today I start from scratch to look at the knowledge of big data. Start from the most basic environment building, slowly to understand what is big data.

Centos system installation

Here is a virtual environment simulated by virtual machine. If you can directly use the real machine, of course, you can also use Docker to deploy three Linux servers. For example, HP computers cannot use Vmware virtual machine and Docker container at the same time, because Vmware and Hyper-V are incompatible. Then choose the one you are most familiar with and install three machines.

  • 1. Download and install Vmware VIRTUAL machine (Baidu solution)
  • 2. Use CenterOS system (Baidu solution)

The above two steps online information is very much, can refer to the information on the Internet to solve easily, here no longer repeat.

Configure the VM network

To communicate with each other, you need to set fixed IP addresses for each centos machine. If you are a VMware VM, you can view the gateway address by using the VM toolbar > Edit > Virtual Network Editor. I have 192.168.253.0. So we should set the IP of the machine to 192.168.253.xxx.

Then turn on the machine and get ready to modify.vim /etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0
HWADDR=00:0C:29:BF:58:0C
TYPE=Ethernet
UUID=c05ca7fc-a6e2-4eb5-b723-c93d7ae68634
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=static
IPADDR=192.168253.130.
NETMASK=255.255255.. 0
GATEWAY=192.168253.2.
DNS1=192.168253.2.
Copy the code

I changed the IP address here to 192.168.253.130. You can configure it according to your situation.

After the configuration is complete, restart the service to make the configuration take effectservice network restartRestart the service.

Install required software

  • Zip is used to decompress compressed packages
  • Vim is used to edit files, generally the default system is only VI, we choose Vim, you do not install vi can also
  • Openssh-server and openssh-clients are used to generate key files so that clusters can use SSH to log in to each other without password.

1. Configure the local YUM source

In /etc/yom.repos. D folder, make a backup of the original files, in case we make mistakes after modification, which may cause irreversible problems. Repo centos-base. Repo. Bak Next modify the file vi centos-media.

[c6-media]
name=CentOS-$releasever - Media
baseurl=file:///media/
gpgcheck=0
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
Copy the code

Prepare to mount mount /dev/dvd/media Note the space before /media, if prompted

Check connected device status under CD/DVD in vm Settings and retry the command.

Indicates success

2, installation,

Let’s start installing some of the tools we need,

yum clean all
yum install -y vim zip openssh-server openssh-clients
Copy the code

With the installation complete, we will use these tools to complete our cluster in the next article. Big data development: the construction of a cluster environment, you can preview the subsequent article.

Come all come, click “like” and then go!

Follow WangScaler and wish you a promotion, a raise and no bucket!