Build a deep learning environment based on Ubuntu+Python+Tensorflow+Jupyter Notebook

[TOC]

preface

Nowadays, advanced knowledge such as artificial intelligence and deep learning has gradually been integrated into everyone’s vision, and deep learning is indispensable to the identification of small and large captcha codes and unmanned driving technologies. To meet the pace of The Times and keep up with the trend of technology, we need to build a deep learning environment first. There are many ways to build an environment, this article is mainly introduced, based on Ubuntu+Python+Tensorflow+Jupyter Ntebook to build an environment, so as to start the journey of deep learning. This paper is mainly introduced from the following parts:

  • Environment to prepare
  • Xshell connects to Ubuntu remotely
  • Configuration and remote access of the Jupyter Notebook server
  • Deep learning remote environment testing

Okay, without further ado, let’s get back on track. (# ` O ‘)

First, environmental preparation

Environment introduction

  • Prepare two computers, one serving as the server and the other serving as the client, and remotely connect to the deep learning environment of the server through the client.
  • Server: Linux system (Ubuntu). You can also use other Linux systems as required.

Ubuntu is introduced: Ubuntu is a desktop application based open source GNU/Linux operating system. Ubuntu is based on Debian GNU/Linux and supports x86, AMD64 (X64) and PPC architectures. Built by a global professional development team (Canonical Ltd). The biggest difference between Linux system and Windows system is that it mainly uses terminal Shell command to carry out a series of operations, so it is often used to build a server in actual use. The system is built in VMware VIRTUAL machine.

  • Client: Windows10

I’m not going to introduce you to this because it’s so familiar.

  • Anaconda

Anaconda is an open source distribution of Python that contains more than 180 science packages and their dependencies, including Conda, Python, etc. So once you download Anaconda, you don’t need to download any other Python packages. There are a number of large Python tools integrated into Anaconda, including Jupyter Notebook, the subject of this article.

  • Xshell

This is not much to say, it is mainly a remote connection tool, you can use SSH protocol to connect to Ubuntu system remotely, and it is often used in conjunction with the remote file transfer tool Xftp.

Software download

Here to give you two ways to download, one is the official website download, if the official download is slow can be downloaded in the following network disk, you can download according to their own convenience.

Website:

  1. Anaconda:www.anaconda.com/download/, to…
  2. Xshell:xshell.en.softonic.com/?ex=BB-682….
  3. Ubuntu:www.ubuntu.com/download/de…
  4. VMware: www.vmware.com/, download it from the server

The cloud:

  1. Anaconda:pan.baidu.com/s/1k8JdX5vo… Password: BHFV download it from the server
  2. Xshell:pan.baidu.com/s/1KKqsFrhb… Password: fi0y download it from the client
  3. Ubuntu:pan.baidu.com/s/1Zwz1GGlE… Password: 50yw download it from the server
  4. VMware:pan.baidu.com/s/1-h0HLJGr… Password: HJNQ download from the server

Install Ubuntu in VMware

Note: The following is configured on a server computer

Installing Ubuntu on a VMware VIRTUAL machine is a bit of a step and takes a while, but it’s not complicated. Considering the cost of time and the value it brings to everyone, the blogger described it in words, not stickers for the time being. But you don’t have to worry, the key steps will be illustrated, everyone can set up the installation according to the process below (youdaoplaceholder0) ~→.

  • After downloading VMware and Ubuntu on the server. First, open VMware and you will find that it will ask you to enter VMware secret keys. The following will give you some current valid secret keys, which may be invalid when you use them. You can find the available secret keys by yourself:
FF31K-AHZD1-H8ETZ-8WWEZ-WUUVA
CV7T2-6WY5Q-48EWP-ZXY7X-QGUWD
ZY5H0-D3Y8K-M89EZ-AYPEG-MYUA8
ZC5XK-A6E0M-080XQ-04ZZG-YF08D
ZY5H0-D3Y8K-M89EZ-AYPEG-MYUA8
Copy the code
  • Enter the secret key, enter the VMware page, this time we need to use the Ubuntu image file downloaded before, create a virtual machine in VMware, the process is as follows:

This is the whole process of installing Ubuntu on VMware. If you have any questions during the installation process, please feel free to contact the author or leave a comment below.

Install Anaconda in Ubuntu

If you are familiar with Java, you will need to install Anaconda. If you are familiar with Java, you will need to install Anaconda. Anaconda is like Maven. After downloading Anaconda from Ubuntu, go to its directory (or CD operation), open the terminal, and run the following command to install:

Bash anaconda3-5.3.1-linux-x86_64. sh # Do not blindly copy and paste anaconda3-5.3.1-linux-x86_64. shCopy the code

After execution there are a number of issues such as the consent protocol, we simply default to the default option (Enter, or yes) until the Anaconda environment variable configuration is displayed (don’t overdo it with yes), and we need to choose to add it to the environment variable. After a series of operations, we close the terminal and then reopen the terminal, and execute the conda list and python commands respectively on the terminal. If your terminal interface displays output similar to the following, congratulations, you have completed the installation of Anaconda.

One more note: If you have a terminal type error (conda not found) or Python version 2.7, you have not added Anaconda to the environment variable (yes is too much), so you need to manually configure Anaconda’s environment variable, as follows:

Open the configuration file for environment variables, and you can see the difference between Linux and Windows (Windows is usually set through the interface, while Linux is mostly set through the terminal and vim).

# Open the configuration file of environment variables, from which we can see the difference between Linux and Windows. On Linux, most of the configuration is done on a terminal using vim sudo vim gedit /root/.bashrcCopy the code

After opening the file we need to add the following content at the end of the file (need to be familiar with vim operation), XXX is your Anaconda bin directory, for example, mine is **/home/lxj/anaconda3/bin**

Export PATH="XXX:$PATH" # XXX is your Anaconda bin directory, e.g. mine is /home/lxj/anaconda3/binCopy the code

Then save (Esc -> shift+: -> wq -> enter source ~/. Bashrc in the terminal to update the Anaconda environment variable. If the Python/conda list command is executed, you will see the conda package list and Python3.7 output.

Add: here is an extra note, the above operation is to use Vim file editing, he is different from the Notepad under Windows, but through specific operations to edit the file. For those familiar with Vim operations, the above configuration of Anaconda environment variables should be understood. However, if you are unfamiliar with Vim, you may be confused. So here are some Vim operations:

After using the vim command, the terminal will enter the vim interface, which cannot modify any content but only allow read only. At this point, typing I will enter Vim’s edit mode, and you can now modify the file. After the content of the file is modified, we need to exit the vim editing mode, the common exit of vim is as follows (first enter Esc key) :

  • q! -> Do not save the file changes and exit
  • Wq -> Save the changes and exit
  • wq! -> Sometimes we need root permission to edit files, using this command can force the modification and exit.

Vim’s familiarity with the configuration of the Anaconda environment variables is enough, and other commonly used commands are introduced for your reference

  • In read mode, double-click D -> delete the current line
  • Ctrl+ B -> Move the content forward one page
  • CTRL + F -> Move the content backward grandpa
  • Shift + G -> Mouse pointer to the end

Commonly used is some of the above, other operations need to learn by themselves

Note: All of the above is configured on the server computer

Now that we have set up the environment, how can we use Xshell to remotely connect to Ubuntu

Xshell connects to Ubuntu remotely

Note: The following is configured on the client computer

With Anaconda installed, we need to use Xshell to connect to our Ubuntu system remotely, at which point we need to move on to the client.

First download Xshell by following the link in the software download above, then install Xshell for Windows foolproof (by the way, install Xftp, which is used with Xshell, although not used in this article). After the operation, there will be a few pits, but don’t worry, I will take you one by one to fill out O (*≧▽≦), ツ Devoured by bill.

Pit 1: connection failed Let’s double click to open Xshell and clickfileCreate a new one, and then operate according to the following figure:

In addition, the host property in the top is the Ubuntu IP address, which can be obtained by running the ifconfig command on the Ubuntu terminal (Windows is ipconfig).

After filling in the above information, enter your Ubuntu login password on the screen that appears. You will find that the connection fails after execution. At this point, we need to check whether Ubuntu and the client are in the same network segment. We can set Ubuntu to bridge mode (right-click Ubuntu vm and set it, which is generally used for wired connection) or use Nat mode (for wireless connection). Then check whether the client (Windows10) can ping the server (Ubuntu) again. Run the following command on the client:

Ping xxx.xxx.xxx.XXX # xxx.xxx.xxx.XXX is the IP address of the serverCopy the code

If the ping fails, run sudo WFW disable on the Ubuntu terminal to disable the firewall.

Pit 2: Connection failure (Blue translation ￣,)

After doing the above, we try to reconnect again. We can find that the connection is still failed and display a message like reject the connection. This is because Xshell connects to Ubuntu through SSH protocol, but Ubuntu does not enable SSH service by default, so we need to run the following command on its terminal to enable SSH service:

sudo service ssh restart
Copy the code

Then we try to connect again.

Pit 3: Connection failure (Blue translation ￣,) (Blue translation ￣,)

By default, Ubuntu does not have SSH installed. In this case, we need to install SSH in Ubuntu and run the following command to install it:

sudo apt-get install openssh-server
Copy the code

Once it is installed, let’s connect to Ubunut again. If the apt process cannot be installed, you need to execute apt-get update to update it. If any error message related to lock cannot be obtained before the update, run the following command to disconnect the APT process and update it again:

# see apt process related to ps aux | grep apt # and then kill kill 9 apt process # # to update the apt - get apt - get the updateCopy the code

The above pit after stepping on the successful connection bar. Indeed, you will have successfully connected to Ubuntu, and then Xshell will open a terminal, which is the terminal under Ubuntu, that is, we can use this terminal to control Ubuntu, and Shell operations on it. Remember, (゚▽゚) Blue

3. Configuration and remote access of the Jupyter Notebook server

Since we had Anaconda installed earlier, Ubuntu was already integrated with Ipython and Jupyter-Notebook (Anaconda is so powerful). We will set up Jupyter Notebook’s server and access it remotely by using Xshell to connect to Ubuntu remotely.

  • Start ipython or Python on a terminal and execute the following command
from IPython.lib import passwd
passwd()
Copy the code

After the above command is executed, the terminal will display the password setting. For convenience, we set the password to 123 here, and then enter and confirm to complete the password setting.

Note: the password is in dark text and will not be displayed after you enter it. Please remember the password you entered, because we will need it to log in when we remotely access the Jupyter Notebook server

After entering the password, we will see that there is a long string, which is an encrypted form of the password above. We will need to copy it and use it in the Settings of the ipython_notebook_config.py file later.

  • Then to facilitate the above recording of the encrypted password string, we open another terminal here (which is where Xshell comes in handy) and create a server name using the following command, for example, here we set the name to: XXX
ipython profile create XXX
Copy the code

Py, ipython_kernel_config.py), and then use the following CD command to enter the **.ipython path CD./.ipython**. The specific operation diagram is as follows:

  • After entering the **. Ipython directory, the ls** command will list the two py files created above, but we need to create an additional PY configuration file and set it. In this case, we use vim to do this. Forgotten friends can go back and reacquaint themselves. After getting familiar with it, we complete the following steps:
cd .ipython/profile_txj     
vim ipython_notebook_config.py
Copy the code

After entering the vim environment, we will edit the following contents in the ipython_notebook_config.py file to configure the notebook login password (encrypted form) and the open port of the service:

c = get_config()
c.IPKernelApp.pyalb = "inline"
c.NotebookApp.ip = "*"
c.NotebookApp.open_browser = False
c.NotebookApp.allow_root = True
c.NotebookApp.password = U "Encrypted password"    # here we need to use the encrypted password above, visible on another terminal
c.NotebookApp.port = 8888   # Here, we need to set a Jupyter-notebook port, as rare as possible to avoid port conflicts
Copy the code

After editing, run the wq command to save the changes and exit.

  • With all that done, let’s now start the server, create a new terminal, and execute the following command
Jupyter notebook --config= [path of your ipython_notebook_config.py file] # Such as/home/LXJ /. Ipython profile_XXX/ipython_notebook_config. PyCopy the code

After the preceding command is executed, if the following information is displayed, it indicates that the server is started normally

  • After the server startup is complete, we open the browser on the client side and visit XXX.XXX.xxx.xxx :8888 (Ubuntu IP plus open port) to try and see if the request can be normal. If a Jupyter Notebook login page appears as shown above, you are now configured and able to access the Jupyter Notebook server remotely. To begin your deep learning journey, enter your password (the password you set in step 1, 123) in the form.

This is all about the configuration and remote access of the Jupyter Notebook server, but we will test it with a few simple examples.

Four, remote environment test

Installation of the Tensorflow software library

TensorFlow™ is an open source software library that uses Data Flow Graphs for numerical calculations. The Nodes represent mathematical operations, and the lines in the diagram represent arrays of multidimensional data related to each other between the Nodes, the tensor. Its flexible architecture allows you to scale computing across multiple platforms, such as one or more cpus (or Gpus) on desktop computers, servers, mobile devices, and more. TensorFlow was originally developed by researchers and engineers in the Google Brain Group (part of the Google Machine Intelligence Research Institute) for machine learning and deep neural networks, but the system’s versatility makes it widely usable in other computing fields.

After that, we need to install the Tensorflow package in Anaconda. Here we can install it in Anaconda virtual environment:

  1. Create a virtual environment using the create command and Python version in Anaconda
  2. Conda env list can see the current environment
  3. The activate command activates the required virtual environment
  4. Tensorflow2: tensorFlow2: tensorFlow2: TensorFlow2:
Create a virtual environment using the 'create' command in # Anaconda and the Python version. Tensorflow2 conda create --name tensorflow2 python=3.6 # 'conda env list Conda activate tensorflow2 PIP install -u tensorflow -i tensorflow2 PIP install -u tensorflow -i tensorflow2 PIP install -u tensorflow -i tensorflow2 PIP install -u tensorflow -i https://pypi.tuna.tsinghua.edu.cn/simple - the default - timeout = 1000 # GPU version installed: Conda create -n tf2 tensorflow -GPU. CUDA,cuDNN, and TensorFlow GPU are automatically installed after the command is executedCopy the code

After Tensorflow2 is installed in the virtual environment, to use the Jupyter Notebook, we also need to install ipkernel in the virtual environment. The CD/anaconda envs/tensorflow2 / bin, then execute. / python) :

conda install ipykernel
python -m ipykernel install --name tensorflow2 --display-name tensorflow2
Copy the code

After successful installation, run the Jupyter notebook command in the TensorFlow2 virtual environment to start. After login, you can specify the TensorFlow2 kernel environment to write code on the service page. By doing the above, we have done all the work. Let’s verify this to see if third-party packages under Anaconda work properly. To this end, we verify it through the following small cases:

  • Simple crawler for baidu and its sublinks (requests, BeautifulSoup)
  • Data visualization operations (numpy, Matplotlib, skimage)
  • Recognition of Fashion_MNist Images Based on Neural Network (Tensorflow, Keras)

Simple crawler

import requests
from bs4 import BeautifulSoup

def get_page(url, headers=None) :
    return requests.get(url).text
    
if __name__ == "__main__":
    baidu_url = "https://www.baidu.com"
    baidu_soup = BeautifulSoup(get_page(baidu_url), "html.parser")
    son_links = [biaoqian_a.attrs["href"] for biaoqian_a in baidu_soup.find_all("a")]
    for index, son_link in enumerate(son_links):
        print("Requesting page {}".format(str(index)))
        print(get_page(son_link))
Copy the code

Data visualization

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib as mpl
% matplotlib inline

for module in np, pd, mpl:
    print(module.__name__, module.__version__)

if __name__ == "__main__":
    ax =pd.DataFrame(np.random.randn(1000.6), columns=list('ABCDEF')).cumsum().plot(secondary_y=["D"."E"."F"])

    ax.set_ylabel('ABC plot')
    ax.right_ax.set_ylabel('DEF scale')
    ax.legend(loc='upper left')
    ax.right_ax.legend(loc='upper right')

    plt.show()
Copy the code

Recognition of fashion_mnist images is realized based on neural network

import tensorflow as tf
import numpy as np
from matplotlib import pyplot as plt
import matplotlib as mpl
%matplotlib inline

for module in tf, np, mpl:
    print(module.__name__, module.__version__)

if __name__ == "__main":
    (x_train_all, y_train_all), (x_test, y_test) = tf.keras.datasets.fashion_mnist.load_data()
    x_valid, x_train = x_train_all[:50000], x_train_all[50000:]
    y_valid, y_train = y_train_all[:50000], y_train_all[50000:]
    
    model = tf.keras.models.Sequential()
    model.add(tf.keras.layers.Flatten(input_shape = [28.28]))  Define the input layer
    model.add(tf.keras.layers.Dense(300, activation="relu")) Define the full connection layer (the most common neural network) with two hidden layers in between
    model.add(tf.keras.layers.Dense(100, activation="relu")) Define a full connection layer
    model.add(tf.keras.layers.Dense(10, activation="softmax")) Define the output layer
    
    model.compile(loss = "sparse_categorical_crossentropy",
                 optimizer = "adam",
                 metrics = ["accuracy"])
    
    history = model.fit(x_train, y_train, epochs=10,
             validation_data=(x_valid, y_valid))
    
    pd.DataFrame(history.history).plot(figsize=(8.5))
    plt.grid(True)
    plt.gca().set_ylim(0.1)
    plt.show()
Copy the code

conclusion

Through the above demonstration, we have completed the remote connection to Linux system and remote access to jupyter-notebook service using xshell in windows10, and verified the use of anaconda in Linux through several small examples. In this way, two PC-level coordination can be realized, one is used as a server to provide the environment, the other as a client to remote access and write code, and the advantages are obvious, not only can reduce the pressure of the computer to a certain extent, but also convenient management and easy to operate. In addition, the above environment setup is only a case, not the only one, for example, you can use Linux Centos, Aliyun cloud server, etc. If this helps you, please remember.゚. (. ◕ ‿ ◕.) ノ ゚. :. + ゚

2018-09-10, By Tao Tao, In Shanghai