A deep learning environment is built based on NVIDIA GPU and Docker container
GPU cloud host:
Operating System: Ubuntu 16.04 64-bit GPU: 1 x Nvidia Tesla P40
1. Install CUDA Driver
The Pre – 1.1 installation Actions
Install GCC, g++, make:
# sudo apt-get install GCC g++ make # GCC --version GCC (Ubuntu 5.4.0-6Ubuntu 1~16.04.10) 5.4.0 20160609 Copyright (C) # sudo apt-get install GCC g++ make # GCC --version GCC (Ubuntu 5.4.0-6Ubuntu 1~16.04.10) 5.4.0 20160609 Copyright (C) 2015 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.If not, install Linux-Headers:
# sudo apt-get install linux-headers-$(uname -r)
1.2 Install NVIDIA Driver
There are two ways to install CUDA: 1.Package install 2.Runfile install
This article chooses the RunFile installation.
First disable Nouveau:
# lsmod | grep nouveau nouveau 1495040 0 mxm_wmi16384 1 nouveau wmi20480 2 mxm_wmi,nouveau video 40960 1 nouveau i2c_algo_bit 16384 1 nouveau ttm94208 1 nouveau drm_kms_helper155648 1 nouveau drm 364544 3 ttm,drm_kms_helper,nouveau # vi /etc/modprobe.d/blacklist-nouveau.conf blacklist nouveau options nouveau modeset=0 # sudo update-initramfs -u Update-initramfs: Generating /boot/ initrd.imG-4.4.0-62-Generic W: mdadm: /etc/mdadm/mdadm.Reboot cloud host:
# reboot
Check Nouveau drivers not loaded after restart:
# lsmod | grep nouveau
#
Login: http://developer.nvidia.com/c… Download the corresponding runfile:
Wget # https://developer.nvidia.com/compute/cuda/10.0/Prod/local_installers/cuda_10.0.130_410.48_linuxStart installing CUDA Driver:
# chmod +x cuda_10.0.130_4100.48_Linux # sudo sh./ cuda_10.0.130_4100.48_Linux Logging to/TMP /cuda_install_1699.log Using more to view the EULA. Do you accept the previously read EULA? Accept/Decline/Quit: Accept Install NVIDIA Accelerated Graphics Driver for Linux-x86_64 410.48? (y)es/(n)o/(q)uit: y Do you want to install the OpenGL libraries? (y)es/(n)o/(q)uit [ default is yes ]: y Do you want to run nvidia-xconfig? This will update the system X configuration file so that the NVIDIA X driver is used. The pre-existing X configuration file will be backed up. This option should not be used on systems that require a custom X configuration, (y)es/(n)o/(q)uit [Default is no]: Install the CUDA 10.0 Toolkit? (y)es/(n)o/(q)uit: y Enter Toolkit Location [default is /usr/local/cuda-10.0]: Do you want to install a symbolic link at /usr/local/cuda? (y)es/(n)o/(q)uit: Y Install The Cuda 10.0 Samples? (y)es/(n)o/(q)uit: y Enter CUDA Samples Location [ default is /root ]: Installing the NVIDIA display driver... Installing the Cuda Toolkit in /usr/local/cuda-10.0... Missing recommended library: libGLU.so Missing recommended library: libX11.so Missing recommended library: libXi.so Missing recommended library: libXmu.so Installing the CUDA Samples in /root ... Coping samples to /root/ nvidia_cuda-10.0_samples now... Finished copying samples. =========== = Summary = =========== Driver: Installed Toolkit: Installed in/usr/local/cuda 10.0 Samples: Installed in /root, But missing recommended libraries Please make sure that -path includes /usr/local/cuda-10.0/ bin-library_path The includes/usr/local/lib64 / cuda 10.0, or, Add /usr/local/cuda-10.0/lib64 to /etc/lda.so.conf and run ldconfig as root to uninstall the Cuda Toolkit, add /usr/local/cuda-10.0/lib64 to /etc/lda.so.conf and run ldconfig as root to uninstall the Cuda Toolkit, Run the uninstall script in /usr/local/cuda-10.0/bin To uninstall the NVIDIA Driver Run nvidia-uninstall Please see cuda_installation_guide_linux.pdf in /usr/local/cuda-10.0/doc/ PDF for detailed information on setting up CUDA. Logfile is /tmp/cuda_install_1699.logInstallation successful!
Reboot cloud host:
# reboot
Equipment verification:
# ls /dev/nvidia* ls: cannot access '/dev/nvidia*': No such file or directory # vi nvidia-probe.sh #! /bin/bash ### BEGIN INIT INFO # Provides: jd.com # Required-Start: $local_fs $network # Required-Stop: $local_fs # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: nvidia service # Description: nvidia service daemon ### END INIT INFO /sbin/modprobe nvidia if [ "$?" -eq 0 ]; then # Count the number of NVIDIA controllers found. NVDEVS=`lspci | grep -i NVIDIA` N3D=`echo "$NVDEVS" | grep "3D controller" | wc -l` NVGA=`echo "$NVDEVS" | grep "VGA compatible controller" | wc -l` N=`expr $N3D + $NVGA - 1` for i in `seq 0 $N`; do mknod -m 666 /dev/nvidia$i c 195 $i done mknod -m 666 /dev/nvidiactl c 195 255 else exit 1 fi /sbin/modprobe nvidia-uvm if [ "$?" -eq 0 ]; then # Find out the major device number used by the nvidia-uvm driver D=`grep nvidia-uvm /proc/devices | awk '{print $1}'` mknod -m 666 /dev/nvidia-uvm c $D 0 else exit 1 fi # chmod +x nvidia-probe.sh # ./nvidia-probe.sh # ls /dev/nvidia* /dev/nvidia0 /dev/nvidiactl /dev/nvidia-uvmDevice found successfully under /dev!
Configure bootstrap from startup:
# cp nvidia-probe.sh /etc/init.d/
# sudo update-rc.d nvidia-probe.sh defaults 95
1.3 Post – installation Actions
Configure environment variables:
# vi /etc/profile ...... The export PATH = / usr/local/bin/cuda 10.0 ${PATH: + : ${PATH}} export LD_LIBRARY_PATH = / usr/local/lib64 / cuda 10.0 ${LD_LIBRARY_PATH: + : ${LD_LIBRARY_PATH}}Persistence Daemon:
# vi /etc/rc.local
......
/usr/bin/nvidia-persistenced --verbose
exit 0
1.4 CUDA Driver verification
View Driver Version:
# cat /proc/driver/nvidia/version NVRM version: NVIDIA UNIX x86_64 Kernel Module 410.48 Thu Sep 6 06:36:33 CDT 2018 GCC Version: GCC Version 5.4.0 20160609 (Ubuntu 5.4.0- 6Ubuntu 1~16.04.10)Verify with the DeviceQuery example:
# CD ~/ nvidia_cuda-10.0_samples /1_Utilities/deviceQuery/ # make "/usr/local/cuda-10.0"/bin/ nvcc-ccbin g++ -i.. /.. /common/inc -m64-gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_75,code=compute_75 -o deviceQuery.o -c deviceQuery.cpp "/usr/local/cuda-10.0"/bin/ nvcc-ccbin g++ -m64 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_75,code=compute_75 -o deviceQuery deviceQuery.o mkdir -p .. /.. /bin/x86_64/linux/release cp deviceQuery .. /.. /bin/x86_64/linux/release # cd .. /.. /bin/x86_64/linux/release/ # ls deviceQuery # ./deviceQuery ./deviceQuery Starting... CUDA Device Query (Runtime API) version (CUDART static linking) Detected 1 CUDA Capable device(s) Device 0: "Tesla P40" CUDA Driver Version/Runtime Version 10.0/10.0 CUDA Capability Major/Minor Version number:6.1 Total amount of global memory: 22919 MBytes (24032378880 bytes) (30) Multiprocessors, (128) CUDA Cores/MP: 3840 CUDA Cores GPU Max Clock Rate :1531 MHz (1.53GHz) Memory Clock Rate: 3615 MHz Memory Bus Width: 384-bit L2 Cache Size: 3145728 bytes Maximum Texture Dimension Size (x,y,z) 1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384) Maximum Layered 1D Texture Size, (num) layers 1D=(32768), 2048 layers Maximum Layered 2D Texture Size, (num) layers 2D=(32768, 32768), 2048 layers Total amount of constant memory: 65536 bytes Total amount of shared memory per block: 49152 bytes Total number of registers available per block: 65536 Warp size: 32 Maximum number of threads per multiprocessor: 2048 Maximum number of threads per block: 1024 Max dimension size of a thread block (x,y,z): (1024, 1024, 64) Max dimension size of a grid size(x,y,z): (2147483647, 65535, 65535) Maximum memory pitch: 2147483647 bytes Texture alignment: 512 bytes Concurrent copy and kernel execution: Yes with 2 copy engine(s) Run time limit on kernels: No Integrated GPU sharing Host Memory:No Support host page-locked memory mapping: Yes Alignment requirement for Surfaces:Yes Device has ECC support:Enabled Device supports Unified Addressing (UVA): Yes Device supports Compute Preemption:Yes Supports Cooperative Kernel Launch:Yes Supports MultiDevice Co-op Kernel Launch: Yes Device PCI Domain ID / Bus ID / location ID: 0 / 0 / 7 Compute Mode: < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) > deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 10.0, CUDA Runtime Version = 10.0, NumDevs = 1 Result = PASSReference:
https://github.com/NVIDIA/nvi…
https://docs.nvidia.com/cuda/…
2. Install Nvidia – docker
2.1 installation Docker
Install the docker – ce:
#sudo apt-get remove docker docker-engine docker.io # sudo apt-get install \ apt-transport-https \ ca-certificates \ curl \ software-properties-common # curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - # sudo add-apt-repository \ "deb [arch=amd64] https://download.docker.com/linux/ubuntu \ $(lsb_release -cs) \ stable" # sudo Apt-get update # sudo apt-get install docker-ce # docker version Client: version: 18.06.1-ce API version: 1.38 Go Version: Go1.10.3 Git Commit: E68FC7a Built: Tue Aug 21 17:24:56 2018 OS/Arch: Linux/AMD64 Experimental: False Server: Engine: Version: 18.06.1-ce API Version: 1.38 (minimum Version 1.12) Go Version: go1.10.3 Git commit: e68fc7a Built:Tue Aug 21 17:23:21 2018 OS/Arch: linux/amd64 Experimental: false2.2 install nvidia – docker
Install nvidia – docker:
# Add the package repositories curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | \ sudo apt-key add - distribution=$(. /etc/os-release; echo $ID$VERSION_ID) curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | \ sudo tee /etc/apt/sources.list.d/nvidia-docker.list sudo apt-get update # Install nvidia-docker2 and reload the Docker daemon configuration sudo apt-get install -y nvidia-docker2 sudo pkill -SIGHUP dockerdVerify the nvidia – docker:
# docker run --runtime=nvidia --rm nvidia/cuda:9.0-base nvidia-smi Thu Oct 25 09:03:27 2018 + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- + | NVIDIA SMI 410.48 Driver Version: 410.48 | | -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- + | | GPU NamePersistence - M Bus-IdDisp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = | | 0 Tesla P40 On | 00000000:00:07. 0 Off |0 | | N/A 20CP8 9W / 250W | 0MiB / 22919MiB | 1% Default | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+2.3 Configure Docker default Runtime
cat /etc/docker/daemon.json
{
"default-runtime": "nvidia",
"runtimes": {
"nvidia": {
"path": "nvidia-container-runtime",
"runtimeArgs": []
}
}
}
Restart service:
# systemctl restart docker
# systemctl status docker
2.4 Run the TensorFlow convolutional neural Model
Docker run:
# docker run - rm - name tensorflow - ti tensorflow/tensorflow: r0.9 devel - gpu root @ bd0fb3758da2: ~ # python -- version Python 2.7.6 root @ bd0fb3758da2: ~ # Python - m tensorflow models. Image. Mnist. ConvolutionalReference:
https://docs.docker.com/insta…
https://github.com/NVIDIA/nvi…