Software and Hardware Environment

  • Ubuntu 18.04 64 – bit
  • NVidia GTX 1070Ti
  • Anaconda with python 3.7
  • CUDA 10.1
  • CuDNN 7.6
  • Opencv 3.4.2
  • Caffe 1.0.0

Introduction to the

Let’s start with the environment, using Anaconda’s Python virtual environment, support for OpencV, support for CUDA and cuDNN acceleration, and support for calling Caffe in Python. For the basic components, refer to the previous article, which will not be covered here

  • Anaconda Basic use
  • Ubuntu installs CUDA and cuDNN
  • Opencv source compilation, support CUDA

Basic Environment Preparation

Install dependency packages and tools

sudo apt install build-essential cmake git ffmpeg libatlas-base-dev libtiff-dev pkg-config python3-dev libavcodec-dev libavformat-dev libswscale-dev libtbb-dev libjpeg-dev libpng-dev libavcodec-dev libavformat-dev libswscale-dev libv4l-dev libx264-dev libboost-all-dev libhdf5-serial-dev libleveldb-dev liblmdb-dev libhdf5-dev
pip install protobuf
Copy the code

opencv

I’ve singled out OpencV here because there are so many ways to install OpencV

  • apt install python3-opencv
  • conda install opencv
  • The source code to compile

Using apt Install is the easiest and least error-prone way to install; The second is conda install, the most likely problem is to compile their own source code, compilation parameters are complex, many dependent libraries, but also version differences.

After the installation is complete, it is recommended to use the opencv_version command to view the current version, the default Ubuntu 18.04 source is 3.2.0, conda will be higher, here is 3.4.0, if you install the source code, Note that sudo ldconfig is executed after sudo make install. This article is installed as conda.

Compile the caffe

Caffe is now ready to compile

git clone https://github.com/BVLC/caffe.git
cd caffe
cp Makefile.config.example Makefile.config
Copy the code

Edit the makefile.config file, mainly some path changes, and paste the changed

## Refer to http://caffe.berkeleyvision.org/installation.html
# Contributions simplifying and improving our build system are welcome!

# cuDNN acceleration switch (uncomment to build with cuDNN).
# 启用cuDNN加速
USE_CUDNN := 1

# CPU-only switch (uncomment to build without GPU support).
# CPU_ONLY := 1

# uncomment to disable IO dependencies and corresponding data layers
# 启用opencv
USE_OPENCV := 1
# USE_LEVELDB := 0
# USE_LMDB := 0
# This code is taken from https://github.com/sh1r0/caffe-android-lib
USE_HDF5 := 1

# uncomment to allow MDB_NOLOCK when reading LMDB files (only if necessary)
#  You should not set this flag if you will be reading LMDBs with any
#  possibility of simultaneous read and write
# ALLOW_LMDB_NOLOCK := 1

# Uncomment if you're using OpenCV 3
# opencv大版本号是3，这里一定要注意
OPENCV_VERSION := 3

# To customize your choice of compiler, uncomment and set the following.
# N.B. the default for Linux is g++ and the default for OSX is clang++
# CUSTOM_CXX := g++

# CUDA directory contains bin/ and lib/ directories that we need.
CUDA_DIR := /usr/local/cuda
# On Ubuntu 14.04, if cuda tools are installed via
# "sudo apt-get install nvidia-cuda-toolkit" then use this instead:
# CUDA_DIR := /usr

# CUDA architecture setting: going with all of them.
# For CUDA < 6.0, comment the *_50 through *_61 lines for compatibility.
# For CUDA < 8.0, comment the *_60 and *_61 lines for compatibility.
# For CUDA >= 9.0, comment the *_20 and *_21 lines for compatibility.
# -gencode arch=compute_20,code=sm_21
# CUDA_ARCH := -gencode arch=compute_20,code=sm_20 
# 这里使用的是CUDA10.1，所以要注释掉前两行
CUDA_ARCH := -gencode arch=compute_30,code=sm_30 \
      -gencode arch=compute_35,code=sm_35 \
      -gencode arch=compute_50,code=sm_50 \
      -gencode arch=compute_52,code=sm_52 \
      -gencode arch=compute_60,code=sm_60 \
      -gencode arch=compute_61,code=sm_61 \
      -gencode arch=compute_61,code=compute_61

# BLAS choice:
# atlas for ATLAS (default)
# mkl for MKL
# open for OpenBlas
BLAS := atlas
# Custom (MKL/ATLAS/OpenBLAS) include and lib directories.
# Leave commented to accept the defaults for your choice of BLAS
# (which should work)!
# BLAS_INCLUDE := /path/to/your/blas
# BLAS_LIB := /path/to/your/blas

# Homebrew puts openblas in a directory that is not on the standard search path
# BLAS_INCLUDE := $(shell brew --prefix openblas)/include
# BLAS_LIB := $(shell brew --prefix openblas)/lib

# This is required only if you will compile the matlab interface.
# MATLAB directory should contain the mex binary in /bin.
# MATLAB_DIR := /usr/local
# MATLAB_DIR := /Applications/MATLAB_R2012b.app

# NOTE: this is required only if you will compile the python interface.
# We need to be able to find Python.h and numpy/arrayobject.h.
# Python头文件路径，再加上numpy的头文件路径
PYTHON_INCLUDE := /home/xugaoxiang/anaconda3/include/python3.7m \
      /home/xugaoxiang/anaconda3/lib/python3.7/site-packages/numpy/core/include
# Anaconda Python distribution is quite popular. Include path:
# Verify anaconda location, sometimes it's in root.
# ANACONDA_HOME := $(HOME)/anaconda
# PYTHON_INCLUDE := $(ANACONDA_HOME)/include \
      # $(ANACONDA_HOME)/include/python2.7 \
      # $(ANACONDA_HOME)/lib/python2.7/site-packages/numpy/core/include

# Uncomment to use Python 3 (default is Python 2)
# 默认是python2，这里使用python3，一定要改，不然后面会报错相应没人在用python2了吧
PYTHON_LIBRARIES := boost_python3 python3.7m
# PYTHON_INCLUDE := /usr/include/python3.5m \
#                 /usr/lib/python3.5/dist-packages/numpy/core/include

# We need to be able to find libpythonX.X.so or .dylib.
# libpython*.so库的路径
PYTHON_LIB := /home/xugaoxiang/anaconda3/lib
# 如果设置了ANACONDA_HOME环境变量，可以使用下面的设置方法，作用一样
# PYTHON_LIB := $(ANACONDA_HOME)/lib

# Homebrew installs numpy in a non standard path (keg only)
# PYTHON_INCLUDE += $(dir $(shell python -c 'import numpy.core; print(numpy.core.__file__)'))/include
# PYTHON_LIB += $(shell brew --prefix numpy)/lib

# Uncomment to support layers written in Python (will link against Python libs)
# WITH_PYTHON_LAYER := 1

# Whatever else you find you need goes here.
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /home/xugaoxiang/anaconda3/include
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/x86_64-linux-gnu/hdf5/serial/

# If Homebrew is installed at a non standard location (for example your home directory) and you use it for general dependencies
# INCLUDE_DIRS += $(shell brew --prefix)/include
# LIBRARY_DIRS += $(shell brew --prefix)/lib

# NCCL acceleration switch (uncomment to build with NCCL)
# https://github.com/NVIDIA/nccl (last tested version: v1.2.3-1+cuda8.0)
# USE_NCCL := 1

# Uncomment to use `pkg-config` to specify OpenCV library paths.
# (Usually not necessary -- OpenCV libraries are normally installed in one of the above $LIBRARY_DIRS.)
# 启用pkg_config，方便caffe找到opencv
USE_PKG_CONFIG := 1

# N.B. both build and distribute dirs are cleared on `make clean`
# 默认编译的目录，所有的目标文件、可执行文件、库都存放在这里
BUILD_DIR := build
DISTRIBUTE_DIR := distribute

# Uncomment for debugging. Does not work on OSX due to https://github.com/BVLC/caffe/issues/171
# 是否打开debug信息
# DEBUG := 1

# The ID of the GPU that 'make runtest' will use to run unit tests.
TEST_GPUID := 0

# enable pretty build (comment to see full commands)
Q ?= @
Copy the code

When complete, execute

make all -j12
Copy the code

The -j parameter refers to the number of CPU cores to be used. The purpose is to speed up compilation

In order to be able to call caffe in Python, you need to execute

make pycaffe -j12
Copy the code

At this point, the entire compilation is over.

validation

Use the Ipython environment for testing

If you are careful, you will notice that if you open a new terminal, you will open ipython and import caffe, but you will get an error. Why?

In Terminal, which does not report errors, you can see this by looking at the environment variables

We export the environment variable PYTHONPATH during compiling caffe, so we need to do the same before using it

For simplicity, you can write the declaration to ~/.bashrc so you don’t have to execute it every time

export PYTHONPATH=/home/xugaoxiang/Works/github/caffe/python:$PYTHONPATH
Copy the code

Q & A

Q1

A TIFF error was encountered during compilation

This is caused by the previous opencv source compilation. It is important to note that if you are compiling opencv from the source, you must add -d BUILD_TIFF=ON when configuring opencv. Also, try not to have both apt and Conda installed environments at the same time, it is more error prone for beginners.

Q2

When importing caffe into ipython, an error occurs

Add libhdf5_Hl.so. 100 to LD_LIBRARY_PATH

export LD_LIBRARY_PATH=/home/xugaoxiang/anaconda3/lib:$LD_LIBRARY_PATH
Copy the code

Q3

When importing caffe into ipython, an error occurs

Change makefile. config to PYTHON_LIBRARIES

PYTHON_LIBRARIES: = boost_python3 python3.7 mCopy the code

The default is PYTHon2

Q4

In caffe, opencV4 is compiled using source code, due to opencV4 version differences, will report errors

This is because in OpencV4, the original CV_LOAD_IMAGE_COLOR and CV_LOAD_IMAGE_GRAYSCALE macros have been changed to CV ::IMREAD_COLOR and CV ::ImreadModes::IMREAD_GRAYSCALE, Therefore, you need to find and replace the caffe source directory to compile successfully

Q5

Hdf5 header file not found during compilation? This is true even after the installation command has been executed

sudo apt install libhdf5-dev
Copy the code
src/caffe/layers/hdf5_data_layer.cu:10:10: fatal error: hdf5.h: No such file or directory
 #include "hdf5.h"
          ^~~~~~~~
compilation terminated.
Makefile:604: recipe for target '.build_release/cuda/src/caffe/layers/hdf5_data_layer.o' failed
make: *** [.build_release/cuda/src/caffe/layers/hdf5_data_layer.o] Error 1
make: *** Waiting for unfinished jobs....
Copy the code

Add /usr/include_dir /hdf5/serial to INCLUDE_DIR and /usr/lib/x86_64-linux-gnu/hdf5/serial to LIBRARY_DIR

The resources

  • Anaconda Basic use
  • Ubuntu installs CUDA and cuDNN
  • Opencv source compilation, support CUDA
  • Github.com/BVLC/caffe/…