[toc]
1 the Python upgrade
Federated Learning unified Python version is 3.6.8, so it is required in
1 Check the Python version
When Python is installed on Linux (the default), you can view the Python version number by typing a simple command:
# python -Vå
Python÷ 2.7.5
Copy the codeOr:
# python --version
Python 2.7.5
Copy the codeAs you can see, the built-in Python version is 2.7.5.
2 Download the new version
Go to the Python download page and select the required version.
In this case, THE version I chose was 3.6.8.
Wget # https://www.python.org/ftp/python/3.6.8/Python-3.6.8.tgzCopy the code
3 the decompression
Once the download is complete, unzip it:
# tar ZXVF Python - 3.6.8 TGZCopy the code4 Installation and Configuration
ModuleNotFoundError: No module named ‘_ssl’
The solutions are as follows:
(1) First go to the python-3.6.3 directory, which you unzipped to install Python3.6.
(2) CD to the Modules directory, go to the Setup and setup. dist files, and uncomment the following lines
Change under Setup:
Vim Setup (approximately 51% position)
Changes under setup.dist
Vim setup.dist (approximately at 50%)
(3). Compilation and installation
Go to the decompressed directory and install and configure:
yum install openssl openssl-devel
Copy the codeIf an error occurs when executing./configure
configure: error: no acceptable C compiler found in $PATH
The proper compiler is not installed. At this point, GCC and other dependencies need to be installed/upgraded.
# yum install make gcc gcc-c++
Copy the codeWhen done, re-execute:
# ./configure --prefix=/usr/local/python3 --enable-optimizations
Copy the code
5 Compilation and Installation
Once configured, it is ready to compile:
# make
Copy the codeThe long wait… When complete, install:
# make install
Copy the code6 validation
After the installation is successful, you can view the Python version:
Python 2.7.5 # python3 -v python 3.5.2Copy the codeOne is the old version 2.x and the other is the new version 3.x.
** Note: ** there is a python3 link under /usr/local/bin/to Python 3.5 in the bin directory.
7 Set 3.x to the default version
Check the Python path, under /usr/bin. You can see that Python is linked to Python 2.7, so executing Python is equivalent to executing Python 2.7.
# the ls - al/usr/bin | grep python - rwxr-xr-x mto. 1 root root on December 1, 11216, 2015 abrt - action - analyze - python LRWXRWXRWX. 1 root Root 7 August 30 12:11 python -> python2 LRWXRWXRWX. 1 root root 9 August 30 12:11 python2 -> python2.7 -rwxr-xr-x. 1 root root 7136 November 20 2015 python2.7Copy the codeRename the original Python soft link:
# mv /usr/bin/python /usr/bin/python.bak
Copy the codeLink Python to python3:
# ln -s/usr/local/python3 / bin/python3.6 / usr/bin/pythonCopy the codeAt this point, look at the Python version:
# python -V
Python 3.5.2
Copy the codeThe output is 3.x, indicating python3 is already in use.
Update the PIP
ln -s /usr/local/python3/bin/pip3 /usr/bin/pip
Copy the code8 configuration yum
After upgrading Python, the default Python is pointed to python3, so yum cannot be used properly. You need to edit the yum configuration file:
# vim /usr/bin/yum
Copy the codeModify at the same time:
# vim /usr/libexec/urlgrabber-ext-down
Copy the codeThe #! The/usr/bin/python to #! /usr/bin/python2.7, save and exit.
9 Troubleshooting SSL problems
If the SSL installation is incorrect or SSL is not installed, you can pass the following check
Note SSL is not installed
Solution to python3 installation directory/usr/local/python3 / Python – 3.6.8 / Modules/Setup file, remove the four lines below
recompile
/configure –prefix=/usr/local/python make make install
C :72:25: FATAL error: Openssl /rsa.h: No such file or directory:
Yum install openssl-devel. /configure –prefix=/usr/local/python3 make && make install
2 Bazel installation
Address: www.jianshu.com/p/bc542266a…
Bazel version 0.26.1.
1 Use the binary installation program
The binary installer is located on Bazel’s GitHub distribution page.
The installer contains Bazel binary 1. Several other libraries must be installed to make Bazel work.
2 Step 1: Install the required packages
First, install prerequisites: pkg-config, zip, g++, zlib1g-dev, unzip, and python.
sudo apt-get install pkg-config zip g++ zlib1g-dev unzip python
Copy the codeStep 2: Download Bazel
Version is limited to 0.26.1.
Next, bazel-
-installer-linux-x86_64.sh downloads the Bazel binary installer from the Bazel distribution page on GitHub.
Enter the following content in Terminal to download the latest version (before September 08, 2018)
Wget HTTP: / / https://github.com/bazelbuild/bazel/releases/download/0.26.1/bazel-0.26.1-installer-linux-x86_64.shCopy the code150 megabytes of files. Take a break after you start downloading.
4 Step 3: Run setup
Run Bazel installer as follows:
//
chmod +x bazel-<version>-installer-linux-x86_64.sh
./bazel-<version>-installer-linux-x86_64.sh --user
// The following is the corresponding version
chmod +x bazel0.261.-installer-linux-x86_64.sh
./bazel0.261.-installer-linux-x86_64.sh --user
Copy the codeThe –user flag installs Bazel into a directory on the $HOME/bin system and sets the. Bazelrc path $HOME/.bazelrc. Use the –help command to view additional installation options.
Step 4: Set up your environment
If you run the Bazel installer with the flag above –user, the Bazel executable will be installed in your $HOME/bin directory. It is a good idea to add this directory to the default path, as follows:
export PATH="$PATH:$HOME/bin"
Copy the codeYou can also add this command to your ~/.bashrc file.
6 Important step: Modify bazel’s cache path
This command does not work the build - disk_cache = / export/dubaokun/tools /. Bazel_cache try environment variables, Join the environment variable in/etc/profile export TEST_TMPDIR = / export/dubaokun/tools /. Bazel_cacheCopy the codeSince I didn’t set bazel’s cache path beforehand, I changed it manually later.
lrwxrwxrwx 1 root root 108 Mar 30 19:51 bazel-bin -> /root/.cache/bazel/_bazel_root/27509bc6e2111f3da3abc402b7d8c310/execroot/org_tensorflow/bazel-out/k8-opt/bin lrwxrwxrwx 1 root root 108 Mar 30 19:51 bazel-genfiles -> /root/.cache/bazel/_bazel_root/27509bc6e2111f3da3abc402b7d8c310/execroot/org_tensorflow/bazel-out/k8-opt/bin lrwxrwxrwx 1 root root 97 Mar 30 19:51 bazel-out -> /root/.cache/bazel/_bazel_root/27509bc6e2111f3da3abc402b7d8c310/execroot/org_tensorflow/bazel-out lrwxrwxrwx 1 root root 87 Mar 30 19:51 bazel-tensorflow -> /root/.cache/bazel/_bazel_root/27509bc6e2111f3da3abc402b7d8c310/execroot/org_tensorflow lrwxrwxrwx 1 root root 113 Mar 30 19:51 bazel-testlogs -> /root/.cache/bazel/_bazel_root/27509bc6e2111f3da3abc402b7d8c310/execroot/org_tensorflow/bazel-out/k8-opt/testlogs -rw-r--r-- 1 root root 122 Mar 30 16:16 BUILD ----- rm bazel-bin ln -s /export/dubaokun/tools/.bazel_cache/_bazel_root/27509bc6e2111f3da3abc402b7d8c310/execroot/org_tensorflow/bazel-out/k8-op t/bin/ bazel-bin rm bazel-genfiles ln -s /export/dubaokun/tools/.bazel_cache/_bazel_root/27509bc6e2111f3da3abc402b7d8c310/execroot/org_tensorflow/bazel-out/k8-op t/bin/ bazel-genfiles rm bazel-out ln -s /export/dubaokun/tools/.bazel_cache/_bazel_root/27509bc6e2111f3da3abc402b7d8c310/execroot/org_tensorflow/bazel-out bazel-out rm bazel-tensorflow ln -s /export/dubaokun/tools/.bazel_cache/_bazel_root/27509bc6e2111f3da3abc402b7d8c310/execroot/org_tensorflow bazel-tensorflow rm bazel-testlogs ln -s /export/dubaokun/tools/.bazel_cache/_bazel_root/27509bc6e2111f3da3abc402b7d8c310/execroot/org_tensorflow/bazel-out/k8-op t/testlogs /export/dubaokun/tools/.bazel_cache/_bazel_root/27509bc6e2111f3da3abc402b7d8c310/external /export/dubaokun/tools/.bazel_cache/_bazel_root/27509bc6e2111f3da3abc402b7d8c310/external/eigen_archive/49177915a14a.tar .gz: All mirrors are down: [GET returned 404 Not Found, Read timed out]Copy the code3 Protocol Buffer installation
Version 3.8.0, specific version
Address: www.jianshu.com/p/00be93ed2…
1 introduction
Protocol Buffers (Protobuf) is a cross-language, cross-platform, scalable serialization tool developed by Google. Compared with XML and JSON, Protobuf provides high performance. Therefore, Protobuf encoding format (file suffix.proto) has been widely used in network development. Protoc, as a compiler of Protobuf, can generate multiple languages from Protobuf definition file.proto (e.g. C++, Java, Go, Python, etc.) type definition file and codec operation code,
This article focuses on how to install and use Protoc.
2 Source code compilation and installation
Official source: github.com/google/prot…
Protoc has the following dependencies: autoconf, automake, libtool, curl, make, g++, unzip, gmock. Gmock depends on libtool
(2) Compilation and installation:
git clone https://github.com/google/protobuf
./autogen.sh Create configure file
./configure
make
make check
sudo make install
sudo ldconfig
Copy the codeLibprotobuf, libprotobuf-lite, libprotoc will be generated in /usr/lib after installation. Then, we need to execute LDConfig to update the lib path. If protoc execution still fails after the above steps are completed, we can manually link as follows
/etc/ld.so.conf.d/libprotobuf.conf
/usr/local/lib
Copy the codeLn -s/usr/lib/libprotobuf. So the 10.0.0 / usr/lib/libprotobuf. SoCopy the code3 Binary Installation
If you don’t want to compile and install it, you can download the binary package to install it. We can get the compiled Protoc binary directly from here.
For example, download the following two packages:
protoc-3.3.0-win32.zip
protoc-3.3.0-linux-x86_64.zip
Copy the codeDecompress the package to PATH to complete the installation
Sudo unzip protoc-3.0.0-beta-3-linux-x86_64.zip sudo chmod a+x protocCopy the codeView user Help
protoc --help
Copy the codeThe official documentation
4 Example
With the Protoc installed, we can install examples from the official manual to try it out
1) Compile.proto definition file
syntax = "proto3"
package tutorial;
message Person {
string name = 1;
int32 id = 2;
string email = 3;
enum PhoneType {
MOBILE = 0;
HOME = 1;
WORK = 2;
}
message PhoneNumber {
string number = 1;
PhoneType type = 2 [default = HOME];
}
repeated PhoneNumber phone = 4;
}
message AddressBook {
repeated Person person = 1;
}
Copy the code- compile
.protoFile to generate the definition and operation file of the desired language
For example, if we want to generate C++ source code here, execute:
protoc --cpp_out=. addressbook.proto
Copy the codeThe generated files are addressbook.pb.h, addressbook.pb.cc
- Put the above generated
.pb.ccFiles and our application C++ files are compiled
g++ addressbook.pb.cc readinput.cpp -lprotobuf
Copy the code5 Protoc usage and options
Protoc –help can be used to see how to use it
Protoc usage format:
protoc [OPTION] PROTO_FILES
Copy the code-IPATH, --proto_path=PATH Specify the directory in which to search for imports.
May be specified multiple times;
If not given, the current working directory is used.
--plugin=EXECUTABLE Specifies a plugin executable to use.
Normally, protoc searches the PATH for
plugins, but you may specify additional
executables not in the path using this flag.
Additionally, EXECUTABLE may be of the form
NAME=PATH, in which case the given plugin name
is mapped to the given executable even if
the executable's own name differs. --cpp_out=OUT_DIR Generate C++ header and source. --csharp_out=OUT_DIR Generate C# source file. --java_out=OUT_DIR Generate Java source file. --javanano_out=OUT_DIR Generate Java Nano source file. --js_out=OUT_DIR Generate JavaScript source. --objc_out=OUT_DIR Generate Objective C header and source. --php_out=OUT_DIR Generate PHP source file. --python_out=OUT_DIR Generate Python source file. --ruby_out=OUT_DIR Generate Ruby source file.Copy the codeExample:
protoc -I .. /.. /protos --grpc_out=. --plugin=protoc-gen-grpc=`whichgrpc_cpp_plugin` .. /.. /protos/route_guide.proto protoc -I .. /.. /protos --cpp_out=. .. /.. /protos/route_guide.protoCopy the code4 Installing Software Packages
You must be the Root user.
1 Installation Commands
sudo pip install -i pypi.tuna.tsinghua.edu.cn/simple numpy grpcio Keras-Applications Keras-Preprocessing h5py requests enum mock
Sudo PIP install -i pypi.mirrors.ustc.edu.cn/simple/ numpy = = 1.18.5 grpcio Keras – Applications Keras – Preprocessing h5py requests enum mock
2 Mirror Address
Image address, change the download path of PIP
-
Ali cloud mirrors.aliyun.com/pypi/simple…
-
Pypi.mirrors.ustc.edu.cn/simple/ China university of science and technology
-
Douban pypi.douban.com/simple/
-
Tsinghua university pypi.tuna.tsinghua.edu.cn/simple/
-
University of science and technology of China pypi.mirrors.ustc.edu.cn/simple/
Run CMD as an administrator and enter the following command
pip install tensorflow -i pypi.tuna.tsinghua.edu.cn/simple/ pip install opencv-python -i pypi.tuna.tsinghua.edu.cn/simple/
5 Tensorflow source code compilation
The reason why I tried to compile TF source code is very simple. First, the following warning appears every time it runs, which makes me feel a bit “uncomfortable”.
The preparatory work
1. View the information that can be optimized
Install the normal TF directly with PIP, and then run a random program. If there are no performance tuning hints, then “Congratulations”, nothing else to do. This is as fast as your computer can go. Note the names of the instruction sets mentioned in the warning for later use if there is room for optimization.
2. Install the Bazel
Bazel is a tool for compiling TensorFlow. First of all, in accordance with theThis web pageTo download the REPO file.
cp vbatts-bazel-epel-7.repo /etc/yum.repos.d/
yum install bazel
Copy the codeOther dependencies may be installed in the meantime, so just wait. This is done as follows.
3.Git TensorFlow
Enter a command.
git clone --recurse-submodules https://github.com/tensorflow/tensorflow
Copy the codeThe above command will create a folder called “tensorflow” in your current folder, where the downloaded files will be stored. As shown in the figure below.
yum install gitCan.
Compile TensorFlow
1. Configure and compile
Enter the following command to switch the current directory to Git’s tensorflow and run the configuration program.
cd tensorflow/
./configure
Copy the code
2. Start compiling
Enter a command.
bazel build --jobs 10 --config=opt //tensorflow/tools/pip_package:build_pip_package =========== bazel build -c opt - copt = - msse4.1 copt = - msse4.2 - copt = - mavx / / tensorflow/tools/pip_package: build_pip_packageCopy the codeThe above command generates a script called build_pip_package. The parameters of this step are the key to optimization, and the optimization instructions are specified here. It is important to note that what is used is not fixed. The code here is optimizable on my VPS, not on your computer. For example, the following command is another server supported optimization directive.
Bazel build -copt --copt= -MSse4.1 --copt= -MSse4.2 --copt=-mavx --copt=-mavx2 --copt= -mfMA //tensorflow/tools/pip_package:build_pip_packageCopy the codeA better way is to install a public TF with PIP as mentioned before, and then run casually, look at the Warning information, write it down, and then configure it here. Otherwise the CPU does not support, but compiled, when the time to use it may report an error. Note that this is a very, very slow process, please be patient. The compilation screen is as follows.
And it could break in the middle. If the network is not good, it is recommended to open a screen compiler, otherwise the middle of the compilation for a long time result SSH disconnection, but do not worry too much, or will start from the breakpoint. Here’s what a restart looks like, with a hint “Found 1 target… “.
Run the script to create the PIP installation package in the/TMP /tensorflow_pkg folder, which will be generated quickly.
. / bazel - bin/tensorflow/tools/pip_package/build_pip_package/TMP/tensorflow_pkg_python3. 6.8 = = = = = = = = = = ./bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkgCopy the code
Install TensorFlow
Before installing this, uninstall the TF installed with PIP first.
pip uninstall tensorflow
Copy the codeThen use PIP to install the wheels we generated.
PIP install/TMP/tensorflow_pkg/tensorflow - 1.5.0 rc0 cp27 - cp27mu - linux_x86_64. WHLCopy the codeDepending on the platform, the package name may be different. In this way, the installation is complete.
Test TensorFlow
Using the same code as before, the output is as follows.
6 Related Materials
-
Fancyerii. Making. IO / 2020/11/14 /…
-
www.jianshu.com/p/1d314fefe… Centos 7 Tensorflow-GPU-1.8.0 source code compiled
-
Zhaoxuhui. Top/blog / 2018/0… Install TensorFlow
-
www.jianshu.com/p/db943b0f1… Tensorflow source code compiled and installed
-
Tensorflow. Juejin. Im/install/ins… Install TensorFlow from the source code
-
Fancyerii. Making. IO / 2020/11/14 /… Compile Tensorflow in CentOS
-
Fancyerii. Making. IO / 2020/11/14 /… Compile Tensorflow in CentOS