2.1 Mariadb deployment – All Controller
Install controller160 for each controller
yum install -y mariadb mariadb-server python3-PyMySQL
Copy the code# Install the galera plugin and build the cluster using Galera
yum install -y mariadb-server-galera mariadb-galera-common galera xinetd rsync
Copy the code# Start the database service and configure it to start at startup:
systemctl enable mariadb.service
systemctl start mariadb.service
Copy the code# Reset the password by running the mysql_secure_installation script. I have set devops==:
mysql_secure_installation
Copy the code# to create and edit/etc/my CNF. D/it. CNF file (if need be, backup/etc/my CNF. D/the existing configuration files), the binding address is set to the control node management IP address, to allow other nodes through the management network access:
[mysqld] binlog_format = ROW bind-address = 172.16.1.160 default-storage-engine = innodb innodb_file_per_table = on max_connections = 4096 collation-server = utf8_general_ci character-set-server = utf8 [galera] bind-address = 172.16.1.160 wsrep_provider = / usr/lib64 / galera libgalera_smm. So wsrep_cluster_address ="gcomm://controller160,controller161,controller162"Wsrep_cluster_name = openstack-ussuri-cluster wsrep_node_name = controller160 wsrep_node_address = 172.16.1.160 wsrep_on=ON wsrep_slave_threads=4 wsrep_sst_method=rsync wsrep_sst_auth=root:devops default_storage_engine=InnoDBCopy the codeStop mariadb services on all controller160 nodes
[root@controller160 ~]# systemctl stop mariadb.service
Copy the code# Start mariadb service on any controller node as follows: select controller160
[root@controller160 ~]# /usr/libexec/mysqld --wsrep-new-cluster --user=root & 2020-06-16 14:36:28 0 [Note] /usr/libexec/mysqld (mysqld 10.3.17-mariadb) starting as process 1785... [root@controller160 ~]# ps -ef |grep mysql root 553 132 0 16:05 pts/2 00:00:00 /usr/libexec/mysqld --wsrep-new-cluster --user=root root 643 132 0 16:08 pts/2 00:00:00 grep --color=auto mysqlCopy the codeAdd other controller nodes to mariadb cluster, for example, controller161; # started to join the cluster, from controller160 controller161 node synchronization data, also can examine mariadb synchronous log/var/log/mariadb/mariadb. The log
[root@controller161 ~]# systemctl start mariadb.service [root@controller161 ~]# systemctl status mariadb.service ● Mariadb. Service - mariadb 10.3 database server the Loaded: the Loaded (/ usr/lib/systemd/system/mariadb. The service; enabled; vendor preset: disabled) Active: active (running) since Tue 2020-06-16 16:10:07 UTC; 9s ago Docs: man:mysqld(8) https://mariadb.com/kb/en/library/systemd/ Process: 803 ExecStartPost=/usr/libexec/mysql-check-upgrade (code=exited, status=0/SUCCESS) Process: 587 ExecStartPre=/usr/libexec/mysql-prepare-db-dir mariadb.service (code=exited, status=0/SUCCESS) Process: 563 ExecStartPre=/usr/libexec/mysql-check-socket (code=exited, status=0/SUCCESS) Main PID: 624 (mysqld) Status: "Taking your SQL requests now..." Tasks: 38 (LIMIT: 32767) Memory: 200.5m CGroup: / system. Slice/mariadb. Service └ ─ 624 / usr/libexec/mysqld -- basedir = / usr Jun 16 16:10:05 controller161 rsyncd [738] : Connect from Controller160 (172.16.1.160) Jun 16 16:10:05 Controller161 rsyncd[738]: Rsync to rsync_sst/./ Performance_schema from controller160 (172.16.1.160) Jun 16 16:10:05 Controller161 rsyncd[738]: receiving file list Jun 16 16:10:05 controller161 rsyncd[738]: sent 48 bytes received 229 bytes total size 61 Jun 16 16:10:05 controller161 rsyncd[740]: Connect from Controller160 (172.16.1.160) Jun 16 16:10:05 Controller161 rsyncd[740]: Rsync to rsync_sst/ from Controller160 (172.16.1.160) Jun 16 16:10:05 Controller161 rsyncd[740]: receiving file list Jun 16 16:10:05 controller161 rsyncd[740]: sent 48 bytes received 185 bytes total size 41 Jun 16 16:10:05 controller161 rsyncd[674]: sent 0 bytes received 0 bytes total size 0 Jun 16 16:10:07 controller161 systemd[1]: Started MariaDB 10.3 Database Server.Copy the code# Restart controller160; Delete data from controller160 before starting
[root@controller160 ~]# pkill -9 mysql
[root@controller160 ~]# rm -rf /var/lib/mysql/*
Copy the codeNote the permissions to start the Mariadb service in system Unit mode
[root@controller160 ~]# chown mysql:mysql /var/run/mariadb/mariadb.pid
Copy the codeController160 synchronizes data from controller162
[root@controller160 ~]# systemctl start mariadb.service
[root@controller160 run]# systemctl status mariadb.serviceLow mariadb. Service - mariadb 10.3 database server the Loaded: the Loaded (/ usr/lib/systemd/system/mariadb. The service; enabled; vendor preset: disabled) Active: active (running) since Fri 2020-06-19 00:55:22 CST; 11s ago Docs: man:mysqld(8) https://mariadb.com/kb/en/library/systemd/ Process: 21791 ExecStartPost=/usr/libexec/mysql-check-upgrade (code=exited, status=0/SUCCESS) Process: 21305 ExecStartPre=/usr/libexec/mysql-prepare-db-dir mariadb.service (code=exited, status=0/SUCCESS) Process: 21281 ExecStartPre=/usr/libexec/mysql-check-socket (code=exited, status=0/SUCCESS) Main PID: 21489 (mysqld) Status:"Taking your SQL requests now..."
Tasks: 38 (limit: 23978) Memory: 302.0M CGroup: / system. Slice/mariadb. Service └ ─ 21489 / usr/libexec/mysqld -- basedir = / usr Jun 19 00:55:16 controller160 rsyncd [21645] : sent 618 bytes received 2111179 bytes total size 2108561 Jun 19 00:55:16 controller160 rsyncd[21705]: Connect from Controller162 (172.16.1.162) Jun 19 00:55:16 Controller160 rsyncd[21705]: Rsync to rsync_SST /./ Placement from Controller162 (172.16.1.162) Jun 19 00:55:16 Controller160 rsyncd[21705]: receiving file list Jun 19 00:55:16 controller160 rsyncd[21705]: sent 48 bytes received 216 bytes total size 61 Jun 19 00:55:19 controller160 rsyncd[21715]: Connect from Controller162 (172.16.1.162) Jun 19 00:55:19 Controller160 rsyncd[21715]: Rsync to rsync_sst/ from Controller162 (172.16.1.162) Jun 19 00:55:19 Controller160 rsyncd[21715]: receiving file list Jun 19 00:55:19 controller160 rsyncd[21715]: sent 48 bytes received 186 bytes total size 43 Jun 19 00:55:22 controller160 systemd[1]: Started MariaDB 10.3 Database Server.Copy the codeCheck the cluster status
[root@controller160 ~]# mysql -uroot -p
Enter password:
MariaDB [(none)]> show status like "wsrep_cluster_size";
+--------------------+-------+
| Variable_name | Value |
+--------------------+-------+
| wsrep_cluster_size | 3 |
+--------------------+-------+
1 row in set(0.001sec) MariaDB [(none)]> SHOW status LIKE'wsrep_ready';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| wsrep_ready | ON |
+---------------+-------+
1 row in set (0.001 sec)
Copy the codeVim /usr/bin/clustercheck # Create user clustercheck_user on any controller node and assign it to the user clustercheck_user; Note That the account/password corresponds to the account/password in the script. The default account/password is used here. Otherwise, modify the clusterCheck script file
[root@controller160 ~]# mysql -uroot -p
MariaDB [(none)]> GRANT PROCESS ON *.* TO 'clustercheckuser'@'localhost' IDENTIFIED BY 'clustercheckpassword! ';
MariaDB [(none)]> FLUSH PRIVILEGES;
Copy the codeCheck the configuration file
Add /etc/xinetd.d/mysqlchk to all controller nodes, for example, controller160
[root@controller01 ~]# wget https://codeload.github.com/olafz/percona-clustercheck/zip/master
[root@controller01 ~]#unzip master
[root@controller01 ~]#cd percona-clustercheck-master/
[root@controller160 percona-clustercheck-master]#cp clustercheck /usr/bin/
[root@controller160 percona-clustercheck-master]#touch /etc/xinetd.d/mysqlchk
[root@controller01 ~]# vim /etc/xinetd.d/mysqlchk
# default: on
# description: mysqlchk
service mysqlchk
{
port = 9200
disable = no
socket_type = stream
protocol = tcp
wait = no
user = root
group = root
groups = yes
server = /usr/bin/clustercheck
type = UNLISTED
per_source = UNLIMITED
log_on_success =
log_on_failure = HOST
flags = REUSE
}
Copy the codeChange /etc/services to tcp9200 port, for example, controller160 node
[root@controller160 ~]# vim /etc/services
#wap-wsp 9200/tcp # WAP connectionless session service
mysqlchk 9200/tcp # mysqlchk
For example, start xinetd on controller160
[root@controller160 ~]# systemctl daemon-reload
[root@controller160 ~]# systemctl enable xinetd
[root@controller160 ~]# systemctl start xinetd
Copy the code# Test heartbeat detection script
[root@controller160 ~]# /usr/bin/clustercheck
HTTP/1.1 200 OK
Content-Type: text/plain
Connection: close
Content-Length: 40
Percona XtraDB Cluster Node is synced.
Copy the code2.2 RabbitMQ Deployment – All controller
OpenStack uses message queues to coordinate operations and status information between services. Message queue services typically run on control nodes. OpenStack supports multiple message queue services, including RabbitMQ, Qpid, and ZeroMQ. If you want to implement different message queuing services, refer to the relevant documentation. # # to add the rabbitmq – Erlang source vim/etc/yum repos. D/rabbitmq – Erlang. Repo
[rabbitmq-erlang]
name=rabbitmq-erlang
baseurl=https://dl.bintray.com/rabbitmq-erlang/rpm/erlang/22/el/8/
gpgcheck=1
gpgkey=https://dl.bintray.com/rabbitmq/Keys/rabbitmq-release-signing-key.asc
repo_gpgcheck=0
enabled=1
Copy the code# # add the rabbitmq source vim/etc/yum repos. D/rabbitmq. Repo
[bintray-rabbitmq-server] name=bintray-rabbitmq-rpm baseurl=https://dl.bintray.com/rabbitmq/rpm/rabbitmq-server/v3.8.x/el/8/ gpgcheck = 0 repo_gpgcheck = 0 enabled = 1Copy the code# to install the rabbitmq
yum install -y erlang rabbitmq-server
Copy the codeUse controller160 as an example to start RabbitMQ and configure it to start at startup:
systemctl enable rabbitmq-server.service
systemctl start rabbitmq-server.service
systemctl status rabbitmq-server.service
Copy the codeBuild a RabbitMQ cluster
Start the RabbitMQ service on any controller node. Select controller160 node
[root@controller160 ~]#systemctl start rabbitmq-server.service
[root@controller160 ~]#rabbitmqctl cluster_status
# distribution. Erlang. Cookies
[root@controller160 ~]# SCP/var/lib/rabbitmq /. Erlang. The cookie [email protected]: / var/lib/rabbitmq /
[root@controller160 ~]# SCP/var/lib/rabbitmq /. Erlang. The cookie [email protected]: / var/lib/rabbitmq /
# Modify user/group in controller161/162 node.erlang. cookie
[root@controller161 ~]#chown rabbitmq:rabbitmq /var/lib/rabbitmq/.erlang.cookie
The default value is 400. You cannot change the default value
[root@controller161 ~]#ll /var/lib/rabbitmq/.erlang.cookie
Start The RabbitMQ service on controller161/162
[root@controller161 ~]#systemctl restart rabbitmq-server
[root@controller162 ~]#systemctl restart rabbitmq-server
# Build the cluster. Controller161/162 nodes join the cluster as RAM nodes
[root@controller161 ~]#rabbitmqctl stop_app
[root@controller161 ~]#rabbitmqctl join_cluster --ram rabbit@controller160
[root@controller161 ~]#rabbitmqctl start_app
Copy the codeCheck whether the creation is successful
[root@controller160 ~]# rabbitmqctl cluster_status
warning: the VM is running with native name encoding of latin1 which may cause Elixir to malfunction as it expects utf8. Please ensure your locale is set to UTF-8 (which can be verified by running "locale" inyour shell) Cluster status of node rabbit@controller160 ... Basics Cluster name: rabbit@controller160 Disk Nodes rabbit@controller160 RAM Nodes rabbit@controller161 rabbit@controller162 Running Nodes rabbit@controller160 rabbit@controller161 rabbit@controller162 Versions rabbit@controller160: RabbitMQ 3.8.5 on Erlang 22.3.4.1 rabbit@controller161: RabbitMQ 3.8.5 on Erlang 22.3.4.1 rabbit@controller162: RabbitMQ 3.8.5 on Erlang 22.3.4.1 Network Alarms (None) Listeners: rabbit@controller160, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication Node: rabbit@controller160, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0 Node: rabbit@controller161, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication Node: rabbit@controller161, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0 Node: rabbit@controller162, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication Node: rabbit@controller162, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0 Feature flags Flag: ImplICIT_default_bindings, state: enabled Flag: quorum_queue, state: implICIT_bindings enabled Flag: virtual_host_metadata, state: enabledCopy the codeAdd user openstack and set password to ==rabbitmq.123==:
rabbitmqctl add_user rabbitmq rabbitmq.123
rabbitmqctl set_user_tags rabbitmq administrator
Copy the codeAllow configuration, write, and read access for openstack users:
rabbitmqctl set_permissions -p "/" rabbitmq ". *" ". *" ". *"
# Check account
rabbitmqctl list_users
Copy the code# Mirror queue ha
Set the high availability of the mirror queue
rabbitmqctl set_policy ha-all "^" '{"ha-mode":"all"}'
# View the mirroring queue policy
rabbitmqctl list_policies
Copy the codeInstall the Web administration plug-in
# Install the Web management plug-in on all controller nodes. Take controller160 as an example
rabbitmq-plugins enableRabbitmq_management access any node, such as: http://172.16.1.160:15672Copy the code
2.3 Memcached Deployment – All controller
# The identity service authentication mechanism uses Memcached to cache the token. The Memcached service typically runs on the control node.
yum install -y memcached python3-memcached
Copy the codeConfigure the service as the management IP address of the controller node. This is to allow other nodes to access through the management network: #vim /etc/memcached.conf
sed -i 's | 127.0.0.1, : : 1 | 0.0.0.0 | g' /etc/sysconfig/memcached
Copy the code# Start Memcached service and configure it to start at startup:
systemctl enable memcached.service
systemctl start memcached.service
systemctl status memcached.service
Copy the code2.4 Etcd Deployment – All Controller
#OpenStack services can use Etcd, which is a distributed reliable key-value store for distributed key locking, storage configuration, tracking service lifecycle, and other scenarios
yum install -y etcd
Copy the codeChange the configuration file to the management IP address of the controller node so that other nodes can access it through the management network: vim /etc/etcd/etcd.conf
ETCD_NAME="controller160"
ETCD_DATA_DIR="/var/lib/etcd"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-01"
ETCD_INITIAL_CLUSTER="Controller160 = http://172.16.1.160:2380, controller161 = http://172.16.1.161:2380, controller162 = http://172.16.1.162:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://172.16.1.160:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://172.16.1.160:2379"
ETCD_LISTEN_PEER_URLS="http://0.0.0.0:2380"
ETCD_LISTEN_CLIENT_URLS="Http://172.16.1.160:2379, http://127.0.0.1:2379
Copy the code# # modified etcd. Service vim/usr/lib/systemd/system/etcd. Service
[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
User=etcd
# set GOMAXPROCS to number of processors
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/bin/etcd \
--name=\"${ETCD_NAME}\" \
--data-dir=\"${ETCD_DATA_DIR}\" \
--listen-peer-urls=\"${ETCD_LISTEN_PEER_URLS}\" \
--listen-client-urls=\"${ETCD_LISTEN_CLIENT_URLS}\" \
--initial-advertise-peer-urls=\"${ETCD_INITIAL_ADVERTISE_PEER_URLS}\" \
--advertise-client-urls=\"${ETCD_ADVERTISE_CLIENT_URLS}\" \
--initial-cluster=\"${ETCD_INITIAL_CLUSTER}\" \
--initial-cluster-token=\"${ETCD_INITIAL_CLUSTER_TOKEN}\" \
--initial-cluster-state=\"${ETCD_INITIAL_CLUSTER_STATE}\ ""
Restart=on-failure
LimitNOFILE=65536
Copy the code# Start the ETCD service on each node in sequence and configure it to start at startup:
systemctl enable etcd
systemctl start etcd
systemctl status etcd
Copy the code# validation etcd
[root@controller160 ~]# etcdctl cluster-health
member c8750a07a2008c34 is healthy: got healthy result from http://172.16.1.161:2379
member ddc8bcfe3f80cd5d is healthy: got healthy result from http://172.16.1.160:2379
member fc456eb6d491b837 is healthy: got healthy result from http://172.16.1.162:2379
cluster is healthy
[root@controller160 ~]# etcdctl member listC8750a07a2008c34: name=controller161 peerURLs=http://172.16.1.161:2380 clientURLs=http://172.16.1.161:2379 isLeader=falseDdc8bcfe3f80cd5d: name=controller160 peerURLs=http://172.16.1.160:2380 clientURLs=http://172.16.1.160:2379 isLeader=trueFc456eb6d491b837: name=controller162 peerURLs=http://172.16.1.162:2380 clientURLs=http://172.16.1.162:2379 isLeader=false
Copy the codeSo far, the basic service has been deployed, if you have any questions, please contact me to correct, thank you!
X. Problems encountered in the process
eg.1.Error: Failed to synchronize cache for repo 'AppStream'DNF upgrade --releasever=8 eg.2.2020-06-16 13:58:50 [ERROR] mysqld: Server GSSAPI error (major 851968, minor 2529639093) : gss_acquire_cred failed -Unspecified GSS failure. Minor code may provide more information. Keytab FILE:/etc/krb5.keytab is nonexistent or empty. 2020-06-16 13:58:50 0 [ERROR] Plugin'gssapi' init function returned error.
2020-06-16 13:58:50 0 [ERROR] Can't init tc log 2020-06-16 13:58:50 0 [ERROR] Aborting rm -rf /var/lib/mysql/tc.log eg3. Troubleshooting: When the power failure occurs in the equipment room, all galera hosts shut down abnormally and start up after an incoming call. As a result, the Galera cluster service cannot be started properly. How to deal with it? Step 1: Enable the Mariadb service on the group host of the Galera cluster. Step 2: Enable mariadb services for the member hosts of the Galera cluster. Exception: Mysql services on the host and member hosts of the Galera cluster cannot be started. What can I do? Solution a: step 1, delete garlera group manager host/var/lib/mysql/grastate dat state file/bin/galera_new_cluster start the service. The startup is normal. Log in and view the WSREP status. Step 2: remove galera member host in the/var/lib/mysql/grastate dat state file systemctl restart mariadb resume service. The startup is normal. Log in and view the WSREP status. Solution 2: step 1, modify garlera group manager host/var/lib/mysql/grastate dat state file of 0 to 1 / bin/galera_new_cluster start the service. The startup is normal. Log in and view the WSREP status. Step 2: modify galera member host in the/var/lib/mysql/grastate dat state file of 0 to 1 systemctl restart mariadb resume service. The startup is normal. Log in and view the WSREP status. Eg4.[root@controller160 percona- ClusterCheck -master]# /usr/bin/clusterCheck HTTP/1.1 503 Service Unavailable Content-Type: text/plain Connection: close Content-Length: 44 vim /usr/bin/clustercheck MYSQL_USERNAME="${MYSQL_USERNAME:=-clustercheckuser}" MYSQL_PASSWORD="${MYSQL_PASSWORD-clustercheckpassword! ${MYSQL_USERNAME: -clustercheckUser}" MYSQL_PASSWORD="${MYSQL_PASSWORD: -clustercheckPassword! }"Copy the code