1. Error symptoms

Received database alarm, the alarm information is as follows. The database address and port are blurred for security reasons.

Instance type: MYSQL Physical address: 10.1.1.1(for security ambiguity) Listening port: 4(for security ambiguity) Instance name: MYSQL Alarm message: PORTTEST-PORTTEST-{['Error', \"Can't connect to MySQL server on '10.1.1.1']"} 10.1.1.1(for security blurring) Listening port: 5(for security blurring) Instance name: MYSQL PORTTEST-PORTTEST-{['Error', \"Can't connect to MySQL server on '10.1.1.1']"} 10.1.1.1(for security blurring) Listening port: 6(for security blurring) Instance name: MYSQL PORTTEST-PORTTEST-{['Error', \"Can't connect to MySQL server on '10.1.1.1']"}Copy the code

2. Log in to the server

2.1 Viewing Processes

There are six MySQL instance processes running on this server. Ps - ef | grep mysql found that 4 - the process of the port 6 has hang up. Neither does the mysqLD_safe process.Copy the code

2.2 Trying to Start processes on Ports 4-6

Mysqld_safe --defaults-file=4.cnf & Read-only file system XXXX mysqld[52940]: XXXX mysqld[52940]: rm: cannot remove '/XX/XXX/XXXX/ xxxx4.pid' : file system XXXX mysqld[52940]: XXXXX mysqld_safe Fatal error: Can't remove the pid fileCopy the code

2.3 Error Analysis

Read-only File system This is the first time this error is encountered in a read-only file system. I haven't come across it before. The server uses SSDS to store database data. Mount /data/ separately to store data. Search and query through Baidu. You can check whether the /proc/mounts directory is mounted. If you check whether the mounts directory is ro, the information about the mounts directory is RW. Otherwise, the mounts directory is ro. Therefore, the database instance cannot be started. A file system error is reported. cat /proc/mounts /dev/sda1 /boot xfs rw,relatime,attr2,inode64,noquota 0 0 /dev/mapper/app /app xfs rw,relatime,attr2,inode64,noquota 0 0 /dev/mapper/data /data xfs ro,relatime,attr2,inode64,noquota 0 0Copy the code

3. SA treatment

Because the server’s resources are managed by SA. Therefore, it was reported to SA colleagues to deal with it. After the investigation of SA colleagues, they replied to me that the disk system was damaged and needed to be repaired. Fix the file system and disk before you fix it. Let me try again. After I tried to start the database, it worked fine, but after a few minutes the database process still hung up. After feedback to SA colleagues, SA feedback to the original factory to repair and replace the disk. After the maintenance, the MySQL database instance is deployed again and the server runs normally.