background

In the last article, we covered how to install Zabbix and how to use Zabbix-Java-Gateway to monitor JVM progress, but this approach has the following disadvantages:

  • If you have multiple JVM processes to monitor on a single machine, you can do a lot of repetitive manual configuration work, which is tedious and error-prone.
  • Depending on zabbix-Java-Gateway, installation on multiple machines is a bit of work.

Zabbix’s low level discovery mechanism is a good solution to this problem.

Some of the previous research programs were found during the course of the pits

Whether Google or Baidu, there are basically several articles about the application of Low Level Discovery mechanism in automatic monitoring of multiple Tomcat or JVM processes that can be searched:

There are several problems with the methods introduced in these articles:

  1. Installation steps are not described in great detail.
  2. The bash script has some problems.
  3. The corresponding template file is missing (or cannot be downloaded).

Fortunately, I found a reliable and complete piece of information on Github: Qiueer/Zabbix

Here, I would like to thank qiueer for providing a very good learning material.

Description of experimental environment

All Lab machines are built under Mac OS X using Parallel Desktop, Linux version is

RPM -- query centos - release # output centos - release - 7-2.1511 el7. Centos. 2.10 x86_64 uname -r # output centos 7 (3.10.0-327.4.5. El7. X86_64)Copy the code
Host Name IP Hardware Memo
lab-ci-monitor 10.211.55.6 2 core, 4GB memory, 10GB disk Deploy Jenkins and Zabbix,And as the external network only access to the board
lab-mysql01 10.211.55.7 2 core, 4GB memory, 10GB disk Deploy mysql
lab-web01 10.211.55.8 2 core, 4GB memory, 10GB disk Deploy the Java Web Application
lab-web02 10.211.55.9 2 core, 4GB memory, 10GB disk Deploy the Java Web Application
lab-lb 10.211.55.10 2 core, 4GB memory, 10GB disk Deployed nginx

Install zabbix-Server and Zabbix-Agent

Refer to the previous article

The Active Agent auto-registration mechanism is used to automatically register the Zabbix Agent

In Zabbix, if we need to monitor a large number of machines, there are two ways to do this automatically:

  • The Network discovery: www.zabbix.com/documentati…

  • The Active agent auto – registration: www.zabbix.com/documentati…

Auto-registration is ideal in cloud environments, especially if the target monitoring machine does not know the IP address in advance. Here we will introduce how to use it in detail.

Step 1: Set zabbix_agentd.conf on lab-web01, lab-web02

Take Lab-Web01 as an example:

[vim /etc/zabbix/zabbix_agentd.conf]

ServerActive=10.211.55.6 Hostname=lab-web01 HostMetadataItem=system.unameCopy the code

After configuring zabbix-Agent, restart zabbix-Agent:

systemctl restart zabbix-agentCopy the code

Step 2: Configure actions on the Zabbix Web portal

Step 1: Under the Configuration — > Actions panel, click [Create Action]. Note that the Event source is Auto Registration

Step 2 On the Action TAB page, enter the name of the created rule and the rule for automatic discovery. Here we will automatically discover zabbix agent containing: lab-Web character in all Host name string.

Step 3: On the Operation TAB page, fill in the operations to be performed after the Zabbix Agent is automatically discovered. Here, we perform the following operations:

  1. Add Host: Automatically adds the agent
  2. Add to host groups: automatically Add to lab-Java-web-server host group
  3. Link to Template: The host is automatically added to the template OS Linux template

When this is done, wait approximately a certain amount of time (depending on the value of RefreshActiveChecks=60 in the Zabbix_Agentd. conf configuration file) to see that the corresponding Host has been added automatically.

Automatically monitor multiple Tomcat processes using the Low Level Discovery mechanism:

Some notes:

Qiueer’s solution will be used for configuration. There are a few issues to note:

  • Note The All In One and JVM directories are available. Use the Template and Python script under All In One.

  • In addition to common JVM and Tomcat solutions, All In One also includes Mysql Redis Nginx MongoDB MemoryCache and other solutions. Refer to the Read Me documentation for details

  • For All In One/ SRC /jvm.py and All In One/ SRC /tomcat.py, I found the following error logs when collecting data:

can not find java command, exit.Copy the code

Later, I found that the reason was that the which function in the script could not get the Java installation path correctly, which was probably related to the python version 2.7 I installed. I had not had time to carefully analyze the source code, so I took the solution of directly writing the Java installation path, the specific method is as follows:

Modify the All In One/ SRC /jvm.py script

class JMX(object): def __init__(self, logpath, debug=False): Self. _java_path = "/opt/ JDK /bin/ Java "self._cmdclient_jar = logpath self._logpath = logpath self._debug = debug = get_realpath() + "/" + "cmdline-jmxclient-0.10.3.jar" self._logger = slog(self._logpath, debug=debug, size=5, count=5) def get_item(self, beanstr, key, port, iphost='localhost', auth='-'): Java-jar cmdline-jmxclient-0.10.3.jar -localhost :12345 java.lang:type=Memory NonHeapMemoryUsage parameter:  """ pre_key = key sub_key = None if "." in key: Pos = STR (key).rfind(".") pre_key = key[0:pos] sub_key = key[pos + 1:] # # print "can not find java command, exit." # return None cmdstr = "%s -jar %s %s %s:%s '%s' '%s'" % (self._java_path, self._cmdclient_jar, auth, iphost, port, beanstr, pre_key)Copy the code

Modify the All In One/ SRC /tomcat.py script

class JTomcat(object): def __init__(self, logpath, debug=False): Self. _java_path = "/opt/ JDK /bin/ Java "self._cmdclient_jar = logpath self._logpath = logpath self._debug = debug = get_realpath()+"/" +"cmdline-jmxclient-0.10.3.jar" self._logger = slog(self._logpath, debug=debug, size=5, count=2)Copy the code

Specific installation steps:

Step 1: Import related templates on the Zabbix Web portal

Qiueer-template JMX tomcat With Io-apr.xml and qiueer-template Jvm.xml are qiueer-template JMX tomcat With Io-apr.xml

Step 2: Adjust the Auto Registration action created before and add the link of the above two templates.

Step 3: Download the relevant source code and scripts on zabbix Agent:

cd /usr/local/src/
git clone https://github.com/qiueer/zabbix.gitCopy the code

Step 4: Determine the configuration file path required for the Zabbix Agent extension function, for example, /usr/local/zabbix_agent_extend/conf

To modify the zabbix Agent main configuration file zabbix_agentd.conf and add the following information:

Vim /etc/zabbix/zabbix_agentd.conf # You must allow zabbix-agent to execute as root in order to execute python scripts without permission problems. Include=/usr/local/zabbix_agent_extend/conf/*.confCopy the code

Step 5: Run the zabbix_extend_init.sh script as user root. The first parameter of the script is the directory where the configuration file is saved, and the second parameter is the directory where the source script is saved.

For example, run the following command, where /usr/local/zabbix_agent_extend/conf is the file path determined in step 3:

mkdir -p /usr/local/zabbix_agent_extend/conf/
mkdir -p /usr/local/zabbix_agent_extend/scripts/  
cd /usr/local/src/zabbix/zabbix/All\ In\ One/
bash zabbix_extend_init.sh /usr/local/zabbix_agent_extend/conf /usr/local/zabbix_agent_extend/scriptsCopy the code

Its role is:

  • Copy the confS files to the /usr/local/zabbix_agent_extend/conf directory
  • Copy the files from the SRC directory to the /usr/local/zabbix_agent_extend/scripts directory

Step 6: Restart the Zabbix-Agent

systemctl restart zabbix-agentCopy the code

Once the above steps are complete, we can see that the Relevant Tomcat data is correctly retrieved in the Zabbix Portal and monitored separately by the JMX port number.

If the data cannot be correctly monitored, perform the following tests:

  • Check whether error logs are generated at/TMP /zabbix_jvm_info.log
  • Check whether error logs are generated at/TMP /zabbix_tomcat_info.log
  • Log and/TMP /zabbix_tomcat_info.log are set to 777
chmod 777 /tmp/zabbix_jvm_info.log
chmod 777 /tmp/zabbix_tomcat_info.logCopy the code

For more information, see the Github project’s Read Me

Unresolved issues

  1. The qiueer-template JvM. XML Template for this project is based on JDK 1.7, so memory data related to the permanent generation cannot be retrieved correctly. This Template will need to be adjusted to support Metaspace memory data retrieved under JDK 1.8.

  2. Qiueer-template JMX Tomcat With Io-apr.xml Template data was not obtained correctly and further work is needed to resolve the problem.