1 Preconditions
1.1 Prerequisites
Yum -- y install LRZSZ // yum -- y install LRZSZ // # yum -- y install GCC GCC -- c++ / # yum -- y install GCC GCC -- c++ / # yum -- y install GCC -- c++ / # yum -- y install GCC -- c++ / # yum -- y install GCC -- c++ / # yum -- y install GCC -- c++ / # yum -- y install GCC -- c++ / # yum -- y install GCC -- c++ / # Some may be less pack support, no yum blah blahCopy the code1.2 Disabling the Firewall
Run the following command to disable the firewall: service iptables stop Then permanently disable the firewall: chkconfig iptables off You must disable the firewall; otherwise, you may not be able to access apache on the local VMCopy the code1.3 JDK installation
Run the jdK1.7 jK-7U71-linux-i586. RPM directory in /opt. Run the jdK-7u71-linux-i586. RPM directory in /opt/ JDK to run the RPM -ivh directory Jkg-7u71-linux-i586. RPM: vi /etc/profile # JDK export JAVA_HOME=/usr/ Java /jdk1.7.0_71 export ANDROID_JAVA_HOME=$JAVA_HOME export JRE_HOME=/usr/java/jdk1.7.0_71/jre export CLASSPATH=/usr/java/jdk1.7.0_71/lib export PATH=$JAVA_HOME/bin:$PATH Save the configuration and run it after exit Then type Java -version to check whether the JDK is installed successfully. [root@localhost conf]# Java -version Java version "1.7.0_71" Java(TM) SE Runtime Environment (build Java HotSpot(TM) Client VM (Build 24.71-B01, Mixed mode, Sharing)Copy the code2 Tomcat
2.1 a Tomcat installation
Tomcat Upload apache-tomcat-6.0.37.tar.gz to /opt/tomcat 1.tar -zxvf apache-tomcat-7.0.41.tar.gz 2 Apache -tomcat-7.0.41 tomcat1 3.mv tomcat1 /usr/local/4. CD /usr/local 5.cp -r tomcat1 tomcat2 6.vi /etc/profile Modify the configuration file Add: #tomcat export TOMCAT_HOME=/usr/local/tomcat1 export TOMCAT_HOME=/usr/local/tomcat2 6 Start tomcat/usr/local/tomcat1 / bin/startup. Sh/usr/local/tomcat2 / bin/startup. The sh show Using CATALINA_BASE: /usr/local/tomcat1 Using CATALINA_HOME: /usr/local/tomcat1 Using CATALINA_TMPDIR: /usr/local/tomcat1/temp Using JRE_HOME: /usr/java/jdk1.7.0_71/jre Using CLASSPATH: / usr/local/tomcat1 / bin/bootstrap jar then ps ef | grep tomcat to check the processCopy the codeEnter http://ip: port in the browser. If the following page is displayed, tomcat is successfully installed
3 Nginx
3.1 Nginx installation
Create folder nginx under opt. Upload nginx-1.3.15.tar.gz to tat-xzvf nginx-1.3.15.tar.gz go to nginx-1.3.15 and run./configure - with - http_stub_status_module - with - http_ssl_module perform make && make install to check the configuration/usr/local/nginx/sbin/nginx - t If the following information is displayed, the configuration file is correct: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok the configuration file / usr/local/nginx/conf/nginx. Conf was * successfully launched nginx/usr/local/nginx/sbin/nginx last port type http://ip: in your browser If you see the Nginx page, the installation is successfulCopy the code3.2 Modifying the Nginx configuration file nginx.conf to implement load balancing
#user nobody; Worker_processes 1; #error_log logs/error.log; #error_log logs/error.log notice; #error_log logs/error.log info; #pid logs/nginx.pid; {worker_connections 1024; } # HTTP {include mime.types; default_type application/octet-stream; #log_format main '$remote_addr - $remote_user [$time_local] "$request" ' # '$status $body_bytes_sent "$http_referer" ' # '"$http_user_agent" "$http_x_forwarded_for"'; #access_log logs/access.log main; sendfile on; #tcp_nopush on; #keepalive_timeout 0; keepalive_timeout 65; #gzip on; Upstream web_server {server localhost: 8080weight =1 max_fails=2 fail_timeout=30s; server localhost:9090 weight=1 max_fails=2 fail_timeout=30s; } server { listen 80; server_name localhost; # Web pages, videos, and image files are read locally And defined in the browser cache location ~ 30 days. * \. (HTM | | GIF JPG | | HTML jpeg | PNG | BMP | SWF | | the ioc rar | | zip TXT | FLV | mid | doc PDF | | | PPT XLS | mp3 | wma) ${ expires 30d; } # js, CSS files, read from the local and defined in the browser cache. 1 hour and the location ~ * \. (js) | CSS? $ { expires 1h; } # dynamic forward files to the backend tomcat cluster location. ~ * \. | | | (PHP JSP cgi JHTML)? $ { proxy_pass http://web_server; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Real-IP $remote_addr; } #charset koi8-r; #access_log logs/host.access.log main; location / { root html; index index.html index.htm; } #error_page 404 /404.html; # redirect server error pages to the static page /50x.html # error_page 500 502 503 504 /50x.html; location = /50x.html { root html; } # proxy the PHP scripts to Apache listening on 127.0.0.1:80 # #location ~ \.php${# proxy_pass http://127.0.0.1; #} # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000 # #location ~ \.php${# root HTML; # fastcgi_pass 127.0.0.1:9000; # fastcgi_index index.php; # fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name; # include fastcgi_params; #} # deny access to .htaccess files, if Apache's document root # concurs with nginx's one # #location ~ /\.ht { # deny all; #} } # another virtual host using mix of IP-, name-, and port-based configuration # #server { # listen 8000; # listen somename:8080; # server_name somename alias another.alias; # location / { # root html; # index index.html index.htm; # } #} # HTTPS server # #server { # listen 443 ssl; # server_name localhost; # ssl_certificate cert.pem; # ssl_certificate_key cert.key; # ssl_session_cache shared:SSL:1m; # ssl_session_timeout 5m; # ssl_ciphers HIGH:! aNULL:! MD5; # ssl_prefer_server_ciphers on; # location / { # root html; # index index.html index.htm; # #}}}Copy the code3.3 Modifying Tomcat1 and Tomcat2 configuration files server.xml
<Server port="8005" shutdown=" shutdown "> <Connector port="8080" protocol="HTTP/1.1" connectionTimeout="20000"redirectPort="8443"/> / / tomcat port numberCopy the code3.4 Testing load
To join the engineering test under two tomcat mkdir/usr/local/tomcat1 / webapps/test the mkdir/usr/local/tomcat2 / webapps/test <% @page contentType="text/ HTML; charset=UTF-8"%> <html> <head> <title> Tomcat_test1</title> </head> <body> <font color = "red" size = "20" > <% out.print( "test111" ); %> </font> </body> </html> <%@ page contentType="text/html; charset=UTF-8"%> <html> <head> <title> Tomcat_test2</title> </head> <body> <font color = "red" size = "20" > <% out.print( "test222" ); %> </font> </body> </html>Copy the codeStart tomcat 3.5 + nginx
Nginx start: / usr/local/nginx/sbin/nginx start tomcat: / usr/local/tomcat1 / bin/startup. Sh/usr/local/tomcat2 / bin/startup. Sh checking processCopy the codeEnter IP address: nginx port/project/JSP page in the browser, preferably in a different browser. If the following information is displayed, the setup is successful
4 kafka
4.1 kafka concept
A typical Kafka architecture consists of producters, brokers, consumers, and a Zookeeper cluster.
- Producter: the party that sends the message; Responsible for creating messages and then delivering them to Kafka.
- Consumer: The party that receives the message. The Consumer connects to Kafka and receives the message for the relevant business logic processing.
- Broker: Can be thought of as an independent Kafka service node or service instance. One or more brokers form a Kafka cluster.
4.2 Zookeeper Installation and Configuration
4.2.1 JDK Installation and Configuration
Install the JDK and configure the environment variables.
4.2.2 Installing and Configuring Zookeeper
4.2.2.1. Download from the official website
zookeeper.apache.org/
4.2.2.2. Decompress installation
[root@ray opt]# tar XVF apache-zookeeper-3.6.3-bin.tar.gz [root@ray opt]# cp apache-zookeeper-3.6.3-bin Apache - zookeeper - 3.6.3Copy the code4.2.2.3. Environment Variables
Edit vim /etc/profile to add environment variables
# zookeeper
export ZOOKEEPER_HOME=/opt/apache-zookeeper-3.6.3
export PATH=$PATH:$ZOOKEEPER_HOME/bin
Copy the code4.2.2.4. Modify the configuration
[root@ray opt]# cd /opt/apache-zookeeper-3.6.3/conf/
[root@ray conf]# cp zoo_sample.cfg zoo.cfg
Copy the codeExample Modify the zoo. CFG configuration file
# The number of milliseconds of each tick tickTime=2000 # The number of ticks =10 # initLimit=10 # The number of ticks that can pass between # sending a request and getting an acknowledgement syncLimit=5 # the directory where the snapshot is stored. # do not use /tmp for storage, / TMP here is just # dataDir=/ TMP /zookeeper/data # ZooKeeper log directory dataLogDir=/ TMP /zookeeper/log # External service port of ZooKeeper clientPort=2181Copy the codeBy default, the/TMP /zookeeper/data and/TMP /zookeeper/log directories do not exist in the system.
[root@ray conf]#mkdir -p /tmp/zookeeper/data
[root@ray conf]#mkdir -p /tmp/zookeeper/log
Copy the code4.2.2.5. Creating a Number
Create a myID file in/TMP /zookeeper/data and write a value such as 0. Myid will store the server number.
4.2.2.6. Starting the Service
[root@ray bin]# zkserver. sh start ZooKeeper JMX enabled by default Using config: /opt/apache-zookeeper-3.6.3/bin/.. /conf/zoo.cfg Starting zookeeper ... already running as process 8770.Copy the codeRun the zkserver. sh status command to check the service status as shown in the following example:
[root@ray bin]# zkserver. sh status ZooKeeper JMX enabled by default Using config: /opt/apache-zookeeper-3.6.3/bin/.. /conf/zoo.cfg Client port found: 2181. Client address: localhost. Client SSL: false. Mode: standaloneCopy the code4.3 Kafka Installation and Configuration
4.3.1 Download from the official website
kafka.apache.org/
4.3.2 Decompressing installation
[root@ray opt]# tar-xvf kafka_2.12-2.8.0.tar [root@ray opt]# CD kafka_2.12-2.8.0/ [root@ray kafka_2.12-2.8.0]# PWD / opt/kafka_2. 12-2.8.0 [root @ ray kafka_2. 12-2.8.0] #Copy the code4.3.3 Environment Variables
Edit vim /etc/profile to add environment variables
# kafka export KAFKA_HOME=/opt/ kafka_2.8.0 export PATH=$PATH:$KAFKA_HOME/binCopy the code4.3.4 Configuration File
Modify the kafka configuration file
Ray config], [root @ # vim/opt/kafka_2. 12-2.8.0 / config/server propertiesCopy the codeThe following is an example:
# the serial number of the broker broker. Id = 0 # services entry address listeners = PLAINTEXT: / / : 9092 # storing the addresses of the log file. The log dirs = / TMP/kafka - logs # cluster address zookeeper.connect=localhost:2181/kafkaCopy the codeCreate log directory path/TMP /kafka-logs
[root@ray config]# mkdir -p /tmp/kafka-logs
Copy the code4.3.5 Starting the Service
Run the command in the kafka installation directory
[root@ray opt]# CD kafka_2.12-2.8.0/ [root@ray kafka_2.12-2.8.0]#./bin/kafka-server-start.sh -daemon ./config/server.propertiesCopy the codeRun the JPS -l command to check whether the Kafka service process is started as shown in the following example:
[root @ ray kafka_2. 12-2.8.0] # JPS - 8770 org. L apache. The zookeeper. Server. Quorum. QuorumPeerMain 14890 kafka. Kafka. "14987 Sun. Tools. The JPS. The JPS [root @ ray kafka_2. 12-2.8.0] #Copy the codeThe JPS command is only used to confirm that the Kafka service has started normally. Whether it can provide services correctly needs to be verified by sending and receiving messages.
4.4 Production and consumption
4.4.1 Maven rely on
Kafka itself provides a Java client to demonstrate sending and receiving messages, requiring modifications to Maven dependencies.
4.4.2 producers
4.4.3 consumers
4.5 Configuring Server Parameters
Is mainly kafka kafka_2. 12-2.8.0 / config/server. The properties files
4.5.1 zookeeper. Connect
This parameter is the service address (including the port number) connecting to the ZooKeeper cluster. It has no default value and is mandatory. You can set this parameter to localhost:2181. localhost:2181,localhost:2181,localhost:2181
4.5.2 of listeners
This parameter specifies the list of addresses to which the broker listens for a client connection, that is, the list of entry addresses from which the client connects to the broker. The configuration format is: Protocol1: / / hostname1: protocol2: / / hostname2: port1, port2, protocol3: / / hostname3: port3 protocol on behalf of the protocol type, Kafka currently supports PLAINTEXT, SSL, SASL_SSL, etc. If security authentication is not enabled, PLAINTEXT is used. Hostname indicates a host and port indicates a port. This parameter is null by default.
4.5.3 the dir and the dirs
Kafka stores all messages on disk, and these two parameters are used to configure the root directory for Kafka log files. In general, log.dir is used to configure a single root directory, while log.dirs is used to configure multiple root directories (separated by commas), with log.dirs having a higher priority than log.dir. By default, only the log.dir parameter is configured, which defaults to/TMP /kafka-logs.
4.5.4 message. Max. Bytes
This parameter specifies the maximum number of messages that the broker can receive. The default value is 1000012 (B), which is approximately 967.6KB. If producter sends a message greater than this value, it will raise RecordTooLargeException. If you need to change this parameter, consider the impact of parameters such as max.request.size (the client parameter) and message.max.bytes (the topic parameter). To prevent cascading effects caused by changing this parameter, you are advised to split messages before changing this parameter.
4.6 kafka backlog
4.6.1 Causes of kafka consumption backlog
The reason:
- Message production is faster than consumption
- Consumers may have an exception that makes the message unconsumable
Solution:
- You can add a queue and a number of consumers
- Check whether exceptions are reported on the consumer end
If the backlog of messages is large and the messages can be discarded, it can also be reset by the management to consume the latest messages.