Business goals

Upload mysql data to ElasticSearch using LogStash and display it using Kibana

A. Download and decompress

Wget https://artifacts.elastic.co/downloads/logstash/logstash-7.1.1.tar.gz tar - ZXVF logstash - 7.1.1. Tar. Gz /usr/local/logstashCopy the code

B. Install the /usr/local/logstash/bin plug-in

./logstash-plugin install logstash-input-jdbc
./logstash-plugin install logstash-output-elasticsearch
Copy the code

C. Save the mysql driver package in the bin directory, the same as the configuration file

Yml: load multiple config files in charge at the same time

Vim stdmysql.conf input {JDBC {jdbc_driver_library => "mysql-connector-java-5.1.7-bin.jar" jdbc_driver_class => ". Com. Mysql. JDBC Driver jdbc_connection_string "= >" JDBC: mysql: / / 127.0.0.1:3306 / luoforTest "jdbc_user = >" root" jdbc_password => "092248" schedule => "* * * * *" statement => "SELECT * FROM UserDto WHERE update_time >= :sql_last_value" use_column_value => true tracking_column_type => "timestamp" tracking_column => "update_time" Last_run_metadata_path => "syncpoint_TABLE"}} Output {elasticSearch {hosts => ["127.0.0.1:9200"] index => "userdto" Document_id => "%{id}"}}Copy the code

E. Start logStash (Visualization via Kibana)

./logstash -f stdmysql.conf &

F. Configure vim config/pipelines

- pipeline.id: elk path.config: ".. /bin/stdin.conf" - pipeline.id: mysql path.config: ".. /bin/stdmysql.conf"Copy the code

You do not need to specify the configuration for direct startup

./logstash &
Copy the code