Small knowledge, big challenge! This paper is participating in theEssentials for programmers”Creative activities

preface

As we all know, it is very important for a program to have detailed log output. On the one hand, logs are convenient for debugging code after errors. On the other hand, detailed logs can also be of great help for future data statistics without prior burying points.

But, because our applications tend to be deployed in different locations on different servers, viewing different logs often requires running around in different folders on different servers. So, is there something that makes it easy to see all the program logs in one place on one host? Next, I’ll show you how to use Loki to do this

The principle of

The preparatory work

The program that outputs logs resides on host A, and the host that views logs is host B.

Install and deploy Loki on host A

Move to any directory where you want Loki to run

cd <path_to_loki_to_run>

Download binary (Linux) or exe executable (Windows)

Binary file

Exe executable file

Remember to unzip to the current directory after downloading

Edit the Loki configuration file

# loki-config.yaml
auth_enabled: false

server:
  http_listen_port: 3100

ingester:
  lifecycler:
    address: 127.0. 01.
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 5m
  chunk_retain_period: 30s
  max_transfer_retries: 0

schema_config:
  configs:
    - from: 2018-04-15
      store: boltdb
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 168h

storage_config:
  boltdb:
    directory: /tmp/loki/index

  filesystem:
    directory: /tmp/loki/chunks

limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h

chunk_store_config:
  max_look_back_period: 0s

table_manager:
  retention_deletes_enabled: false
  retention_period: 0s
Copy the code

Important parameter description in the configuration file

  • Server. http_listen_port: port on which the Loki service runs. You’ll need this later when you configure Promtail and Grafana.

Other parameters can be ignored for our current needs.

The deployment of Loki

Linux

./loki-linux-amd64 -config.file=loki_config.yaml

Windows

./loki-windows-amd64.exe -config.file=loki_config.yaml

Install and deploy Promtail on host A

Move to any directory where you want Promtail to run

cd <path_to_promtail_to_run>

Download binary (Linux) or exe executable (Windows)

Binary file

Exe executable file

Edit the Promtail configuration file

# promtail-config.yaml
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://127.0.0.1:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: <path_to_log_file>
Copy the code

Important parameter description in the configuration file

  • Clients. url: indicates the URL of the log push interface of the Loki servicehttp://<ip_of_loki>:<port_of_loki>/loki/api/v1/pushIn this article, because Loki and Promtail are on the same host, the IP is127.0.0.1And in the Loki configuration file above, Loki is deployed in3100Port is 3100. Therefore, the value of this parameter ishttp://127.0.0.1:3100/loki/api/v1/push
  • Scrape_config: log crawler configuration, here uses the simplest file crawler as an example.jon_nameIt’s just the name of a crawler.targetsIs the crawler target host, which is set to localhost in this paper. Labels are labels for crawlers. You can use label filtering to view logs.labels.__path__Promtail will automatically push the log file to the Loki service.

Deploy Promtail

Linux

./promtail-linux-amd64 --config.file=promtail-config.yaml

Loki

./promtail-windows-amd64.exe --config.file=promtail-config.yaml

Install and deploy Grafana on host B

Download and install Grafana

Download link

Run Grafana and configure the Loki service

Enter it in http. URLhttp://<ip_of_host_A>:<prot_of_Loki_of_host_A>Save the configuration.Once configured, you can view our logs in the explor in Grafana

conclusion

This section simply describes how to send logs of one host to another host. To send logs of multiple hosts to one host, perform the same operations.

Promtail and Loki are not covered in depth here. In addition to pushing logs from log files to Loki, Promtail can also push logs from other services such as Docker containers or system services to Loki. In addition to directly pushing logs, logs can also be filtered and modified before being pushed out. For those interested, take a closer look at Promtail and Loki.