Lightmeter ControlCenter (Alpha)

Introduction to the

Supported mail transfer agents

Postfix MTA is currently supported. More MTA support is planned for the future.

Quick start

The installation

Install using Docker

Building from source

The following dependencies are required during development.

• Bash
• Any recent (such as 2020) version of Git.
• Go compiler version 1.15 or later.
• GCC9.3 or later.
• Libc development file. Glibc and MUSL have been successfully tested.
• GNU Make or compatible.
• Ragel6. X version. We have successfully tested version 6.10. Ragel 7 is not currently supported.
• Vue CLI – For web-based user interface.

On Alpine Linux 3.12, for example, they can be installed.

$apk add git make GCC go libc-dev ragel NPM $NPM install -g@vue /[email protected] @vue/[email protected]Copy the code

To create a release, dynamically link to the local liBC and execute.

make release

Copy the code

To create a statically linked version (only supported if your LIBC is statically linked), do this.

make static_release
Copy the code

This will download all the dependencies and create a file called lightMeter, which you can simply copy to your Postfix server and use as described in the Usage section.

If you’re going to contribute code (thank you! :-)), please check out the development.

upgrade

The installation

Install from binary

export
Copy the code

Logs managed by Rsync

If the log directory passed by -watch_dir is not in the same file system that Postfix writes to, but is copied by rsync, you must pass the command line argument -logs_use_rsync, otherwise the new log line received after the first rsync execution will not be noticed.

When using Rsync, be careful not to use any in-place synchronization options, such as –append, because the control center expects the default Rsync behavior to be used when updating files, including first creating a temporary file at the destination and renaming it to the final file when it’s fully migrated.

Docker mirror

$ docker run -p 8080:8080 -v "<path_to_workspace>:/workspace:rw" -v "/var/log/:/logs:ro" \
  registry.gitlab.com/lightmeter/controlcenter -workspace /workspace -watch_dir /logs

Copy the code

NixOS software packages and modules

You can find the published PKG, modules, test platform, and “how to build and install it” instructions on Github.

API

Headless mode (no Web user interface

Headless mode allows ControlCenter to run only in the background, with no Web UI. Currently, this is for development purposes. Headless mode can be very useful for using only the API parts of an application.

Running headless mode requires building the ControlCenter for this purpose.

make devheadless # Build ControlCenter
./lightmeter -stdin -verbose --listen :8003 # Example command to start ControlCenter quickly (same as running a normal build)
Copy the code

certification

Password reset

You can reset the administrator password using the command line.

./lightmeter -email_reset '<registration-email>' -password '<new-password>'

Delete user

• By removing<workspace-name>/auth.db*To delete all users. Such as:rm -rf /var/lib/lightmeter_workspace/auth.db* 。
• Using SQLite to manually delete a single user, usesqlite3 <workspace-name>/auth.db 'delete from users where email = "<admin email address>"'. Such as:sqlite3 /var/lib/lightmeter_workspace/auth.db 'delete from users where email = "[email protected]"' 。

Method of use

See Usage for more information.

• Run lightmeter -help to display a list of all available commands

• After compiling (or downloading) the Lightmeter control center, you should run the binary Lightmeter file, read the logs, and start the local Web server so that you can view the Lightmeter Control center through a Web UI in a browser on port 8080 on the same network, for example: http://localhost:8080/. You can use -listen “:9999”, for example to use different ports or network interfaces, in which case all the interfaces are on port 9999.

• An authentication session for a Web user interface lasts for one week by default.

• To serve logs through stdin instead of logfile location, use the command line argument -stdin, such as lightmeter -stdin < [log-data].

• You can also receive logs listened on Unix sockets or TCP ports, such as -socket “Unix; The/path/to/socket. The sock or TCP – socket “; Localhost: 9999.” It is important to note that this socket communication is unauthenticated and encrypted, so it can only be used in secure environments

• To provide a single log file, use the command line argument – stdin, such as the tail -f/path – to – the file. The log | lightmeter – stdin.

• Mail server data is stored in different workspaces so that different servers can be monitored individually. The default workspace directory is /var/lib/lightmeter_workspace. You can use -workspace /path/to/workspace.

• Since the Postfix log does not contain the year as part of the date per line, when using -stdin, the year of the log being processed is assumed to be the current year. To override this and manually specify a year, use -log_starting_year, for example -log_starting_year 2018

• Lightmeter can also “monitor” the Postfix log directory managed by Logrotate, import existing files (even if they are gzip-compressed), and wait for new log files to occur after import. To use it, start lightmeter with -watch_dir /path/to/dir, which might be /var/log/mail. If these logs have already been imported, Lightmeter does not import them again in case the process restarts.

• You can run it behind a reverse HTTP proxy like Apache HTTPD or Nginx, or even on a different path. No additional configuration is required.

Currently, log files for the following modes are “monitored”.

• mail.log
• mail.warning
• mail.err
• maillog
• zimbra.log

The environment variable

Rotating file

We were able to identify the files archived by Logrotate and import them the first time the application was run. Currently only Gzipped and uncompressed files are supported.

The log files that support archiving are suffixes.

• Mail.log.2.gz and similar files, the older the file, the larger the suffix number.
• Mail.log-20030102.gz, the suffix number is a date, the smaller the number, the older the file.

If you use a different log naming convention, create a problem on Gitlab.

Syslog compatibility

The import logs

The import process can take a long time, depending on how many files you have and how big they are.

It is important not to use -watch_dir with other ways of getting logs, as this behavior will be disabled in future versions of Lightmeter.

If you make a similar mistake.

2020/05/29 13:45:05 Missing file mail.log . Instead, found:  /var/log/mail/mail.log.2.gz

Copy the code

This means you should have a file called mail.log, which means you should check your Postfix installation and make sure it logs correctly.

Read from the Logstash

Use the default JSON-encoded log

Starting with version 1.8, we support default JSON-encoded logs sent via Logstash, which means you don’t need to change your Logstash configuration.

You need to add the command line option -log_format logstash or set the environment variable LIGHTMETER_LOG_FORMAT=logstash.

Then configure the Logstash as follows.

filter {
  if [log][file][path] == "/var/log/mail.log" {
      clone {
        add_field => { "log-type" => "mail" }
        clones => ["lightmeter"]
      }
  }
}

output {
  if [log-type] == "mail" and [type] == "lightmeter"{
        tcp {
            host => "address-of-control-center-host"
            port => 9999
        }
    }
}
Copy the code

Use a custom log format

filter {
  if [log][file][path] == "/var/log/mail.log" {
      clone {
        add_field => { "log-type" => "mail" }
        clones => ["lightmeter"]
      }
  }
}

output {
  if [log-type] == "mail" and [type] == "lightmeter"{
        tcp {
            host => "address-of-control-center-host"
            port => 9999
            codec => line {
                format => "{[@timestamp]} %{[message]}"
            }
        }
    }
}

Copy the code

Characteristics of the file

notice

Integration of slack

Create an app on your Slack account and go to api.slack.com/

• Click on “Create Custom APP”, then you will see a pop-up window, used to configure your application (https://api.slack.com/apps).

Domain name mapping

Domain name mapping is supported. This means that interconnected remote hosts are treated as a whole when necessary (for example, Outlook.com and Hotmail.com).

Currently, the mapping is hard-coded in the application – changing the mapping requires rebuilding the application.

Map is stored in the domainmapping/mapping json, covering the biggest remote host by default. You can easily customize the mapping by editing the file and then rebuilding it.

Consider extending the default mapping to benefit all users by making merge requests

The message the detective

Management view

You can click the “search” icon on the navigation bar to enter the information Detective’s administrative view. Using the sender address, the recipient address, and the time interval you want to check, you can determine the status of any message processed within a given time frame.

The search results will include the status of the information, the queue ID, when the information was processed, and the status code for each delivery attempt.

A message can have one of the following states.

• Send a success message
• Messages rejected by the recipient’s mail provider are returned.
• Messages that were temporarily rejected and retried were delayed
• If the delivery is abandoned after too many delays, it is overdue.
• Return: When a return notification is sent back to the original sender (only if your Postfix is configured to do so).

Public view

Known problems

High risk

Low risk

• Some insight triggers too often (depending on usage) and can fill the home page with repetitive details (planned fix: #231,#157).

• Clicking on the chart section on the home page may cause misleading statistics to be reported (Planned fix: #63

The development of

Front-end development using VueJs

ControlCenter’s Web UI uses the VueJS framework and related tools. The front end is essentially a stand-alone Javascript application that interacts with the back end through an API. When built for production, the ControlCenter back end serves front-end files for convenience.

Foreground files, such as javascipt and CSS, need to be compiled and moved to the correct directory before they can be used by Web browsers. This process is handled automatically by the make command/build script and the VueJS terminal tool for development.

The most efficient workflow for developing/modifying front-end files is to run the front-end and back-end separately (as opposed to production mode), provide services using the Vue CLI and automatically rebuild the files in development mode. This allows you to see changes to foreground files almost immediately, without having to execute any commands or manually rebuild or compile any files.

instructions

Browser Automation testing

These tests (also known as user acceptance tests) are found in the Acceptance_tests directory and are performed by Gauge and Taiko. These tests are part of the CI/CD and are performed on each GitLab commit.

Refer to the acceptance_tests/ readme.md file for details on how to run these tests manually.

The user interface can be translated (I18n).

Make back-end strings translatable (i18n

Translatable strings can be found in different files, such as the Go code used on the back end, or the Vue/ HTML/JS files used in the Web UI.

To update translatable strings, make them available to translators with commands.

$ make messages
Copy the code

Example.

package
Copy the code

troubleshooting

Problems after the upgrade from VERSION X to version Y

• Many of the problems with ControlCenter loading the UI can be solved by clearing browser cookies and caches.

Support unit

License and copyright information

This project is licensed under AGPL-3.0. However, some files are under different licenses, please check individual files and files for accurate information. Reuse /dep5.

Copyright 2021, Lightmeter[email protected]