Ant Financial lightweight monitoring and analysis system SOFALookout server open source

preface

New technology directions such as containers, K8S, microservices, Mesh, and Serverless are fundamentally revolutionizing the way we run software. We build systems that are more distributed and, because of the container, have shorter life cycles and become perishable. SOFALookout wanted to provide a lightweight solution to these changes. Previously SOFALookout has open-source client capabilities. Today, the server-side Metrics section of SOFALookout is open source. This article introduces the main features and usage of SOFALookout server.

What is a SOFALookout

SOFALookout (SOFALookout) is an open source lightweight middleware service that solves system measurement and monitoring problems. It provides burial, collection, processing, storage, and query of Metrics. The open source project consists of two separate parts, client side and server side services.

SOFALookout aims to build a platform of lightweight Observability real-time tools to help users solve monitoring and analysis problems of infrastructure, applications and services. SOFALookout (now open source) is a project that uses multi-dimensional metrics to measure and monitor target systems. SOFALookout’s multidimensional metrics refer to the Metrics2.0 standard.

SOFALookout: github.com/sofastack/s…

SOFALookout installation documentation: www.sofastack.tech/sofa-lookou…

Key features of SOFALookout server:

• ADAPTS to major Metrics data source protocols (e.g., Prometheus, Metricbeat, etc.);
• Elasticsearch is supported by default in the open source version, and is transparent and automated.
• Compliance with Prometheus query API standards and PromQL support with appropriate improvements;
• Built-in console for data query and support for Grafana data visualization;
• Simple to use, support a single process to run the entire server – side module.

With open source server-side code for SOFALookout (Metrics), the processing of metrics data has become closed loop. In the future, we will further open source the service capabilities related to Trace and Event, please stay tuned.

SOFALookout project structure

The server-side code includes two parts: Gateway module and Server module. As shown below (showing the module outline structure of the SOFALookout source project)

├── Heavy Exercises ── heavy ExercisesCopy the code

The function of boot module in the project is to facilitate the integration and running of the Server module. It can run the Gateway and Server services independently, or complete the integration of All in One (Gateway and Server) into a single process with the help of SOFAArk.

SOFALookout working mechanism

The figure below shows the complete process path from SOFALookout to metrics data collection, reporting, storage, and final presentation.

SOFALookout currently supports flexible metrics data storage selection. For the time being, we only support Elasticsearch as a storage solution. We hope that more students can participate in the construction and support of other storage adaptors. Support for Elasticsearch was prioritized because ELK solutions are already widely used in the industry, especially for logging data.

For those of you who are not familiar with Elasticsearch, SOFALookout has built in an automated tool for storing metrics data, eliminating the need to create your own Index and maintain ES Index. More details will follow.

Open source module is added this time

SOFALookout Gateway module

SOFALookout Gateway is a lightweight data pipeline that provides rich protocol access support, including its own SDK (SOFALookout Client) reporting protocol and support for Prometheus data protocols (push and pull modes). Metricbeat protocol (version 6), OpenTSDB write protocol. Each data source corresponds to the concept of an Importer.

SOFALookout Gateway supports local hard disk buffering for remote (push mode) reporting. The overall design of Gateway is based on the Pipeline form of data processing, including the pre and post data filter to facilitate the data processing of developers. By default, Elasticsearch and Standard Exporter(which are used for data forwarding between gateways) are supported by the Gateway. Developers can also customize other types of storage or Kafka.

2. SOFALookout Server module

SOFALookout Server provides compatibility and enhancements to Prometheus’ RESTful apis for data and metadata queries. We also implemented compatibility and enhancement for PromQL (excluding Alert syntax). SOFALookout’s PromQL parsing logic was ported from Prometheus with some optimizations and improvements. Thanks to Prometheus for open-source the QL implementation of golang, which is so easy to use and powerful.

For the convenience of developers to do data exploration and experimentation, we also provide our own Web-UI support, which can meet the basic functions.

We still recommend that you use Grafana for data presentation. Grafana’s integration with SOFALookout is simple by selecting Prometheus as the data source protocol (SOFALookout’s default query port is also 9090). The following figure shows the Grafana new data source configuration:

The recent plan

Here’s the near-term Roadmap:

More students are welcome to participate in SOFALookout, especially to support more Metrics repositories.

Financial Class Distributed Architecture (Antfin_SOFA)