The main work of 2019 is to do something around Flink, which is divided into the following aspects:

I. Real-time streaming platform

Second, real-time monitoring

Three, real-time count warehouse

4. Real-time business development

Next, I will elaborate on some things I have done in these aspects and how to solve some problems I have encountered and what I will do.

I. Real-time streaming platform

First, take a look at the overall architecture diagram of Flink. The task execution mode is per-job on YARN, which is convenient for dynamic adjustment of cluster resources. The logs of each task can also be separated to facilitate troubleshooting. Flink calculation results are output to external storage, and real-time service calculations are output to MySql/HBase. The upper-layer Unified Data Service query interface queries data for visualization platform data display. Data of some monitoring classes will be output to Influxdb, and Grafana will perform data visualization and alarm. In addition, some data is exported to HDFS, and hour-level data analysis is performed using Hive/Spark. Some data quality monitoring is also performed on the output service data to detect data that does not conform to specifications in a timely manner. For us, we focus on the Flink computing framework to create a real-time streaming platform integrating task development, management, monitoring and cluster management. The architecture diagram is as follows:

For the whole platform, the goal is to allow business developers who do not know real-time computing to complete their own real-time business development through SQL to realize real-time business data. For this purpose, the focus is on SQL programming, providing source table, result table DDL, dimension table association, and also abstracting some common UDF to provide use. For some services that cannot be completed using SQL, jar mode submission tasks are also provided. You can write DataStream/Table layer apis to package submission tasks. Throughout the task in the development process, found that for some use of external data sources Kafka/MySql/Hbase, etc. It is difficult to manage, if there is a change in the source, it is very difficult, so the unified management of all external data sources, foreign only provides a data source ID, you can through the data source ID to acquiring data source information.

For target acquisition, started by invoking the rest API provided, regularly polling way to get and then through the platform to provide a visual display, but along with the increase in the late tasks, can cause a certain delay, cause the polling mode need to collect index more platform also need to be adjusted, the corresponding selection report way, InfluxdbReport is used when indicators are output to the influxDB. However, the YARN per-job mode is used, so the collected jobManager/taskManager indicators do not have job identity, so the source code is rewritten. Get the applicationId from the task-level metric and add it to the timed report.

In order to facilitate the troubleshooting of user logs, log information is written to Kafka through the customized log4J Appender mode, and collected to ES through logStash. In ES, task related logs are queried through applicationId. At the same time, the log for writing files is kept, but there are often some detailed data for processing printed in UDF or code, which leads to disk bursting. Therefore, some specifications are made. User logs can only use the specified logger name, and a filter is defined to filter it in file Logger. Make it output only to Kafka.

Because we are many area, many cluster scene, so deployment to upgrade the frame or task deployment will have trouble, on the platform made more clustering task synchronization automatically, there is no need to repeat in each cluster operation, also can avoid the inconsistency of the code, to upgrade framework provides a cluster configuration function and framework package upload function, Automated deployment through the platform.

Second, real-time monitoring

Real-time monitoring refers to real-time link monitoring, such as the number of API call requests, success rate, and time consumption, rather than service monitoring. The initial architecture is as follows:

This should be a common log link approach, where application log data is collected into Kafka, processed by Flink, written to InfluxDB, and displayed and alerted by Grafana. This method has long links, time-consuming and difficult troubleshooting, so another method is developed. The architecture diagram is as follows:

Provide a client SDK that encapsulates common metrics, such as: The client only needs to call the corresponding API, and then the SDK asynchronously sends the indicators to the middle layer. In the middle layer, a pre-aggregation is performed. On the one hand, the indicators are sent to Kafka, and on the other hand, some application information and indicators corresponding to the indicators are written to the influxDB. Application indicators were demonstrated by Grafana. After the metric is sent to Kafka, it is processed by the common Flink program, which outputs the metric data to the InfluxDB. In this way, users only need to access SDK, and the downstream processing is universal. For us, there is no need to do secondary development, which shorts the whole cycle and saves costs.

Three, real-time count warehouse

Since Flink itself provides SQL programming interface, one of the application scenarios we saw a lot of Flink in 2013 is real-time data warehouse. We are also trying real-time data warehouse based on business requirements. The current real-time data warehouse architecture is shown as follows:

In the real-time data warehouse construction process is mainly completed by SQL+UDF, the data source is mainly binlog and terminal log, and then by the Flink program to complete the cleaning, the data source into JSON format, sent to the ODS layer Kafka; The DIM layer data comes from two parts: one is obtained by real-time Flink program processing ODS layer, and the other is obtained by offline task.

At present, the following issues are mainly focused in specific business construction:

  1. In Flink Forward, it is mentioned that FirstValue can be used for real-time deduplication, but the current version 1.8 does not provide this function. Therefore, the FirstValue function was implemented in 1.8 to do exact weight removal.

  2. At present, many scenarios need to be withdrawn, such as in the number of devices corresponding to the statistical product, but the product to which the device belongs may change, so it is necessary to withdraw the previous statistical results. Fortunately, Flink SQL itself supports the withdrawal function, so some research has been done on this convenience. A typical example is that Kafka provides tableSink of the Append type and implements kafkaTableSink that accepts Retract streams.

  3. The biggest problem of join between streams and streams is the cross-window problem, which will lead to the failure of late data to be associated. In addition, global join will bring state storage problems. Therefore, in the process of use, join between streams should be converted into join between streams and dimension tables as much as possible. Another way to do global joins is to use StreamQueryConfig to set a TTL as large as possible to do timed data cleaning;

4. Real-time business development

Real-time business development is mainly to do some scenarios that SQL cannot meet, such as the need to do delayed data processing, mainly talk about several focus points in business development:

  1. Delayed data processing, in the use of event time semantic window processing, can not avoid delayed data processing, sideoutput can be used to do delay processing;

  2. The guarantee of Exactly-Once semantics, Flink itself is the guarantee of Exactly-Once semantics that supports output to Kafka/HDFS, but we use output terminals such as MySQL/HBase more often, so different schemes of guaranteeing semantics are implemented for different scenarios:

    A. Idempotent. For example, the window output is unique, so it only needs to be overwritten at design time

    B. Transactional. According to Flink’s two-phase commit, the transaction mechanism of writing MySql is guaranteed

    C. Final consistency, with the help of Flink itself, can ensure Exactly-Once, save all the results in the state, only need to output the result data in the state to the external

  3. Timing quantification output, timing quantification output is mainly to reduce the pressure on external writing, quantification will store the intermediate result data in the cache, and then use the state as a fault tolerance mechanism, timing with the timing mechanism in Flink to complete;

  4. Event time skewness: In the business logic processing, the group processing will be conducted according to specific business fields, namely keyBy operation. However, if a task has no data for a long time, the time cannot be advanced in the downstream processing, thus the corresponding operation cannot be triggered. Therefore, in the implementation process, both event time and processing time can be triggered.

  5. To ensure the sequence of data, some businesses are logically associated before and after processing, will require upstream to send the business associated data to the same partition of Kafka topic when sending data;

V. Things to do

The real-time streaming platform is improved, which is mainly divided into the following points:

  1. Provide data verification function, that is, to let the business recognize the results of our calculation data is correct
  2. The SQL verification function is provided. At present, only after the task is submitted can you know whether the SQL is correct. It is hoped that SQL verification can be carried out during the development process
  3. The platform supports the test function, provides the test entrance and the result data output function. At the same time, it also does the technology selection and application of OLAP. In addition, it also covers more scenarios, such as the application of CEP.