First, record the original information

In many production environments, recording the raw data flow through the ingestion pipeline is an undesirable behavior because it may result in the disclosure of sensitive data or security-related configurations, such as keys to Flume log files. By default, Flume does not record such information. On the other hand, if the data pipeline breaks, Flume will try to provide clues to debugging problems. Debug event pipeline problem method

  • Method: Set up an additional Memory Channel connected to Logger Sink, which outputs all event data to the Flume log. However, in some cases, this approach is not sufficient.

  • In addition to the LOG4J properties, some Java system properties must be set to ensure that event and configuration-related data are logged.

    • A > Ensure that the configuration is logged,

      • Set on the command line is executed Java system properties – Dorg apache. The flume. The printconfig = true.
      • You can also set it in the JAVA_OPTS variable of flume-env.sh.

    • B > Ensure that data is recorded \

      • Please click the above Settings in the same way as the Java system properties – Dorg apache. The flume. The rawdata = true.
      • For most components, the log4J logging level must also be set to DEBUG or TRACE for event-specific logging to appear in Flume logs.

Here is an example of enabling configuration logging and raw data logging, with Log4j logging level set to DEBUG for console output:

bin/flume-ng agent --conf conf --conf-file example.conf --name a1 -Dflume.root.logger=DEBUG,console -Dorg.apache.flume.log.printconfig=true -Dorg.apache.flume.log.rawdata=true
Copy the code