This paper will analyze in detail how Canal locates the synchronization site when it starts for the first time. The behavioral thinking is explained by the source code first and then by the flow chart. In the summary part, mind mapping is used to summarize, trying to trigger discussion among you. \

This program recording

    • 1. Canal locates the startup site
        • 1.1 Search Site
        • 1.2 Search for binlog loci based on timestamp
    • 2, summarize

1. Canal locates the startup site

When a Canal Instance is started, before sending the dump command to MySQL, the first step is to calculate where the binlog should be synchronized from, and how to find the loci when the Canal Instance is first started. The code is shown below:



From this point of view, the findStartPosition method will be called to find the position from which bingLog needs to start. This method is an abstract method implemented in its subclass, and we will focus on its subclass MysqlEventParser.



There are two ways to locate binlog logs in MySQL: gTID and binlog file name +position. Therefore, Canal searches for position in two ways. Due to space problems, gtid will not be considered in this section.

Here mainly call findStartPositionInternal method to find the loci, there is also a tag needTransactionPosition, found out the site said whether to start or end of a transaction.

Next, we will focus on how Canal locates the parsing point during startup.

1.1 Search Site



Step1: View parsed point data using the point storage manager. Canal provides a variety of log management implementations, more on this later.

Step2: There are two cases as follows

  • If the log Location Manager does not store relevant location information, such as processing logic at initial startup.
  • Processing logic if the log point manager has stored the relevant point information.

Since the log site manager does not store its site information at first startup, let’s look at the case where the log site manager does not store sites.



Step3: If the current connection is the master node, try to use masterPosition. If the current connection is the slave node, that is, standbyPosition, then where did the information of the two sites come from? It turns out that before the Canal Instance starts, you can manually set the start resolution site through the Positions property.

Step4: If the initial parsing site is not manually set at startup, then the synchronization starts from the last site of the current binlog. The implementation principle is to send the show master status\G command to the MySQL server. The command output is as follows:



Next, look at the processing logic for finding a point in place from the log point manager, and take a look at the entity class representing the point to get a glimpse of its structure before diving into the process.



LogIdentity records which slaveId the log site is by and the MySQL server it is connected to.



Step5: if you query the location point from the log point manager, you need to determine whether the address of the currently connected server is consistent with that recorded in the log point. If the address is inconsistent, it indicates that a failover occurs. To ensure data is not lost, a rollback time mechanism is provided.

  • If the number of times of parsing dump exceeds its wide value, it may be that the dump occurs in VIP mode. In this case, you can determine whether and how the dump occurs according to the serverId. In this case, you can go back to find the site again by time.
  • If sites link to find the information and the information does not accord with the current connection, illustrate the switch, you need to back the specified time, namely according to the time zone to locate sites, as for the back how long time, can be set up through parameter fallbackIntervalInSeconds, default is 60 s.

The procedure for locating the synchronization point when Canal Instance starts is described here, and then Canal uses the timestamp to locate the binlog point.

For process integrity, before learning how to look up binlog sites based on timestamps, let’s look at the process of looking up the corresponding site information from the site manager.

If the information of the point in place is queried from the site manager, first determine whether the currently connected MySQL server (master or slave) is consistent with the information of the site. If the information is inconsistent, it indicates that a master/slave switchover has taken place. In order to ensure the integrity of data, the site needs to be moved forward, which is reverted to the site before 60s by default.

1.2 Search for binlog loci based on timestamp

The implementation method of finding binlog loci based on timestamp is findByStartTimeStamp of MysqlEventParser. Next, let’s take a look at its implementation principle.



Step1: First query the maximum and minimum loci. For the minimum loci, send SQL: show binlog events limit 1.



Step2: then start from the last file and try to do log lookup based on the start timestamp. More on how to locate endposition from a binlog later.



Step3: if a suitable endposition is found, end the search. If a suitable endPosition is not found, try to parse the previous file. First parse the name of the smallest file to be searched, for example (mysql-bin.000036), and then reduce the file name by 1. Check whether the file name is smaller than the smallest file name to be searched. The selection continues, otherwise the search ends and NULL is returned.

Now let’s see if we can find the appropriate loci in a binlog file based on the timestamp.



The MySQL Master sends the dump command to set up a connection, parses the events one by one from the binlog log, matches each logevent from the Master, calls the seek method of SinkFunction.



Step1: If justForPositionTimestamp parameter is true, it means that only the timestamp is considered and transactions are not considered when the site is queried. In the method searched by the start timestamp, this parameter is false, that is, the method will not be entered.



Step2: obtain the basic information about the current log, such as the binlog log file, log offset, log write timestamp, and master serverId.



Step3: if the timestamp of the log is greater than or equal to the timestamp to be searched, return false, stop the stop in the file, whether to continue to search for other files depends on whether the log (LogEvent) has been found in the current file, that is, whether to find the timestamp is less than or equal to the timestamp to be searched.

Tips: Search by timestamp. The design concept is to find logEvents that are less than the maximum timestamp in the timestamp to be searched.



Step4: if the current resolved log offset is smaller than the maximum offset to be searched, the search for this file is also ended (for the first file searched). Because the current maximum offset, that is, the snapshot, is queried first, the new content is not included in the search scope.



Step5: Search for TRANSACTIONEND and TRANSACTIONBEGIN events, and store them in logPostion, which represents the events in the file that meet the search criteria. However, you do not exit the file as long as you find one event, but continue to search for the next. Until you find the right event.

Because the source code analysis is not intuitive, in order to better understand the search log point according to the timestamp, and then give its flow chart, as follows:

2, summarize

Reading the source code is not so intuitive, so first to a flow chart for a summary, to assist you to understand the core steps of the positioning site.