1. Basic introduction

DataX is an offline synchronization tool for heterogeneous data sources. It provides stable and efficient data synchronization between heterogeneous data sources, including relational databases (such as Mysql), HDFS, Hive, Hbase, FTP, and ElasticSearch. And has a strong scalability.

In solving the synchronization problem of heterogeneous data sources, DataX changes the complex mesh synchronization link into star data link. It does not need to write data synchronization code through complex development between data sources, but can rely on DataX to connect different data sources. Each data source plug-in can only consider the data it needs to synchronize to the target data source, DataX can help you do this efficiently.

DataX currently supports the following data sources:

2

  • Heterogeneous data sources: Data between different database management systems. Due to the influence of factors such as business, usage scenarios, data volume and different requirements, enterprises will use a variety of different storage methods for business data, including data management systems, from simple file databases to complex network databases, which constitute heterogeneous data sources of enterprises.

  • Offline: Online and offline usually refer to whether a function or work (such as detection, control, or assignment) is performed during or before (or after) system operation. Offline is usually the production process is not connected with the computer, and not controlled by the computer, but by people to contact and do the corresponding operation called offline mode or offline mode.

3. Main features

  • The data synchronization between different data sources is realized efficiently
  • Full memory operation, not read or write disk, therefore high efficiency
  • The framework is open and extensible, enabling developers to develop new plug-ins to support new data sources in a short period of time
  • Using Framework + Plugin architecture construction, Framework is mainly responsible for caching, flow control, concurrency, monitoring, statistics, context loading and other high-speed and efficient data synchronization of most of the technical problems, provides a large number of interfaces can interact with the plug-in, plug-in only to achieve access to the data processing system
  • Single-process, multithreaded running mode.

4. Application scenarios

  • DataX is preferred when data synchronization between different data sources is required.
  • DataX transmission is efficient and stable, suitable for full data synchronization scenarios.
  • Large-scale migration of data volumes between different or identical data sources.

5. Advantages and disadvantages

Advantages:

  • Data quality monitoring is perfect, so that the data can be intact and intact transmission to the destination
  • Data synchronization between a large number of data sources is supported and is highly extensible
  • The speed can be controlled and the number of concurrent requests can be configured through the configuration file
  • DataX3.0 provides three flow control modes, including channel (concurrent), record stream and byte stream. You can control your job speed at will, so that your job can achieve the best synchronization speed within the range of the library can bear
  • The synchronization performance is very efficient. Each reader has one or more sharding strategies, which can reasonably divide the job into multiple tasks for parallel execution. The single-machine multi-thread execution model enables the DataX speed to increase linearly with concurrency
  • Easy to use and install, out of the box, easy to use
  • The handling of dirty data is thorough and thoughtful
  • Good stability, log perfect and humanized

Disadvantages:

  • Single-node, not distributed.
  • Running full memory operation, high stand-alone pressure.
  • Json files need to be configured for each synchronization, which is difficult to use in complex scenarios.

6. Summary

DataX is a data synchronization tool widely used in Ali. The principle and use of DataX will be introduced in detail in the follow-up. If you want to know more about the core advantages of DataX, you can go to the official documents.

Github.com/alibaba/Dat…