Introduction: This course will introduce the background of the birth of the Data center, the architecture of the data center, and how to select the appropriate technology. It will also expand the design idea of Spark based heterogeneous data synchronization tool with the implementation of a universal relational database synchronization tool suite to HDFS. In this chapter, we will successively introduce how the system architecture of Central Taiwan was born, the characteristics and existing problems of each type of architecture, how the data central Taiwan architecture solves these problems, and the evolution process of central Taiwan, to help you have a preliminary understanding of central Taiwan.

1-1 Course Introduction (05:14)

1-2 The Origin of Central Taiwan (08:09)

1-3 Evolution of Central Platform (04:21)

Chapter 2 Data center system

In this chapter, we will first review the birth process of data center from the perspective of data application. Then introduce the core capability that a mature data center should have, and decompose the architecture of the data center according to the module, introduce the function of each module, and select the respective technology.

2-1 Development stages of Data Application (06:25)

2-2 Capabilities of mature Data Medium (05:17)

2-3 Data Center Architecture (05:20)

2-4 Data center technology selection (05:50)

Chapter 3 Multi-source heterogeneous data synchronization suite based on Spark

In this chapter, the technical architecture of data collection and convergence will be introduced successively. A mature data synchronization suite architecture and functions, and lead you to implement a fully functional relational database synchronization to HDFS tool suite, expand Spark based on the general data synchronization suite design ideas.

3-1 Technical Architecture of Data collection and convergence (10:40)

3-2 Building a Synchronization Suite for Heterogeneous Data Sources (05:29)

3-3 DataX Data Synchronization Cases (15:30)

3-4 Porting DataX configuration management functionality (07:30)

3-5 Configuration Management Function Point code Demo (09:32)

3-6 Function development of Relational Database Synchronization to HDFS (07:44)

3-7 Synchronization process abstracted into three modules (06:04)

3-8 Overall Architecture Implementation of three Modules (11:02)

3-9 Concrete function realization of the three modules (10:29)

3-10 Plug-in management of Data Conversion Module (12:37)

3-11 Spark Based Data Synchronization Suite Design (04:33)

3-12 Summary of this Chapter (02:35)

Source code free download address: github.crmeb.net/u/defu