Data is the gold is widely recognized, data production is like gold panning, the production steps are cumbersome and need to consume a lot of computing resources. The diversity and difference of data make the data production have to continuously iterative processing logic, the design of hierarchical data warehouse simplifies the design of single-layer data model, but increases the consumption of additional computing resources, when the data quality problem is a nightmare for quality control. Sort out the source of the data, the processing steps, the references and dependencies between the data, and these information can be summed up as the blood relationship of the data. SQL powerful description ability is widely used in a variety of data tools, analysis of SQL AST is an important source to obtain data blood relationship. I developed a small tool to analyze the kinship of SQL data: www.sqllineage.com/demo
Due to the use of personal spare time, time is short, many functions still need to be improved, welcome to put forward valuable suggestions for improvement. Usually busy work, please do not call, but very welcome email and wechat message.
At present, the function provided is very simple, enter SQL and click “SQL analysis” button to graphically display the blood relationship of data.
Target functions:
- Collect, store and analyze data kinship.
2. Analyze consanguinity to optimize data production and save computing resources.
3. Trace data and quickly locate problem data.
4. Use reference relationships of data to aggregate data and complete complex incremental data production.