TiDB Hackathon 2020 TiDB Hackathon 2020 TiDB Hackathon 2020 TiDB Hackathon 2020 TiDB Hackathon 2020 This year is the fourth time that TiDB Hackathon has been held. The scale of participating teams is the largest ever, with a total of 45 teams from all over the world signing up, realizing global linkage for the first time. After 2 days of extreme challenge, many exciting projects emerged in the competition. In order to let more friends know the stories behind these teams, we will start TiDB Hackathon 2020 Excellent project sharing series. This article will introduce the wonderful stories behind HundunDM team before the competition.
We invited nolouch, HundunDM, CheneyDing, lhy1024, and the judge xu chengxuan to talk about the so-called SQL “vision”.
“Observability” of SQL
TiDB 4.0 not only improves the performance, but also makes more efforts to monitor its own performance, adding a lot of memory tables, so that DBAs can monitor the data distribution and real-time status of TiDB cluster in time. Nolouch found that the distribution of data is an important indicator in the daily analysis of scheduling problems, but operations personnel often need to enter a long SQL or find the right script to perform more granular analysis, and the results are not very intuitive.
Query the data distribution of the table
The query results are not intuitive
So he came up with the idea of creating a visualization tool.
The Hackathon gave them a chance to put their ideas on the ground. TiDB, as an HTAP database, has both efficient transaction processing capabilities and powerful real-time data analysis capabilities. We can use the advantages of TiDB itself to combine BI tools into TiDB ecosystem.
** They hope to learn from Metabase and other open source tools to present the obtained data in forms, line charts, bar charts, pie charts, etc. Combined with TiDB Dashboard, users can directly perform data visualization analysis in TiDB. ** And TiDB has a lot of internal and its own state of some statistics, but also through the tool directly to TiDB some problems for diagnosis.
Metabase can visualize data
Practice: Work together and perform their respective duties
Since TiDB already had many built-in system tables for analysis, we discussed the demo that could be displayed, including table region distribution, leader distribution, abnormal log entry time distribution, and traffic distribution, but these were not enough to give play to its imagination. Further, HundunDM hopes to display the distribution data of the underlying engine SST. In this way, one person is responsible for a module, and HundunDM team starts to take action.
The project is divided into two steps: data acquisition and data visualization.
- Data acquisition
The user selects specified tables, filter criteria, and aggregate functions and columns from the TiBI component to automatically generate the corresponding SQL. The Dashboard obtains the TiDB address and query data from the Client.
- Data visualization
Refer to Metabase and other open source tools to obtain data in tables, line charts, bar charts, pie charts, etc.
In addition, they want to integrate TiBI into the TiDB Dashboard:
- pluggable
Like a diagnostic plug-in, TiBI is implemented in Dashboard as a plug-in. If TiDB can support the plugin market in the future, it can be open to community maintenance and development as a third party.
By a strange coincidence, each of them happens to be working on a language: Nolouch is responsible for the TiKV layer API and uses Rust; Lhy1024 is responsible for processing data and associated built-in tables from TiDB, using the Go language; CheneyDing is responsible for the statistics of SST data in RocksDB, using C++; Hundundm is responsible for the front-end work, using JavaScript.
Finally, they completed the interface realization of the expected five modules. In addition, TiBI can also carry out data health distribution statistics and rapid disaster recovery analysis. Log analysis, performance analysis; Displays machine and rack distribution at each topological level, and displays hot and cold data distribution within Engine, hierarchy characteristics, etc.
SST distribution of LSM-tree in distributed scenario
“The visual effect of this project is very good,” Mr. Xu Chengxuan, the judge, commented highly on this project. “They can show the level distribution of RocksDB SST table, which is different from general visualization projects. “It’s very valuable to visualize something that is difficult to understand about RocksDB’s storage hierarchy and data distribution.”
About the future: reduce the learning cost of TiDB for operation and maintenance personnel
Due to the limitations of various conditions, they also left a lot of regrets in this Hackathon. For example, they did not completely comb through the interface, but used external tools to transmit the data in TiDB, and there is a lot of room for improvement in performance and security.
When it comes to their expectation for TiBI in the future, their idea is very simple, that is, to integrate into TiDB Master and reduce the cost of TiDB database learning for operation and maintenance personnel through the realization of multidimensional data visualization. In addition, it can help TiDB operation and maintenance personnel to more quickly understand the workload of various dimensions of the system and quickly locate problems.
In order to do this, they not only need to solve the problems exposed in Hackathon, but also need to make more extensions to TiBI’s features, such as dynamic observation ability, flexible adjustment of multiple dimensions, rich custom templates built in, simplified guidance of custom templates, plugins open to the community…
Judge Xu Chengxuan was also optimistic about TiBI’s future. He said, “BI is a concept in the field of data mining, and AI often makes some statistically related decisions based on data. ** Now that AI+ databases are a trend, TiBI sees the possibility of BI+ databases. ** Now the configuration and index of the database itself is more and more complex, we also pay more and more attention to the data generated by the database itself, a better, more intelligent tool is very valuable. The introduction of BI concepts into the field of database monitoring, unlike traditional monitoring tools, is profound.”
Now TiBI has entered TiDB Incubator, promising future!
TiDB Hackathon: An amazing journey
In TiDB Hackathon 2020, HundunDM team consists of several Hackathon veterans, as well as newcomers who participated in Hackathon for the first time. For both the first and NTH time, Hackathon was a fun experience.
For example, there were all kinds of unexpected situations in the actual competition. The students in charge of Rust, the core part of the project, had to solve customer problems temporarily, which made their poor development time worse. They had to find another way to export data utilization tools and then process them through the front end. The students in charge of the front end were not professional front-end engineers, and the React update and iteration speed was very fast. HundunDM also felt helpless when he mentioned this part: “It seems that one of the meanings of Hackathon every year is to learn the changes of React in this year.”
Teacher Xu Chengxuan is also an old Hackathon player. In 2019, he attended the competition as a contestant. When he mentioned his feelings about participating in Hackathon as a judge, he praised repeatedly: “I thought it would be more interesting to be a contestant, but the projects in this competition were very creative and completed, and it was exciting to see the full picture of each project for the first time as a judge.” At the same time, he also expressed his love for TiDB Hackathon: “Hackathon is a very interesting activity. In these two days, I have no work burden, and I can easily realize my brainstorm ideas. It’s also a geeky event that allows you to meet like-minded people.”
In this Hackathon, due to the epidemic situation, many contestants took a remote approach. When asked about their expectations and suggestions for Hackathon, contestants and judges agreed that they hoped that next time Hackathon could provide more opportunities for offline communication: drinking wine and singing songs, let the code fly ~