Two small things happened during the National Day holiday, one was that I spent my 35th birthday in silence, the other was that I played Grafana and was praised by engineer M on the first day after the holiday: quite professional.

1. Why am I playinggrafana?

Several months ago, I submitted a data background requirement to engineer M. He selected and deployed grafana, a third-party open source tool, and completed the requirement in only two or three days. This was pretty fast, and I was impressed by M and Grafana. However, I am still not satisfied with the finished product submitted by M, where all the data is graphical and I prefer tabular data. To further integrate the other data for statistical analysis, I manually exported the data and wrote a complex Python script to do statistical analysis with PANDAS, which I ran at least once a week. I often come up with new data requirements. They are not big, but going through the requirements review, scheduling, development and acceptance process can seem cumbersome.

All these are just paving the way, the real trigger was the sprint summary meeting before the National Day holiday, which mentioned that the data of several of our products would be integrated into grafana unified implementation. The CTO mentioned to me that if I knew some SQL statements, I could write them myself. – huh? How to use python to interact with mysql and use common SQL statements? I was in “SQL but so, release your horse and come!” fearless state -_ – | |

National Day holiday is coming. Let’s have fun.

2. How to get started in a short timegrafana ？

In order to develop my requirements, M has completed the deployment of Grafana and set up the data source. This time he created a practice dashboard and gave it to me to edit. In particular, I only have the right to query the data source without the right to add, delete or modify it, which is safe for the data source. In addition, the existing implementation of M also allows me to imitate the model for the first time. These are some of the advantages I had over the grafana novice who was completely unprepared.

I don’t want to re-implement the requirements THAT M has already developed, I want to implement my own data requirements. My mind was full of requests for what data I wanted to present in what form.

The learning process is divided into three parts:

  • A: Familiar and understandinggrafanaTo learn how to add and edit charts and other modules on dashboard is to understand how the tool works.
  • B: Be familiar with and understand data sources (our product database), which tables, which fields and so on.
  • C: How to write SQL statements to fulfill data requirements.

In part A, I searched for two articles on how to use Grafana. Always follow other people’s organized step by step diagram, learning experience is not good. Most of the time, I do it directly. This open source visual tool, try it yourself can quickly become familiar with.

B) It is OK to ask engineers to export all forms of describe for help. M gave me a class definition file of the table. Through the exploration in Part A, I quickly found that when creating A module on Dashboard, if the type of line chart is selected, the SQL statement editing area can select the table name, so that the data tables are clear. If you select the table type and use select * from table_name limit 50, you can render some of the table’s data so that it is clear what fields the table has.

In part C, I separately copied the SQL statements realized by M to The Jupyter Notebook, and disintegrated them into more basic knowledge points and got familiar with them. One tip is that the legibility of SQL statements is important to beginners and can directly reduce complexity. So I used the markdown syntax as follows, which makes the syntax clear:

The above three sections do not need to be done in order. Start whichever module you are more interested in; The process can also be cross-rotated. The next step is to repeatedly consolidate and deepen the above three parts by realizing their own data needs until they produce satisfactory output. 24~48H is enough time to spend, without missing holidays with family, sleeping late, watching movies.

3. Be aware of the division of labor

Finally, IT’s important to note that I know I don’t want to take the place of an engineer in the back-end of data development. Such as:

  1. Some complex requirements, I write SQL, difficult and slow to learn. I will collect and list them one by one, arrange them and ask engineers for help. I will not drill into them by myself.

  2. Even if I ended up with a lot of diagrams, I was really only thinking about implementation, not performance (and not yet ability), so even the features I wrote had to be reviewed and optimized by engineers.

Even so, there are significant benefits to going directly to the data source and implementing it with Grafana:

  1. I have a better understanding of which original data have been collected, which indicators I can define and count, and which need further support from engineers.

  2. I can directly implement some relatively simple data monitoring/statistics that are helpful to business. There is no need to sort out and describe the requirements – communicate with the engineer – realize it after the engineer understands – and then accept such a complicated process.

  3. As the initiator of requirements, my immature requirements can also be very efficient in the process of iteration.

Of course, these ideas need to be communicated clearly with engineers and superiors, so as not to have misunderstanding. If my notes are helpful, please like them or let me know in the comments!