background
Cloud Integral is an Internet company deeply engaged in the e-commerce industry. Its business includes some data statements and data analysis of e-commerce, and burial point has become an essential part of its business. And because of the complexity of the e-commerce industry, for the buried point this piece, also has its unique complexity.
For example, a customer, for an interactive activity held by himself, needs the data of PV, UV, number of new recruits, number of members, consumption amount, order conversion rate and so on.
Compared with traditional burial point providers, the burial point of cloud integration has common places, such as PV, UV, the number of new people and other indicators, which are better statistical. However, if it involves the data of consumption amount, order transformation and other dimensions, it is difficult to do specific index analysis. And each business, each activity, focus on different data indicators. Even for the same index, such as the number of members, some people understand that it is clicking the button to join, even if the membership, some people understand that it is through the interface to join, after the return of success is counted as the number of members. These gap, need to have a very practical tool, to help you solve these problems.
Pain points analysis
Based on the development history of cloud integration business, we can objectively view the current situation and future of the business.
Data report is a strong demand, which is the point that every business pays great attention to. At first, in order to meet the demands of customers. We define a set of rules, and big data provides interface query capabilities based on the reported data. This set of rules is actually collecting user behavior data, defining each behavior as a type of event, and each event is reported with some common fields, such as project name, page URL, user OpenID, and so on.
This design, while seemingly fine, led to a series of development disasters in practice.
1. The coupling of big data and front desk business, the upper layer business and the bottom layer big data cannot form a unified normative cognition
After each interactive demand comes in, the front end carries out the burial point according to the specification, and then the big data is processed after the burial point. But business needs, it is often adjusted, demands from customers there is often a understanding differences, and to develop, also has the understanding differences, front-end think I gave to the big data, core data of big data can be calculated, think big data, front-end data field transfer is not complete, there is something dirty data. In return, the customer side is that the data statistics of cloud integration are often inaccurate.
2. Serious internal human resources consumption, every time about the report function, we need to re-understand the data requirements for development
Data burial point should be a set of standard, quantifiable solutions. However, with the increase of different demands from customers, both the front end and big data are deeply involved in the understanding of the business. A small change in the link, from the user, to the product, to the front end and to the big data, everyone needs to understand it again. And for such a small appeal, the cost of manpower is very high.
Tentative plan
We should have a new approach to these business pain points.
1. Complete decoupling of big data and front desk business.
In principle, big data itself should not need to focus on specific businesses. Even if it participates in some business understanding, it should focus on a specific capability and service as much as possible within a controllable scope.
The initial assumption is that big data does not focus on specific businesses. It is only the aggregation, statistics and processing of data. Provides common interface capabilities for the upper layer.
The front desk business needs to be responsible for specific data. Big data is responsible for the accuracy of services, and the front desk is responsible for the accuracy of business.
2. Need to abstract the business model
According to the above requirements, the separation of big data and foreground requires good concept abstraction ability.
Our burial site design abstruses the four-layer concept, ensures the separation of big data and front desk business, and defines the standard format of their respective concerns. The four concepts are: application, indicator, event, and attribute.
1. Application: Every new project is an application, and every application will be associated with relevant indicators, which are the data that customers are concerned about.
2. Indicators are the core data that customers pay attention to. Each indicator will be associated with relevant events.
3, event, event is a user’s behavior, each user makes a behavior, will produce a piece of data, by the big data for statistics.
4. Attributes. Attributes define each event, what parameters are passed, and what each parameter represents.
It is divided into four layers model. Big data only needs to pay attention to specific events and relevant statistics. By itself, it provides only upper-layer queries for the statistical results of the event data. Last time, the service focused on indicators. Indicators need to be associated with events. The specific indicators are calculated by events and determined by upper-layer services.
3. Reports should be custom
Reports do not need to be redeveloped every time. We should have such a platform that can create customized reports through applications, because each application is associated with indicators, and we can visualize these indicators. As for big data, it still provides the underlying basic event query capability. We decide what kind of data we need.
Plan details
There are too many details about the plan, the implementation. We’re not going to get caught up in one small point and tell you what the solution is, we’re going to give you one of our solutions, our technical framework.
The overall architecture diagram is shown below
Buried point management platform
It contains three parts: buried point specification definition constraints, custom report, data market analysis.
Among them, the buried point specification defines the constraint, which is the field standard to define THE SDK for data reporting. Such as application name, code, event name, event code, and so on.
You can customize reports, configure application-related reports, read event statistics from standard interfaces of big data, and aggregate the data.
Data market analysis, that is, user behavior analysis.
In terms of technology, our buried point management platform uses a Node service as the server and react as the front end for page display.
The SDK part
The project of cloud integration is mainly composed of two parts: 1. All kinds of small programs without platform. (Jd.com, Taobao, Tiktok, etc.) 2. PC background management system and H5 page on the Web.
Therefore, in the design of SDK, there are two key points: the compatibility of small programs and H5, data field verification.
Data receiving Service
This is a Java service that is a reporting interface called internally by the SDK and has only one function: to push the reported data to Kafka.
Big data
The main functions of big data are as follows:
1, big data from Kafka consumption reported data, the specific event into the number of stores.
2. Big data runs tasks regularly every day, and collects and calculates the event data.
3. Provide standard event query service.
Ability to open to the outside world
From beginning to end, the purpose of the burial platform is twofold:
1. Provide the ability to customize reports. Solve the pain points in the business that need to re-develop the report every time.
2. Open up the API. After we manage events uniformly and generate data, we can conduct unified behavioral data analysis of users’ behaviors. Enabling business value.
Based on the buried point platform, it can do various data analysis. In addition, in our system, there is an abstraction of indicator capabilities, which are of great business value and can assist our business development and decision-making.