What is front end burial point?

General Manager Ma once said that now is the era of DT (big data era). Data has become the most valuable wealth of a company, and more and more Internet companies begin to pay attention to the application of data. The process of data application is: data collection -> data collation (data synchronization) -> data analysis -> data visualization. Front-end burying point is a very important means in the field of user behavior data acquisition. It refers to the relevant technology and implementation process of capturing, processing and sending specific user behaviors or events. Of course, front-end buried point reporting is not only the collection of user behavior data, but also a very important field — front-end health analysis. Including collection page loading performance, JS error collection, interface error reporting, custom speed measurement and so on.

Why do you want to bury the front end

Buried points can help analysts get the business data they really need and the ancillary information, which plays a very important role in product development, business decision making and operation guidance. Business people may focus on different information and perspectives in different scenarios. Typical application scenarios include Taobao activity operation, PD and UED. The former focuses on source channels and advertising effects, while the latter two care more about the optimization of product processes and experience. It is necessary to collect the corresponding data through buried points and make decisions and judgments based on the data. For example, to increase the pass rate at a junction, a city must readjust the logic of traffic lights. It is not enough to just look at it. We need to get the vehicle flow momentum at the intersection at all times and in all directions at the same time, which requires the installation of cameras and traffic flow counters on the corresponding nodes. Same thing with buried points.

DataWorks products and buried site needs

DataWorks is the largest and most excellent one-stop big data platform within Alibaba Group, integrating big data design and development, operation and maintenance monitoring, data integration, data management, data security, data quality and other products, and through the algorithm platform PAI, forming a complete big data closed loop.

Demand background

At present, all front-end products of DataWorks lack a unified burying point scheme, which results in the product and UED not being able to accurately get user data to assist decision-making. Given this background, we faced some challenges in burying and reporting data for DataWorks.

Data to be reported

  • UV/PV (support single page application)
  • Click events for all button links
  • Navigation click event
  • Table Filter form usage
  • Operating system and browser information
  • Front-end performance data
  • Front-end error record
  • Interface return time
  • Error information, such as errorCode and requestId, is reported on the interface

Problems encountered

  1. First of all, DataWorks is a data platform with 15+ products, involving nearly 50 pages, which is a huge amount of work if you use traditional buried sites.
  2. Due to historical problems, the front-end technology stack of the product is not unified, including jquery, Requirejs, React, Angular1. x, and some self-developed front-end frameworks. As a result, unified buried point rules cannot be used for data collection and reporting.
  3. Many of DataWorks’ products are complex single-page applications, and the interaction and iteration requirements are particularly frequent, which have high requirements for the expansion and flexibility of user behavior data burying point reporting.

Other Business requirements

  1. The service needs to analyze user roles and BU information. Therefore, the reported data must contain the current user information and some product customized information.
  2. One place to see PV/UV for all products, online time, attrition rate, etc.
  3. To meet the requirements of customization, it can easily extract and analyze the reported data and output reports.
  4. Support the reporting of buried points of private clouds.

Burial site design

Design principles

  1. Supports basic PV/UV and front-end health report, and supports user – defined report.
  2. Try to be traceless, or make as few code changes as possible. Reduce the workload of accessing buried sites.
  3. Can provide fast data extraction analysis methods.
  4. It allows developers to pass in custom data each time they collect data for different products. For example, user information, project information and so on.
  5. By default, all records can be set to report sampling, and provide a switch to close the buried point function.
  6. Unified entrance, such as unified access to the buried point in the public head, and then external exposure method, so that each product set its own buried point configuration.

Technology selection

Two mature solutions within the group:

  • Aplus: it can collect users’ PV/UV, and statistics the online duration of the page, the loss rate and other indicators, and provides a visual display of the page data.
  • Retcode: More focused on front-end health monitoring and alarm, and provides flexible custom reporting interface. In addition, Retcode’s source log data is relatively open, which is more convenient for subsequent data analysis.

For the data reporting requirements of DataWorks, we decided to use Retcode for the main data reporting and aPlus will also be accessed by default (Ali’s Nginx will insert APlus JS into the page by default).

But one of the buried requirements that neither APlus nor Retcode can implement is “click events for all button links”. This requirement will need to use the “non-trace buried point” (under the “non-trace buried point” scenario, data monitoring tools tend to capture and send as much as possible when monitoring events and information, and trigger condition match after data processing and statistical calculation work and so on, to better support your focus changes and historical data back.) .

However, due to the disunity of the product technology stack of DataWorks, especially the current industry is basically blank in the direction of the traceless burying point of the React and Angular frameworks. Therefore, I designed a set of burying point mechanism to achieve traceless burying point under different front-end frameworks, and reported the collected events and data through Retcode.

Architecture design

From the structure, divided into base and plug-in two large.

  • Base: responsible for plug-in loading, data encapsulation, processing, and reporting.
  • Plug-in: Buried point schemes of different front-end frameworks develop plug-ins based on specified data formats, and insert them into the base block as needed to achieve buried point data collection.

In terms of functions, the base is divided into three layers: data collection layer, data processing layer, and data transmission layer.

The base design

It says the base is divided into three layers:

  1. Data acquisition layer, provides a complete set of plug-in access mechanism. The base itself does not provide data collection. It is only a container that allows plug-ins to be inserted into the base according to certain rules to provide data collection.
  2. Data processing layer: Provides the data specification according to which each plug-in wraps the collected data (such as business data added to each product configuration) and passes it to the data transport layer.
  3. Data transmission layer: Encapsulates retcode to report buried points.

The advantage of this design is that the base is not responsible for the collection, the collection work is implemented by the plug-in. The plug-in mechanism also ensures the extensibility of the whole burying point mechanism, and even other developers can develop various front-end framework burying point reports based on this base and specification in the future.

Plug-in design

At present, we have developed buried point collection plug-ins of jQuery, React and FETCH. They front-end event and request data collection schemes, mainly through Hack common front-end framework, on the event handler. The specific Hack methods are described below.

  • Advantages: Can accurately locate specific selection elements and their functions, and can combine the advantages of the framework to transfer business data, which can better meet the needs of processing data in complex business scenarios.
  • Disadvantages: We must execute our buried code before executing our user code.

JQuery buried points:

  • Step 1: Rewrite jQuery’s event binding methods (on, Click, delegate, etc.)
  • Step 2: Determine if it is click, record the selector and wrap the callback.
  • Step 3: When callback encapsulation is performed, the selector is passed into the function and reported.

Note: It’s important to distinguish between selector and find, because it can be found by parent or find, and we need to look for the exact element based on selector and prevObject.

React buries:

React.createElement: returns the Property passed by the component. If there is an onClick event, the Property will be used to report the data. The Component name and parent relationship of the Component can be reported throughout the process.

Request collection scheme:

Current Ajax for jQuery. The main approach is to modify ajax’s beforeSend and complete functions for uniform ajax processing and reporting. In the case of FETCH, the hack browser’s native FETCH method wraps the request and reports the request.

Other frameworks Angular, Vue, backbone….

If we want to use our buried point mechanism, any front-end framework just needs to follow the data format of the transmission and implement their own click event collection.

Solve problems and optimize

Optimization: 1. Optimization of PV/UV data collection, because the page will not be closed when users use DataWorks, resulting in inaccurate PV/UV data. Therefore, PV/UV collection will also be carried out when users activate the page for the first time every day. 2. JQuery plugin to reduce unnecessary global events, such as tied to the body to hide events in the right-click drop-down menu.

1. Retcode must be introduced during page document stream loading, otherwise PV/UV and page loading performance data will not be reported. This is a serious problem because our base library is uniformly introduced through the common header and the document stream is already executed, so we implement PV/UV reporting by manually calling the SPM reporting interface provided by Retcode. In addition, performance data is manually collected and reported through the PerformanceTiming API of the browser. 2. The on event binding method of jQuery needs to extract the GUID during hack and assign it to the hack function, otherwise the corresponding function will not be found during OFF. 2. Promote Seven Star Array to support Retcode data reporting.

The late plan

The main goal of this period is to produce data, that is, to collect, report and summarize the data of the point of purchase. The follow-up work can be developed from the two directions of front-end and back-end data processing.

Front end direction:

  1. Make browser plug-in: Based on buried data, develop browser plug-in similar to Udata, which can help PD, UED and operation to have an intuitive understanding of user behavior data on the page, so as to support their product design and decision-making. The data source of this plug-in can be real-time data or offline data.
  2. Data visualization: Through the integration with some customized report tools within the group, it can quickly and conveniently display data visualization.

1. Automatic generation of offline tasks: DataWorks itself is a set of big data platform, which can realize data extraction, analysis, operation and maintenance. In the future, we will consider building a mechanism that can automatically generate scheduling tasks to help users with data analysis.

That is to form a closed loop of data collection -> data consolidation (data synchronization) -> data analysis -> data visualization.

​​