Front-end early chat conference, a new starting point for front-end growth, jointly held with the Nuggets. Add wechat Codingdreamer into the conference exclusive push group, win at the new starting line.

The 14th | front-end growth promotion, 8-29 will be broadcast live, 9 lecturer (ant gold suit/tax friends, etc.), point I get on the bus 👉 (registration address) :

The text is as follows

This article is the fifth session – front-end Monitoring System construction special lecturer – voice sharing:

The opening is introduced





Hello, everyone, I am Qingyin, from the political cloud front-end team. Since last year, I have been responsible for the construction of user behavior collection and analysis system. I am very glad to have the opportunity to share with you our experience in building user acquisition and analysis system from 0 to 1.

The construction of value

First of all, why do we want to make such a user behavior analysis system?





Business value of data burial points

Students in the full department want more accurate positioning, and users frequently ask questions, reduce the amount of consultation, and improve user satisfaction. A series of functional modules have been launched to help users answer questions, but the effect is impossible to measure.





Operation wants to know which AD space, whichResource bits are more valuable? Who are their ownThe target customer?







Product and design students want toImprove user experience. Development students in the continuous iteration of the functional version online. So what are the effects of what we thought were optimization points? How do you measure it? Need to beData index supportNext job.



We will be the named user behavior acquisition and analysis system for the armillary sphere, data collection service launched a year and a half, the number of logs is armillary sphere platform has reached 1.6 billion, the number of every working day to collect around 10 million, front internal dotted line is established interest groups, from gathering requirements, design plan to fall to the ground, The total manpower cost is around 90 person days.





Later, I will introduce the Armillary sphere system in four parts. The first part is the general introduction of the key architecture, and then it is divided into three modules of data acquisition, analysis and application.

The key structure





The overall flow of armillary sphere system can be summarized into three steps. First, the data is collected, then the data is processed, and finally, the data is presented.



To support these three departments, four functional modules are realized:

  • SDK for data acquisition
  • Data processing and data storage services
  • Pit-level data display Chrome plugin
  • A site for displaying system-level data




Overall architecture of the armillary sphere

Overall Architecture Preview

First, let’s take a look at the overall architecture diagram of the Armillary Sphere. We collected data from three terminals, PC, H5 and APP.





Two SDKS are used on PC and H5 to listen for different events. Then the monitored events are reported to the data processing service through the REST interface and stored in the Ali Cloud log service. We carried out test environment and true line environment, data isolation of the two environments. The SDK of the test environment will refresh in the form of negotiated cache, and the true line will use strong cache, and carry out version control; Due to the large amount of true line data, we regularly store the log database’s historical data of more than one and a half years to OSS. There is a scheduled task every day to filter a portion of the data to be stored in the database. It also interacts with many external systems.

  • Authority system: mainly control the access authority of armillary sphere site;
  • Performance system: output some high-frequency access pages for performance detection;
  • Operation and maintenance platform for system deployment; There will be some data interaction with the big data platform. We will push the behavioral data to the big data platform, and also extract some business-related data from the platform.

System Critical Architecture

See the detailed architecture diagram of the key modules above. The module on the left is for user-oriented behavior collection, while the module on the right is for internal users. Provide users with very rich data visualization. In addition to the visual site, it also provides Chrome plug-ins for more and more data display. It also acts as a pass platform to provide some SQL queries, Excel reports export, and API data pulling reports. It can also carry out secondary development based on existing data.

The data collection





First of all, let’s talk about the realization of data acquisition module. We collected page entry and exit events, user click and scroll events, and some standard custom events. We can automatically collect the events of page entry, departure and scrolling. Click and custom events need to be implemented with the cooperation of front-end students and code implantation.



So how do we achieve automatic collection of zero code?





  • First of all, in the process of compiling the project in the page, we implanted the project ID for the project and mounted the head part of the page.
  • Then, when entering the page, according to the path of the page, to automatically generate the page ID, mounted on the body node.
  • Finally, when the user enters and leaves the page, the SDK of data collection can automatically get the project ID and page ID to locate a unique page and automatically report the entry and departure events.


Take the click event as an example. The DOM point that currently triggers the click event is called a pit. The DOM nodes that are wrapped around the pit are called blocks. Both need to be mounted in an intrusive manner, and to reduce the cost of mounting, we have provided tools to help them do so.







In the unified site, we provide the function of purchase application. After the application is completed, click generate code, but the location and ID to be mounted will be listed in detail immediately. Developers only need to paste the generated code to the location to be buried.



In addition, we talked about building in the last session. The page generated by the building system will also automatically implant these location ids. With these data, we can start a complete report.



Analysis of collected data





Here are some of the figures we presented at the conference. The xy coordinates here can be used to generate a thermal map of the user’s click area, reflecting the user’s main concerns on a page. The diagram below:





  • Operating system, resolution, and browser information allow us to analyze some of the mainstream browsers of the current platform’s mainstream users to determine platform compatibility.
  • If it is the APP side, there will be some other data such as the client version, the phone model, the current network situation;
  • LogType Classifies events.
  • Evt to specify the event type;
  • CreateTime is the time at which the event occurred.
  • UtmCnt is the trigger location used to locate the current event to a specific DOM node;
  • The UUID is the unique identifier of the visitor, and every time the user enters the page, the SDK will look for it in the cookie. If it doesn’t exist, we will generate it and set the expiration time to permanent. The IP address of the user can be traced back to the province or city of the user.
  • UserId can be associated with ** UUID ** to associate meaningless visitors with platform users to form detailed user portraits.
  • Utmfrom marks the location of the source, more on the jump to a link later, this field is used to concatenate the links before and after.
  • The reported information in the green box can be summarized as the information related to the browser page, the information related to the event in the red box, and the capital in the blue box can be summarized as the information related to the user.

Event-specific interception





We will beAll four target events are delegated to documentAll of these events are blocked as soon as they are triggered, but only after certain filteringEvents triggered by targets with pit ids and block ids are reported. This makes the data collected more accurate. If we do not conduct screening here, the reported data will become large and complete, and the amount of data will be extremely large. Therefore, the current code intrusive buried point will be switched to full buried point, which will also make the data analysis more difficult.







As you can see from the image above, a push request is sent by default when you first enter the page. There is also a push request for each click, but it is categorized in other.



This is because in the process of leaving, sending HTTP request is usually cancelled, and sendBeacon is a way to ensure that the data is sent. It can initiate an HTTP request in the unload or beforeunload event handler to send data, ensuring that our request to leave the page is recorded.







We also use it here in data reportingThe labelMainly for the sake ofEnsure browser compatibility. At present, IE does not support sendBeacon method, and some users on our platform are still using IE, and we are constantly paying attention to the proportion of IE use, so the data of this part will be very important. In order to collect more IE usage data, we will first determine whether the sendBeacon method is available. If not, we will use **<Img>** to send the request. Cors is the most common way for us to send cross-domain requests. It can use POST requests to successfully send some data reported in batches within the length limit.



Besides being able to send some reported data in a single dimension, we also need to be able to connect the whole browsing process of the user with the reported data. One of the most common is the jump of links.







Why not refer to connect the whole process? If I could, but his dimensions are still too thick. He can only go to the source page and he can’t record where he came from on the page.

  • When we enter the page, we generate a unique ID for the next entry
  • When the user clicks a link, we will subscript the current project LED page ID block ID and the pit of the current LINK A
  • Finally, the ID generated when the page is generated is composed of five tag ids into a uTM value. After the page jumps, the SDK will obtain the uTM value from the URL of the page and report the uTM value

So that we can confirm that the source of the current page is from a page on which the location of the link, also let the link click behavior into an automated acquisition event, as long as the current was implanted the block ID, all of the below it a link click can be recorded, and the whole process used in tandem.

The data analysis





In the data processing part, we use the LOG Service of Ali Cloud. One of its great benefits is that it can provide the real-time consumption interface of logs, and the query means are very rich. Ability to add real-time indexes; It currently meets most of our query requirements.





In the process of data analysis, a very important point is how to convert the collected data into understandable indicators.



One of the main processes in the figure above is to conduct data analysis. Indicators should be set up first, then data analysis should be carried out after data of points to be analyzed can be obtained, and finally data is applied to the actual scene.



It’s easy to calculate basic metrics like PV, UV, CTR, and exposure based on the data we’ve collected, but how do we translate that into a funnel model?







And that’s to splice the underlying data. Take the flow loan process of our platform as an example:

  • After entering the application details page, we can use the UV that enters the page as the first data.
  • Click immediately apply to open the protocol popup, the number of clicks as the second data, and so on, to assemble into a funnel model
  • We can calculate the percentage of each step relative to the previous one, and this is an unordered funnel;
  • Assuming that some users may enter directly from step 3, there may be cases where the percentage exceeds 100%.
  • If it’s an ordered funnel, it filters on a user basis, so we’re going to filter from front to back and the users at each step are going to say, well, we’re only going to keep the users from the first step to the last step, not counting the users from the middle. In this way, the conversion percentage must be less than or equal to 100%, which makes the data more valuable for reference.


Here is a scenario for setting confidence intervals





  • Page stay time is actually the time difference between page entry and page departure. The fastest case is that the user leaves the page in 1s
  • However, it is also possible that the user interrupts the operation in the middle but does not close the window, resulting in a very long stay time;
  • In the case of small sample size, such special data may push up the average page access time to more than 4 hours, and the average access time may lose reference value
  • The median here refers to 22 minutes and 8 seconds after sorting the user access duration from small to large. At this time, the median can better reflect the actual usage of the page.

So how do we make the average time worth something? We can set confidence intervals to exclude users who spend more than 5 or 8 hours on the page, depending on the situation, and then look at the average time spent on the entire page, or with the help of a bar chart, to see the overall distribution and trend of time spent.

Data applications





The current analysis model is mainly divided intoEvent analysis, page analysis conversion analysis and user analysis, and internally divided into 10 kinds of small indicators; Tools are also provided in custom Settings to help establish metrics;





  1. Event analysis is the basis of user behavior analysis and the most commonly used function, including frequency, distribution, and interval. You can create various analysis reports by analyzing events. Page based on the behavior data of each page, the page layout of the landing page is optimized to increase the attraction of the landing page. Thermogram analysis, by visualizing user behavior, helps us to deeply analyze user attention to content and features.
  2. Transformation analysis is the most important analysis model in user behavior analysis. Through transformation analysis, we can find out the transformation path and funnel of user behavior and improve the overall conversion rate of the platform. Conversion occurs all the way from the moment the user is reached to the moment the user completes the conversion.
  3. User analysis can well help us determine the target user group of the product, user behavior habits, user activity and retention characteristics, through user segmentation can achieve fine user operation.






From the perspective of a single page, we can get many dimensions of data, which can be displayed on the page through plug-ins;







This is a data kanban to view custom core metrics and reports in real timeReal-time grasp of key indicatorsTo help users discover problems in real time.

Operating system, browser ratio, you can click to check;

  • Number of visits, number of visits
  • Heat map: heat map is obtained by coordinate and resolution conversion
  • Path analysis: can see where and where;
  • Here are the capabilities for custom Settings: for creating events and funnels;

other





The Armillary sphere is a fairly basic data acquisition and analysis product, with a lot of extensible points behind it.



If the company does not need to build such a system by itself at the current development stage, but the business needs such a capability, it can also consider some third-party products. For the sake of platform data security, some of them also support localized deployment.









The end of the

Q&A