Discuss the application and design of front-end log library in SaaS products

Three netease article | yuan enterprise senior front-end development engineer

One, foreword

The author’s company mainly provides full-process enterprise services and a full set of SaaS solutions for enterprises. For enterprise services SaaS products, the completion of a customer purchase does not mean that the product value has been fully delivered, because the customer often needs to go through a series of training and use to truly generate value when they first purchase a product. Therefore, in essence, helping customers solve problems in the process of use is a paid service provided by b-end products, which is a very important link in the product value chain.

This article will focus on the application of front-end log library under the scenario of developer troubleshooting customer feedback problems, and how to design and develop front-end log library suitable for this scenario.

Two, SaaS business under the front-end challenges

The author used to do C-terminal business, naturally with c-terminal business thinking, think the front-end product interaction experience to the extreme is enough. When I apply this approach to do SaaS business, really eat a lot of losses. B end product and C end products in the difference of payment method, purchase decision and the actual users, the product use different (B end customers buy products in order to help the enterprise to make money, is the root cause of the c-terminal product purchase decisions may be just on impulse fun), leads to SaaS business focus on indicators and C end product is put in bigger difference, Indirectly leading to the different orientation of the R&D side. C-end business front-end may invest r&d resources for user experience regardless of cost and optimize webpage performance to improve user stickiness, which is manifested in mobile Internet, e-commerce and other industries tend to focus on indicators such as DAU, MAU and GMV. The core focus of b-end products is whether they can help customers improve efficiency. Whether the products can help customers achieve their work goals and achieve their goals quickly is much more important than whether the product interface is beautiful. There is an important indicator to measure an excellent SaaS enterprise — NDR (revenue retention rate). For front-end development of SaaS business, the primary challenge to solve is how to improve product ease of use, task efficiency, service efficiency and other indicators through software research and development. In this way, enterprises can improve the molecular value of NDR (renewal of existing customers + additional purchase).

B terminal and C terminal are compared

It is worth mentioning that in this regard, “Cloud Computing Software Product Use experience Quality Measurement Model and Measurement Method” also puts forward five index dimensions to measure product use experience, which is very reference. These dimensions include ease of use, task efficiency, satisfaction, consistency, and page performance. Among them, ease of use includes easy operation, easy learning and clear, while task efficiency includes function utilization, task completion rate and task completion time. Considering the sustainability of revenue from SaaS products, one of the goals of SaaS enterprises is to increase the proportion of output value relying on software products and reduce the proportion of output value relying on manual services, because only the marginal cost of output value of software products is the lowest, the efficiency of products and services can be continuously improved. At this point, purely relying on human services ultimately has a very high marginal cost, so relying on technological innovation to improve problem solving efficiency in SaaS business scenarios is a very large product value that the front end can provide.

Iii. How to solve customer feedback problems

When we focus on the solution of customer feedback problems, we can first divide customer feedback problems into functional consulting, problem reporting and pre-sales consulting. Among them, developers mainly focus on problem reporting, and some functional consulting problems that technical support cannot answer will flow to developers. For such problems, we can generally build an internal problem screening system to solve them. The main feedback problems encountered by front-end developers are not only from consultations such as SDK access, but also from problems reported by customers that product functions do not meet expectations.

In view of this, in order to effectively and quickly locate the causes of these problems, on the one hand, the front end can log on the client side and report to the problem detection system, on the other hand, for class S class A customers (based on the hierarchical model of the customer enterprise scale based on SaaS enterprises) emergency problems, if necessary, can quickly communicate with customers. Use tools such as remote assistance to reproduce and locate the cause of the problem on the customer’s device. For the latter, we designed and developed woodpecker-Remote, the best remote debugging solution based on Chrome DevTools protocol, which can support web developers to perform remote debugging directly on web users’ Chrome browsers. For the former, we designed and developed a front-end log library ** Woodpecker-Log to support persistent storage of client running status and other information for developers to debug and troubleshoot problems.

The concept of front-end logging

The concept of front-end logging is introduced here. Generally speaking, the concept of logging is often heard in back-end development. To the backend, logging refers to a file used to record the startup and running status of the server. Front-end logs here refer to log files that are stored on the client or uploaded to the server to record the running status of the client. In the development and test environment, the Console is enough to record the running status of the front end. In the production environment, the client logs need to be sent to the server for storage, which is convenient for troubleshooting and locating problems reported by users.

5. Problems and challenges of traditional front-end logging

1. Front-end log library penetration is not high, poor perception, and exceptions, performance monitoring, etc.
2. Play log is not standard, various log format chaos.
3. The log library itself occupies the front-end performance budget. In terms of performance, it is necessary to minimize the occupation of the JS main thread and ensure that the main thread is as idle as possible. Some front-end log libraries store logs on the server. After logs are generated, they need to be uploaded to the server immediately, occupying bandwidth and occupying the maximum number of concurrent requests between the browser and the domain name, thus slowing down normal services. Ajax requests need to be reported in a balance between the reporting frequency and the size of each log.
4. Large applications (for example, applications with tens of millions or hundreds of millions of users) tend to generate a large number of logs. The correlation between logs and bugs is low, and the search and analysis operation costs are high. For example, The kibana query efficiency is low when Elasticsearch is used to store massive logs.
5. Log development and deployment are not convenient, there are some problems, front-end developers need to pre-bury the log printing code in the code. Otherwise, logs do not exist and cannot be analyzed when faults are located.
6. The log library lacks the context of the problem to trace a complete session. For example, the user interface interaction operations before and after a page access problem and page redirection trace cannot be recorded.
7. If the problem feedback link is long, if it cannot be deeply integrated with the product, it is easy to lose clues in the feedback link, resulting in high communication cost. The solution can be to automatically bring the current log information when the customer reports the problem, and associate the internal work order, Jira and other problem feedback solution system.

6. Front-end log library design based on b-end services

The first problem to be solved is the correlation between logs and user feedback. The core idea is to use the client to store logs. When problems occur, users or programs discover logs and report them to the server. Here are two questions: How do users find problems? How does the program find the problem? In addition, performance problems and JS anomalies are also sources of customer feedback, but from the statistics of daily SaaS business customer feedback sources, these two are not the main sources. The other two areas of JS anomaly monitoring and performance monitoring have mature front-end infrastructure support. Therefore, customer feedback problems caused by non-JS exceptions and performance issues are mainly covered and solved by the front-end logging library.

6.1 Typical Front-end logging Scenarios

Before you start designing, think about how the front-end will use the log library. There are typical scenarios where front-end logging may be required.

1. Call a third-party service and assume the worst if it becomes unavailable.
2. Performance data needs to be reported for pages with a low performance budget and high performance sensitivity.
3. If the result of JS assertion is false, record the reason for the assertion failure.

For the third scenario, here is a simple list of demos that log using the front-end log library when the program assertion is false:

6.2 Maintainability of the log library SDK

Rather than maintaining thousands of lines of code in a single file, developing the SDK as a standalone project with a front-end engineering approach is much more maintainable. Front-end engineering refers to the adoption of modular, componentized, standardized and automated technical solutions to solve the quality and maintainability problems of engineering from the perspective of software engineering. Here are some key technology selections:

1. A programming language

The type documentation provided by the Typescript language is readable and easy to read relative to the underlying code of the SDK, static type checking helps users of the framework or library find errors before the code runs, and intelligent syntax awareness provides useful API type cues.

2. build

It is necessary to consider supporting multiple JS module specifications for SDK users. Taking rollup as an example, the configuration is as follows:

3. Automated testing

Automated testing of the SDK during development focuses on unit testing and integration testing. Unit testing is used to verify the correctness of modules, functions or classes, and can use the JEST framework. It is worth noting that fake-IndexedDB can be used to Mock a database when unit testing indexedDB stores and queries. The purpose of integration testing is to assemble modules between systems and verify the correctness of the code using true external dependencies and access to a true indexedDB database. In this example, we use Karma + Mocha + chai to test ChromeHeadless, FirefoxHeadless, and Safari browsers.

4. Version control

Versioning is based on semver.

5. The Demo and document

6.3 Using indexedDB to Store Logs on clients

LocalStorage is suitable for key-value storage of a small amount of data. IndexedDB is more suitable for client log storage. It has the following advantages:

• Store and query structured data, binary supported
• Support transactions
• asynchronous
• Processing large amounts of data

Assume that 34,952 logs with a capacity of 10 MB and 300bytes can be saved. Supports a maximum of 4369 cycles per day for 8 days.

6.4 Performance Overhead

• Network performance (latency, request failure rate) – log length, request volume
  • Use an independent DNS server to process log requests Chrome limits the number of concurrent connections to the same domain name
  • DNS Prefetch dns-prefetch
  • String compression is performed before log storage
    • Using JS based on Gzip algorithm to achieve lzma-js, Level6, 300bytes, compression rate 79%
• sendBeacon
  • Data is reliably transferred asynchronously without delaying page unloads or affecting the loading of the next navigation
• Merge request
  • Multiple small-volume logs are merged and reported on one 1 MB page
• HTTP/2 header compression
• Running performance
  • All-asynchronous non-blocking operation, storage, retrieval, and reporting
  • Maintenance storage queues Support batch log storage operations

6.5 API design

SDK initialization Settings

Instance methods

6.6 What should I do if there is an online problem but no log is entered in the code

It is often necessary to design the logs that need to be printed as business-critical processes are executed in code prior to release. Otherwise, when we need to locate the problem, we find that we do not output relevant logs, so we will be passive. At this point, you can only temporarily change the code and log, and re-publish. Is there a way to add a log to the code in case of a problem, so that the user can print the log immediately after the business process is changed, without having to go through the release process again? This paper introduces an implementation in Woodpecker-proxy, using MutationObserver interface to listen for script insertion DOM events, rewrite browser JS request, and proxy it to the target server. In order to achieve the target server to modify JS to add log code can be logged in the user browser. Demo Address: Demo.

6.7 Log Specifications – Logs are printed by application and classification

Logs that comply with good standards help you quickly filter logs based on information levels and applications during troubleshooting.

Ratings of

Points and application

Because the client storage is restricted by the same origin, logs can be accessed only under its own domain name. Multiple applications may record logs under the same domain name. Log information of different applications can be stored by application name.

6.8 Context What Information needs to be collected

• Device, browser, page information
• Associated user interface interactions for a session
• A page jump associated with a session

6.9 How can I Quickly Find Related Logs when RECEIVING Customer Feedback

1. The user ID and session ID are stored in logs and indexed.
2. The log reporting function is integrated into SaaS applications, and the current session logs are automatically queried and reported when customers report problems.
3. Write user ID and session ID into internal work order system when customer feedback problems, and bring them to Jira system when raising bugs.

7. The direction of future efforts

• Enhanced reliability
  • How can I ensure that the front-end log library still works properly and record logs when the web page crashes? Use Service Worker to monitor web crashes.
• More intuitive problem context
  • Use the browser to record the user interface before and after the fault occurs.
• Friendlier customer notifications and alerts
  • Use Notification desktop Notification.

Viii. Reference materials

• Cloud computing software product use experience quality measurement model and measurement method
• Front-end logging specification
• woodpecker-log
• woodpecker-remote
• woodpecker-proxy