background

In recent months, I have been working on a log collection system. Since this system belongs to the inheritance project, I also want to make some landmark changes in the system, which will be passed on as a baton.

From the front end to the server side, there are some major changes to the logging system, such as virtual list, electron hot update, SQL optimization, adding user concept, requiring login, updating database to Elasticsearch, etc. I’ll share some of the things I find funny later.

The first change I made was the display of the log list. The list is just a regular heap node, and the page will get more and more DOM nodes over time, which is a problem that has to be changed.

Analyze requirements

First, it’s best not to make major changes to the experience because colleagues are used to this list for a long time. To do this, I need to list the previous general features:

  • Every time a new message comes into the list, you want to see the latest data.
  • When the user clicks on one of the items in the list, the list needs to stop updating, that is, scrolling.
  • When the list is locked and the scroll bar reaches the bottom, the list resumes scrolling automatically.
  • When there is data in the selected state in the list, you can move the focus by pressing the up, down, left, and right keys.

After summarizing the previous features, I need to review my requirements:

  • As the list grows longer over time, you need to control the number of nodes displayed.
  • The list length increases with WebSocket communication, and the data update frequency is too high. Therefore, a buffer pool is required.
  • Added a list lock prompt that can be unlocked manually.
  • When moving focus, log details will be displayed immediately, because moving speed is too fast, so you need to increase the anti-shake.

After combing through, I decided to use a virtual list instead of the existing long list, which was the beginning of the road to the pit.

To start developing

Long list to virtual list

You’ve probably heard of this before in some way or other or even developed virtual lists, I don’t care. But before WE get started, let me just give you an idea. (Knock on the blackboard!!)

Why a virtual list

We know that the more DOM nodes the browser renders the page, the more performance is affected each time it is redrawn.

So let’s say we need to show a large amount of information, hundreds of thousands of pieces of data. In this case, there are many solutions. The most common solution is similar to the next page or the previous page on PC, but this solution is not friendly in terms of experience. Most users would prefer to scroll down continuously to see new content, but this will encounter a problem, keep loading data, resulting in the page pile up more and more nodes, memory consumption continues to increase, and finally even scrolling will be stalled.

We analyze again, you will find there are a lot of data are in most cases we don’t need to see, if only consider we can see the data, actually need to render the amount of data that will be very less, very good to improve the rendering efficiency, reduce because a lot of redrawing according to the influence of unnecessary.

Such a comb, the answer is simply clear —- virtual list.

What is a virtual list

There’s nothing particularly magical about a virtual list. It’s basically the idea of presenting a list, creating a container on the page as a viewable area where part of a long list is displayed, rendering the list in viewable area.

As shown in the figure, it is a simple virtual list model. There are a few concepts in the figure that you need to understand a little bit:

  • Visible area.
  • Real list.
  • StartIndex.
  • EndIndex.
Visual area

You can think of it this way: we now have a

, and we’re going to give this element some style. 
.show-box{
    width: 375px;
    height: 500px;
    margin: 0 auto;
    position: relative;
}
Copy the code

From this style we can see that the height of the visual area container is 500px.

Real list

A real list is a list that will be rendered, for example: there are currently 1000 lists that need to be rendered, but the actual number of lists that need to be rendered on the page (the data that needs to be seen) is only 100, which is called a real list.

< div class = "list - the body - box" @ scroll = "listScroll" > -- -- -- -- -- real list < div class = "list - body" > -- -- -- -- -- - carrier < / div > < / div > -------------------------- style -------------------------------- .list-body { min-height: 10px; position: absolute; width: 100%; }Copy the code

In this case, it is recommended that the length of the real list be longer than the height of the viewable area. If you have a scroll bar, then you can use scroll to listen for other operations.

There may be a point to explain to you why my

is absolute.

When one of your elements changes frequently, it is best to remove the module from the document stream by absolutely positioning it to reduce the impact of backflow.

Let’s take a look at the browser rendering mechanism

  • Parse HTML, generate DOM tree, parse CSS, generate CSSOM tree
  • Combine DOM Tree and CSSOM Tree to generate Render Tree
  • Layout(backflow): According to the generated rendering tree, backflow (Layout), get the geometry information of nodes (position, size)
  • Painting(repainting): Get absolute pixels of nodes based on the rendered tree and geometry information from backflow
  • Display: Send the pixel to the GPU and Display it on the page.

Absolute positioning or floating is out of the normal flow of documents, just storing a token on the node and mapping through that token, so if you use absolute positioning, only that element will be redrawn.

startIndex

As mentioned earlier, the actual list is actually only a small part of the total list, and there are many more lists that need to be rendered. Therefore, you can think of a real list as a fragment. The index of the first element to be rendered is the position of the first element in the fragment in the total list, which is the index in the array.

For example: my total list (array) is 1000 in length, and the list fragment I need to render is 100-200, so the index of the array at the starting position is 99.

edIndex

The index of the last element is 199.

The implementation of virtual list

It should be mentioned here that my framework uses VUE, so virtual lists are also relatively easy to implement.

<div class="list-body-box">
  <div class="list-body">
    <div 
      v-for="(item, idx) in list" 
      v-if="idx >= startIdx && idx <= endIdx" 
      :key="idx"
      class="list-row">
      <div class="col-item col-1">{{item.col_1}}</div>
      <div class="col-item col-2">{{item.col_2}}</div>
      <div class="col-item col-3">{{item.col_3}}</div>
      <div class="col-item col-4">{{item.col_4}}</div>
      <div class="col-item col-5">{{item.col_5}}</div>
      <div class="col-item col-6">{{item.col_6}}</div>
      <div class="col-item col-7">{{item.col_7}}</div>
    </div>
  </div>
</div>
Copy the code

There is nothing special about the template. The main thing is to use V-if to control the display of the list, and to use startIdx and endIdx to increase or decrease the display of data in different positions.

We’ll talk about the implementation of auto-scrolling in code below, mainly through an active event that triggers frequent increments and decrement of startIdx and endIdx.

let time = null; . autoScroll(){ time = setTimeout(()=>{ let listLen = this.list.length - 1; this.endIdx = listLen; this.startIdx = (listLen + 1) <= 100 ? 0 : listLen - 100; this.autoScroll(); }, 300); }Copy the code

As shown in the code above, I just need another method to trigger autoScroll(), which will call itself under setTimeout, startIdx and endIdx will increment and the list will scroll automatically, with an expression here

this.startIdx = (listLen + 1) <= 100 ? 0 : listLen - 100;
Copy the code

In other words, startIndex needs to be 0 until the total length of the list reaches a value.

At this point, a simple virtual list is implemented.

WebSocket buffer pool

We use WebSocket to pass data, a lot of data. Therefore, it is likely that the data will be updated too often, and the page will change as the data is updated, which will have an impact on performance. So we need to control this frequency. The current solution is to add a buffer pool.

The idea behind this buffer pool is that when WebSocket passes data, we store it in an array and then push it to the full list every 500ms. This solution might involve throttling.

let socketPool = []; // Store data for a period of time. SocketFun ((data) => {// Create an interval to cache socket data socketPool.push(data); // Push the current data to list if(! socketTimer){ socketTimer = setTimeout(()=>{ this.appendRecord(socketPool); socketPool.length = 0; this.scrollToBottom(); socketTimer = null; }, 500); }});Copy the code

In this case appendRecord() is a method that processes the data and puts it in the list, while scrollToBottom() is a method that displays the latest data when the data is pushed to the list, that is, scrolls the page to the bottom of the list.

Buffer pools are actually a way to improve performance,The heart and soul of the projectisReduce the number of page renders. You can understand: so there may be 10 per second data need to be rendered, if I honestly to render every time, so time of 10 seconds I will render 10 times, is not necessary, so we can consider to render once every 2 seconds, 10 s of time so I render times will be reduced to 5 times.You can think of it as double performance.

The list of lock

As required, the list should stop scrolling when the user clicks on one of the items in the list. So I added a scroll lock called autoScrollLoack, which means that when I click on an item in the list, autoScrollLoack = True will not scroll. The lock determination will be placed in this.scrolltobottom ().

scrollToBottom(){
  if(autoScrollLoack){
    return;
  }
  ...do something
},
Copy the code

The autoScrollLoack in the page with a two-way binding box, so the user can change the radio button selected to control the state of the lock, actually after with this lock, page if because demand to stop rolling, the user can also be perception, not scroll stopped suddenly, looks like a bug.

Focus on the mobile

The function of focus movement has been mentioned before, that is, when a message is selected, the focus can be pointed to the previous one or the next one through the up and down key, which is actually easier to achieve

<div 
  v-for="(item, idx) in list" 
  v-if="idx >= startIdx && idx <= endIdx" 
  :key="item.id"
  :class="{'active':curIdx==idx}"
  class="list-row"
  @click="showDetail(item.id)">
Copy the code

Here, you can see that active is what the list looks like when it’s in focus. Logically, the index of the current selected item is assigned to curIndex, and the style is controlled by the binding of vUE to class on the front-end template. The judgment condition is curIndex == index.

Focus function has been achieved, so the next step is to achieve the effect of moving focus through the up and down keys in the keyboard. This function is very simple, we can completely through the monitoring events provided by VUE to achieve the specific implementation you can search keyUp on the official website.

<div class="list-body-box" @scroll="listScroll" @keyup="moveFocus"> ... . moveFocus(e){ let keyCode = Number(e.keyCode); switch(keyCode){ case 38: this.curIdx -= 1; this.showDetail(); break; case 40: this.curIdx += 1; this.showDetail(); break; }},Copy the code

This code moves the focus up and down. According to the requirements, we need to display the log details corresponding to the focus items every time we focus, and the details need to be obtained by sending Ajax requests. Here comes the problem, there is a scene: I want to move the current focus down 10 times through the keyboard, and I will trigger the request 10 times in the process of continuous focus, which is not necessary, I do not care about the details in the process of fast movement, I just need to show the target details. In summary, we need to add another anti – shake.

let detailTimer; showDetail(id){ if(detailTimer){ clearTimeout(detailTimer); } detailTimer = setTimeOut(() => { $.post('... ',{ id:id }).then((res) => { do something... }); }, 300); }Copy the code

As we can see from the logic above, the request will not be triggered when the user is moving the focus quickly, and in fact this change will greatly improve the user’s fluency.

conclusion

In fact, the development of virtual list is relatively simple, but the practical significance is relatively large, in this process will involve a lot of page optimization, interested in children’s shoes can try.