• 原文 address: How GitHub Actions address large-scale logs
  • Written by Alberto Gimeno
  • The Nuggets translation Project
  • Permanent link to this article: github.com/xitu/gold-m…
  • Translator: felixliao
  • Proofread by: Tong-h, KimHooo, and Hoarfroster

How do Github Actions render large logs

Rendering logs on the Web side looks simple: they’re just lines of plain text. But it can also have a lot of user-friendly add-ons: coloring, sorting, searching, permaculture, and so on. But most importantly, the user interface for displaying logs should work, whether it’s just ten lines or tens of thousands of lines. This has been a priority for us since we opened GitHub Actions to the public in 2019. We didn’t have usage metrics back then, but it was obvious that we needed to deal with very long logs. Browsers can get stuck on the first load, or they can become completely unusable if mishandled. We have to apply a technology called virtualization.

Virtualization means rendering only a portion of the list. This allows the interface to work smoothly without the user noticing that the content outside the visible window is not being rendered. It needs to update the currently visible content as the user scrolls, and even calculate the position of the layout before the content is rendered to ensure a smooth scrolling experience.

The preliminary implementation

When we first launched GitHub Actions, we tested a React based library and a native JavaScript based library. Some libraries are very limited because of the way they are implemented. For example, many libraries require that all rendered items have a fixed height. After all, this limitation makes them much easier to calculate, because if a user wants to scroll to a particular item (whether visible or not), they just need to calculate item_index * items_height to figure out its position and scroll over. Again, they only need to calculate items_count * items_height to calculate the height of the entire scrollable region. Of course, in many cases not all entries have the same height, which makes this limitation unacceptable. In GitHub Actions, we want to have lines that are very long, which means we need to support highly variable log entries.

We ended up with a library based on native JavaScript that covered as much functionality as we needed: support for rendering highly variable elements, support for scrolling to specific items, and so on. However, for a variety of reasons, we started to find bugs and user experience flaws:

  • The scrollable area needs to have a fixed height, which makes it easy to implement internal functions, but is also a typical limitation. But in our case, this makes for a poor user experience, especially in our log rendering, where a task has several steps and each step has its own list of steps that need to be rendered virtually. This means that each step on the page needs to have its own scroll area, as well as a scroll bar that serves the entire page.
  • The ability to switch the visibility of virtualized lists from hidden to visible has not been well tested. In GitHub Actions, we allow users to expand and collapse steps, and log automatically. We found a bug when the steps started to unfold. When the log is automatically expanded to become visible, but the page tag is not visible at the time, users will sometimes not see the previously expanded log when they cut back to the page.
  • Users cannot scroll and select text at the same time, because the selected text is removed from the DOM for virtualization.
  • Sometimes we need to render the log lines in the background to calculate their height, which makes the experience very stagnant. Virtualization didn’t help us much, in fact some rows were rendered twice instead of not being rendered at all. But we have to do this because the wrong height calculation can cause the log line to be truncated by the interface.

Revisit the logging experience based on actual usage

For these reasons, we decided to improve the logging experience. Let’s start with a question: Do we still need virtualization? As mentioned earlier, we don’t have metrics to use, but now we can make decisions based on actual usage. For example, if most users’ logs are small enough to render without virtualization, we can remove virtualization but allow larger log files to be downloaded separately.

Our data shows that 99.51% of existing tasks are no more than 50,000 lines, but we know that browsers start to struggle to render logs that exceed 20,000 lines. At the same time, we found that even a small number of log lines can take up a lot of memory. Armed with this information, we decided that we didn’t need data virtualization, but we still needed user interface virtualization. Applying data virtualization would have allowed us to load only part of the log into memory and continue to load more information as the user scrolled, but we found that this level of complexity was unnecessary. In edge cases where the log file is very large but the number of lines is small, we truncate the log and provide a link to download the log.

Implementing our own virtualization library from scratch

We immediately tried other libraries that we hadn’t used before, but none of them met our needs. We need to implement one of our own from scratch. Our goals are:

  • You can render at least 50,000 lines of logs, preferably on a mobile phone.
  • Allows the user to select text without limitation.
  • Keep the user interface/experience smooth for the most part. This includes jumping between search results or permalink and streaming logs.
  • Use only one scrollable area and the height does not need to be fixed.
  • Use a viscosity meter head on the steps.
  • Make calculations as fast as possible and use less memory. Less than perfect calculations are acceptable.

To achieve these goals, we need to approach other libraries differently in several ways:

  • Predict height before rendering: Other virtualization libraries rely on precise calculations or fixed heights and use absolute locations to make internal implementations easy. We decided to estimate the height before rendering to avoid complex calculations. We also use relative positions to avoid truncation of log lines due to inaccurate calculations.
  • DOM structure: Another challenge is how to organize the DOM to allow sticky headers with only one scrollable region. Not only must the rolling container satisfy these two conditions, but the sticky header cannot be virtualized.

We did a quick implementation to verify our strategy. After some testing to generate large logs, we found that while we proved that we could achieve all our goals with our own implementation, we had to be very careful because it was very easy to make mistakes and ruin the entire user experience. For example, we quickly learned that while it’s important to change as few DOM nodes as possible, it’s also important to minimize DOM changes when scrolling. As the user scrolls, we need to add nodes that become visible and remove those that are no longer in the visible view. If the user scrolls quickly, especially on mobile devices, it can easily lead to too many DOM changes and a bad experience.

However, there are several ways to solve this problem. For example, you can use throttling to batch slow updates. But we found that this approach made the user interface clunky. The final solution we came up with was to block the log lines, so instead of working on individual rows when adding and removing content, we operate on blocks of N rows. After some testing, we figured out how many rows a block should have: 50.

Production ready

It took us about a week to complete the initial implementation, which has already seen a number of user experience improvements. At this point we know we’re on the right path. Over the next few weeks, we continued to refine the user interface and experience, and we knew we would have a lot of edge cases to deal with.

After a lot of internal work, we delivered it to our users and are happy to deliver a better logging experience: faster, smoother, better to use, and more complete and robust. Most of the time you don’t need to reinvent the wheel, but sometimes the best solution is to implement your own solution from scratch so that you have complete control over the experience and performance of your product.

If you find any mistakes in your translation or other areas that need to be improved, you are welcome to the Nuggets Translation Program to revise and PR your translation, and you can also get the corresponding reward points. The permanent link to this article at the beginning of this article is the MarkDown link to this article on GitHub.

The Nuggets Translation Project is a community that translates quality Internet technical articles from English sharing articles on nuggets. The content covers Android, iOS, front-end, back-end, blockchain, products, design, artificial intelligence and other fields. If you want to see more high-quality translation, please continue to pay attention to the Translation plan of Digging Gold, the official Weibo, Zhihu column.