In the spring of the RTC 2020 Programming Challenge held by Sonnet, the pixel-level, low-flow screen sharing tool “Syncit” won the first place in the second track. Yanzhen Yu is also the author of rrWeb, an open source project with 6500+Star. In CSDN’s previous interview with the winners, we learned that Yu Yanzhen, who is engaged in front-end development, uses Teamviewer and other remote desktop tools in his daily work. Most, however, require only remote control of each other’s browsers and do not require operating system-level control. Starting with a browser, he developed Syncit, a screen sharing and remote control tool that focuses on privacy and ease of use without consuming too much network bandwidth. Syncit passes snapshots and op-logs in text format. The ability to share and reverse control by observing interactions and changes in web views. Data transmission is based on WebRTC + Agora RTM SDK, which can reduce the delay to a lower level and ensure smooth use.
On June 20 (This Saturday), Yu Yanzhen will share in RTC Meetup the difficulties and practical experience encountered during the development of Syncit project in detail, including how to realize the new privacy protection ability of perceivable content, how to sandbox remote control ability, and the realization of low delay transmission layer, etc.
Before that, we can briefly understand the basic functions and implementation ideas of “Syncit” through this article.
The basic function
Remote screen sharing
We need pixel-level synchronization, low latency, and ultra-low traffic occupancy. The idea is to convert the changes and interactions of Web views into serialized snapshots and op-logs, which are transmitted from the local end to the peer end and then played back. The peer end is always in the sandbox to avoid security problems.
Privacy protection
In order to protect privacy, it is necessary to be able to block some content flexibly during screen sharing. Syncit blocks content, not location. Each part of the view is a structured DOM that not only precisely screens the view, but also blocks interaction events within the corresponding area.
Reverse remote control
We want to implement reverse remote control similar to Teamviewer for the Web, but without installing a client. Syncit collects interaction events from the peer side and plays them back at the source side, and control cannot escape from the current Web page for better security.
Which scenarios are suitable?
The remote service
You can access Syncit from a Web application, and when a user encounters a problem, a customer service person can provide remote assistance through a Web browser. This function can also be combined with functions such as remote customer service videos to achieve a better application experience.
Web sharing on weak networks
Due to the complex network environment, the user may be in a weak network environment, and the packet loss rate is high. So Syncit can be used for all kinds of terminals in all kinds of network environments, and can communicate and share information through Web browsers without degrading image quality. Similar scenarios include online education, telecommuting, etc.
Syncit is suitable for all Web scenarios, and as Web applications evolve, there will be more of them.
Syncit’s functional design
A live model
In simple terms, Syncit is based on serializing a Web view into a text-formatted snapshot, and then logging all the operations that might change the Web view into an op-log. Simply rebuild the snapshot and replay the operations in the OP-log again to achieve accurate playback of the Web view. Recording, serialization, playback, sandbox, etc. We’ll talk more about sandboxed implementation in a live broadcast on June 20. You can read a full description on Github at that time.
Further, Syncit also includes encoder, buffer, transporter and other components to achieve the effect of live broadcast, as shown in the diagram below:
Each component is pluggable, for example:
-
Encoder /decoder can be implemented to encrypt and compress the transmitted data. (No extra encoder enabled in the game version because deflate compression is included in the Agora Real-time Messaging SDK)
-
Buffer can be implemented to cache data in transit and retry and sort when network anomalies or timing disorders occur. (Buffer in the match version is an implementation that is non-persistent in memory)
-
You can implement a transporter that exposes a standard interface for data communication between the source and peer ends. (The competition version uses Agora real-time messaging SDK to encapsulate the Transporter, and includes a Transporter simulation based on localStorage for quick testing.)
Since the op-log portion of the transferred data is very sequence-sensitive, retry and sorting in buffer are important implementations to ensure screen sharing stability.
Buffer has a one-second (controllable) delay on the user’s side, which means that as long as the delay of transferred data is less than one second, Buffer can buffer it to a state with no delay for viewing.
Control mode
The control mode is realized on the basis of the live broadcast mode. In the live broadcast mode, we have built a sandbox view on the peer end by rebuilding the snapshot. When the control mode is enabled, we will start to monitor the interaction of the peer user in the sandbox, and then transmit it to the source end for implementation.
A fully synchronized DOM map is maintained on both the source and the peer side so that each interaction record can be transferred as a serialized data.
For example
Assume the source page is a button that alerts a message when clicked
-
When you recreate the snapshot on the peer side, the same button appears, but it is in the sandbox and clicking it does not trigger an alert.
-
Listen for interactive events in the peer sandbox. After the peer user clicks a button in the sandbox, click events and click objects are collected.
-
Click events and click objects are transferred to the source, and corresponding click events are triggered by the program.
-
Source alert INDICATES a message.
-
The latest view changes on the source end are synchronized to the peer end.
The schematic diagram is as follows:
Author, live and source
- Github @yuyz0112: github.com/Yuyz0112
The author will share details of Syncit’s implementation during Saturday’s live broadcast. At the same time, open Syncit source code. You are welcome to scan the code to register and exchange more technical details with the author in the live broadcast room. Interested students can scan the code to register.
