As an indie developer or team using WebRTC, how do you know your App’s call quality is “up to scratch”? How to conduct reasonable weak network simulation test? Introduce developers to the deployment, usage, advantages and disadvantages of the three open source tools.

Welcome to the RTC developer community to exchange experience with more real-time audio and video developers.

If you are a veteran developer or tech enthusiast who has been following WebRTC for a long time, you may have noticed that some well-known WebRTC tech bloggers have been making their voices heard.

Here’s the thing.

Recently, Jitsi published a video call comparison test of WebRTC versus the Zoom Web client on their official blog [1]. Tests showed that WebRTC’s video call quality was even better than Zoom’s. A stone stirred a thousand waves, many bloggers expressed their views.

It may seem like a provocation, but Jitsi has a very good reason for doing so.

Jitsi is an open source project that enables developers to implement real-time voice and video calling applications on the Web, Windows, Linux, Mac OS X, and Android platforms. There are plenty of indie developers working on their own video calling apps based on this code. All these are achieved on the basis of WebRTC. However, Jitsi saw video conferencing service Provider Zoom repeatedly stating in some places since 2015 that not only did it not use WebRTC, but that “WebRTC is a very limited solution” :

How does Jitsi test WebRTC weak network transport?

They tested it twice, using WebRTC and Zoom for one-on-one video calls on the same Wi-Fi environment and the same Mac. During the first 10 seconds of the two calls, only normal calls were made. After 10 seconds, network loss was increased and uplink and downlink bandwidth was limited to 500kbps. It then measures how long it takes each scheme to adjust to make the ongoing video call stable to the current network bandwidth changes. As shown in the figure below, blogger Tsahi Levent-Levi in his blog [2] uses a comparison diagram to depict the change in bit rate during the test process.



The result was that WebRTC’s video call took 20 seconds to fully adjust to the proper bit rate after the bandwidth was artificially limited, while Zoom took 156 seconds.

Rather than comparing the results, we are more concerned with how useful this approach is to WebRTC developers. Can WebRTC developers use this method to accurately test the gap between their apps and others?

The answer is no, and it’s not a rigorous approach.

Based on the experience of acoustic network, it is not a common quality testing method to limit the same bandwidth threshold for both upstream and downstream. They usually limit the upside in one direction or limit the downside. But the test itself is fair. The Jitsi does not debug for this particular scenario and gives this comparison result. It is possible that Zoom has a weakness in this scenario.

From the perspective of communication architecture, Zoom adopts the MCU/SFU server access communication mode, and adopts the piecewise bandwidth adaptive strategy. The Jitsi’s 1-to-1 communication is believed to follow WebRTC’s end-to-end feedback. So, they are different. Full-link feedback has certain advantages in this scenario, because the bottleneck on the link can be quickly reflected to the sender for rapid self-adaptation. However, segmental strategy, which requires estimation of upstream and downstream bandwidths respectively, depends on the delivery decision mechanism of the server, and policy configuration is a difficulty of QoE.

Tsahi Levent-Levi also said in a blog post that the way human tools intervene in network transport is not enough to fully reproduce the real user scenario. But we can use tools to get as close to the user’s real world as possible.

Real user scenarios and weak network environments

What is a real user scenario? A person who accesses the Internet via Wi-Fi at home at night can play movies online smoothly, but once he or she makes a video call in the evening during peak Internet hours, the picture will be blurred, which may not only limit bandwidth, but also have a high packet loss rate.

Compared with wired network communication, wireless network communication is more affected by the environment, such as high-rise buildings, user movement, environmental noise, closed environment, etc. The network service quality is relatively unstable, resulting in users often communicate in weak network environment. Video calls in the garage, for example, are often not as good as those outside.

In addition to the environment, network coverage, overload control, and neighboring area leakage may also cause call failure and service quality deterioration. These are real user scenarios.

All the Jitsi does is test to simulate a weak network environment. Generally, this test is simulated by modifying bandwidth, packet loss and jitter parameters. In terms of data, weak networks are defined differently for different applications. Therefore, the lowest rate and service scenarios of different network types should be considered comprehensively. Take mobile scenarios as an example. Generally 2G, 3G with low speed, and Wi-Fi with weak signal are considered weak networks and need to be included in weak network test scenarios.

Weak mesh simulation tests correct posture

In fact, this incident also revealed a very common problem, many independent developers new to WebRTC may not know how to simulate weak network scenarios. Here we share some of our experiences with The Agora Audio and Video Lab, and recommend three weaknet environment simulation testing tools that WebRTC developers can use.

The following is a detailed description of how to build each tool, how to use it, and the advantages and disadvantages of the three tools:

Linux Traffic Control(TC)

The Linux kernel has a built-in Traffic Control framework, which can implement Traffic limiting, Traffic shaping, and policy application. It can inject delayed faults, packet loss faults, packet repetition faults, and out-of-order faults, as well as simulate intermittent network disconnection. TC also has some requirements on hardware and system:

Hardware requirements

  • PC – CPU i3, 4 GB memory, and 64 GB hard disk are recommended

  • Dual NIC – One PCI-E nic (such as Intel 82574L) is required in addition to the original ONBOARD NIC.

  • Router – Supports bridge mode

  • Network cable – several

    System requirements

    • Fedora, OpenSuse, Gentoo, Debian, Mandriva, or Ubuntu are required. If the Linux kernel is older than 2.6, TCS are built in.

    • System module

    • Iproute2 is required for Ubuntu/Debian systems

    • Iproute-tc is required for Fedora/RHEL

    • iptables

    • Linux kernel module : sch_netem

    In addition, the DHCP server needs to be installed. For details about how to install Ubuntu, see the official documentation [3].

    Start the deployment

    • Nic-0 connects to the Internet through a network cable. Assume that the NIC corresponds to Net Device eth0

    • Nic-1 connects to the WAN port of the router through a network cable. Assume that NIC 1 corresponds to Net Device eth1

    • Router: Enable the bridge mode and disable the DHCP service

      Enter the command line on the PC:

      1vi /etc/default/isc-dhcp-serverCopy the code

      Add:

      1INTERFACESv4="eth1"Copy the code

      Restart the service:

      1sudo /etc/init.d/isc-dhcp-server restartCopy the code

      After the restart, run the following command:

      1echo "1" > /proc/sys/net/ipv4/ip_forward2iptables -F3iptables -P INPUT ACCEPT4iptables -P FORWARD ACCEPT5iptables -t nat -A POSTROUTING -o eno1 -j MASQUERADE6modprobe ifb7ip link set ifb0 upCopy the code

      At this point, you have completed deployment.

      How to use TC

      Weak network test is basically done according to the following four steps:

      1. The device successfully connects to the Wi-Fi hotspot and obtains the IP address, for example, 192.168.3.101.

      2. Start Linux Terminal and run the TC command to add network damage for the device whose sender IP address is 192.168.3.101.

      3. At this point the mobile phone is running in the weak network environment.

      4. After the test is complete, run the TC command to cancel the weak network.

      For example, if you want to limit uplink packet loss by 5% on a device whose IP address is 192.168.3.101, run the following command:

      1sudo tc qdisc add dev ifb0 root handle 1: prio bands 32sudo tc qdisc add dev eth1 ingress3sudo tc filter add dev eth1 parent ffff: protocol ip u32 match u32 0 0 flowid 1:1 action mirred egress redirect dev ifb04sudo tc qdisc add dev ifb0 parent 1:3 handle 30: netem loss 5 limit 10005sudo tc filter add dev ifb0 protocol ip parent 1:0 prio 3 u32 match ip src 192.168.3.101 flowid 1:3Copy the code

      To limit downstream packet loss by 20% for the device whose IP address is 192.168.3.101, run the following command:

      1sudo tc qdisc add dev eth1 root handle 1: prio bands 32sudo tc qdisc add dev eth1 parent 1:3 handle 30: netem loss 20 limit 10003sudo tc filter add dev eth1 protocol ip parent 1:0 prio 3 u32 match ip dst 192.168.3.101 flowid 1:3Copy the code


      It can be said that the TC framework can implement many scenarios, but only if developers learn to use the TC command line. If you want to learn more about TC commands, you can study the official documentation [4].

      Augmented Traffic Control(ATC)

      ATC is actually a set of web testing tools that Facebook opened source in 2015. ATC is tC-based encapsulation.

      After deploying the ATC weak network controller, you can switch between different network environments at any time on the mobile phone through the Web interface. Multiple mobile phones can be connected to the same Wi-Fi, multiplexing the same weak network controller, and the simulated network environment between multiple devices do not affect each other. This means that once the testing tool is deployed, anyone on the team can test on their own through the Web without interfering with each other.

      The deployment method of ATC is relatively complex, but as long as according to the official document [5], it can be successfully built. After completing the setup, you need to configure the HOST address with the following commands, and then you are ready to run.

      Open the Setting:

      1vi atcui/atcui/settingsCopy the code

      Add HOST address:

      1ALLOWED_HOSTS = [The '*']Copy the code

      Start command:

      1atcd --atcd-wan eth0 --atcd-lan eth1Copy the code

      Method of use

      1. The device is connected to the corresponding Wi-Fi

      2. Open http://192.168.3.1:8000 (assuming eth1 IP address: 192.168.3.1)

      3. After the corresponding weak network parameters are input, press [Update Shaping] to take effect. The weak network takes effect only on the machine

      When the test is complete, click the [Turn Off] button to clear the weak network Settings.


      Network Link Conditioner(NLC)

      Maybe some iOS developers have already recognized it. NLC is an official network simulation tool provided by Apple. It can be installed on macOS and iOS.

      MacOS endThe installation

      • Open Xcode and select Xcode -> Open Developer Tool -> More Develop Tools.

      • Use your Apple account to log in to the Site, search for Additional Tools for Xcode, and download Additional Tools for the version of Xcode.

      • Open the downloaded file and double-click Network Link to install in the Hardware folder. Once installed, the tool will appear in the last row of system Settings.


      The iOS sideThe installation

      Network Link is enabled by enabling Developer options.

      • Data cable to connect the phone to Mac, Xcode -> Windows -> Devices -> select the current phone device, right click popup

      • Menu -> Select Show Provisioning Profiles… A certificate list window pops up

      If the phone already has the necessary developer credentials, just click the Done button in the window. Otherwise, click the + sign in the lower left corner to import the certificate downloaded from the Internet and click the Done button to close the window.

      After entering the developer option, Network Link Conditioner option is displayed.

      Method of use

      NLC is much simpler to use and does not require the command line. If the configuration in NLC does not meet your requirements, you can manually add more configurations. On a Mac or iOS, do as follows.

      Mac

      The iOS side



      Note that interface is set to Cellular when iOS serves as weak network control of access devices by sharing Wi-Fi hotspots.

      Contrast and summary

      Relatively speaking, TC parameters are the most abundant, can control more details, can simulate a variety of different network conditions, but the operation is too complex, developers need to be familiar with TC commands and network model. NLC is the most simple and easy to operate, and parameter configuration can meet common development requirements.

      WebRTC 1.0 focuses on giving developers more control over media and data channels. According to the previous proposal [6], the next version of WebRTC will make it possible to remove data processing from the main thread. Using RTCDataChannels for data transfer provides better congestion control than using WebSocket.

      According to the analysis of WebRTCHacks blogger Philipp Hancke [7], Zoom’s Web Client does not use WebRTC. The Client uses WebSocket for media transmission, which is similar to Turn/TCP in WebRTC. Although it is beneficial to traverse the firewall, if packets are lost in real-time communication, they will be retransmitted, resulting in accumulated delay. From that perspective alone, the next version of WebRTC’s plan is better than Zoom’s.

      As we mentioned above, the development of policy configuration for WebRTC servers is the difficulty of QoE. As a result, poor quality of multiplayer communication is one of the most common complaints about native WebRTC applications. In fact, The Agora Web SDK of Sonnet is also developed based on WebRTC and has been optimized in many aspects based on the native WebRTC. Agora Web SDK has always focused on improving the communication quality, optimized to the current version, has been able to support 17 people video calls. We optimized WebRTC gateway in many aspects, such as transmission quality assurance, native WebRTC QoS tuning, and different optimization strategies for different scenarios.

      We provide global services covering video conferencing, online healthcare, online education, social live streaming, social game audio and video, finance, IoT and other real-time audio and video communication scenarios. At present, Agora Web SDK is the largest weBRTc-based real-time communication SDK in the world’s commercial services. In many cases WebRTC will not be considered as a large channel solution. The Agora Web SDK now supports millions of concurrent large channel calls.

      More WebRTC developers to exchange development issues and experience, welcome to visit
      RTC developer communityWebRTC section