Real-time audio and video data processing, transmission process can be briefly summarized as: collection, coding, pre-processing, transmission, decoding, post-processing, rendering, the process as shown in the following figure, we have more than once explained this process, I believe we have long been familiar with the heart. The Agora VIDEO/audio SDK can help developers solve the development problems in this process, and quickly realize a variety of real-time audio/video interactive scenes.
There are also many teams in the industry, who have the ability to develop their own encoding and decoding, or use special encoding format, custom encryption encoding, non-mainstream encoding format, such as:
- Developing IoT products
There are many intelligent hardware products related to RTC, the most typical of which are smart glasses, smart cameras and so on. There are generally two codec methods. One is to use the API provided by chip manufacturers for codec, which has the advantage of high speed, but is bound to the hardware platform and lacks flexibility. The other is to use X264, X265 and other software, the advantage is high flexibility of customization, but the speed is slow. After solving the codec problem, the development team will also need to deploy their own servers to enable audio and video communication between devices.
- Developed based on WebRTC
WebRTC for many developers to provide a good entry ladder, WebRTC has its own codec, noise reduction, echo cancellation module, to solve the developers need to develop problems on the client side. Of course, some teams have their own optimized codec algorithms, and they can make more optimization in the codec process. Many development teams build their own apps based on WebRTC, such as online education and social streaming. However, it is a lot of work to extract the good part of WebRTC for secondary development. In any case, it is necessary to set up signaling server and media server, deploy nodes reasonably and optimize transmission strategy.
The above are just two typical situations. However, the most troublesome problem for such a development team is having to step through the “server build” pit:
- Complex network response
Different from common applications, applications with real-time audio and video calls or real-time message commands have higher requirements on the server, including packet loss, jitter, and delay. Taking packet loss as an example, packet loss resistance is indispensable during real-time data transmission on the public network. Packet loss, that is, packets that do not arrive on time, usually occurs in two transfer phases: Server to Device and Server to Server.
First, Server to Device has two pathways:
A. Server-base-base-device. To consider different types of base stations (3G/4G/WDCDMA/TD), similar base stations in different locations (such as near the concert), similar base stations in different locations at different times (such as network peak), the transmission of base stations in different countries under several circumstances.
B. Server-route-device. Routes are divided into two bands: 2.4g and 5G. Although 2.4g covers a wide area, it is prone to interference and packet loss; Compared with 5G band, there is less interference and packet loss, but the coverage area is small. When multiple people connect to the same route, bandwidth competition is severe and packet loss is sometimes high. Moreover, poor quality routes are prone to network jitter or other bugs. These also need to be taken into account by the server team.
Server to Server phase: backbone network and edge network. In the backbone network, traffic may also encounter network congestion. You need to pay attention to the transmission quality between the same carrier in the same country, because packet loss can occur between the same carrier; You also need to pay attention to the transmission between different operators in the same country, because in many cases, the settlement between operators and limited bandwidth can make the network unstable. If your business is trans-regional and transnational, such as application going to sea or online education, then the transmission problem in different countries will be more responsible. You should choose good computer rooms in different countries and monitor them in real time. Network quality varies greatly in different countries, such as Southeast Asia and some Middle East countries. This stage of transmission optimization is a special technical field, if self-research needs to invest a lot of manpower, energy and development costs.
- operations
To build a reliable toC or toB service, 24×7 real-time monitoring and operation is essential. Not only do they need to be able to monitor and fix data transmission problems in a timely manner, but they also need effective means to make quality problems transparent to their r&d teams and users. In addition, self-developed developers need to prepare disaster recovery and backup policies, links, and fault recovery mechanisms.
- Team building and development costs
Team building is definitely the foundation. In addition to the team that develops its own codec algorithm, a large front-end development team is also needed to implement multi-platform clients based on Web, Native or cross-platform solutions. More importantly, a server team is required to solve complex network architecture issues.
In addition, from the choice of where to deploy the server, to the actual development, real-time deployment, real-time monitoring, if you want to step over these “pits”, need to pay research and development, time costs.
Sound net officially launched real-time code stream acceleration SDK. Self-developed codec developers can access sD-RTN ™ real-time network through the API of real-time code stream SDK after the processing of audio and video data on the client side. With the help of SDK, all audio and video data can be transmitted with low delay and high quality. For WebRTC developers, you can develop audio and video functions on the end based on WebRTC, and then transfer audio and video data through the INTERFACE of THE RTSA SDK. Developers do not need to build their own servers. For more details: www.agora.io/cn/rtsa/