2020 will be an extraordinary year. Agora and its developers have faced many unknowns and unexpected changes. Fortunately, RTC technology connects you and me, so that work and life will not stop. In the past year, we have reached our goals, passed a milestone and set foot on the next journey.

New technology landing

Ai-based bel Canto, sound and noise reduction

On the audio side, Agora this year launched the industry’s first AI-based real-time bel Canto solution. The program is suitable for language chat scenes including voice call, interactive live broadcast, language chat room, board game werewolf killing and other scenes, as well as interactive live broadcast, karaoke room, online KTV, FM radio and other singing scenes. At the same time, in the above scene, sound net also provides four real-time sound effects: sound changes, music sound, space shaping, electric sound. Developers can modify the interface parameters to adjust the sound effects of voice and music.

We previously shared the technical principles behind AI bel Canto in detail in our article “Using AI+ Big Data to Make The User’s voice more appealing”.

Since we were founded to do real-time audio and video, so we have been deeply engaged in audio algorithm design and implementation. In 2020, we developed the Residual Fully Convolutional Auto Encoder speech noise reduction model RFCAE (Residual Fully Convolutional Auto Encoder) based on deep learning for the pain points of traditional noise reduction that could not deal with the sudden noise well (such as the wheat blast, which is the typical sudden noise). Based on this model, a real-time noise reduction scheme was implemented this year. However, we have not had time to share the technology and algorithm of this part in detail, and we will have an opportunity to talk about it later.

Multi-algorithm implementation to optimize video hd quality and experience

At the video level, we always focus on video quality and interactive experience. Combined with artificial intelligence algorithm and advanced coding technology and transmission algorithm, we optimized sD-RTN ™ transmission strategy and Last Mile transmission strategy, especially for Southeast Asia, India, Indonesia, Egypt and other regions to further reduce the time of first frame drawing and channel cutting drawing. And effectively improve the rate of bitstream climbing, optimize the user experience.

At the same time, we combined THE AI algorithm to implement the Agora Perception Augment Suit (APAS), which was specially built for RTE scenes, including acquisition, pre-processing, coding, transmission, anti-weak mesh and rendering, and optimized the end-to-end processing of audio and video through a variety of self-developed algorithms.

Develop a new transport protocol

At the network Transport layer, THE Agora Universal UDP-based Transport Protocol (AUT) is developed by sonnet. The protocol meets the following technical requirements:

  • Versatility: the use of a set of protocols designed to meet the needs of different scenarios, not only RTC, but also reliable data channels;

  • Native Stream support in transport protocol: multiplex, flexible priority management, piggyback custom Stream Meta information to the Stream, for the user to manage the Stream decision;

  • Flexible congestion control module interface, can be extended to achieve different congestion control algorithm;

  • The underlying network interface can support SD-RTN™, UDP Socket and any virtual network, etc.

AUT protocol has been verified as the underlying transport technology in Agora RTC SDK, providing high-quality transport guarantee and flexible control mechanism for upper applications. Our lead architect shared the protocol live at RTE 2020, a replay of which can be viewed here.

Open source Demo and cross-platform framework support

Cross-platform, multi-language support

Cross-platform development has been one of the most important technology trends for many years. From React Native to now Google’s Flutter, frameworks have been evolving year after year, but their popularity has never waned. Adopting cross-platform development technology allows software developers to build applications for different platforms using almost the same set of code, saving time, manpower, and unnecessary work.

We may be aggressive in our cross-platform development support. We currently support multiple languages, cross-platform frameworks, and game development engines, including cocos2D-X, Cocos Creator, Electron, Flutter, Python, React Native, Rust, Unity, Unreal, and UWP. Some of them were originally developed by SJS community developers and maintained together with officials, such as React Native Version Agora SDK developed by @Shaoyan, UWP Version Agora SDK developed by Chen Shi, etc. Some are officially developed and maintained, such as the SDK for Flutter. Blue Is the blue

The SDKS and QuickStart demos that support different languages, frameworks, and game engines are uploaded to AgoraIO and Agoraio-Community, respectively. Friends who are familiar with different frameworks and languages can have a try and ask questions at any time.

In addition, a wave of SDKS supporting other languages or cross-platform frameworks, such as Vue and Go, are on the way to further speed up and simplify multiple scenarios for developers to interact in real time.

API Examples to facilitate the implementation of basic audio and video functions

With the increase in real-time interactive scenarios and the constant addition of features to the product, there is more and more sample code, making it difficult for developers to find. Therefore, we launched APIExamples project in AgoraIO warehouse, which contains all basic functions, single scenario sample code, such as one-to-one video call, one-to-one voice call, etc.

In plain English, the code is easier to find. Once new features are available, we will update them to APIExamples in a more timely manner. You can click 👁Watch to stay tuned!

Open source Demo, fast implementation of a variety of audio and video scenes

In addition, we opened source official Demo with relatively complete functions for several mature scenarios this year, including:

  • Agora Voice, this open source Demo is for developers who want to develop high quality Voice social applications.
  • Agora Live, the open source Demo, can directly achieve the mai Live, single anchor Live, PK Mai Live, virtual anchor four popular scenes.
  • E-education, the open source Demo, contains three scenarios of one-to-one interactive teaching, a pair of N online small classes and low-delay large classes.
  • Agora Meeting, the open source Demo, is a sample project designed for enterprise video conferencing and includes whiteboard functionality.

All of the above demos are available for download in the Apple App Store. You can find them by searching the developer “Agora IO, Inc” in the App Store. The App names are Agora Voice, Agora Live, Sonnet Cloud Classroom and Agora Meeting.

In addition to these demos, we have many other demos and sample code, which you can also access from the following three Github sites:

  • AgoraIO, which provides some basic code examples, as well as some officially maintained versions of the SDK for cross-platform, third-party frameworks.
  • Agoraio-community, with demos developed by Community developers, SDKS based on different languages or frameworks, etc.
  • Agoraio-usecase, which mainly contains some mature scenario demos.

Of course, if you have also silently developed a Demo or practice using Agora audio/video SDK, Agora RTM SDK or other SDKS, please leave a comment and let us know.

Detailed documentation and upgraded experience

It’s not enough just to provide a Demo, because many of the interface parameters need to be documented. This year we helped developers of all levels better use the SDK by adding zero-base running Demo documentation for more than 10 platforms/frameworks and new types of documentation such as scenarios, best practices, and terminology.

At the same time, sonnet document center was revised in November, which improved user experience by adding product home page, API area and platform selection while comprehensively optimizing UI.

Develop problems and solve them in time

Under the pandemic, a lot of our communication and interaction with developers has moved from offline to online.

I’m sure you all know the rTCdeveloper.com domain, which is our original community. A lot of the conversations between developers and soundnet engineers take place here. More than 1,700 questions from developers were answered here last year. Other problems that rose to work orders were resolved within an average of 15 hours.

In addition, we launched the Soundnet Developer Community this year. Here you can learn more about audio and video technology knowledge, development experience, and how to use the SDK, including our long-running Agora Talk, project review of this year’s RTE 2020 Innovation Challenge Spring and Fall, and essay contest.

Give me a little something

You’ve all had a very special year. Leave a comment in the RTC Developer Community about your changes and what you’ve learned in 2020, and what new technologies you’ve learned. What obstacles have you crossed? What new flags do you want to set for 2021?

We will give out 21 gifts ranked by the number of likes on your message

  • The first 5 to leave a message, can get the sound net custom motorcycle fly from the kettle
  • The first 6-10 people to leave a message will get a customized sonnet backpack
  • The first 11-21 to leave a message will receive a customized canvas bag

Click here to join the event

Write in the last

We also introduced a number of new products and made improvements to existing products this year, including:

  • XLA, the industry’s first experience quality standard, brings clarity to what was once a black box.
  • Extreme live, for low delay, strong synchronization of the “light interaction” live scene.
  • Low code HD products, under the same picture quality, reduce video bit rate by 50%, save the pressure of live bandwidth.
  • Launch of a new Web audio/video SDK that supports TypeScript with the Promise (async/await) asynchronous approach for efficient development.
  • The RTSA Code Stream Acceleration SDK now supports THE RTOS system, which means that cameras, microphones, and robots can take advantage of sonnet’s real-time interactive network for low-latency data transmission.
  • … There’s a lot more

In 2020, we will have many typical scenarios implemented, including video interview, smart Watch, smart homework lamp, Watch Party, space killing and cloud invigilation.

We do not describe these, if you are interested, you can see our wechat push.

2020 is full of complexity and uncertainty. In the face of an imperfect world, what we can do is to keep thinking, temper our strength and grow in this process. We hope that 2021 will move forward with you, respecting the value of developers, unleashing the power of technology, and pushing the boundaries of real-time interaction together.