With the change of lifestyle in modern society, the way of social entertainment has gradually changed. Traditional face-to-face social entertainment activities are gradually changing, and more and more interaction behavior is gradually transferred to the network. The progress of RTC technology also promotes the change of online entertainment forms. Single-direction information transmission methods, such as movies, songs and videos, account for a declining proportion of entertainment methods, while more interactive methods, such as interactive live broadcasting, voice calls and online KTV songs, are gradually rising.

The necessity of audio processing

As one of the most important forms of human communication, sound processing is crucial. On the one hand, human beings are extremely sensitive to sound, and the transmission of sound is affected by the characteristics of human physiological structure. As vision is limited by illumination and orientation, it is not a source of information that can be relied on all the time. In many cases, hearing becomes the most important channel for human beings to perceive environmental information. On the other hand, the communication mode of sound separate from the picture also has the scene of independent application.

The RTC interactive communication function, as an extremely important function, has the following requirements for the processing of audio calls:

  • Ultra-low latency, real-time interaction, zero distance

  • Super high call quality. The factors that affect the listening sensation, such as echo and noise, must be properly handled to ensure no interference during the call

The characteristics of social entertainment put forward new requirements for audio processing. For example, users want to get high-quality music, good sense of presence, interesting audio effect, high-quality audio content sharing and so on. Therefore, this requires us to optimize the audio from different aspects to achieve the optimal effect. Today we are going to share audio.

The concept of audio sharing

Audio sharing refers to sharing audio sounds with other participants so that both parties can hear the same sound, for example, listening to a song together.

In some cases, it is important for users to feel more present when they hear the same voice on the phone. There is a direct way to let the opposite user hear the local voice from the microphone channel, ** but in many cases this effect is not very good. Distortion in the acquisition and playback process and the specific processing of the voice in the microphone channel can damage the quality of the audio.

Providing an audio sharing feature that bypasses front-end processing and is flexible for various scenarios becomes a real requirement.

Implementation scheme of netease Yunxin audio sharing

In order to meet users’ requirements for audio sharing in multiple scenarios, netease Yunxin implements flexible audio sharing schemes.

There are multiple shared sound sources available. You can use source files, as well as network audio sources.

Through the built-in decoder decoding after mixing, can be compatible with common Mp3, AAC and other formats of data files, this is the most simple and common way.

What happens when users like the sound of third-party software? We provide the capture and processing of playback data based on the system interface, so that users do not have to suffer from the inability to obtain data sources, so that the sources of audio sharing are more diversified.

The architecture here appears to be a bit different from the usual RTC architecture, with the addition of an echo cancellation module and a change in the source of the reference signal. This is the place where the architecture special, a echo cancellation module for the basic conversation below, due to the Shared voice to be heard and the other at the same time, the microphone to voice might also contain this part of the signal, the need to eliminate the part to include not only to the sound, but also the end play.

Here, the actual broadcast signal is used as the reference input to ensure that the local voice input is cleaner. An additional echo cancellation is used to cancel the voice of the opposite end. When we use third-party playback as a shared source, we get a signal that contains the entire content. In this way, the peer voice can be removed from the shared source, so that high-quality audio calls can still be maintained during the sharing process.

Application scenarios of audio sharing

The above audio sharing scheme is a unified architecture, which can be used for game opening, audio sharing, online KTV and other scenes. It covers multiple scenes of entertainment and office.

With this basic processing framework in place, functions can be implemented by flexibly setting up internal processes with appropriate external logic. Here are some examples:

Replace the above third-party audio content with a game, music player, or browser, and you can easily turn your game on, listen to music together, or share audio with others.

If you think this example is a little simple, then the following is an online KTV chorus implementation example.

On the left is the lead end, which provides accompaniment music that is transmitted to the side singers via RTC audio after local vocals are added.

The voice of the singer on the right will be transmitted to the lead singer through RTC for the synchronization of the two singers’ chorus. At the same time, the voice of the side singer will be mixed with the song passed by the lead singer to form a complete chorus, which will be pushed to the live audience.

The above is the realization of an online KTV scenario. Of course, the realization of the online KTV scene involves many aspects, and the problems are far beyond the audio sharing part. Lyrics delivery, synchronization, and end-to-end audio latency are all obstacles that need to be overcome to provide a better experience

conclusion

SDK products of netease Yunxin provide a complete audio sharing solution, supporting two-channel full channel, which can cover a series of scenes including game opening black, listening to songs together, online KTV and so on. If you are interested, you can log in to the official website of netease yunxin to download Demo for experience.