Welcome to yunjia community, access to more Tencent mass technology practice dry goods oh ~
Author: Tencent Game Cloud
Introduction: In the first Tencent User Open Day, Tencent Audio and Video Lab presented its 3D location sound solution to all users, providing users with immersive audio experience in 360 degrees three-dimensional space. Then how does this technology improve users’ listening sense and experience in combination with specific scenes? This article will elaborate.
Based on game development engine (such as Unreal, Unity) implementation of 3 d sound effects in the application of the game is very popular, through the game engine simulation reproduce sound source location in space, such as shooting guns, shot in CS sound, nearby enemies when light footsteps, can significantly improve the immersive of the game, create immersive gaming experience.
Here we can listen to the audio comparison before and after the 3D sound algorithm is processed in the lab (please listen with headphones) :
Before processing: m.v.qq.com/play/play.h…
After treatment: m.v.qq.com/play/play.h…
It can be heard that the processed audio carries more spatial and positional information. By expanding the extension of the music, the sound field is bigger, the sound position is more accurate, the sound movement track is more vivid, and the sense of presence and immersion of the music is improved.
However, 3D sound has not been applied in the field of real-time voice communication on a large scale before, which is mainly due to the fact that in previous heavy games, the positioning of voice ability in gameplay is more a way of communication than innovative gameplay, and the use scene does not have strong appeal to real-time voice carrying location.
In 2017, a game, PUBG: Battleground, won the favor and popularity of global players in a short period of time through the innovative play of 100 people fleeing to kill and the winner eating chicken. In addition to the creativity of the gameplay itself, the game has also taken a radical approach to real-time voice gameplay. During the game, players can set [All Mode] to shout, challenge and chat up nearby players. This voice mode can carry out interesting communication and interaction with strangers. There are already plenty of funny videos on social media based on the voice mode.
With the introduction of 3D positional voice, players will reveal their location and location during the shout, and their voice will change in real time based on their position. Arguably, 3D sound makes the player-to-player communication and combat experience in Battle Royale feel more real, more immersive, and more immersive.
In addition to the use of 3D audio in Chicken Eater games, it can also be combined with multiplayer audio scenes such as online card rooms to significantly improve the user experience.
Multiple voices are now mixed into mono in a multiplayer conversation, with the source concentrated in a single spot. With the application of 3D location sound, the listener can virtualize the different sender to different positions in the space, making it easier for the user to know who is speaking through location information. This technique is ideal for fixed location chat rooms.
For example, when 3D sound is applied to the voice room of board and card games, players can clearly perceive whether other players’ voices are coming from their left, right or front by virtual each voice to the corresponding position on the table, creating an immersive experience of online chess and card rooms and making players more willing to open voice communication.
The following is the real-time voice effect in a board and card game that simulates player orientation processing:
M.v.qq.com/play/play.h…
Another important application of 3D sound is VR audio.
VR is all about presence, and realistic 3D sound effects can be the finishing touch. While it is important for VR devices to track objects through head movements, it is equally important to be able to hear the sound that carries the location information of objects. Realistic 3D sound effects can convey the location information of other players to the player, and the environment information, and achieve the best effect through the mutual reinforcement of visual and auditory effects.
From the orientation, each sound in VR can be virtual to a certain position in the space, so that users can feel that the sound comes from a specific position. When the user turns his head, the sound of the virtual orientation just now in VR will change the corresponding orientation, creating a 360-degree stereo sound positioning effect, making the player more convinced that the sound source is in reality. This is critical for VR gaming experiences. At present, the 3D sound related technical proposal of av Lab has been submitted to AVS (National Audio and Video Codec Technical Standard) VR Audio National standard, and it has been launched in a VR game “Hello World” released by Tencent this year.
3D audio can also be used for audio processing of HD video, allowing users to experience the immersive stereo experience of cinema at home with a two-channel headset.
We know that in order to create stereo effect in the process of cinema viewing 5.1 channels, 7.1 channels, the sound of each channel through the speakers in different positions to play out, with the film, the audience experience is more shocking. When wearing headphones to watch the movie, as a result of the conventional method is simple to mix all depend on the left side of the channel in the left channel, on the right side of the channel on the right track, track of left and right sides are mixed, resulting in either 5.1 or 7.1 all audio and video eventually become left right blurs many details lead to some voice changes embody unreal, For example, moving from behind left to left to front left. More serious is, if other sound channel is silent, only before left sound channel has sound, became only left earphone ring, this kind of feeling is very bad.
3D audio can be used to virtual each channel to the location of each speaker, making the sound more like that of each speaker in space. Using 3D sound technology, users can experience immersive cinema stereo effects with dual-channel headphones at home.
The following is the original sound of the music captured on Tencent Video and processed by 3D sound effects:
M.v.qq.com/play/play.h…
So how does three-dimensional virtual hearing work? Processing normal mono sounds into immersive audio requires understanding the following question: How does the human ear tell where the sound is coming from? Let’s take a look at the picture below:
As can be seen from the figure, the sound sequence heard by the left and right ears, the size of the sound heard by the left and right ears, and the actual frequency spectrum of the sound heard by the left and right ears are different.
Our 3D sound technology is to use “head associated transmission function” (HRTF algorithm) to model the time difference and spectrum difference between the ears, so as to form a sound with a sense of source location. At the same time, we also sampled reverb to fill in the ambient reflection sound missing from HRIR, so as to build a complete sound field model. In a small room, for example, you can not only hear the conversation between two people, but also hear the effect of the sound reflected off the wall, adding to the sense of immersion. In the next article on 3D sound technology, we will reveal more about how this technology works, so stay tuned!
3D positional voice solution is applied to real-time voice communication in the game. Players can “listen to voice and identify position” according to the voice of their teammates, and have an “immersive” game experience. At present, 3D sound technology has been integrated into TMG (Tencent Mobile Gaming) solution for real-time game audio and video launched by Tencent Audio and Video Lab. TMG provides real-time multiplayer voice, real-time video, voice message, voice to text and other functions, covering a variety of game types such as casual social, MOBA, MMORPG and so on. It is a real-time game audio and video SDK specially customized for game scenes, and the access threshold is very low. A single SDK can meet diverse game audio and video appeals.
Real-time audio and video game Mobile Gaming (Tencent) service has officially on Tencent cloud, game makers developers to provide service to our customers, detailed information can browse cloud.tencent.com/product/tmg…
Want to know more about Tencent cloud GAME industry solutions and cases, immediately sign up for Tencent cloud game-tech Salon hangzhou station on January 19, we discuss together
Live to make an appointment: www.itdks.com/eventlist/d…
reading
“Tencent cloud game developer technology salon” On January 19 game acceleration theme waiting for you to come about! Tencent Game Cloud Dolphin launched the game update easy mode one minute configuration of the “black technology”, Tencent cloud exclusive open cloud pressure measurement ability
This article has been authorized by the author yunjia community published, reproduced please indicate the source of the original