Abstract: Help developers quickly get AI visual features in audio and video scenes — beauty, filters, background matting, etc.

Native SDK development team

Generation Z, as a new consumer force in the society, pursues individuality, loves novelty, favors cool and novel ways of playing, and follows the entertainment trend closely. The combination of AI+ audio and video, as a supporting point for gameplay innovation in the third scene besides audio and video, just caters to the trend property.

In the current market, it is difficult to find stable and reliable suppliers with AI vision and audio and video capabilities. Due to the respective technical threshold of AI vision and audio and video capabilities, many enterprises have to connect with more suppliers, which also delays the arrival of integrated services that are more suitable for user needs.

Slow access?

High maintenance costs?

Unresponsive?

ZEGO Architecture Technology launched its AI visual product “ZEGO Effects” in June this year, providing a convenient and efficient one-stop service with stable and reliable AI+ audio and video capabilities to solve industry pain points. To meet the needs of users for AI visual functions such as beauty, filters and AI effects in audio and video scenes.

In this article, we will show you how to use Zego’s Express SDK and Effects SDK products to quickly build audio and video + AI scenes on mobile devices.

First, concept introduction

Before we introduce the audio and video +AI scene construction, we will briefly introduce the basic concepts of Express SDK and Effects SDK.

1, Express the SDK

Express SDK refers to ZEGO real-time audio and video SDK, which can provide developers with convenient access, hd fluency, multi-platform communication, low latency, high concurrency audio and video services, and can realize one-to-many, many-to-many real-time audio and video interaction, live shows, video conferences and other scenes.

2, the Effects to the SDK

Effects SDK refers to ZEGO AI visual SDK, which provides multiple intelligent image rendering and algorithm capabilities, including intelligent beauty, AR Effects, image segmentation, etc., and can be widely used in entertainment live broadcast, online education, photo tools and other scenes.

Fully meet all kinds of gameplay in the pan-entertainment scene, so that enterprises can quickly acquire AI+ audio and video innovation ability:

• Facial beauty — intelligent facial beauty, advanced facial beauty based on accurate key point monitoring and 3D model, natural beauty makeup;
• AR Effects — ZEGO Effects offers a wide range of AR Effects, including SUPPORT for 2D/3D stickers, face stickers, and strong robustness to complex backgrounds, lighting changes, and exaggerated gestures.
• Intelligent segmentation — Through accurate segmentation to change the background of the picture, users can realize the free change of video and live scene, supplemented by a variety of background materials, can achieve more interesting gameplay;
• Variety of filters – provide a variety of exquisite filters, including fresh, Japanese, soft, forest, light effect and other different themes, you can easily change the image style.

AI vision in audio and video code implementation logic

Next, we will take the iOS and Android terminals as the starting point to take you to understand the entire implementation process of Express SDK + Effects SDK.

Initialize the Effects SDK

Initialize the Effects SDK to create objects and set up resources.

The objective-C code is implemented as follows:

. // Set the resource [ZegoEffects setModels:@[faceDetectionModelPath, segmentationModelPath]]; [ZegoEffects setResources: @ [pendantBundlePath, whitenBundlePath]]. ZegoEffects *effects = [ZegoEffects create:@license]; Self. effects = effects; // Initialize effects object [self.effects initEnv:CGSizeMake(width, height)];Copy the code

Java code implementation is as follows:

. // Set the resource zegoEffects.setModels (aiModeInfos); ZegoEffects.setResources(aiResources); ZegoEffects Effects = ZegoEffects.create(license, getApplication()); // Initialize effects.initEnv(width, height);Copy the code

2. Initialize Express SDK

Initialize the Express SDK to set the video frame data format, enable the custom pre-processing function, and register callback objects. It is important to note that video frame formats vary from platform to platform.

The objective-C code is implemented as follows:

/ / create express instances [ZegoExpressEngine createEngineWithAppID: appID appSign: appSign isTestEnv: YES scenario:ZegoScenarioGeneral eventHandler:self]; / / select CVPixelBuffer type video frame data ZegoCustomVideoProcessConfig * processConfig = [[ZegoCustomVideoProcessConfig alloc] init]; processConfig.bufferType = ZegoVideoBufferTypeCVPixelBuffer; / / open the custom before handling the [[ZegoExpressEngine sharedEngine] enableCustomVideoProcessing: YES config: processConfig channel:ZegoPublishChannelMain]; / / set the video pretreatment callback object [[ZegoExpressEngine sharedEngine] setCustomVideoProcessHandler: self].Copy the code

Java code implementation is as follows:

/ / create instances engine = express ZegoExpressEngine. CreateEngine (appID appSign, true, ZegoScenario. GENERAL, getApplication (), null); / / select GL_TEXTURE_2D type video frame data ZegoCustomVideoProcessConfig config = new ZegoCustomVideoProcessConfig (); config.bufferType = ZegoVideoBufferType.GL_TEXTURE_2D; / / open the custom before processing engine. EnableCustomVideoProcessing (true, config, ZegoPublishChannel. MAIN); / / set before the video processing engine. The callback object setCustomVideoProcessHandler (myHandler);Copy the code

3. Express SDK + Effects SDK implementation logic

The Express SDK provides custom video pre-processing that works well with the Effects SDK. The custom video preprocessing function does not require the developer to manage the device input source. Instead, the developer can manipulate the raw data in the Express SDK’s collection callback function and send the data back to the Express SDK. The Effects SDK takes care of the raw data in this process.

The objective-C code is implemented as follows:

- (void)onCapturedUnprocessedCVPixelBuffer:(CVPixelBufferRef)buffer timestamp:(CMTime)timestamp channel:(ZegoPublishChannel)channel { ... / / to send data to Express the SDK acquisition Effects the SDK for processing [self. The Effects processImageBuffer: buffer]; / / throw Effects SDK after processing the data back to Express the SDK [[ZegoExpressEngine sharedEngine] sendCustomVideoProcessedCVPixelBuffer: the output timestamp:timestamp channel:channel]; . }Copy the code

Java code implementation is as follows:

public void onCapturedUnprocessedTextureData(int textureID, int width, int height, long referenceTimeMillisecond, ZegoPublishChannel channel) { ZegoEffectsVideoFrameParam param = new ZegoEffectsVideoFrameParam(); param.format = ZegoEffectsVideoFrameFormat.BGRA32; param.width = width; param.height = height; Int processedTextureID = Effects. ProcessTexture (textureID, param); / / throw Effects SDK after processing the data back to Express the SDK Express. SendCustomVideoProcessedTextureData (processedTextureID, width, height, referenceTimeMillisecond); }Copy the code

4. AI visual effect adjustment

Effects_SDK also provides functions such as beauty, beauty, background segmentation, face detection, widgets, filters, etc. Effects_SDK is widely used in entertainment live, online education, photo tools and many other scenarios. Developers can call the corresponding interface according to their needs.

The objective-C code is implemented as follows:

[self.effects enableWhiten:YES]; ZegoEffectsWhitenParam *param = [[ZegoEffectsWhitenParam alloc] init]; ZegoEffectsWhitenParam alloc = [ZegoEffectsWhitenParam alloc] init] param.intensity = 100; [self.effects setWhitenParam:param];Copy the code

Java code implementation is as follows:

Effects. enableWhiten(true); ZegoEffectsWhitenParam param = new ZegoEffectsWhitenParam(); ZegoEffectsWhitenParam(); param.intensity = 100; effects.setWhitenParam(param);Copy the code

conclusion

So that’s how to build audio and video +Al on mobile by using Express SDK and Effects_SDK.

As an AI visual product, ZEGO Effects provides multiple intelligent image rendering and algorithm capabilities, including intelligent beauty, portrait detection, image segmentation, etc. It is widely used in entertainment live broadcast, online education, photo tools and other scenes.

This is an organic combination of AI capabilities and audio and video, and we look forward to more innovative applications of audio and video and AI in the future.