The annual Double Eleven is coming. Yijie sold 330 million yuan in sales alone on Last year’s Singles’ Day, creating a sales myth in the industry. In the past two years, many e-commerce platforms have begun to pay attention to live streaming interactive e-commerce, hoping to increase interaction in the live streaming. For example, in the process of live streaming, limited amount of preferential commodities are offered, and real-time news of buying is sent to the audience. So we did a simple Demo.

The general idea of Demo is as follows: Based on the interactive model of live video, it is designed with the function of voice transfer, so as to get rid of the tedious operation of Windows terminal and realize the function of quick question Posting. The host enters the question (question and answer, only yes or no) by voice, and sends the text of the question to all the audience in the room after confirming it. After receiving the question, the App automatically pops up the box and gives the audience the selection result.

1.1 Functional Disassembly:

  • Only anchors have the entry for Posting topics.
  • Automatic Speech Recognition (ASR) is required, and there are two solutions: online real-time translation and local offline translation.
  • The ASR result must be confirmed by the anchor.
  • After confirming the ASR result, the ASR anchor must notify all non-anchor users.
  • When the non-anchor users receive the topic information, they need to take the initiative to pop up the window and give the user the selection result.

1.2 Scheme determination:

  • In order to ensure the accuracy of ASR, online real-time translation is selected, and sogou Zhiyin open platform is finally selected through comparison.
  • The topic information is also a text type. You can use the real-time message channel of group chat to add special characters before the topic information. When non-anchor users receive the message, they can determine whether it starts with special characters. Special characters can be defined with extensibility in mind, and other similar functions can be implemented through this scheme in the future.

2.1 Live video DEMO

A simple live video Demo can be achieved according to the following steps, you can find several Android devices run to see the effect, or quite easy drops.

Step1 SDK integration

The SDK supports maven dependencies. Add a line to the Dependencies module in build.gradle:

dependencies {
...
implementation 'the IO. Agora. RTC: full - SDK: 2.8.1'
}
Copy the code

Step2 Create a live streaming engine

Sound network SDK has an important class RtcEngine, responsible for live function management, provides on/off, status monitoring, sound/video Settings and other rich Api, encountered problems, the first check this class on the right. When creating an engine, parameter APP_ID is the application ID created by the acoustic network development platform.

private RtcEngine mRtcEngine;
try {
    mRtcEngine = RtcEngine.create(context, LiveDefine.APP_ID,
mRtcEventHandler);
    mRtcEngine.setChannelProfile(Constants.CHANNEL_PROFILE_COMMUNICATION);
    mRtcEngine.enableAudio(); // Enable the audio function
    mRtcEngine.enableVideo(); // Enable the video function
} catch (Exception e) {
    e.printStackTrace();
}
Copy the code

Step3 live broadcast View association

The roles are divided into anchor and viewer, and there are some differences when associating views. ANCHOR_UID indicates the id of the host user. The anchorman associates a View with his own user ID, and the audience associates a View with the user ID of the watching anchorman. The user system needs to be managed by the application itself. The SDK of Sonnet does not provide user management.

SurfaceView surface = RtcEngine.CreateRendererView(this);
// Host side View association
mRtcEngine.setClientRole(Constants.CLIENT_ROLE_BROADCASTER);
mRtcEngine.enableLocalAudio(true); // The host side needs to open the local audio
mRtcEngine.setupLocalVideo(new VideoCanvas(surface,
VideoCanvas.RENDER_MODE_HIDDEN, ANCHOR_UID)); // The host side is set to local video
mRtcEngine.startPreview();  // Anchors need to enable video preview
 
// Viewer View association
mRtcEngine.setClientRole(Constants.CLIENT_ROLE_AUDIENCE);
mRtcEngine.enableLocalAudio(false); // The viewer does not need to open local audio
mRtcEngine.setupRemoteVideo(new VideoCanvas(surface,
VideoCanvas.RENDER_MODE_HIDDEN, ANCHOR_UID)); // The audience is set to the remote end, i.e. anchor video
Copy the code

Step4 join the room

The token of the first parameter is the token of the current login account, which is managed by the application itself and can be sent to null during the test. The second parameter, the channel ID, is also managed by the application itself. The third parameter is the channel name. The last parameter is the id of the currently logged in account

mRtcEngine.joinChannel("", CHANNEL_ID, "CHANNEL_NAME", uid);
Copy the code

Step5 leave the room

// Anchor end leave
mRtcEngine.setupLocalVideo(null);
mRtcEngine.stopPreview();
mRtcEngine.leaveChannel();
// Audience away
mRtcEngine.setupRemoteVideo(null);
mRtcEngine.leaveChannel();
Copy the code

2.2 Message Function

The function of live room message can be said to be relatively basic and simple. We choose the REAL-TIME information SDK of Sonnet, which is an independent tool class SDK. Sonnet decoules the real-time message function and can provide message support for each scene. The following steps are available for real-time group chat messages:

Step1 configure dependencies

dependencies {
...
implementation 'IO. Agora. RTM: RTM - SDK: 1.0.1'
}
Copy the code

Step2 create a message engine

// APP_ID should be the same as the video interaction SDK
private RtmClient mRtmClient;
mRtmClient = RtmClient.createInstance(context, LiveDefine.APP_ID, listener);
Copy the code

Step3 initialize the room message

Before creating a message channel, perform a login operation. The first parameter is the application account token and the second parameter is the account ID.

mRtmClient.login("", userId,
new ResultCallback<Void>() {
    @Override
    public void onSuccess(Void aVoid) {
        Log.d(TAG, "rtmClient login success");
    }
    @Override
    public void onFailure(ErrorInfo errorInfo) {
        Log.d(TAG, "rtmClient login fail : "+ errorInfo); }});Copy the code

Create a message channel. CHANNEL_ID is an identifier. It can be different from a live channel, but it is recommended to keep the same:

RtmChannel mRtmChannel;
 
RtmChannelListener rtmListener = new RtmChannelListener(){
    @Override
    public void onMessageReceived(RtmMessage var1, RtmChannelMember var2){
        RtmChannelMember can check whether the sender of the message is RtmChannelMember. If the sender of the message is RtmChannelMember, the message is not processed.
    }
    
    @Override
    public void onMemberJoined(RtmChannelMember var1){
        // If a user has joined, it can be used to handle user login messages
    }
 
    @Override
    public void onMemberLeft(RtmChannelMember var1){
          // If a user leaves, it can be used for user offline message processing}}; mRtmChannel = mRtmClient.createChannel(CHANNEL_ID, rtmListener );;Copy the code

Step4 send the message

RtmMessage rtmMessage = mRtmClient.createMessage();
rtmMessage.setText(msg);
mRtmChannel.sendMessage(rtmMessage, callback);
Copy the code

Step5 exit the message channel

This method can be called when you exit the live room.

mRtmChannel.release();
Copy the code

2.3 Online speech translation

First of all, it is also necessary to register an account and create an application. For details, see Sogou Zhiyin Documentation Center. To achieve this, please refer to the following steps:

Step1 initialization

Call the init method to initialize

// The following information is obtained from zhiyin platform application
private static final String BASE_URL = "api.zhiyin.sogou.com";
private static final String APP_ID = "";
private static final String APP_KEY = "";
private SogoSpeech mSogouSpeech;
private DefaultAudioSource mAudioSource;
private OnSogouAsrListener mListener;
 
public void init(Context context) {
    ZhiyinInitInfo.Builder builder = new ZhiyinInitInfo.Builder();
    ZhiyinInitInfo initInfo = builder.setBaseUrl(BASE_URL).setUuid(UUID).setAppid(APP_ID).setAppkey(APP_KEY).create();
    SogoSpeech.initZhiyinInfo(context, initInfo);
 
    SogoSpeechSettings settings = SogoSpeechSettings.shareInstance();
    settings.setProperty(SpeechConstants.Parameter.ASR_ONLINE_AUDIO_CODING_INT,
1);
    settings.setProperty(SpeechConstants.Parameter.ASR_ONLINE_VAD_ENABLE_BOOLEAN,
false); 
    settings.setProperty(SpeechConstants.Parameter.ASR_ONLINE_VAD_LONGMODE_BOOLEAN,
true); // Long time ASR
    settings.setProperty(Parameter.ASR_ONLINE_LANGUAGE_STRING,
ASRLanguageCode.CHINESE); // English ASR asrlanguagecode.englis is also supported
 
    mSogouSpeech = new SogoSpeech(context);
    mSogouSpeech.registerListener(mSpeechEventListener);
 
    mAudioSource = new DefaultAudioSource(new AudioRecordDataProviderFactory(context));
    mAudioSource.addAudioSourceListener(mAudioSourceListener);
}
 
private EventListener mSpeechEventListener = new EventListener() {
    @Override
    public void onEvent(String eventName, String param, byte[] data, int offset, int length, Object extra) {
        if (TextUtils.equals(SpeechConstants.Message.MSG_ASR_ONLINE_LAST_RESULT,
eventName)) {
            if (null != mListener) {
                mListener.onSogouAsrResult(param);
            }
            stopTransform();
        }
    }
 
    @Override
    public void onError(String errorDomain, int errorCode, String errorDescription, Object extra) {
        // 9002 The user cancelled the service
        if (9002! = errorCode &&null! = mListener) { mListener.onSogouAsrResult(""); } stopTransform(); }};private IAudioSourceListener mAudioSourceListener = new IAudioSourceListener() {
    @Override
    public void onBegin(IAudioSource iAudioSource) {
        Log.d(TAG, "AudioSource onBegin");
        mSogouSpeech.send(SpeechConstants.Command.ASR_ONLINE_START, "".null.0.0);
    }
 
    @Override
    public void onNewData(IAudioSource audioSource, Object dataArray, long packIndex, long sampleIndex, int flag) {
        final short[] data = (short[]) dataArray;
        mSogouSpeech.send(SpeechConstants.Command.ASR_ONLINE_RECOGIZE, "", data, (int) packIndex, 0);
    }
 
    @Override
    public void onEnd(IAudioSource audioSource, int status, Exception e, long sampleCount) {
        Log.d(TAG, "AudioSource onEnd");
        mSogouSpeech.send(SpeechConstants.Command.ASR_ONLINE_STOP, "".null.0.0); }};public interface OnSogouAsrListener {
    void onSogouAsrResult(String result);
}
Copy the code

Step2 start speech recognition

public void startTransform(OnSogouAsrListener listener) {
    mListener = listener;
    mSogouSpeech.send(SpeechConstants.Command.ASR_ONLINE_CREATE,
null.null.0.0);
    new Thread(mAudioSource, "audioRecordSource").start();
}
Copy the code

Step3 stop speech recognition

Normally this method is not required; it is already called in the EventListener callback, or it can be called manually when exiting the room to ensure that the status is normal.

public void stopTransform(a) {
    mListener 
= null;
    if (null != mAudioSource) {
        mAudioSource.stop();
    }
}
Copy the code

We have uploaded this Demo to Github, you can download it directly. Android:github.com/AgoraIO-Com… IOS:github.com/AgoraIO-Com…

Finally, I’ll show you what the Demo looks like. (1) Host end live broadcast questions (voice to text) :










If you have any questions, feel free to discuss them with the authors in the RTC Developer community.

Author: Agora Evangelist Jin Yeqing