Author: Shan Hui, Senior development engineer of Agora.

As we all know, Qt is a cross-platform C++ graphical user interface application development framework, it has cross-platform, rich API, support 2D/3D graphics rendering, support OpenGL, open source and other excellent features. Many common applications or games in the market, such as VLC, WPS Office, Need for Speed, etc., are developed based on Qt.

This article will show you how to develop an audio and video calling application using Qt.

1 Use Qt Quick

Qt currently has two ways to create user interfaces:

  • Qt Widgets
  • Qt Quick

Qt Widgets is a traditional desktop interface library, while Qt Quick is a new generation of advanced user interface technology, which can be easily used in mobile terminals, embedded devices and other interface development.

Qt Widgets are currently in maintenance, stable and mature. Qt Quick is the main direction of future development, its development is more simple and convenient, better user experience.

So this article chooses Qt Quick as the way to create the user interface, the development environment is as follows:

  • Qt: 5.12.0
  • Qt Creator: 4.8.2
  • Agora Video SDK: 2.4.0

2 design the interaction flow

First, we design a simple VIDEO call UI interaction process.

There are two main UI interfaces:

  • JoinRoom: Login channel interface;
  • InRoom: Video call interface;

And three auxiliary UI interfaces:

  • Splash: Welcome interface;
  • VideoSetting: Video parameter setting interface;
  • DeviceSetting: Device setup interface;

The interaction logic between the UIs has been marked with the corresponding red box.

3 Create a Qt project

Open Qt Creator and select Create a new project.

  1. Select Qt Quick Application-Empty;

  1. Enter the project name AgoraVideoCall and select the project path;

  1. Select qmake to compile;

  1. Select the least supported Qt version, which Rimmer considers Qt 5.9;

  1. Select the native Qt version, which uses 5.12.0;

4 Importing Resources

4.1 Importing images Resources

We will first prepared ICONS and other resources, import into the project.

  1. Copy the images folder to the project directory.
  2. In the Project view of Qt Creator, right click on the Resources/ QML. QRC file;
  3. Select add existing path;
  4. Select the images folder;
  5. All resources in the images folder are automatically added to the QML. QRC file.

4.2 Importing Controls Resources

There are two ways to use controls such as buttons in Qt Quick:

  1. Use controls defined by Qt Quick; Advantage is not their own development, can be quickly integrated use.
  2. Use user-defined controls; The advantage is that styles can be customized, and you can define more controls that are not officially provided.

We are using some pre-prepared controls here, so follow the steps to import them into the project.

  1. Copy the controls folder to the project directory
  2. In the Project view of Qt Creator, right click on the Resources/ QML. QRC file;
  3. Select add existing path;
  4. Select the controls folder;
  5. All controls in the controls folder are automatically added to QML. QRC file.

Note that controls are not imported by default, requiring the developer to import them in the UI to use. For example:

4.3 Importing the Agora. IO Audio and Video Call SDK

To use the audio and video call function, you need to import the SDK corresponding to agora. IO. You can register the developer account of Agora. IO and obtain the SDK of the corresponding platform from the SDK download address.

After downloading, copy the corresponding header files to the include folder in your project, static libraries to the Lib folder in your project, and dynamic libraries to the DLL folder in your project.

Then modify the Qt project file, specify the linked dynamic library, open the agoravideocall. pro file, and add the following:

INCLUDEPATH += ? PWD/lib win32: LIBS += -L? PWD/lib/ -lagora_rtc_sdk

5 UI and UI service logic

After creating the project and importing the resources, we first need to implement the five UIs we designed earlier.

5.1 create the UI

  1. Right-click on the project, select Add New, and select the QtQuick UI File template.

5.2 UI Service Logic

After the UI is complete, add the service logic triggered by the corresponding button. When creating a QtQuick UI File, for example when creating a Splash UI, two QML files are created by default:

  • Splashform.ui.qml: declaration description of UI;
  • Splash. QML: response of UI corresponding events and part of UI business logic;

So, for example, Button click events, mouse events, etc., are associated with the id of the corresponding control.

For example, in splashForm.ui.qml, we expect users to return to the login screen if they click anywhere, and add a mouse event listening area to splashForm.ui.qml:

MouseArea {
    id: mouseArea
    anchors.fill: parent
}
Copy the code

Add business logic to Splash. QML:

mouseArea.onClicked: main.joinRoom()
Copy the code

Finally, add joinRoom’s response function in main. QML:

Loader {
    id: loader
    focus: true
    anchors.fill: parent
}
​
function joinRoom() {
    loader.setSource(Qt.resolvedUrl("JoinRoom.qml"))}Copy the code

This completes a basic UI business logic. Other UI business logic, such as opening the Settings window and logging in to the channel, will not be listed one by one.

Of course, the actual triggering of the core business logic, such as logging in to the channel for audio and video calls, setting parameters to take effect, can be left blank, after completing all UI interaction responses, this part of the logic will be filled in.

5.3 QML interacts with C++

Once the basic UI business logic is complete, logical interaction between QML and C++ is typically required. For example, after pressing the Join button to enter the channel, we need to call the audio and video related logic of Agora in C++ to enter the channel and make a call.

There are two ways to use C++ classes and objects in QML:

  1. Define a subclass of QObject in C++, register it in QML, create the object of this class in QML;
  2. Create an object in C++ and set it to a QML context property to use in QML;

Here we use the second approach, defining the MainWindow class that loads main.qml as the core form and sets itself as a QML context property in its constructor:

setWindowFlags(Qt::Window | Qt::FramelessWindowHint);
resize(600.600);
​
m_contentView = new QQuickWidget(this);
m_contentView->rootContext()->setContextProperty("containerWindow".this);
m_contentView->setResizeMode(QQuickWidget::SizeRootObjectToView);
m_contentView->setSource(QUrl("qrc:///main.qml"));
​
QVBoxLayout *layout = new QVBoxLayout;
layout->setContentsMargins(0.0.0.0);
layout->setSpacing(0);
layout->addWidget(m_contentView);
​
setLayout(layout);
Copy the code

6 Video Rendering

The Agora SDK provides interfaces that allow users to define their own rendering methods. The interfaces are as follows:

agora::media::IExternalVideoRender *
AgoraRtcEngine::createRenderInstance(
    const agora::media::ExternalVideoRenerContext &context) {
  if(! context.view)return nullptr;
  return new VideoRenderImpl(context);
}
Copy the code

VideoRenderImpl need to inherit the agora: : media: : IExternalVideoRender class, and implement the related interface:

virtual void release(a) override {
  delete this;
}
​
virtual int initialize(a) override {
  return 0;
}
​
virtual int deliverFrame(const agora::media::IVideoFrame &videoFrame,
    int rotation, bool mirrored) override {
  std::lock_guard<std::mutex> lock(m_mutex);
  if (m_view)
    return m_view->deliverFrame(videoFrame, rotation, mirrored);
  return - 1;
}
Copy the code

We’ll use OpenGL to render and define the renderFrame:

int VideoRendererOpenGL::renderFrame(const agora::media::IVideoFrame &videoFrame) {
  if (videoFrame.IsZeroSize())
    return - 1;
​
  int r = prepare();
  if (r) return r;
​
  QOpenGLFunctions *f = renderer();
  f->glClear(GL_COLOR_BUFFER_BIT);
​
  if(m_textureWidth ! = (GLsizei)videoFrame.width() || m_textureHeight ! = (GLsizei)videoFrame.height()) { setupTextures(videoFrame); m_resetGlVert =true;
  }
​
  if (m_resetGlVert) {
    if(! ajustVertices()) m_resetGlVert =false;
  }
​
  updateTextures(videoFrame);
​
  f->glDrawElements(GL_TRIANGLES, 6, GL_UNSIGNED_BYTE, g_indices);
  return 0;
}
Copy the code

UpdateTextures:

void VideoRendererOpenGL::updateTextures(
    const agora::media::IVideoFrame &frameToRender) {
  const GLsizei width = frameToRender.width();
  const GLsizei height = frameToRender.height();
​
  QOpenGLFunctions *f = renderer();
  f->glActiveTexture(GL_TEXTURE0);
  f->glBindTexture(GL_TEXTURE_2D, m_textureIds[0]);
  glTexSubImage2D(width, height,
                  frameToRender.stride(IVideoFrame::Y_PLANE),
                  frameToRender.buffer(IVideoFrame::Y_PLANE));
​
  f->glActiveTexture(GL_TEXTURE1);
  f->glBindTexture(GL_TEXTURE_2D, m_textureIds[1]);
  glTexSubImage2D(width / 2, height / 2,
  frameToRender.stride(IVideoFrame::U_PLANE),
  frameToRender.buffer(IVideoFrame::U_PLANE));
​
  f->glActiveTexture(GL_TEXTURE2);
  f->glBindTexture(GL_TEXTURE_2D, m_textureIds[2]);
  glTexSubImage2D(width / 2, height / 2,
  frameToRender.stride(IVideoFrame::V_PLANE),
  frameToRender.buffer(IVideoFrame::V_PLANE));
}
Copy the code

This allows you to draw the Frame in the Agora SDK callback on the specific Widget.

7 Core service logic

We need to simply encapsulate the Agora SDK logic to provide audio and video calling capabilities.

7.1 Callback Event

Agora SDK will provide a lot of event callback information, such as the remote user to join channel, the remote user exit channel, etc., we need to inherit the Agora: : RTC: : IRtcEngineEventHandler event callback class, pay equal attention to writing part need to function, to the incident response.

class AgoraRtcEngineEvent : public agora::rtc::IRtcEngineEventHandler {
 public:
  AgoraRtcEngineEvent(AgoraRtcEngine &engine)
    :m_engine(engine) {}
​
  virtual void onVideoStopped(a) override {
    emit m_engine.videoStopped();
  }
​
  virtual void onJoinChannelSuccess(const char *channel, uid_t uid,
                                    int elapsed) override {
    emit m_engine.joinedChannelSuccess(channel, uid, elapsed);
  }
​
  virtual void onUserJoined(uid_t uid, int elapsed) override {
    emit m_engine.userJoined(uid, elapsed);
  }
​
  virtual void onUserOffline(uid_t uid,
                             USER_OFFLINE_REASON_TYPE reason) override {
    emit m_engine.userOffline(uid, reason);
  }
​
  virtual void onFirstLocalVideoFrame(int width, int height,
                                      int elapsed) override {
    emit m_engine.firstLocalVideoFrame(width, height, elapsed);
  }
​
  virtual void onFirstRemoteVideoDecoded(uid_t uid, int width, int height,
                                         int elapsed) override {
    emit m_engine.firstRemoteVideoDecoded(uid, width, height, elapsed);
  }
​
  virtual void onFirstRemoteVideoFrame(uid_t uid, int width, int height,
                                       int elapsed) override {
    emit m_engine.firstRemoteVideoFrameDrawn(uid, width, height, elapsed);
  }
​
 private:
  AgoraRtcEngine &m_engine;
};
Copy the code

Here we emit events from AgoraRtcEngine’s signaling functions and respond in the UI without complicated processing logic.

7.2 Resource Management

Define the AgoraRtcEngine class and, in the constructor, initialize the audio and video calling engine: agora:: RTC ::IRtcEngine:

AgoraRtcEngine::AgoraRtcEngine(QObject *parent)
    : QObject(parent), m_rtcEngine(createAgoraRtcEngine()),
  m_eventHandler(new AgoraRtcEngineEvent(*this)) {
  agora::rtc::RtcEngineContext context;
  context.eventHandler = m_eventHandler.get();
​
  // Specify your APP ID here
  context.appId = "";
​
  if (*context.appId == '\ 0') {
    QMessageBox::critical(nullptr, tr("Agora QT Demo"),
    tr("You must specify APP ID before using the demo"));
  }
​
  m_rtcEngine->initialize(context);
  agora::util::AutoPtr<agora::media::IMediaEngine> mediaEngine;
  mediaEngine.queryInterface(m_rtcEngine.get(), agora::AGORA_IID_MEDIA_ENGINE);
  if (mediaEngine) {
    mediaEngine->registerVideoRenderFactory(this);
  }
  m_rtcEngine->enableVideo();
}
Copy the code

Note: Please refer to the official Agora documentation for information on how to obtain the Agora APP ID.

On App exit, the audio/video call engine resources should be released in the destructor of the AgoraRtcEngine class, which we manage automatically by specifying the unique_ptr release function:

struct RtcEngineDeleter {
  void operator(a)(agora::rtc::IRtcEngine *engine) const {
    if(engine ! =nullptr) engine->release(); }};std: :unique_ptr<agora::rtc::IRtcEngine, RtcEngineDeleter> m_rtcEngine;
Copy the code

7.3 Logging In to channels

With most of the logic taken care of, this is the most important step.

Add AgoraRtcEngine QML context properties to MainWindow:

AgoraRtcEngine *engine = m_engine.get();
m_contentView->rootContext()->setContextProperty("agoraRtcEngine", engine);
Copy the code

When the user enters the channel name and clicks the Join button to trigger the logon logic, we add event processing in JoinRoom.qml:

btnJoin.onClicked: main.joinChannel(txtChannelName.text)
Copy the code

In main. QML, call the joinChannel function of AgoraRtcEngine. If successful, switch to the InRoom interface:

function joinChannel(channel) {
    if (channel.length > 0 && agoraRtcEngine.joinChannel("", channel, 0) === 0) {
        channelName = channel
        loader.setSource(Qt.resolvedUrl("InRoom.qml"))}}Copy the code

7.4 the local flow

After entering the InRoom interface, local stream (generally the image collected by the camera) needs to be rendered. Add onCompleted to inroom.qml:

Component.onCompleted: {
    inroom.views = [localVideo, remoteVideo1, remoteVideo2, remoteVideo3, remoteVideo4]
​
    channelName.text = main.channelName
    agoraRtcEngine.setupLocalVideo(localVideo.videoWidget)
}
Copy the code

In AgoraRtcEngine, set the local stream rendering Widget as the canvas to draw on:

int AgoraRtcEngine::setupLocalVideo(QQuickItem *view) {
  agora::rtc::view_t v =
    reinterpret_cast<agora::rtc::view_t> (static_cast<AVideoWidget *>(view));
​
  VideoCanvas canvas(v, RENDER_MODE_HIDDEN, 0);
  return m_rtcEngine->setupLocalVideo(canvas);
}
Copy the code

7.5 the remote flow

AgoraRtcEngine throws the onUserJoined and onUserOffline events when it receives them:

virtual void onUserJoined(uid_t uid, int elapsed) override {
  emit m_engine.userJoined(uid, elapsed);
}
Copy the code

At this point, in the InRoom interface, capture the event and process it:

Connections {
    target: agoraRtcEngine
​
    onUserJoined: {
        inroom.handleUserJoined(uid)
    }
    onUserOffline: {
        var view = inroom.findRemoteView(uid)
        if (view)
            inroom.unbindView(uid, view)
    }
}
​
function findRemoteView(uid) {
    for (var i in inroom.views) {
        var v = inroom.views[i]
        if(v.uid === uid && v ! = =localVideo)
            return v
    }
}
​
function bindView(uid, view) {
    if(view.uid ! = = 0)return false
    view.uid = uid
    view.showVideo = true
    view.visible = true
    return true
}
​
function unbindView(uid, view) {
    if(uid ! == view.uid)return false
    view.showVideo = false
    view.visible = false
    view.uid = 0
    return true
}
​
function handleUserJoined(uid) {
    //check if the user is already binded
    var view = inroom.findRemoteView(uid)
​
    if(view ! == undefined)return
​
    //find a free view to bind
    view = inroom.findRemoteView(0)
​
    if (view && agoraRtcEngine.setupRemoteVideo(uid, view.videoWidget) === 0) {
        inroom.bindView(uid, view)
}
}
Copy the code

We designed the UI to display a maximum of four remote streams, so beyond that, no bindView processing is done.

In AgoraRtcEngine, set the remote stream rendering Widget as the canvas to draw on:

int AgoraRtcEngine::setupRemoteVideo(unsigned int uid, QQuickItem* view) {
  agora::rtc::view_t v =
      reinterpret_cast<agora::rtc::view_t> (static_cast<AVideoWidget *>(view));
  VideoCanvas canvas(v, RENDER_MODE_HIDDEN, uid);
  return m_rtcEngine->setupRemoteVideo(canvas);
}
Copy the code

At this point, the basic core service logic is complete, and the call results are as follows:

8 summarizes

Qt as a very mature graphical interface library, it is very simple to use, and has a large number of documents and solutions, I think is the desktop under the development of graphical interface library one of the first choice. This Demo is developed to help those who want to add audio and video calls to their applications.