The three authors of this article, Zhengyang, Haiyang and Ali, are engineers from different companies. They developed a smart car that can see houses remotely with real-time video at the Agora RTC Hack Shanghai hackathon. This article will share their development experience from solution design, hardware development, Android side development and server side deployment in detail.
Welcome to the RTC developer community to share your experience with more RTC developers.
Creative idea
I’ve heard a lot about hackathons, where a group of people from different places get together, team up, conceive, develop, and build a prototype in 48 hours. The three of us took part in the competition with the idea of listening to others’ ideas and focusing on participation. I had a general idea of what I wanted to do before the game:
- The direction of the idea is to combine and splice according to the parts we are good at, which has to say my two powerful teammates Ocean and Ali; Hai Yang is embedded software engineer, automotive electronics direction, write a drive to do a car easily. Alex is a back-end engineer, capable of handling both server-side and front-end pages.
- Therefore, the team has two parts of embedded and cloud capabilities, and the direction of the technical idea is that the cloud enables embedded. Hope to have a car, the car can be transmitted back to the video image, video image can be real-time to multiple users, users after getting authorization, remote control of the car.
Program design and division of labor
With this in mind, the implementation architecture is shown in the figure above. Now that we have a basic architecture and a clear idea of what we want to implement, it’s time to split the roles. Taking into account our respective strengths, the division of labor is as follows:
| Division of labor | figure | Achieve Goal 1 | Achieve Goal 2 |
|---|---|---|---|
| Car drive and camera end | ocean | Use the Android system to complete the camera driver | Add drive in Android system, analyze motion control signal, realize car movement |
| Car APP and user APP | zhengyang | Transmit the video collected by the camera, parse the server signaling, and send the motion signal to the car | Users can view videos of the car and control the car through their mobile phones |
| Server-side and Web front end | The force | After the video transmission server is connected to the public network, users can view car videos by inputting web addresses | Provide web front end, the user can control the car through web page |
Start with hardware development
The car is driven by a 4-wheel servo motor, which is equipped with a video capture module, servo motor drive module, STM32 control module and camera holder module. The overall effect picture after installation is as follows:
Users need to connect the car to the Internet through wifi before remotely controlling various movements of the car. The user can use the upper computer (Android APP or web front) to control the car to move around or control the head to adjust the camera direction.
The video capture module includes wifi module, which can be connected to wifi hotspot to provide network basis for video transmission. Also provides HDMI interface and monitor connection, convenient user debugging. The camera is connected with the video acquisition module through USB. We use the no-drive Tianmin 6602 camera, which can reach 640*480 resolution and can automatically focus.
The STM32 control module uses Arduino interface to connect with servo motor drive module. The STM32 module is responsible for the generation of motor and PTZ signal, and the servo motor drive module directly drives the motor to work. Input voltage of servo motor is 6~12V, dc drive.
The working principle of
The video acquisition module in the car adopts the customized Android system to provide network connection, command forwarding and video stream acquisition and transmission functions. When the upper computer is connected to the car through remote service, the upper computer can request the video information on the camera of the current car. At the same time, the video acquisition module also analyzes the control signal from the upper computer into the protocol data with specified format and function, and sends it to the STM32 control module through the serial port.
After receiving relevant control signals, the STM32 control module on the trolley adjusts the duty ratio of the output pulse signal, and the drive board converts the output level to directly control the servo motor or PTZ module to make corresponding actions, so as to complete the control function desired by the upper computer user.
Control signal protocol
For simple car control, we only need to send control signals to the STM32 control module through the serial port. The simple control signal protocol is as follows:
| Type the command | baotou | type | The command | data | Package the tail |
|---|---|---|---|---|---|
| stop | FF | 00 | 00 | 00 | FF |
| forward | FF | 00 | 01 | 00 | FF |
| back | FF | 00 | 02 | 00 | FF |
| Turn left | FF | 00 | 03 | 00 | FF |
| Turn right | FF | 00 | 04 | 00 | FF |
| Yuntai up and down | FF | 01 | 01 | Angle value | FF |
| Yuntai around | FF | 01 | 02 | Angle value | FF |
The Android SDK custom
The development tools
In order to realize the real-time video and the remote control function of the car we want, we need to adopt the video SDK of Sonnet and run it on the Android development board. For the development board, we choose Firefly’s RK3128 platform, which adopts Corcortex A7 architecture quad-core 1.3ghz processor, Mali-400MP2 GPU, onboard gigabit Ethernet port, 2.4ghz Wi-Fi and Bluetooth 4.0, and supports Android and Ubuntu dual systems.
Customized serial Port Driver
In order to realize the control of the car by RK3128, we need to realize the communication between RK3128 and STM32 control module through USB to serial port module. So we first need to reconfigure the RK3128 kernel to support USB-to-serial driver.
Download the RK3128 Android SDK and verify the MD5 value of the file:
Md5sum/path/to/fireprime_android5. 1 _git_20180510. Tar. Gz fce0e6d65549939167923260142b2c1e Fireprime_android5. 1 _git_20180510. Tar. GzCopy the codeAfter confirmation, decompress:
Mkdir -p ~/proj/ FirePrime CD ~/proj/ FirePrime tar XVF /path/to/fireprime_android5.1_git_20180510.tar.gz git reset --hard git remote add bitbucket https://bitbucket.org/T-Firefly/firenow-lollipop.git git pull bitbucket fireprime:fireprimeCopy the codeConfigure and compile the kernel:
cd ~/proj/fireprime/kernel
make rk3128-fireprime_defconfig
make menuconfig
make -j8 rk3128-fireprime.img
Copy the codeFor make Menuconfig, check Device Drivers > USB Support > USB Serial Converyer Support > USB Serial Console Device Support/USB Generic Serial Driver and select CP210x, CH341, FTDI, PL2303 and other common Serial port tools.
Compiling Android:
cd ~/proj/fireprime
. build.sh
make -j8
./mkimage.sh
Copy the codeFinally, after compiling, burn the partition image and insert the USB-to-serial tool to check whether the following log information appears in the system DMESG:
[2213.003173] USB 1-1.3: New full-speed USB device Number 6 using Rockchip_EHct [2213.113759] USB 1-1.3: New full-speed USB device Number 6 using Rockchip_EHct [2213.113759] New USB device found, idVendor= 10C4, idProduct=ea60 [2213.113839] USB 1-1.3: New USB device strings: Mfr=1, Product=2, SerialNumbe3 [2213.113883] USB 1-1.3: Product: CP2102 USB to UART Bridge Controller [2213.113921] USB 1-1.3: Manufacturer: Silicon Labs [2213.113956] USB 1-1.3: Manufacturer: Silicon Labs SerialNumber: 0001 [2213.120813] CP210X 1-1.3:1.0: CP210X Converter Detected [2213.209852] USB 1-1.3: Reset full-speed USB device number 6 using Rockchip_et [2213.320161] USB 1-1.3: cp210x converter now attached to ttyUSB0Copy the codeIf the serial port device is attached to ttyUSBx, the serial port driver is successfully customized.
The above is the method of fully compiling the Android SDK, which requires the Android system to be compiled, which is more time-consuming than just compiling the kernel. We can check the required serial port driver as M in the above make Menuconfig, compile the driver into a. Ko file through make Modules method, and then automatically load the driver when the Android system starts up:
First copy the. Ko driver file to the Android file system
adb shell su mount -o remount ,rw / mkdir /modules chmod 777 /modules chown -R nobody:nobody /modules exit exit adb push ./xxxx.ko /modulesCopy the codeWrite the startup script /data/serial.sh
#! /system/bin/sh
insmod /modules/xxxx.ko
mknod /dev/ttyUSB c 240 0
Copy the codeModify init.rc and add a script to run your own
service serial /system/bin/sh /data/serial.sh
user root
oneshot
Copy the codeRealize video transmission in App end
Video transmission and signaling transmission are realized by Agora SDK of acoustic network. As it relates to the combination with the embedded development board, we mainly refer to the audio network in Github to provide a variety of cases in the doll machine demo. The structure diagram in the sample code looks like this:
The sample code has the video transmission part, and the control signaling needs to be done by itself.
| Two APP | video | Control signal |
|---|---|---|
| Android App on the car end | Send video signal | Receiving control signal |
| The user has an Android App on his mobile phone | Video reception | Send control signal |
Sound net SDK briefMethod of useAs follows:
- First of all, to apply forAppID
Add the following to res/values/strings_config. XML in the AndroidAPP to set agora_app_id
<resources>
<string name="agora_app_id">1a486ee31a30xxxxxxxxxx</string>
</resources>
Copy the code- Copy the.jar file to liBS /. Since the signaling and video transfer parts are used, two.jar files are required: agora-RtC-sdk.jar and agora-sig-sdk.jar
- Add armeabi-v7a and its.so file to SRC /main/jniLibs
In build.gradle, you can specify the following description:
dependencies { compile fileTree(dir: 'libs', include: [' *. Jar ']) testCompile junit: junit: '4.12' compile 'com. Android. Support: appcompat - v7:23.4.0'}Copy the codeAt this point, the video transfer function can be turned on smoothly using the sample code.
Let App control the car with signaling
The specific usage of signaling is described in the Reference of Agora. The main functions used are as follows:
/ / initialize the signaling SDK m_agoraAPI = AgoraAPIOnlySignal. GetInstance (context, appID); M_agoraapi. login2(appId, Account, token, uid, deviceID, retry_time_in_s, Retry_count) / / / / / / / / / / / / / / point to point test / / / / / / / / / / / / / / / send a point-to-point message m_agoraAPI messageInstantSend (account, uid, MSG, MsgID) / / set to receive news callback (m_agoraAPI. OnMessageInstantReceive (uid account, MSG) {/ / code there} / / / / / / / / / / / / / channel test / / / / / / / / / / / / / / / / / join channel m_agoraAPI channelJoin (channelName) / / channel to send message M_agoraAPI. MessageChannelSend (channelName, MSG, msgID) / / set the channel to end receives the message callback m_agoraAPI. OnMessageChannelReceive (channelID, Account, uid, MSG) {/ / code there} / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / exit m_agoraAPI Agora signaling system. The logout ()Copy the codeAndroid App operation serial port
The serial port is required to send data after the Android App at the car end gets the signaling. Therefore, how to implement Android App operation serial port. Here are two schemes:
- Using the architecture given by the Android system for processing, Android with SerialPort demo code, named SerialPort. Two points to note here are that the code here relies on the JNI tools and NDK and can cause problems when using project code if it is not fully installed. In addition, serial port operation is not convenient to use the Android simulator for testing. For devices without serial ports, when the serial port is opened, an error will be reported and the program may exit and crash.
- Example echo ‘aa’ > /dev/ttyusb0 will send AA to the serial port ttyUSB0. The advantage of doing this is that the code itself is simple and the serial port directly calls the bottom layer.
For a short time to achieve the function, scheme 2 is easier to achieve the method, here need to pay attention to a point, need to re-compile the Android framework layer to give app root authority when the signaling analysis is completed, serial port debugging through, can achieve remote control of the car.
Finally: server-side deployment
In order to realize the convenience of online real-time house viewing through mobile phones or computers, we need to connect the Android App terminal of the car through the Web terminal to obtain the real-time transmitted video content. In our vision, users can control the car remotely, which makes it convenient for users to know all aspects of the house. To sum up, we need to implement the following two functions:
- It has video connection function
- With remote control function
Fortunately, with the services provided by Sonnet, we can easily build these two services. In this project, we use the video SDK of Sonnet to realize the video connection between the web page and the car APP, and send messages through the signaling SDK to control the car’s front, back, left, right walking and the camera’s up, down, left, right swing.
To list the tools we use:
-
The Web SDK and documents of video call are used to realize the remote video interaction function.
-
Signaling SDK and documents, used to achieve remote control intelligent car;
-
Server for deploying static pages;
Realize video connection and message sending
First introduce the SDK for video and signaling on the page. And then let’s do the video connection first.
// Create AgoraRTC instance and join channel
const client = AgoraRTC.CreateClient({mode:"interop"})
client.init(appId, function () {
console.log("AgoraRTC client initialized");
client.join(channel_key, CHANNEL_NAME, null.function (uid) {
console.log("User " + uid + " join channel successfully")
console.log(new Date().toLocaleTimeString())
// do something}}Copy the codeSubscribe to the remote video stream and play it.
let stream = AgoraRTC.creatStream(merge(defaultConfig.config))
localStream.init((a)= >{
client.on('stream-added'.function (evt) {
var stream = evt.stream;
console.log("New stream added: " + stream.getId());
console.log("Subscribe ", stream);
client.subscribe(stream, function (err) {
console.log("Subscribe stream failed", err);
});
});
client.on('stream-subscribed'.function (evt) {
var stream = evt.stream;
console.log("Subscribe remote stream successfully: " + stream.getId());
if ($('div#video #agora_remote' + stream.getId()).length === 0) {$('div#video').append('<div id="agora_remote' + stream.getId() + '" style=" width:810px; height:607px;" >');
}
stream.play('agora_remote' + stream.getId());
});
})
Copy the codeImplement sending information.
// Create the signaling object
const signal = Signal(appId)
// In experimental conditions, no token is set
const token = '_no_need_token'
/ / login
const session = signal.login(account, token)
session.onLoginSuccess = (uid) = > {
// Send the message to the specified account
signal.sendMessage(reciveAcount, message)
}
Copy the codeAfter completing the above steps, set the same appId and token as the car side (if necessary), set the corresponding parameters, and we can remotely control the car and get the video.
The hackers demonstrate that RTC technology can be used for more than just entertainment, social, and education. Until recently, Agora RTC Hack was in the works in other cities around the world. Individuals and teams from startups entered and won awards. We will invite several of the winning teams from around the world to attend the RTC 2018 Real-time Internet Conference, which will be held on September 7-8. There will not only be Google WebRTC product manager, Huawei Multimedia Laboratory chief scientist, Northwestern Polytechnical University Intelligent Acoustics and Immersive Communication Center chief scientist, Twitch chief R&D engineer, AVS standard working group leader and other technical experts brought dry goods, but also these small projects full of creativity and development enthusiasm. Want to chat with them and make friends?
Click here to sign up for a live date!