Abstract: The “Summer leisure Team” from University of Electronic Science and Technology of China won the third place in the “2nd Huawei Cloud ARTIFICIAL Intelligence Contest · Unmanned Car Challenge Cup” held by Huawei Cloud and Shanghai Jiao Tong University Student Innovation Center.
The final of “The 2nd Huawei Cloud ARTIFICIAL Intelligence Contest · Unmanned Car Challenge Cup” held by Huawei Cloud and Shanghai Jiao Tong University Student Innovation Center has come to a successful conclusion. The “Summer leisure Team” from University of Electronic Science and Technology of China, with solid knowledge of pattern recognition, robotics and other disciplines, developed and deployed unmanned vehicle model with the help of ModelArts, huawei’s one-stop AI development and management platform, and HiLens, the end-to-end cloud collaborative solution, and finally won the third place.
School of Automation Engineering, University of Electronic Science and Technology of China GUI Xingtai
“Summer Leisure Team” wrote and shared their experience, including how to use Huawei Cloud ModelArts and HiLens to achieve target detection of unmanned vehicles, multi-source data fusion and driving control capabilities.
I. Background story of the team
1 “Why do we participate?”
We are from university of electronic science and technology college of automation engineering and computer science and engineering of five new graduate, because graduate research direction for more in areas such as pattern recognition, robot, and very interested in artificial intelligence technology, so in July when organize a signed up for this year’s huawei cloud unmanned vehicle challenge. The five members have rich scientific research competition experience and are all loyal fans of Huawei. They are full of confidence and enthusiasm for this new competition.
2. “What difficulties did we meet?”
First, because this particular outbreak reasons, we play the five previous cannot get together to discuss and plan of the project, the official start of the case is in early September, so our preparation time is only about 15 days, but with the support of the school, college, we can have a good debugging field, from the school is also directly into the development of the high strength work, Basic functions have been implemented before leaving for Shanghai.
In the process of semi-final and final, the main reason was that the car tour line was sensitive to the light environment, and ran out of track in the final. However, our team members in charge of the tour line solved this problem in time through rich experience of on-the-spot car adjustment, and finally achieved good results.
3 “Our Happy Hour”
Looking back on the days when we prepared for the competition, the happiest time was the days and nights when five people tuned the car in the campus. They faced the technical difficulties together and enjoyed the happiness of inspiration realization together. They watched a car realize all kinds of functions from nothing to finally. Of course, in the final, we were very happy to finish third.
Ii. Introduction to the competition questions
The car needs to pass layers of checkpoints on the specified track, including traffic light recognition (starting & parking), curve driving, zebra crossing pedestrian recognition, speed recognition, automatic parking, racing and so on. The car must not only finish all the match points, but also run fast. The test here is the comprehensive ability of participants, including the body structure, chassis control, sensors, network communication, cloud data processing, artificial intelligence and other multidisciplinary knowledge, and the difficulty is to integrate these knowledge. Team members need to develop the car by combining Huawei Cloud AI platform (Huawei Cloud ModelArts one-stop AI development and management platform, HiLens End-cloud collaborative AI development and application platform) and ROS operating system.
Iii. Overall Solution:
In this unmanned vehicle competition, the vehicles mainly have three sensors: lidar and two cameras, one is the car’s own camera and the camera on HiLens Kit. The multi-source data fusion solution of lidar and camera is the most mainstream technology in the autonomous driving field. Our team summarized the role of each sensor:
Lidar is used for passable area detection and dynamic obstacle avoidance, car camera is used for lane line detection and HiLens Kit is used for target detection and corresponding decision control.
3.1
Laser radar by acquiring point cloud data to drive the car in front of effective barriers within the sector average coordinates calculation, giving a barrier-free condition to realize close emergency obstacle detection, pedestrians obstacle detection, so the default of one of the biggest bright spot is that when a moving process of any state to detect obstacles can be timely brake, in case the greatly convenient for us in the process of debugging.
The average point of feasible advance region is obtained by detecting the boundary of feasible advance region by lidar.
Then through the Ackerman turf movement boundary to determine the forward point, and get the desired direction.
We used the above two technologies to complete the two tasks of detecting obstacles on the sidewalk and passing the intersection smoothly
3.2
Lane line detection is carried out by using the camera in the front part of the car, and the detection scheme is as follows:
Step1: Gaussian filtering, mean filtering and Otsu method are used for image processing.
Step2: search boundary points line by line in the image.
Step3: filter out unreasonable boundary points.
Step4: Use the least square method to repair the left and right edges.
Step5: Calculate the center line of the image.
Step6: Use the least square method to repair the center line.
Step7: Use PID to calculate the rotation Angle.
Lane line detection results are as follows:
3.3
HiLens Kit is a multi-mode AI development Kit developed by Huawei Cloud. The AI chip used is Ascend 310 developed by Huawei, and the specific computing power is 16 TONS. Meanwhile, developers can not only use the device’s own camera, but also external cameras, and handle 10-16 channels of video analysis at the same time. After HiLens Kit detects various targets, it sends them to ROS terminal through socket protocol, and ROS terminal makes a series of decision control to complete a series of operations including red light stop, green light travel, sidewalk stop, speed limit sign speed limit, speed limit sign speed limit and yellow light parking.
3.4
The overall implementation scheme of the trolley is shown as follows:
ROS communication is shown as follows:
Iv. Use of Huawei Cloud ModelArts platform
The competition needs to use ModelArts platform to carry out target detection tasks, including detection of red, green and yellow lights, detection of speed limit resolving speed limit signs and sidewalk detection.
The process of model building using ModelArts platform includes:
Data acquisition and enhancement, data set preparation, model selection and training, model transformation
4.1 Data acquisition and enhancement
In addition to the official training set provided by the competition, we also added training sets at the test site and competition site in the final stage. The training set was first captured by HiLens Kit camera and saved to/TMP directory. From HiLens via TCP to local using OpencV to periodically capture images from the video as our original data set.
In addition, three data enhancement methods, namely Horizontal Flip, Cutout and Mix Up, were used to solve the over-fitting problem caused by the small amount of data.
Horizontal flip is the horizontal mirror flip, which is relatively simple to implement, but it should be noted that only when an image distribution does not have visual chirality, we can use the mirror flip to enhance the data set without changing the original image distribution. Because the sidewalk and the traffic lights of the preliminary race do not have visual chirality, we use mirror flipping to enhance the data of such images, and the conversion effect is as follows:
Cutout and Mixup are two commonly used data enhancement methods in computer vision. Cutout fills the random area of the image with noise, while Mixup is a hybrid enhancement algorithm applied in computer vision. It can mix images of different classes to expand the training data set, and the enhancement effects are as follows:
4.2 Data Set Preparation
Step1: download OBS Browser, configure OBS bucket, upload original collected data to OBS bucket, and create data set.
Object Storage Service is a stable, secure, efficient, and easy-to-use cloud Storage Service. It provides standard Restful apis to store unstructured data in any quantity or form
Object storage service OBS configuration documentation: support.huaweicloud.com/obs/index.h…
OBS Console documentation:
Support.huaweicloud.com/usermanual-…
Then enter the data annotation module of ModelArts platform, create data set, select object detection and add label set.
Data set creation Reference:
Support.huaweicloud.com/engineers-m…
Step2: mark in ModelArts
Click the Unlabeled TAB on the Annotated Data page to display all unlabeled image data. Click any image to enter the annotation page. Select the area where the object in the picture is located with the mouse frame, then select the label color in the pop-up dialog box, Enter the label name, and press “Enter” to finish adding the label. After the annotation is complete, the status of the image in the left picture directory will be displayed as “annotated”. The following figure
Annotation of ModelArts data:
Support.huaweicloud.com/exemlug-mod…
4.3 Model selection and training
In ModelArts, we select the model and create the training task. The model we choose is YOLO V3. YOLO is the algorithm of one stage. YOLO V3 adds network complexity over V2, sacrificing a little speed in exchange for an increase in accuracy that is accurate enough to meet competition requirements. After model selection, wait for the training task to complete.
After the training is completed, the training can be observed through visualization
4.4 Model Transformation
After ModelArts training to obtain the ideal model, it is necessary to complete the model transformation in HiLens Studio to generate the model (OM model) that can be used on Centerm chip (this step can also be carried out in HiLens Studio). Please refer to:
ModelArts Platform model conversion tool Guide:
Support.huaweicloud.com/engineers-m…
HiLens Studio model Transformation Tool Guide:
www.huaweicloud.com/ascend/doc/…
5. Use of HiLens, Huawei’s cloud collaborative AI development platform
5.1 HiLens configuration
Before using the HiLens Kit, enable the HiLens service. Register a Huawei cloud account and complete real-name authentication. Then, log in to the HiLens management console and apply for permission for HiLens. After networking with HiLens Kit on a PC, you can log in to the HiLens Kit management platform on the webui to connect to a wireless network in one-click mode.
5.2 Use of HiLens Studio
To enter HiLens Studio from HiLens Console, we first need to create a project. In the skill template, we can select the template skills of unmanned car competition and modify them based on this template. Py, socket_config.py and main.py are mainly used in project, in which utils.py is used to preprocess images, detect the labeling of frames and output categories and coordinate information of labeled frames. Socket_config. py mainly encodes detection results and constructs socket communication protocol for information interaction with ROS end. Main. py is used to call these Python files and call the trained model. In the main.py file, we use multithreading to avoid blocking the program due to socket communication.
The actual results of model testing are as follows:
Six, highlight
(1) Good robustness. Our data are collected in multiple scenes, so the detection effect is relatively stable in different environments.
(2) Continuous inter-frame detection is stable. We set confidence threshold value separately for each category to prevent one frame from being detected and another frame from not being detected. In addition, strict conditions are ensured during decision-making before control instructions are given to avoid misidentification caused by external objective factors.
(3) the ability to patrol car, as the chart, in the process of driving cars in the intersection, and there is no complete lane line, but still can realize patrol line operation, and the image processing speed, so our team entry vehicles can be close to full speed running all the way, so our team in the semi-finals and finals are also to the judges and the audience left a very deep impression.
(4) Multi-source data fusion improves decision-making effect, as shown in the following two examples:
By identifying walls, it helps the radar find viable areas
After identifying the sidewalk, inform the turning direction (for example, in the picture above, inform the left turn) and help the car to patrol the line
Vii. Impressions of joint development between ModelArts and HiLens
With the rapid development of artificial intelligence today, how to cross artificial intelligence algorithm with multi-disciplines and achieve the real intelligence era has become the common goal of the majority of scholars and researchers. The combination of ModelArts and HiLens developed by Huawei Cloud platform provides a more convenient and efficient development environment for both artificial intelligence beginners and experienced engineers. In this competition, the solution jointly used by ModelArts and HiLens provided very high quality service for the competition. Its advantages are as follows:
(1) Data operation including data upload, data annotation and data set release is very convenient.
(2) In the algorithm market, Huawei Cloud and the majority of developers provide algorithms for various tasks, including but not limited to image classification, target detection, semantic segmentation and other tasks, which users can directly adjust and use.
(3) ModelArts supports model deployment to a variety of production environments, which can be deployed as cloud online reasoning and batch reasoning, or directly deployed to the end and side.
(4) The model trained by ModelArts can be transformed into the model and develop relevant skills in cooperation with HiLens, so that the skills can be directly deployed to the end-to-end devices and the algorithm can be implemented.
(5) After registering the HiLens device, the device management office can manage the device, including viewing and deregistering the device, and upgrading the firmware version of the device with one click.
(6) HiLens provides a large number of skill templates, based on which developers can develop and deploy capabilities.
(7) The acceleration of Centerm chip makes the model reasoning faster and the delay lower.
References:
DeVries, T.; Taylor, G. W. Improved Regularization of Convolutional Neural Networks with Cutout. ArXiv :1708.04552 [CS] 2017.
Zhang, H.; Cisse, M.; Dauphin, Y. N.; Lopez-paz, D. Mixup: Empirical Risk Minimization. ArXiv :1710.09412 [CS, STAT] 2018.
Lin, Z.; Sun, J.; Davis, A.; Snavely, N. Visual Chirality. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2020, 12292-12300. Doi.org/10.1109/CVP… .
Redmon, J.; Farhadi, A. YOLOv3: An Incremental Improvement. 2018.
Zheng, Z.; Wang, P.; Liu, W.; Li, J.; Ye, R.; Ren, D. Distance-iou Loss: Faster and Better Learning for Bounding Box Regression. ArXiv :1911.08287 [CS] 2019.
Yamauchi, Brian. “Frontier-based exploration using multiple robots.” Proceedings of the second international conference on Autonomous agents. 1998.
Lopez Perez, Jose J., et al. “Distributed multirobot exploration based on scene partitioning and frontier selection.” Mathematical Problems in Engineering 2018 (2018).
Click to follow, the first time to learn about Huawei cloud fresh technology ~