Project background and topic introduction

Satellites are so widely used that many industries rely on them every day. We generally categorize the orbits of man-made earth satellites into low orbit (200 to 2,000 km), medium orbit (2,000 to 20,000 km) and high orbit (more than 20,000 km) according to altitude.

The geostationary orbit belongs to the high orbit, which is 0 degrees Angle from the equatorial plane. The geostationary orbit satellite moves in the same direction as the earth’s rotation and runs in the same period as the earth, keeping relatively stationary with us on the earth. It runs large communication satellites of many countries. Limited geostationary orbit resources are precious. Therefore, in order to avoid possible collisions of satellites in operation, timely detection, tracking, early warning and cataloguing of unknown space objects such as space debris and natural objects in operation space is an important task for “space protection soldiers” to ensure the normal and healthy activities of satellites.

The project is based on the SpotGEO Challenge in geostationary orbit (GEOstationary Orbit) and nearby target detection competition organized by ESA in collaboration with the University of Adelaide in 2020 to realize the “Space Guard Soldier” – space space target detection system to detect faint space targets in images collected by low-cost telescopes. Including space debris, non-cooperative aircraft, and so on.

The project link: aistudio.baidu.com/aistudio/pr…

Introduction to data set

At present, the international space target detection and recognition system generally includes two parts: ground-based system and space-based system, and observation equipment includes sky survey telescope and CCD camera. The data set presented in this competition was captured at night by a low-cost CMOS camera mounted on a ground-based telescope with an exposure time of 40s and a sequence of 5 consecutive frames.

Schematic diagram of ground-based telescope data acquisition

Background stars other than targets or bodies in geostationary orbit have relative velocities with Earth during the exposure time of the camera, so the background stars appear striped in the image. While GEO targets remain relatively static and appear mostly in the form of spots or short stripes in the image. During the shooting process of frame sequence, the telescope where the camera is located rotates uniformly, so the movement track of space target is straight line.

Sample dataset

This is the structure of the data set provided by the competition. The decompressed data set includes two folders train and test and the corresponding two annotation files in the same directory. The training set and test set respectively contain a two-level directory composed of several sequences and the frame images that comprise the sequence. The image format is PNG, and the size is 640×480. The total number of images in the whole data set is 32000, among which the training set contains 1280 sequences 6400, and the test set contains 5120 sequences 25600.

Annotation file format: Each image annotation includes four groups of values, namely, frame serial number, frame image serial number, target number and corresponding position coordinates (X, Y).

Sample data set composition

Target characteristics

The problem we need to solve is further concretized, summed up is sequence image small target detection under the starry sky background.

Target characteristics: In the image, the target we pay attention to appears mostly in the form of spots or short stripes, rather than completely in the form of point targets. The overall brightness of the target is also dark, which is due to the pixel dispersion phenomenon caused by longer exposure time, atmospheric distortion, sensor defects and other reasons. In addition, conditions such as cloud cover, atmospheric/weather effects, light pollution, stellar occlusion (background stars that happen to cross the orbit of an object) add to the problem. There are some problems in the data set, such as abnormally highlighted pseudo objects, or the target objects only appear in a few frames of the 5 frames in the sequence.

Target motion characteristics: We use the annotation information to match the target position of 5 frames of images in a sequence into a picture, presenting an obvious uniform linear motion track.

Problem resolution

problem

Combined with the target characteristics, the position of the annotation point in the data set is regarded as the central position of the BBox annotation box, and the width of the left, right and left of the central point is 3 pixels to obtain the target annotation box. The coordinates of the upper left corner and the coordinate values of the lower right corner are calculated according to the length and width offset, thus transforming the problem into a target detection problem with class number 2.

With graphic

Overall development process

PaddleX is used in this project. The whole project mainly includes three parts: data preprocessing, model training and export, and model deployment.

The development process

The data preprocessing process mainly includes:

1. Clean data

Firstly, the frame sequence without any target is deleted.

Import json def read_annotation_file(path): annotation_list = json.load(open(path)) # Transform list of annotations into dictionary annotation_dict = {} for annotation in annotation_list: sequence_id = annotation['sequence_id'] if sequence_id not in annotation_dict: annotation_dict[sequence_id] = {} annotation_dict[sequence_id][annotation['frame']] = annotation['object_coords'] #just pull out the object_coords return annotation_dict anopath = 'work/train_anno.json' train_annotation= Read_annotation_file (Anopath) ori_seq = len(train_annotation) print('\n Number of original sequences: For I in range(1, ori_seq): if len(train_annotation[I][1]) == 0: Del train_annotation[I] # train_annotation is data numpy real_seq = len(train_annotation) print('\n valid number of sequences: ', real_seq)Copy the code

2. Data set annotation format conversion

To facilitate the use of the PaddleX data interface, the original annotation file was converted back to VOC format

from pycocotools.coco import COCO import os, cv2, shutil from lxml import etree, objectify from tqdm import tqdm from PIL import Image CKimg_dir = './SpotGEOvoc/VOCImages' CKanno_dir = './SpotGEOvoc/VOCAnnotations' def catid2name(coco): Dict () for cat in coke. dataset['categories']: classes[cat['id']] = cat['name'] # print(str(cat['id'])+":"+cat['name']) return classes def get_CK5(origin_anno_dir, origin_image_dir, verbose=False): dataTypes = ['train'] for dataType in dataTypes: annFile = '{}_anno.json'.format(dataType) annpath = os.path.join(origin_anno_dir, annFile) print(annpath) coco = COCO(annpath) classes = catid2name(coco) imgIds = coco.getImgIds() # ImgIds =imgIds[0:1000]# for imqdm in TQDM (imgIds) img = coco.loadImgs(imgId)[0] showbycv(coco, dataType, img, classes, origin_image_dir, verbose=False) def main(): Base_dir = './SpotGEOvoc' # step1 Image_dir = os.path.join(base_dir, 'VOCImages') # Create images in the above folder, Annotations Two subfolders anno_dir = os.path.join(base_dir, 'VOCAnnotations') MKR (image_dir) MKR (anno_dir) origin_image_dir = './SpotGEOv2' # Step 2 SpotGEOv2 = './SpotGEOv2' # step 3 Get_CK5 (origin_anno_dir, origin_image_dir, verbose)Copy the code

3. Divide the original training data into training set and verification set

! Paddlex --split_dataset --format VOC -- dataset_DIR MyDataset --val_value 0.2 --test_value 0Copy the code

For model selection and development: Because the project is ultimately deployed end-to-end, a lightweight and efficient inspection model is preferred. Here, I use the characteristic flying paddle model PP-YOLOv2, which is optimized, improved and upgraded by Baidu flying paddle team based on THE PP-YOLO model. The network structure is shown in the figure below. Baseline Model is PP-YOLO, backbone network is RESnet50-VD, Combination add Optimization components include Deformable Conv, SSLD, CoordConv, DropBlock, SPP, Larger Batch Size, EMA, IoU Loss, IoU Aware, and Grid Sensitive 10 Tricks

The training body of the model just needs this code, configure the format and path of the dataset, and then modify the num_classes parameter. The number of classes is minus the background classes, which is the total number of classes -1. Adjust and set a series of training parameters, and then start the model training.

num_classes = 1 model = pdx.det.PPYOLOv2(num_classes=num_classes, backbone='ResNet50_vd_dcn') model.train( num_epochs=3600, train_dataset=train_dataset, train_batch_size=4, Eval_dataset = eval_DATASET, Learning_rate = 0.001/8, WARMup_steps =1000, WARMup_START_LR =0.0, save_interval_epochs=36, lr_decay_epochs=[216, 243], save_dir='output/PPyolov2_r50vd_dcn')Copy the code

Then wait for model training and evaluation results before model export and deployment.

! paddlex --export_inference --model_dir=/home/aistudio/output/PPyolov2_r50vd_dcn/best_model --save_dir=/home/aistudio/inferenceCopy the code

Finally, a practical demonstration system was implemented based on Nvidia Jetson Nano platform.

The whole process from preliminary preparation to inference program verification for the deployment of the flying propeller model on the Jetson NANO development board mainly includes the following parts.

Software part, side – to – side reasoning program is mainly the following three parts, have to praise API again is really sweet!

The specific reasoning code is as follows:

Import glob import numpy as NP import threading import time import random import OS import base64 import cv2 import json import paddlex as pdx os.environ['CUDA_VISIBLE_DEVICES']='0' predictor = pdx.deploy.Predictor(model_dir='./infer', use_gpu=True, gpu_id=0, use_trt=True) def get_images(image_path, support_ext=".jpg|.jpeg|.png"): if not os.path.exists(image_path): raise Exception(f"Image path {image_path} invalid") if os.path.isfile(image_path): return [image_path] imgs = [] for item in os.listdir(image_path): ext = os.path.splitext(item)[1][1:].strip().lower() if (len(ext) > 0 and ext in support_ext): item_path = os.path.join(image_path, item) imgs.append(item_path) return imgs def crest_dir_not_exist(path): if not os.path.exists(path): os.mkdir(path) def run(img_path,img_name,save_path): result = predictor.predict(img_file=img_name, warmup_iters=100, Repeats =100) time2= time.time() pdx.det. endeavour (img_name, result, threshold=0.5, save_dir=save_path) time3 = time.time() print("Visual Time: {}s".format(time3-time2)) if __name__ == "__main__": test_path = 'visual/' save_path = 'output/visual/' crest_dir_not_exist(save_path) L = get_images(test_path) N = len(L) print(N) for i in range(N): print(L[i]) run(test_path, L[i],save_path)Copy the code

The actual resulting demo GIF is shown below.

The above is my share, interested partners can click on the project link below, exchange and learn together.

aistudio.baidu.com/aist