The OpenMMLab framework covers almost all aspects of deep learning visual tasks. For each specific task, we provide a corresponding algorithm library, such as MMClassification for task classification, MMDetection for any task, MMSegmentation for task segmentation and so on.

A subsequent question is, if I want to apply the research method to multiple tasks and evaluate it on multiple tasks, do I have to fork each library? Further, what if I want to open source my code that relies on the OpenMMLab framework?

Here, we introduce how to develop a new backbone network quickly, and use MMCLS, MMDET and MMSEG to easily evaluate the classification, detection and segmentation tasks.

To help you get started, we provide a sample repository: github.com/mzr1996/bac…

In the sample repository, we used ConvNeXt as an example to show the files needed to set up a backbone network and benchmark it. So, let’s use this example repository as an example to learn step by step how to build your own backbone network.

Configure the environment

First, we need to configure Python and the PyTorch environment, which I’m sure those of you who are familiar with PyTorch already have. Note that we require Python >= 3.6, PyTorch >= 1.5.0, and newer versions are of course fine.

In addition, we need the following OpenMMLab repositories:

  • MIM: A command-line tool for managing OpenMMLab packages and experiments
  • MMCV: OpenMMLab Computer vision foundation library
  • MMClassification: OpenMMLab Image classification toolbox and benchmarks. In addition to sorting tasks, it is also used to provide a diverse backbone network
  • MMDetection: OpenMMLab instrumentation toolkit and benchmarks
  • MMSegmentation: OpenMMLab Semantic segmentation toolkit and benchmarks

Assuming you have the Python and PyTorch environments ready, you can configure the software environment with the following two commands.

pip install openmim mmcls mmdet mmsegmentation 
mim install mmcv-full 
Copy the code

Data preparation

Suppose we want to benchmark the classification task on ImageNet, the semantic segmentation task on ADE20K, and the detection task on COCO, then we need to organize the data files in the following format.

The data / ├ ─ ─ imagenet │ ├ ─ ─ "train" │ ├ ─ ─ val │ └ ─ ─ meta │ ├ ─ ─ "train". TXT │ └ ─ ─ val. TXT ├ ─ ─ ade │ └ ─ ─ ADEChallengeData2016 │ ├ ─ ─ annotations │ └ ─ ─ images └ ─ ─ coco ├ ─ ─ annotations │ ├ ─ ─ instance_train2017. Json │ └ ─ ─ instance_val2017. Json ├ ─ ─ Train2017 └ ─ ─ val2017Copy the code

Here, we list only the necessary files for training and validation. If you want to benchmark on more data sets or tasks, such as panorama segmentation using MMDetection, you can simply organize the corresponding data sets according to MMDetection’s needs. In addition, the MMSegmentation provides a tutorial on how to organize semantically segmented related data sets.

Implement your backbone network

With everything ready, let’s start implementing and training our backbone network step by step

The first is implementation:

1. Create your trunk network file in the Models folder. For example, models/convnext. Py in the sample repository. In this file, you only need to implement your trunk network using PyTorch, with the following two additional points to note:

  • The backbone network and major modules need to be inheritedmmcv.runner.BaseModule. thisBaseModule 是 torch.nn.ModuleA subclass of, whose behavior is almost identical, except that it supports additional useinit_cfgParameter specifies the initialization method including the pretrained model.
  • Use the following decorator structure to register your trunk network class tommcls.models.BACKBONESRegistry.
from mmcv.runner import BaseModule 
from mmcls.models import BACKBONES 
 
@BACKBONES.register_module(force=True) 
class MyBackbone(BaseModule): 
    ... 
Copy the code

What is a registry? Look at this!

2.[Optional] If you want to add additional modules and components for a particular task, refer to models/det/layer_decay_optimizer_constructor.py to add them to the appropriate folder.

3. Add the trunk network and custom components you added to the models/__init__.py file.

Adding a Configuration File

If you’re not quite sure what configuration files are, this tutorial should help you.

In general, we use several base profiles to organize an experimental profile by inheritance. These base profiles typically include models, data sets, optimization policies, and operational configurations. You can also overwrite the base configuration file in the configuration file, or write it all into one configuration file without using the base configuration file.

In this sample library, we provide a set of common base configuration files. You can find more useful base configuration files by accessing the following links: MMClassification, MMDetection, MMSegmentation.

For each different task, the corresponding configuration files can be placed in different subfolders in the Configs directory. Note that in the Model section of the configuration file, in order to enable MMDetection and MMSegmentation to call the backbone network registered with MMCLs.Models.BACKBONES, We need to specify the location of the backbone network registry in additional configuration files for DET and SEG by adding MMCLS to Type. A prefix, such as:

model = dict( backbone=dict(type='mmcls.MyBackbone', ...) ,...).Copy the code

To learn more about cross-repository invocation modules, see “How to Open configuration files using MMDet Backbone?”

Training and testing

Relying on MIM, the unified experiment management tool provided by OpenMMLab, we can train and test different tasks without writing any Python scripts after we have implemented the model and written the configuration files.

First, we need to add the directory of the current repository to the PYTHONPATH environment variable, so Python can find our model file and run the following command in the repository root:

export PYTHONPATH=`pwd`:$PYTHONPATH  
Copy the code

Single machine single GPU

Check MMCLS $CONFIG -- WORK_DIR $WORK_DIR -- check MMCLS $CONFIG -c $CHECKPOINT --metrics accuracy --metric-options "topk=(1, 5)" mim train mmdet $CONFIG --work-dir $WORK_DIR "mim test mmdet $CONFIG -c $CHECKPOINT --eval bbox mim train mmseg $CONFIG -- WORK_DIR $WORK_DIR # test mmseg $config-c $CHECKPOINT --eval mIoUCopy the code

Description of some parameters:

  • $CONFIG:configs/Configuration files in folders.
  • $WORK_DIR: folder for storing log and weight files
  • $CHECKPOINT: Weight file path

Single-node multiple Gpus (With four Gpus as an example)

Mim train MMCLS $CONFIG --work-dir $WORK_DIR --launcher Pytorch --gpus $CHECKPOINT --metrics accuracy --metric-options "topk=(1, Mim train mmdet $CONFIG --work-dir $WORK_DIR --launcher Pytorch --gpus 4)" --launcher Pytorch --gpus 4 Test mmdet $CONFIG -c $CHECKPOINT --eval bbox seGM --launcher Pytorch --gpus 4 Mmseg $CONFIG --work-dir $WORK_DIR --launcher Pytorch --gpus 4 # mim test mmseg $CONFIG -c $CHECKPOINT --eval mIoU --launcher pytorch --gpus 4Copy the code

conclusion

OpenMMLab has been dedicated to providing you with a toolbox for visual deep learning tasks. At the same time, the libraries for various tasks are organic and unified. This article introduces a practical way to use multiple libraries together. You are welcome to download and try out our sample repository. In addition, we will also be available at github.com/open-mmlab/… Examples of how to use MIM to accomplish various tasks are constantly updated