When you hear “search by map”, do you think of baidu, Google and other search engines search by map function? In fact, you can build a picture search system of their own: their own photo library; Select a picture to search in the library, and get a number of similar pictures.

Milvus, as a similarity search engine for massive feature vectors, aims to help analyze the increasingly large amount of unstructured data and explore the huge value behind it. In order to enable Milvus to be applied to similar image retrieval scenarios, we designed an image search system based on Milvus and image feature extraction model VGG.

The text is divided into seven parts: data preparation, system overview, VGG model, API introduction, image construction, system deployment, and interface display. This chapter describes the data support of the graph search system. The system Overview section shows the overall architecture of the system. The chapter of VGG model introduces the structure, characteristics, block structure and weight parameters of VGG. This chapter describes the working principles of the five basic functions of the SYSTEM. The image Building section describes how to build docker images on both client and server sides from source code. The system Deployment chapter shows how to set up a system in three steps. The page display section shows the search page of the system.

1. Data preparation

This paper uses PASCAL VOC photo collection as an example to build an end-to-end solution to search images by images. The photo collection contains 17,125 images, covering 20 categories: human; Animals (birds, cats, cows, dogs, horses, sheep); Means of transportation (airplane, bicycle, boat, bus, car, motorcycle, train); Indoor (bottles, chairs, dining table, potted plants, sofa, TV).
Data set size: ~2GB
Download address: http://host.robots.ox.ac.uk/pascal/VOC/voc2012/VOCtrainval_11-May-2012.tar
Note: You can also use other image data to load. The supported image formats are. JPG and. PNG.


2. System Overview

In order to allow users to interact on web pages, we adopt a C/S architecture. The WebClient receives the user’s request and sends the request to the WebServer. After receiving the HTTP request, the WebServer performs the calculation and returns the calculation result to the WebClient.
Webserver is mainly composed of two parts, image feature extraction model VGG and vector search engine Milvus. VGG model is responsible for converting images into vectors, while Milvus is responsible for storing vectors and retrieving similar vectors. The architecture of webServer is as follows:

3. VGG model

VGGNet, developed by researchers from the Visual Geometry Group at the University of Oxford and Google DeepMind, is the first location task and second classification task in ILSVRC-2014. Its outstanding contribution is to prove that the effect of the model can be effectively improved by increasing the network depth with small convolution (3*3), and VGGNet has good generalization ability for other data sets. VGG model performs better than GoogleNet in multiple transfer learning tasks, and VGG model is the preferred algorithm for extracting CNN features from images. Therefore, VGG is selected as the deep learning model in this scheme.

VGGNet explored the relationship between the depth of CNN and its performance. By repeatedly stacking 3*3 small convolutional kernels and 2*2 maximum pooling layer, VGGNet successfully constructed CNN with 16-19 layers of depth. In this scheme, the VGG16 model provided by Keras application module (Keras.Applications) is used.

(1) VGG16 structure

VGG16 contains 13 Convolutional layers, 3 Fully connected layers, and 5 Pool layers. Among them, the convolutional layer and the fully connected layer have weight coefficients, so they are also called weight layers. The total number is 13+3=16, which is the source of 16 in VGG16. (The pooling layer does not involve weights and therefore does not belong to the weight layer and is not counted).

(2) VGG16 characteristics

  • The convolution layer adopts the same convolution kernel parameters
  • The same pooling kernel parameters are used for all pooling layers
  • The model is composed of several convolutional layers and pooling layers stacked (stack), which is easy to form a deep network structure

(3) VGG16 block structure

The convolutional layer and pooling layer of VGG16 can be divided into different blocks, which are numbered from front to back from Block1 to Block5. Each block contains several convolution layers and a pooling layer. For example, Block2 contains two convolutional layers (conv3-256) and one pooling layer (maxPool). And in the same block, the number of channels in the convolution layer is the same.
According to the structure diagram of VGG16 shown below, the input image of VGG16 is 224x224X3. During the process, the number of channels doubles from 64 to 128 and then to 256, until 512 remains unchanged and no longer doubles. The height and width were halved from 224→112→56→28→14→7.

(4) Weight parameters

VGG has a simple structure, but it contains a large number of weights, up to 139,357,544 parameters. These parameters include
Convolution kernel weightand
Full connection layer weight. Therefore, it has high fitting ability.

4. The API is introduced

The webserver of the whole system provides five apis: Train, process, count, search and delete. Users can load images, query the loading progress, query the number of Milvus vectors, search images and delete Milvus tables. These five apis cover all the basic functions of the graph search system, each of which is described below.

(1) train

The parameters of the train API are shown in the following table:
methods name type
POST File string
Before similar image retrieval, the image library needs to be loaded into Milvus. At this time, train API is called to pass the path of the image into the system. Because Milvus only supports the retrieval of vector data, images need to be transformed into feature vectors. The transformation process is mainly realized by calling VGG model in Python:


from preprocessor.vggnet import VGGNetnorm_feat = model.vgg_extract_feat(img_path)Copy the code
After obtaining the feature vectors of the image, import these vectors into Milvus using the interface insert_Vectors of Milvus: 
from indexer.index import milvus_client, insert_vectorsstatus, ids = insert_vectors(index_client, table_name, vectors)Copy the code
After importing these feature vectors into Milvus, Milvus will assign a unique ID to each vector. In order to facilitate the search of its corresponding image according to the id of feature vector in the later retrieval, the relationship between the ID of each feature vector and its corresponding image needs to be saved: 
from diskcache import Cachefor i in range(len(names)):    cache[ids[i]] = names[i]Copy the code
When the train API is called, the image is converted into a vector and stored in Milvus through the above three steps.

(2) process

The process API’s methods are GET, and no additional parameters need to be passed in when called. The Process API can view the progress of image loading. When called, it will see the number of images that have been loaded and transformed and the total number of images that have been passed in to the path.

(3) count

The count API’s methods are POST, and no additional arguments need to be passed in when called. The count API allows you to view the total number of vectors in Milvus, each converted from an image.

(4) search

The parameters of the Search API are shown in the following table:
methods Num file
POST topk (int) image file
The Search API is called when you have selected an image to search for similar images. When the image to be searched is passed into the system, the VGG model is called first to transform the image into a vector: 
from preprocessor.vggnet import VGGNetnorm_feat = model.vgg_extract_feat(img_path)Copy the code
After obtaining the vector of the image to be searched, call Milvus’s search_Vectors interface for similar vector retrieval: 
from milvus import Milvus, IndexType, MetricType, Statusstatus, results = client.search_vectors(table_name=table_name, query_records=vectors, top_k=top_k, nprobe=16)Copy the code
After the vector ID similar to the target vector is searched, the corresponding image name is retrieved according to the corresponding relationship between the previously stored vector ID and the image name: 
from diskcache import Cachedef query_name_from_ids(vids):    res = []    cache = Cache(default_cache_dir)    for i in vids:        if i in cache:            res.append(cache[i])    return resCopy the code
When you call the Search API, you can search for images that are similar to the target image by following the above three steps.

(5) delete

The delete API’s methods are POST, and no additional arguments are required to call. The delete API will delete the table in Milvus, emptying previously imported vector data.


5. Build an image

(1) Construct pic-Search-WebServer image

First pull the Milvus Bootcamp code, then build the image using the Dockerfile we provide: 
$ git clone https://github.com/milvus-io/bootcamp.git$ cd bootcamp/solutions/pic_search/webserver# build mirror $docker build -t PIC - search - webserver. # | view image generated $docker images - search - webserver grep PICCopy the code
The docker image of webserver can be constructed through the above steps. Of course, you can also use the image we uploaded to DockerHub directly: 
$docker pull milvusbootcamp/PIC - search - webserver: 0.1.0 fromCopy the code

(2) Build pic-Search-WebClient image

First pull the Milvus Bootcamp code, then build the image using the Dockerfile we provide: 
$ git clone https://github.com/milvus-io/bootcamp.git$ cd bootcamp/solutions/pic_search/webclient# build mirror $docker build -t PIC - search - webclient. # | view image generated $docker images - search - webclient grep PICCopy the code
Through the above steps, the Docker image of WebClient can be constructed. Of course, you can also use the image we uploaded to DockerHub directly: 
$docker pull milvusbootcamp/PIC - search - webclient: 0.1.0 fromCopy the code


6. Deploy the system

We provide a GPU deployment scheme and a CPU deployment scheme for users to choose. A detailed deployment process can be found at:
https://github.com/milvus-io/bootcamp/blob/0.6.0/solutions/pic_search/README.md

Step 1 Start Milvus Docker

For details, please refer to the link:
https://milvus.io/cn/docs/v0.6.0/guides/get_started/install_milvus/install_milvus.md

Step 2 Start pic-search-WebServer Docker

$ docker run -d --name zilliz_search_images_demo \-v IMAGE_PATH1:/tmp/pic1 \-v IMAGE_PATH2:/tmp/pic2 \-p 35000:5000 \-e "DATA_PATH=/tmp/images-data" \-e "MILVUS_HOST = 192.168.1.123"\ milvusbootcamp/PIC - search - webserver: 0.1.0 fromCopy the code

Step 3 Start pic-search-WebClient Docker

$ docker run --name zilliz_search_images_demo_web \-d --rm -p 8001:80 \-eAPI_URL = http://192.168.1.123:35000 \ milvusbootcamp/PIC - search - webclient: 0.1.0 fromCopy the code
The whole map search system can be deployed in just three steps.


7. Interface display

After the deployment is complete, enter localhost:8001 in the browser to access the image search page.
Fill the picture path in the path box for loading, and wait for all the pictures to be converted into vectors and loaded to Milvus before image retrieval:



conclusion


This paper uses Milvus and VGG to build a graph search system, showing the application of Milvus in unstructured data processing. Milvus vector similarity search engine is compatible with various deep learning platforms, searching a billion vectors in milliseconds response. You can explore more AI usage with Milvus!
For more information about the VGG model, visit:
VGG official website:
http://www.robots.ox.ac.uk/~vgg/research/very_deep/
VGG lot:
https://github.com/machrisaa/tensorflow-vgg


Welcome to the Milvus community

Milvus source


github.com/milvus-io/milvus
Milvus website


milvus.io
Milvus Slack community


milvusio.slack.com




Milvus online communication group
Follow the ZILLIZ account
= >Click on the menu
“Online communication” =>Add ZILLIZ to the group