Tag images in Jupyter Notebook, predict new images and visualize neural networks (and share them using Docker Hub!).

Author | Jenny Huang compile | Flin source | towardsdatascience

Authors: Jenny Huang, Ian Hunt-Isaak, William Palmer

Making Repo: github.com/ianhi/AC295…

introduce

Training an image segmentation model on a new image can be daunting, especially if you need to tag your own data. To make this task easier and faster, we’ve built a user-friendly tool that lets you build the entire process in Jupyter Notebook. In the following sections, we will show you how our tools enable you:

  1. Manually mark your own image

  2. An effective segmentation model is established by transfer learning

  3. Visual model and results

  4. Share your project as a Docker image

The main advantages of the tool are that it is easy to use and well integrated with existing data science workflows. Through interactive widgets and command prompts, we built a user-friendly way to tag images and train models.

Best of all, everything can be run on Jupyter Notebook, making it quick and easy to model without much overhead.

Finally, by working in a Python environment and using standard libraries such as Tensorflow and Matplotlib, you can integrate the tool nicely into existing data science workflows, making it ideal for purposes such as scientific research.

In microbiology, for example, segmentation of microscopic images of cells is very useful. However, tracking cells over time is likely to require segmentation of hundreds or thousands of images, which can be difficult to do manually. In this paper, we will use microscopic images of yeast cells as a data set and show how we built tools to distinguish between background, mother and daughter cells.

1. The label

There are many tools available to create labeled masks for images, including Labelme, ImageJ and even the graphics editor GIMP. These are great tools, but they are not integrated into Jupyter Notebook, which makes them difficult to use with many existing workflows.

  • Labelme:github.com/wkentaro/la…
  • ImageJ：imagej.net/Welcome
  • GIMP：www.gimp.org/

Fortunately, Jupyter Widgets make it easy to make interactive components and connect them to the rest of our Python code.

  • Jupyter Widgets: ipywidgets. Readthedocs. IO/en/latest /

To create a training mask in a notebook, we solve two problems:

  1. Select part of the image with the mouse

  2. Easily switch between images and select categories to mark

To solve the first problem, we used the Matplotlib widget back end and the built-in LassoSelector.

  • LassoSelector:matplotlib.org/3.1.1/galle…

LassoSelector processes a line to display your selection, but we need some custom code to draw the mask as an overlay:

import numpy as np
import matplotlib.pyplot as plt
from matplotlib.widgets import LassoSelector
from matplotlib.path import Path

class image_lasso_selector:
    def __init__(self, img, mask_alpha=75., figsize=(10.10)) :
        """ img must have shape (X, Y, 3) """
        self.img = img
        self.mask_alpha = mask_alpha
        plt.ioff() # see https://github.com/matplotlib/matplotlib/issues/17013
        self.fig = plt.figure(figsize=figsize)
        self.ax = self.fig.gca()
        self.displayed = self.ax.imshow(img)
        plt.ion()
        
        lineprops = {'color': 'black'.'linewidth': 1.'alpha': 0.8}
        self.lasso = LassoSelector(self.ax, self.onselect,lineprops=lineprops, useblit=False)
        self.lasso.set_visible(True)
        
        pix_x = np.arange(self.img.shape[0])
        pix_y = np.arange(self.img.shape[1])
        xv, yv = np.meshgrid(pix_y,pix_x)
        self.pix = np.vstack( (xv.flatten(), yv.flatten()) ).T
        
        self.mask = np.zeros(self.img.shape[:2])
        
    def onselect(self, verts) :
        self.verts = verts
        p = Path(verts)
        self.indices = p.contains_points(self.pix, radius=0).reshape(self.mask.shape)
        self.draw_with_mask()
        
    def draw_with_mask(self) :
        array = self.displayed.get_array().data

        # https://en.wikipedia.org/wiki/Alpha_compositing#Straight_versus_premultiplied     
        self.mask[self.indices] = 1
        c_overlay = self.mask[self.indices][...,None] * [1..0.0]*self.mask_alpha
        array[self.indices] = (c_overlay + self.img[self.indices]*(1-self.mask_alpha))

        self.displayed.set_data(array)
        self.fig.canvas.draw_idle()
        
    def _ipython_display_(self) :
        display(self.fig.canvas)
Copy the code

For the second question, we used ipyWidgets to add nice buttons and other controls:

We combined these elements (along with improvements like scrolling and scaling) to make a label controller object. Now, we can take microscopic images of yeast and segment mother and daughter cells:

Lasso selection image tag demo: Youtu.be /aYb17GueVcU

You can view the full object, which allows you to scroll, zoom, right-click to pan, and here (github.com/ianhi/AC295… Select multiple classes.

Now we can mark a few images on the notebook, save them to the correct folder structure, and start training CNN!

2. Model training

model

U-net is a convolutional neural network originally designed for segmentation of biomedical images, but has been successfully used for many other types of images. Based on existing convolutional networks, it can work better with fewer training images and perform more accurate segmentation. This is a fairly recent model and is also easy to implement using the Segmentation_Models library.

  • Segmentation_models library: github.com/qubvel/segm…

U-net is unique in that it combines encoders and decoders through cross-linking (gray arrows in the image above). These skip connections span the same size portion of the lower sampling path to the upper sampling path. This improves your understanding of the raw pixels that are input into the model when upsampling, which has been shown to improve the performance of segmentation tasks.

As great as U-Net is, it won’t work if we don’t give it enough training examples. Considering the tedious work of manual image segmentation, we only marked 13 images manually. It is impossible to train a neural network with millions of parameters with only a few training examples. To overcome this problem, we need both data augmentation and transfer learning.

Data expansion

Naturally, if your model has many parameters, you need proportional training examples to get good performance. Using small datasets of images and masks, we can create new images that are just as insightful and useful to the model as the original.

What do we do? We can flip the image, rotate the Angle, zoom in or out, crop, pan, and even blur the image by adding noise, but most importantly, we can combine these operations to create many new training examples.

Compared with classification, image data enhancement has more complexity in segmentation. For classification, you just need to zoom in on the image because the label will remain the same (0 or 1 or 2…). . However, for segmentation, you also need to synchronize the conversion label with the image (as a mask). To do this, we use the Albumentations library with the custom data generator because, as far as we know, the Keras ImageDataGenerator does not currently support the combination “Image+ Mask.”

import albumentations as A
import numpy as np
from tensorflow.keras.preprocessing.image import ImageDataGenerator

TARGET_SIZE = (224.224)
BATCH_SIZE = 6

def create_augmentation_pipeline() :
    augmentation_pipeline = A.Compose(
    [
        A.HorizontalFlip(p = 0.5), # Apply horizontal flip to 50% of images
        A.OneOf(
            [
                # Apply one of transforms to 50% of images
                A.RandomContrast(), # Apply random contrast
                A.RandomGamma(), # Apply random gamma
                A.RandomBrightness(), # Apply random brightness
            ],
            p = 0.5 
        ),
        A.OneOf(
            [
                # Apply one of transforms to 50% images
                A.ElasticTransform(
                    alpha = 120,
                    sigma = 120 * 0.05,
                    alpha_affine = 120 * 0.03
                ),
                A.GridDistortion()
            ],
            p = 0.5 
        )
    ],
    p = 0.9 # In 10% of cases keep same image because that's interesting also
    )   
    return augmentation_pipeline
  
def create_datagenerator(PATH) :
    options = {'horizontal_flip': True.'vertical_flip': True}
    image_datagen = ImageDataGenerator(rescale=1./255, **options)
    mask_datagen = ImageDataGenerator(**options)
    val_datagen = ImageDataGenerator(rescale=1./255)
    val_datagen_mask = ImageDataGenerator(rescale=1)
    
    # Create custom zip and custom batch_size
    def combine_generator(gen1, gen2, batch_size=6,training=True) :
        while True:
            image_batch, label_batch = next(gen1)[0], np.expand_dims(next(gen2)[0] [:, :,0],axis=-1)
            image_batch, label_batch = np.expand_dims(image_batch,axis=0), np.expand_dims(label_batch,axis=0)

            for i in range(batch_size - 1):
                image_i,label_i = next(gen1)[0], np.expand_dims(next(gen2)[0] [:, :,0],axis=-1)
              
                if training == True:
                    aug_pipeline = create_augmentation_pipeline()
                    augmented = aug_pipeline(image=image_i, mask=label_i)
                    image_i, label_i = augmented['image'], augmented['mask']

                image_i, label_i = np.expand_dims(image_i,axis=0),np.expand_dims(label_i,axis=0)
                image_batch = np.concatenate([image_batch,image_i],axis=0)
                label_batch = np.concatenate([label_batch,label_i],axis=0)
              
            yield((image_batch,label_batch))


    seed = np.random.randint(0.1e5)

    train_image_generator = image_datagen.flow_from_directory(PATH+'train_imgs', seed=seed, target_size=TARGET_SIZE, class_mode=None, batch_size=BATCH_SIZE)
    train_mask_generator = mask_datagen.flow_from_directory(PATH+'train_masks', seed=seed, target_size=TARGET_SIZE, class_mode=None, batch_size=BATCH_SIZE)
    train_generator = combine_generator(train_image_generator, train_mask_generator,training=True)
    
    val_image_generator = val_datagen.flow_from_directory(PATH+'val_imgs', seed=seed, target_size=TARGET_SIZE, class_mode=None, batch_size=BATCH_SIZE)
    val_mask_generator = val_datagen_mask.flow_from_directory(PATH+'val_masks', seed=seed, target_size=TARGET_SIZE, class_mode=None, batch_size=BATCH_SIZE)
    val_generator = combine_generator(val_image_generator, val_mask_generator,training=False)
    
    return train_generator, val_generator
Copy the code

The migration study

Even though we have now created 100 or more images, this is still not enough because the U-Net model has over 6 million parameters. This is where transfer learning comes into play.

Transfer learning allows you to train a model on one task and reuse it for another similar task. It dramatically reduces your training time and, more importantly, it can produce effective models even for small workouts like ours. For example, neural networks such as MobileNet, Inception, and DeepNet learn feature Spaces, shapes, colors, textures, and so on by training large numbers of images. We can then pass on what we have learned by taking these model weights and modifying them slightly to activate patterns in our own training images.

Now how do we use u-NET’s transfer learning? We use the Segmentation_Models library to do this. We use the layers of your chosen deep neural network (MobileNet, Inception, ResNet) and the parameters found on image classification (ImageNet) and use them as the first half of the U-Net (encoder). You then train the decoder layer with your own extended data set.

finishing

We put all of this into the Segmentation model class, which you can find here.

  • Github.com/ianhi/AC295…

When creating a model object, an interactive command prompt appears where you can customize various aspects of U-NET, such as loss functions, backbone, and so on:

vimeo.com/419423808

After 30 weeks of training, we achieved 95% accuracy. Note that it is important to choose a good loss function. We first tried cross entropy loss, but the model could not distinguish between mother cells and daughter cells with similar appearance, and the performance of the model was poor due to the class imbalance of non-yeast pixels more than yeast pixels.

We found that Dice Loss can achieve better results. Dice Loss is associated with IOU and is generally better suited for segmentation tasks because it can narrow the gap between predicted and true values.

3. The visualization

Now that our model is trained, let’s use some visualization techniques to see how it works.

We follow Ankit Paliwal’s tutorial. You can find the implementation in its corresponding GitHub repository. In this section, we will visualize two of his techniques on yeast cell segmentation models, namely, heat maps of interlayer activation and quasi-activation.

  • Refer to the tutorial: towardsdatascience.com/understandi…
  • GitHub Repository: github.com/anktplwl91/…

Mesosphere activation

The first technique displays the output of the middle layer as the network propagates forward on the test image. This allows us to see which features of the input image are highlighted at each layer. After inputting test images, we visualized the first few outputs of some convolutional layers in the network:

In the encoder layer, it should be expected that filters closer to the inputs detect more detail, while filters closer to the model outputs detect more general features. In the decoder layer, we see the opposite pattern, moving from abstraction to more concrete detail, which is not surprising.

Class active heat map

Next, let’s look at the class activation diagram. These heat maps give you an idea of how important each position in the image is to predict the output category. Here, we visualize the last layer of the yeast cell model because the class prediction tag depends heavily on it.

The heat map showed that the cell locations, and some of the image boundaries, were activated correctly, which was somewhat surprising.

We also looked at the last technique in this tutorial, which shows which images each convolution filter responds to to the greatest extent, but the visualizations weren’t very useful for our particular yeast cell model.

4. Create and share Docker images

It can be very frustrating to find a great model and try to run it, only to find that it doesn’t work in your environment due to mysterious dependency issues.

We solve this problem by creating a Docker image for our tool. This allows us to fully define the environment in which the code runs, even the operating system.

For this project, we built a Docker image based on the Jupyter/Tensorflow-Notebook image of Jupyter Docker Stacks. We then added just a few lines of code to install the required libraries and copy the contents of the GitHub repository into the Docker image.

If you’re curious, you can check out our final Dockerfile here. Finally, we push this image to the Docker Hub. You can try this by running the following command:

sudo docker run -p 8888:8888 ianhuntisaak/ac295-final-project:v3 \
-e JUPYTER_LAB_ENABLE=yes
Copy the code

Conclusions and future work

Using this tool, you can easily train segmentation models on new images in a user-friendly manner. While effective, there is still room for improvement in usability, customization, and model performance. In the future, we hope to:

  1. Improve the lasso tool by building custom Jupyter widgets using HTML5 canvas to reduce the lag in manual segmentation
  2. New loss functions and models are explored as the basis of transfer learning
  3. Make interpretation visualization easier and suggest ways to improve results to users

The original link: towardsdatascience.com/how-we-buil…

Welcome to panchuangai blog: panchuang.net/

Sklearn123.com/

Welcome to docs.panchuang.net/