Loading custom image datasets for deep learning models: Part 1

The author | Renu Khandelwal compile | source of vitamin k | forward Data Science

In this article, you will learn how to load custom data and create image sequences and test data sets as inputs to a deep learning model.

Open CV2
PIL

The data set used here is Intel image classification from Kaggle.

Data set links: www.kaggle.com/puneet6060/…

The “Intel Image Classification” dataset is divided into train, test, and Val, and we will only use the training dataset to learn how to load the dataset using different libraries.

Typical steps for loading custom data sets for deep learning models

Open the image file. The file format can be JPEG, PNG, or BMP.
Adjust the image size to match the input size of the input layer of the deep learning model.
Convert image pixels to floating point data types.
Normalize the image so that the pixel value is between 0 and 1.
The image data of a deep learning model should be either a NUMPY array or a tensor object.

Custom folder structure for image data

Each class is a folder containing images for that particular class.

Load image data using CV2

Import the required libraries

import pandas as pd
import numpy as np
import os
import tensorflow as tf
import cv2
from tensorflow import keras
from tensorflow.keras import layers, Dense, Input, InputLayer, Flatten
from tensorflow.keras.models import Sequential, Model
from  matplotlib import pyplot as plt
import matplotlib.image as mpimg
%matplotlib inline
Copy the code

Output five random images from one of the folders

plt.figure(figsize=(20.20))
test_folder=r'CV\Intel_Images\seg_train\seg_train\forest'
for i in range(5) : file = random.choice(os.listdir(img_folder)) image_path= os.path.join(img_folder, file) img=mpimg.imread(image_path) ax=plt.subplot(1.5,i+1)
    ax.title.set_text(file)
    plt.imshow(img)
Copy the code

Sets the image dimension and source folder used to load the dataset

IMG_WIDTH=200
IMG_HEIGHT=200
img_folder=r'CV\Intel_Images\seg_train\seg_train\'
Copy the code

Create image data and labels from images in folders

In the function below

The Source folder is an input parameter that contains images of different classes.
Read the image file from the folder and convert it to the correct color format.
Adjust the image size according to the input size required by the model
Converts the image to a Numpy array of data type FLOAT32
Normalize the image array so that the values are between 0 and 1, which helps with faster convergence.

def create_dataset(img_folder) :
   
    img_data_array=[]
    class_name=[]
   
    for dir1 in os.listdir(img_folder):
        for file in os.listdir(os.path.join(img_folder, dir1)):
       
            image_path= os.path.join(img_folder, dir1,  file)
            image= cv2.imread( image_path, cv2.COLOR_BGR2RGB)
            image=cv2.resize(image, (IMG_HEIGHT, IMG_WIDTH),interpolation = cv2.INTER_AREA)
            image=np.array(image)
            image = image.astype('float32')
            image /= 255 
            img_data_array.append(image)
            class_name.append(dir1)
    return img_data_array, class_name
# extract the image array and class name
img_data, class_name =create_dataset(r'CV\Intel_Images\seg_train\seg_train')
Copy the code

Converts text labels to numeric codes

Create a dictionary for all unique values of the class

target_dict={k: v for v, k in enumerate(np.unique(class_name))}
target_dict
Copy the code

Converts class names to their respective numeric values according to the dictionary

target_val=  [target_dict[class_name[i]] for i in range(len(class_name))]
Copy the code

Create a simple deep learning model and compile it

model=tf.keras.Sequential(
        [
            tf.keras.layers.InputLayer(input_shape=(IMG_HEIGHT,IMG_WIDTH, 3)),
            tf.keras.layers.Conv2D(filters=32, kernel_size=3, strides=(2.2), activation='relu'),
            tf.keras.layers.Conv2D(filters=64, kernel_size=3, strides=(2.2), activation='relu'),
            tf.keras.layers.Flatten(),
            tf.keras.layers.Dense(6)
        ])

encoder.compile(optimizer='rmsprop', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
Copy the code

We eventually tune the data set to train the model. We can use a Numpy array as input

history = model.fit(x=np.array(img_data, np.float32), y=np.array(list(map(int,target_val)), np.float32), epochs=5)
Copy the code

We can also use tF.cast () to convert the input data into tensors to train the model.

history = model.fit(x=tf.cast(np.array(img_data), tf.float64), y=tf.cast(list(map(int,target_val)),tf.int32), epochs=5)
Copy the code

We use different libraries to load the image dataset but we use the same model for further training

Load image data with PIL

Add additional libraries to load image datasets using PIL

from PIL import Image
Copy the code

Use PIL to create image data and labels from images in the folder

In the function below

The Source folder is an input parameter that contains images of different classes.
Use PIL to open the image file from the folder.
Adjust the image size according to the input size required by the model
Converts the image to a Numpy array of data type FLOAT32
Standardize image arrays to speed up convergence.

def create_dataset_PIL(img_folder) :
    
    img_data_array=[]
    class_name=[]
    for dir1 in os.listdir(img_folder):
        for file in os.listdir(os.path.join(img_folder, dir1)):
       
            image_path= os.path.join(img_folder, dir1,  file)
            image= np.array(Image.open(image_path))
            image= np.resize(image,(IMG_HEIGHT,IMG_WIDTH,3))
            image = image.astype('float32')
            image /= 255  
            img_data_array.append(image)
            class_name.append(dir1)
    return img_data_array , class_name
PIL_img_data, class_name=create_dataset_PIL(img_folder)
Copy the code

Converts text labels to numeric codes

Here is the code we used in CV2

target_dict={k: v for v, k in enumerate(np.unique(class_name))}
target_val=  [target_dict[class_name[i]] for i in range(len(class_name))]
Copy the code

Create and compile a simple deep learning model

model=tf.keras.Sequential(
        [
            tf.keras.layers.InputLayer(input_shape=(IMG_HEIGHT,IMG_WIDTH, 3)),
            tf.keras.layers.Conv2D(filters=32, kernel_size=3, strides=(2.2), activation='relu'),
            tf.keras.layers.Conv2D(filters=64, kernel_size=3, strides=(2.2), activation='relu'),
            tf.keras.layers.Flatten(),
            tf.keras.layers.Dense(6)
        ])

encoder.compile(optimizer='rmsprop', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
Copy the code

We eventually tune the data set to train the model. We can use a Numpy array as input

history = model.fit(x=np.array(PIL_img_data, np.float32), y=np.array(list(map(int,target_val)), np.float32), epochs=5)
Copy the code

We can also use tF.cast () to convert the input data into tensors to train the model.

history = model.fit(x=tf.cast(np.array(PIL_img_data), tf.float64), y=tf.cast(list(map(int,target_val)),tf.int32), epochs=5)
Copy the code

The procedure for loading the dataset using CV2 and PIL is the same except for a few steps.

Now this will help you load the dataset using the CV2 and PIL libraries.

The code to load the dataset using CV2 and PIL is provided here: github.com/arshren/Loa…

In the next article, we will use the following libraries to load datasets.

Keras
tf.data

The original link: towardsdatascience.com/loading-cus…

Welcome to panchuangai blog: panchuang.net/

Sklearn123.com/

Welcome to docs.panchuang.net/

Loading custom image datasets for deep learning models: Part 1

Load image data with PIL

Related Posts

Troubleshooting is common in Spark

Gensim training model word vector preservation and invocation

Learning record of Python Deep Learning