directory

 

Abstract

The body of the

What is mixed data?

How does Keras accept multiple inputs?

Housing price data set

Get the housing price data set

The project structure

Load numerical and categorical data

Load the image data set

Defining multilayer Perceptron (MLP) and Convolutional Neural Network (CNN)

Multiple inputs using Keras

Multiple input and mixed data results

conclusion

Keras: Multiple Inputs and Mixed Data, by Adrian Rosebrock.

Abstract

Click here to download the source code: jbox.sjtu.edu.cn/l/NHfFZu

In the background of regression forecast house prices. Housing price data sets include not only numerical and category data, but also image data, known as mixed data of various types. The model needs to be able to accept mixed data inputs of many types and calculate the output values of regression problems.

In the rest of this tutorial, you will learn how to:

  1. Define a Keras model that can accept multiple types of input data simultaneously, including numeric, category, and image data.
  2. Train the end-to-end Keras model on mixed data input.
  3. Use multiple inputs to evaluate our model.

The body of the

In the first part of this tutorial, we will briefly review the concept of mixed data and how Keras accepts multiple types of input data.

What is mixed data?

Figure 1: Using the flexible Keras deep learning framework, it is possible to define a multi-input model that includes CNN and MLP branches to process mixed data separately.

Mixed data refers to the simultaneous use of input data of different data types. For example, let’s say we’re machine learning engineers working in a hospital developing a system that can categorize the health of patients. We have multiple types of input data for a patient, including:

  1. Numerical/continuous values such as age, heart rate, blood pressure
  2. Category values, including gender and race
  3. Image data, such as MRI, X-ray, etc.

Our machine learning model must be able to mix these data and make (accurate) predictions about a patient’s health.

Developing machine learning systems that can handle mixed data is challenging because each data type may require separate pre-processing steps, including scaling, standardization, and feature engineering.

Working with mixed data is still a very open field of research and is often heavily dependent on specific tasks/goals.

How does Keras accept multiple inputs?

Keras can handle multiple inputs (or even multiple outputs) through its function API.

While you have no doubt used Sequential apis before with the Sequential class, functional apis, by contrast, can be used to define much more complex models that are not Sequential, including:

  • Multiple input model
  • Multiple output model
  • A model includes multiple inputs and multiple outputs
  • Directed acyclic graph
  • Models with shared layers

For example, we can define a simple sequential neural network as:

model = Sequential()
model.add(Dense(8, input_shape=(10,), activation="relu"))
model.add(Dense(4, activation="relu"))
model.add(Dense(1, activation="linear"))
Copy the code

The network is a simple feedforward neural network with 10 inputs, the first hidden layer has 8 nodes, the second hidden layer has 4 nodes, and the last output layer is used for regression.

We can define a sample neural network using the Functional API:

inputs = Input(shape=(10,))
x = Dense(8, activation="relu")(inputs)
x = Dense(4, activation="relu")(x)
x = Dense(1, activation="linear")(x)
model = Model(inputs, x)

# define two sets of inputs
inputA = Input(shape=(32,))
inputB = Input(shape=(128,))
 
# the first branch operates on the first input
x = Dense(8, activation="relu")(inputA)
x = Dense(4, activation="relu")(x)
x = Model(inputs=inputA, outputs=x)
 
# the second branch opreates on the second input
y = Dense(64, activation="relu")(inputB)
y = Dense(32, activation="relu")(y)
y = Dense(4, activation="relu")(y)
y = Model(inputs=inputB, outputs=y)
 
# combine the output of the two branches
combined = concatenate([x.output, y.output])
 
# apply a FC layer and then a regression prediction on the
# combined outputs
z = Dense(2, activation="relu")(combined)
z = Dense(1, activation="linear")(z)
 
# our model will accept the inputs of the two branches and
# then output a single value
model = Model(inputs=[x.input, y.input], outputs=z)
Copy the code

We can see that we have defined two inputs to the Keras neural network:

  • InputA: 32 d
  • InputB: 128 d

The visual model architecture is as follows:

Figure 2: This model has two input branches that eventually merge to produce one output. The Keras function API allows this type of architecture, and you can build any other architecture you can imagine.

Notice that our model has two different branches.

The first branch accepts 128-dimensional input, while the second branch accepts 32-dimensional input. These branches run independently of each other before joining, after which they output a value. In the remainder of this tutorial, you will learn how to create a multi-input network using Keras.

Housing price data set

Figure 4: Housing price data set includes numerical data, category data and image data. Using Keras, we will build a model that supports multiple inputs and mixed data types, and use this regression model to predict home values.

In this series of articles, we use a housing price data set from Ahmed and Mustafa’s 2016 paper, House Price Estimates from Visual and Text Features.

This dataset includes numerical data, category data, and image data for 535 sample houses.

Numerical attributes and classification attributes include:

  1. Number of bedrooms
  2. Number of bathrooms
  3. Area (area)
  4. The zip code

A total of four pictures are provided for each house:

  1. The bedroom
  2. The bathroom
  3. The kitchen
  4. The front of the house

In the first article in this series, you learned how to train a Keras regression network based on numerical and categorical data. In the second article in this series, you learned how to use Keras CNN for regression. Today we will use Keras to process multiple inputs and mixed data.

We will take numerical data, category data and image data, process each type of data by defining two branches of the network, and finally combine these branches to get our final housing price forecast. In this way, we will be able to process multiple inputs and mixed data using Keras.

Get the housing price data set

Click here to download the source code: jbox.sjtu.edu.cn/l/NHfFZu

The housing price data set should be in the Keras-multi-Input directory, which we used for this project.

The project structure

Tree - dirsfirst - filelimit $10. ├ ─ ─ Houses - the dataset │ ├ ─ ─ into \ the dataset [2141 entries] │ └ ─ ─ the README. Md ├ ─ ─ Pyimagesearch │ ├─ ├─ ├─ models.py ├─ mixed_trainingCopy the code

The houses-dataset folder contains the housing price dataset we used in this series. When we are ready to run the mixed_train.py script, all you need to do is provide a path as a command-line argument to the data set (I’ll explain how this is done in the results section).

Today we’ll review three Python scripts:

  • Pyimagesearch/datasets. Py: our digital data loading and pretreatment, the category data and image data.
  • Pyimagesearch/Models.py: contains multi-layer perceptron (MLP) and convolutional Neural network (CNN). These components are the input branches of our multi-input mixed data model.
  • Mixed_training. py: First our training script will usepyimagesearchModules to load and split training data sets, add data headers, and connect two branches to our network. The model is then trained and evaluated.

Load numerical and categorical data

# import the necessary packages
from sklearn.preprocessing import LabelBinarizer
from sklearn.preprocessing import MinMaxScaler
import pandas as pd
import numpy as np
import glob
import cv2
import os
 
def load_house_attributes(inputPath):
	# initialize the list of column names in the CSV file and then
	# load it using Pandas
	cols = ["bedrooms", "bathrooms", "area", "zipcode", "price"]
	df = pd.read_csv(inputPath, sep=" ", header=None, names=cols)
 
	# determine (1) the unique zip codes and (2) the number of data
	# points with each zip code
	zipcodes = df["zipcode"].value_counts().keys().tolist()
	counts = df["zipcode"].value_counts().tolist()
 
	# loop over each of the unique zip codes and their corresponding
	# count
	for (zipcode, count) in zip(zipcodes, counts):
		# the zip code counts for our housing dataset is *extremely*
		# unbalanced (some only having 1 or 2 houses per zip code)
		# so let's sanitize our data by removing any houses with less
		# than 25 houses per zip code
		if count < 25:
			idxs = df[df["zipcode"] == zipcode].index
			df.drop(idxs, inplace=True)
 
	# return the data frame
	return df
Copy the code

Load_house_attributes function. This function reads the value and category data from the housing price data set in the form of CSV files via Panda’s PD.

The raw data needs to be filtered to accommodate the uneven distribution of samples. If some zip codes are represented by only 1 or 2 houses, we delete (lines 23-30) any records from less than 25 houses in the zip code. This alleviates the problem of uneven distribution of zip code samples, and the result is a more accurate model.

Define the process_house_attributes function:

def process_house_attributes(df, train, test):
	# initialize the column names of the continuous data
	continuous = ["bedrooms", "bathrooms", "area"]
 
	# performin min-max scaling each continuous feature column to
	# the range [0, 1]
	cs = MinMaxScaler()
	trainContinuous = cs.fit_transform(train[continuous])
	testContinuous = cs.transform(test[continuous])
 
	# one-hot encode the zip code categorical data (by definition of
	# one-hot encoding, all output features are now in the range [0, 1])
	zipBinarizer = LabelBinarizer().fit(df["zipcode"])
	trainCategorical = zipBinarizer.transform(train["zipcode"])
	testCategorical = zipBinarizer.transform(test["zipcode"])
 
	# construct our training and testing data points by concatenating
	# the categorical features with the continuous features
	trainX = np.hstack([trainCategorical, trainContinuous])
	testX = np.hstack([testCategorical, testContinuous])
 
	# return the concatenated training and testing data
	return (trainX, testX)
Copy the code

This function applies min-max scaling to continuous features via Scikit-learn’s MinMaxScaler(lines 41-43). The one-hot encoding of classification features is then calculated using Scikit-Learn’s LabelBinarizer(lines 47-49). The contiguous and classified properties are then concatenated and returned

Load the image data set

Figure 6: One branch of our model receives an image — a Mosaic of four images from the house. Using pieced images combined with numeric, category data, input to another branch, our model then uses Keras framework regression to predict residential values.

The next step is to define a helper function to load the input image. Again, open the data.py file and insert the following code:

def load_house_images(df, inputPath):
	# initialize our images array (i.e., the house images themselves)
	images = []
 
	# loop over the indexes of the houses
	for i in df.index.values:
		# find the four images for the house and sort the file paths,
		# ensuring the four are always in the *same order*
		basePath = os.path.sep.join([inputPath, "{}_*".format(i + 1)])
		housePaths = sorted(list(glob.glob(basePath)))
Copy the code

The load_house_images function has three functions:

  1. Load all photos from the housing price dataset. Recall that we had four photos per house (Figure 6).
  2. Generate a single Mosaic image from four photos. The composited images are always arranged in the order you see them in the diagram.
  3. Add all these main masks to the list/array and return to the calling function.

Continue to:

  • Initialize the list of images (line 61) and populate the list with all the spliced images we created.
  • Loop over the path of the house in the data frame (line 64) to get four photos of the current house

Inside the loop:

	# initialize our list of input images along with the output image
		# after *combining* the four input images
		inputImages = []
		outputImage = np.zeros((64, 64, 3), dtype="uint8")
 
		# loop over the input house paths
		for housePath in housePaths:
			# load the input image, resize it to be 32 32, and then
			# update the list of input images
			image = cv2.imread(housePath)
			image = cv2.resize(image, (32, 32))
			inputImages.append(image)
 
		# tile the four input images in the output image such the first
		# image goes in the top-right corner, the second image in the
		# top-left corner, the third image in the bottom-right corner,
		# and the final image in the bottom-left corner
		outputImage[0:32, 0:32] = inputImages[0]
		outputImage[0:32, 32:64] = inputImages[1]
		outputImage[32:64, 32:64] = inputImages[2]
		outputImage[32:64, 0:32] = inputImages[3]
 
		# add the tiled image to our set of images the network will be
		# trained on
		images.append(outputImage)
 
	# return our set of images
	return np.array(images)
Copy the code

So far, the code has accomplished the first goal discussed above (grabbing four images per house).

  • In the loop, we:

    • Perform initialization (lines 72 and 73). Our inputImages will contain four photos of each record in a list. Our inputImages will be Mosaic images of photos (see Figure 6).

    • Loop 4 photos (line 76):

      • Load, resize, and attach each photo toinputImagesIn (lines 79-81).

    • Create tiling for images of four houses (lines 87-90):

      • Picture of the bathroom at the top left.
      • Picture of bedroom in upper right.
      • Front view in lower right corner.
      • The kitchen is in the lower left corner.

    • Add a spliced outputImage to images(line 94).

  • Breaking out of the loop, we return all images as a NumPy array (line 97).

Defining multilayer Perceptron (MLP) and Convolutional Neural Network (CNN)

Figure 7: The Keras multi-input (mixed data) model has one branch that accepts numeric/category data (left) and another branch that accepts 4-photo Mosaic image data (right).

So far, we have carefully processed the data using multiple libraries: Panda, SciKit-Learn, OpenCV, and NumPy. We have organized and preprocessed two modes of the dataset using datasets.py.

  • Numerical and categorical data
  • Image data

To achieve this, the skills we use are developed through a bit of debugging through experience and practice. Please do not overlook the data processing techniques we have discussed and used so far, as it is key to the success of our project.

Let’s switch gears and discuss how we will build a multi-input and mixed data network using Keras’s functional API.

To build our multi-input network, we need two branches:

  • The first branch is a simple multi-layer perceptron (MLP) for processing numeric inputs.
  • The second branch is the convolutional neural network, which is used to manipulate image data.
  • These branches are then connected together to form the final multi-input Keras model.

We will deal with building the final join multiple input model in the next section, and our current task is to define these two branches.

Open the models.py file and insert the following code:

# import the necessary packages
from keras.models import Sequential
from keras.layers.normalization import BatchNormalization
from keras.layers.convolutional import Conv2D
from keras.layers.convolutional import MaxPooling2D
from keras.layers.core import Activation
from keras.layers.core import Dropout
from keras.layers.core import Dense
from keras.layers import Flatten
from keras.layers import Input
from keras.models import Model
 
def create_mlp(dim, regress=False):
	# define our MLP network
	model = Sequential()
	model.add(Dense(8, input_dim=dim, activation="relu"))
	model.add(Dense(4, activation="relu"))
 
	# check to see if the regression node should be added
	if regress:
		model.add(Dense(1, activation="linear"))
 
	# return our model
	return model
Copy the code

Our category/numeric data will be processed by a simple multi-layer perceptron (MLP). MLP is defined by create_mlp. Our MLP is simple:

  • Fully connected (dense) input layer with ReLU activation
  • A fully connected hidden layer, also with ReLU activation
  • Finally, a linear activation of the optional regression output

Although we used the regression output of MLP in the first article, it will not be used in this multi-input mixed data network. As you’ll see shortly, we’ll explicitly set regress=False, even though it’s also the default. Regression will later be performed on the header of the entire multi-input mixed data network.

According to Figure 7, we have now constructed the upper left branch of the network.

Now let’s define the upper-right branch of our network, CNN:

def create_cnn(width, height, depth, filters=(16, 32, 64), regress=False):
	# initialize the input shape and channel dimension, assuming
	# TensorFlow/channels-last ordering
	inputShape = (height, width, depth)
	chanDim = -1
	
	# define the model input
	inputs = Input(shape=inputShape)
 
	# loop over the number of filters
	for (i, f) in enumerate(filters):
		# if this is the first CONV layer then set the input
		# appropriately
		if i == 0:
			x = inputs
 
		# CONV => RELU => BN => POOL
		x = Conv2D(f, (3, 3), padding="same")(x)
		x = Activation("relu")(x)
		x = BatchNormalization(axis=chanDim)(x)
		x = MaxPooling2D(pool_size=(2, 2))(x)
Copy the code

The create_CNN function processes image data and takes five arguments:

  • Width: The width of the input image in pixels.
  • Height: The height of the input image in pixels.
  • Depth: The number of channels in the input image. For RGB color images, it is 3.
  • Filters: A set of progressively larger filters so that our network can learn more distinguishing features.
  • Regression: A Boolean value indicating whether a fully connected linear activation layer is added to CNN for regression.

From here, we begin to traverse the filter and create a set of CONV => RELU > BN =>POOL layers. Each iteration of the loop accumulates these layers.

Let’s complete the construction of CNN network branch:

# flatten the volume, then FC => RELU => BN => DROPOUT
	x = Flatten()(x)
	x = Dense(16)(x)
	x = Activation("relu")(x)
	x = BatchNormalization(axis=chanDim)(x)
	x = Dropout(0.5)(x)
 
	# apply another FC layer, this one to match the number of nodes
	# coming out of the MLP
	x = Dense(4)(x)
	x = Activation("relu")(x)
 
	# check to see if the regression node should be added
	if regress:
		x = Dense(1, activation="linear")(x)
 
	# construct the CNN
	model = Model(inputs, x)
 
	# return the CNN
	return model
Copy the code

We flatten the next layer, meaning we form all of the extracted features into a one-dimensional feature vector, and then add a fully connected layer with BatchNormalization and Dropout.

Another full connection layer is used to match four nodes from multiple layers of perceptrons. The number of matching nodes is not required, but it does help balance the branches.

Check to see if regression nodes are added, and add them accordingly if needed. In fact, we’re not going to do a regression at the end of this branch. Regression will be performed at the head of the multi-input mixed data network (bottom of Figure 7).

Finally, the model consists of our inputs and all the layers that are assembled together. We can return the CNN branch to the calling function (line 68).

Now that we’ve defined two branches of the multi-input Keras model, let’s learn how to combine them!

Multiple inputs using Keras

Now we are ready to build the final Keras model that can handle multiple inputs and mixed data. This is where the branches gather — where the “magic” happens. Training will also take place in this script.

Create a new file named mixed_training.py, open it, and insert the following code:

# import the necessary packages
from pyimagesearch import datasets
from pyimagesearch import models
from sklearn.model_selection import train_test_split
from keras.layers.core import Dense
from keras.models import Model
from keras.optimizers import Adam
from keras.layers import concatenate
import numpy as np
import argparse
import locale
import os
 
# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-d", "--dataset", type=str, required=True,
	help="path to input dataset of house images")
args = vars(ap.parse_args())
Copy the code

First, let’s import the necessary modules and parse the command-line arguments.

  • datasets: Our three convenient features load/process CSV data from house data sets and load/preprocess house photos.
  • models: our MLP and CNN input branches, which will serve as our multi-input mixed data services.
  • train_test_splitA:scikit-learnFunction to construct our training/test data split.
  • concatenate: a special Keras function that will accept multiple inputs.
  • argparse: Handles parsing command line arguments.

In lines 15-18, we have a command line parameter to parse, dataset, which is the path to your house price dataset.

Next, let’s load our numeric/categorical data and image data:

# construct the path to the input .txt file that contains information # on each house in the dataset and then load the dataset print("[INFO] loading house attributes..." ) inputPath = os.path.sep.join([args["dataset"], "HousesInfo.txt"]) df = datasets.load_house_attributes(inputPath) # load the house images and then scale the pixel intensities to the # range [0, 1] print("[INFO] loading house images..." Load_house_images (df, args["dataset"]) images = images / 255.0Copy the code

Here, we load the housing price data set as the Panda Dataframe (lines 23 and 24). We then load the image and scale it to [0,1] (lines 29-30). If you need a reminder of the underlying functionality of these functions, be sure to check out the load_house_Attributes and load_house_images functions above. Now that our data is loaded, we’ll build our training/test split, adjust the price, and deal with the property:

# partition the data into training and testing splits using 75% of # the data for training and the remaining 25% for testing print("[INFO] processing data..." ) split = train_test_split(df, images, test_size=0.25, random_state=42) (trainAttrX, testAttrX, trainImagesX, testImagesX) = split # find the largest house price in the training set and use it to # scale our house prices to the range [0, 1] (will lead to better # training and convergence) maxPrice = trainAttrX["price"].max() trainY = trainAttrX["price"] / maxPrice testY = testAttrX["price"] / maxPrice # process the house attributes data by performing min-max scaling # on continuous features, one-hot encoding on categorical features, # and then finally concatenating them together (trainAttrX, testAttrX) = datasets.process_house_attributes(df, trainAttrX, testAttrX)Copy the code

Our training and testing is on line 35 and line 36. We allocated 75% of the data for training and 25% for testing.

Based on this, we find maxPrice from the training set (line 41) and adjust the training and test data accordingly (lines 42 and 43). Adjust the value data to the range of [0,1] for better training and convergence.

Finally, we continue processing our house attributes by performing a minimum-maximum scaling for continuous features and a heat coding for classified features.

The process_house_Attributes function handles these operations and concatenates the contiguous and classified attributes together, returning results (lines 48 and 49).

Are you ready to do your magic? Okay, I lied. There’s really no “magic” in the next block of code! But we will connect our network branches to complete our multi-input Keras network:

# create the MLP and CNN models
mlp = models.create_mlp(trainAttrX.shape[1], regress=False)
cnn = models.create_cnn(64, 64, 3, regress=False)
 
# create the input to our final set of layers as the *output* of both
# the MLP and CNN
combinedInput = concatenate([mlp.output, cnn.output])
 
# our final FC layer head will have two dense layers, the final one
# being our regression head
x = Dense(4, activation="relu")(combinedInput)
x = Dense(1, activation="linear")(x)
 
# our final model will accept categorical/numerical data on the MLP
# input and images on the CNN input, outputting a single value (the
# predicted price of the house)
model = Model(inputs=[mlp.input, cnn.input], outputs=x)
Copy the code

Once you’ve organized your code and model, it’s easy to process multiple inputs using Keras.

In lines 52 and 53, we create the MLP and CNN models. Note regress=False — our regression header appears after line 62.

We will then connect the MLP output to the CNN output as shown on line 57. I call this our combinedInput because it is the input to the rest of the network (as you can see from Figure 3, this is concatenate_1, the two branches together).

The combined input for the last layer in the network is the output of the 8-4-1 FC layer based on the MLP and CNN branches (because both branches output the 4-dimensional FC layer and we then connect them to create an 8-dimensional vector).

We anchor a fully connected layer of four neurons on the combinedInput (line 61). We then add the “liner” Activation regression header (line 62), whose output is the predicted price.

Let’s continue compiling, training, and evaluating our newly formed model:

# compile the model using mean absolute percentage error as our loss, # implying that we seek to minimize the absolute percentage difference # between our price *predictions* and the *actual  prices* opt = Adam(lr=1e-3, decay=1e-3 / 200) model.compile(loss="mean_absolute_percentage_error", optimizer=opt) # train the model print("[INFO] training model..." ) model.fit( [trainAttrX, trainImagesX], trainY, validation_data=([testAttrX, testImagesX], testY), epochs=200, batch_size=8) # make predictions on the testing data print("[INFO] predicting house prices..." ) preds = model.predict([testAttrX, testImagesX])Copy the code

Our model was compiled with “mean_absolute_percentage_error” losses and an Adam optimizer that has learning rate attenuation (lines 72 and 73).

The training begins on lines 77-80. This is known as model fitting (where ownership weights are tuned by a process called back propagation).

We can evaluate our model by calling model.predict(line 84) on our test data set to get the predicted home values of the model.

Now let’s evaluate:

# compute the difference between the *predicted* house prices and the # *actual* house prices, then compute the percentage difference and # the absolute percentage difference diff = preds.flatten() - testY percentDiff = (diff / testY) * 100 absPercentDiff = np.abs(percentDiff) # compute the mean and standard deviation of the  absolute percentage # difference mean = np.mean(absPercentDiff) std = np.std(absPercentDiff) # finally, show some statistics on our model locale.setlocale(locale.LC_ALL, "en_US.UTF-8") print("[INFO] avg. house price: {}, std house price: {}".format( locale.currency(df["price"].mean(), grouping=True), locale.currency(df["price"].std(), grouping=True))) print("[INFO] mean: {:.2f}%, std: {:.2f}%".format(mean, std))Copy the code

To evaluate our model, we calculated the absolute percentage (lines 89-91) and used it to arrive at the final metrics (lines 95 and 96). These measures (mean of price, mean of price standard deviation and absolute percentage, and standard deviation) are printed to the terminal in the appropriate format (lines 100-103).

Multiple input and mixed data results

Figure 8: Real estate price forecasting is a difficult task, but our Keras multi-input and mixed-input regression models produce better results on our limited housing price data set.

Finally, we train our multi-input network on mixed data!

Make sure you are prepared:

  1. Follow the first tutorial in this series to configure the development environment.
  2. Use the source code for this tutorial.
  3. Download the housing price data set using the instructions in the “Getting Housing price Data Set” section above.

On this basis, open the terminal and run the following command to start network training:

$ python mixed_training.py --dataset Houses-dataset/Houses\ Dataset/
Copy the code

Our average absolute percentage error started out very high but continued to decline throughout the training. At the end of the training, we had a 22.41 percent test set absolute error, which meant that our network’s home price predictions would drop by about 22 percent on average.

Let’s compare this result with the previous two articles in this series:

  1. Use MLP only for numerical/categorical data: 26.01%
  2. Only CNN was used for image data :56.91%
  3. Using mixed data: 22.41%

As you can see, the method for handling mixed data is as follows:

  1. Combine our digital/L category data with image data
  2. Multi-input model training for mixed data.
  3. With a better performance model!

conclusion

In this tutorial, you learned how to define a Keras network that can accept multiple inputs.

You also learned how to work with mixed data using Keras.

To achieve these goals, we define an acceptable multi-input neural network:

  • The numerical data
  • Categorical data
  • Image data

Before training, the min-max of the numerical data was scaled to the range [0,1]. Our category data is one-hot encoded (make sure the resulting integer vector is in the range of [0,1]).

The numeric and category data are then joined into a feature vector to form the first input to the Keras network.

Our image data is also scaled to the range [0,1] — this data serves as the second input to the Keras network.

One branch of the model contains a strictly fully connected layer (for connected numeric and categorical data), while the second branch of the multi-input model is essentially a small convolutional neural network.

Combine the outputs of the two branches and define one output (regression prediction).

In this way, we were able to train our network of multiple inputs end-to-end to get better accuracy than using just one of them.

Translated from:Keras: Multiple Inputs and Mixed Data, by Adrian Rosebrock.