1. Introduction
Recently, when we want to realize a captcha recognition function, we vacanted between neural network and computer graphics, but the convolutional neural network can realize “end-to-end”, that is, the captcha recognition of input image and output captcha, so we abandoned CV and chose CNN.
I was already familiar with PyTorch, and even had a pyTorch + CUDA + CUDNN environment. However, Tensorflow has tensorflow. js that can be perfectly combined with the cross-platform solution Electron. Move to Tensorflow.
After a while of research (MO) (GAI) official document + Github + StackOverflow + Zhhu + CSDN parties code, finally can use my GTX1660 Dan!
2. Implement
2.1 Image preprocessing
The first step to do is the image pretreatment, at least one gray mode to have it?
Take a look at the image of the captcha to work with:
You can see that there are two lines in there one red, one green, running through the picture. In theory, these two lines can be removed by image manipulation (original picture – red-green).
But now that we’re using convolutional neural networks, let’s not do so much fancy stuff, let’s just go ahead and do it.
A few lines of code grayscale the image and remove the surrounding black lines:
img = cv2.imread(img_file, cv2.IMREAD_GRAYSCALE)
np_img = np.array(img)
flat_img = np_img.flatten()
ind = np.argmax(np.bincount(flat_img))
if(ind < 255):
np_img[0] = ind
np_img[-1] = ind
np_img[:, 0] = ind
np_img[:, -1] = ind
img = np_img / 255.0
Copy the codeOutput picture:
Of course, the image processing here is only because my network model does not require the shape of the input data. In fact, many neural networks require the shape of the input data, such as InceptionV4(299,299),ResNet_18(224,224), etc. It is also possible to extend images based on Opencv-Python. One implementation was to use cv2.copyMakeBorder to make the short side as long as the long side, and then cv2.resize.
Good. Now it’s time to collect enough image captcha for training.
Later, the original binary processing was also done, but later found that only gray performance is also good, so the binary was removed.
2.2 Convolutional neural network construction and training
2.2.1 Training process
After a few days of gathering and hand-marking hundreds of maps (later expanded to 18,000 +) as a training set, it was time to try alchemy.
Have to say that the age of [email protected] is too brilliant, now find most of the 1.x version of the source code.
So I took a look at the github repository and started manual training (magic change) by referring to the tutorial in the official document.
The idea is to feed the captcha images labeled in the training set into the model with one-hot according to the corresponding character set. As you can see from the previous captcha diagram, it has six characters per diagram, with a total of 10 digits +26 uppercase letters =36 possibilities per character. Therefore, it is a multi-class model, the total category is 6*36=216, the loss function categorical_crossentropy, the activation function softmax.
Here is the code to build the model (the body is in a class, of course, omitted)
def generate_model(self) :
self.model = model = models.Sequential()
input_shape=(self.image_height, self.image_width, 1)
# conventional
model.add(layers.Conv2D(96, (3.3), activation='relu',padding="SAME"))
model.add(layers.Conv2D(96, (3.3), activation='relu',padding="SAME"))
model.add(layers.MaxPooling2D((2.2)))
# conv2
model.add(layers.Conv2D(128, (3.3), activation='relu',padding="SAME"))
model.add(layers.Conv2D(128, (3.3), activation='relu',padding="SAME"))
model.add(layers.Conv2D(128, (3.3), activation='relu',padding="SAME"))
model.add(layers.MaxPooling2D((2.2)))
# conv3
model.add(layers.Conv2D(128, (3.3), activation='relu'))
model.add(layers.Conv2D(128, (3.3), activation='relu'))
model.add(layers.MaxPooling2D((2.2)))
# conv4
model.add(layers.Conv2D(128, (3.3), activation='relu'))
model.add(layers.Conv2D(128, (3.3), activation='relu'))
model.add(layers.MaxPooling2D((2.2)))
# flatten
model.add(layers.Flatten())
# fc1
model.add(layers.Dense(384, activation='relu'))
model.add(layers.AlphaDropout(rate=0.2))
# fc2
model.add(layers.Dense(512, activation='relu'))
model.add(layers.AlphaDropout(rate=0.2))
# output
model.add(layers.Dense(self.max_captcha*self.char_set_len, activation='softmax'))
model.add(layers.Reshape((self.max_captcha,self.char_set_len)))
model.summary()
model.compile(optimizer=tf.keras.optimizers.Nadam(learning_rate=1e-5, clipnorm=1),
loss='categorical_crossentropy',
metrics=['accuracy'])
return model
Copy the codeCompile produces a network model that looks like this:
Model: "sequential"
_________________________________________________________________ Layer (type) Output Shape Param # ================================================================= conv2d (Conv2D) (None, 80, 300, 96) 960 conv2d_1 (Conv2D) (None, 80, 300, 96) 83040
conv2d_2 (Conv2D) (None, 80, 300, 96) 83040 max_pooling2d (MaxPooling2D (None, 40, 150, 96) 0
)
conv2d_3 (Conv2D) (None, 40, 150, 128) 110720 conv2d_4 (Conv2D) (None, 40, 150, 128) 147584
conv2d_5 (Conv2D) (None, 40, 150, 128) 147584 max_pooling2d_1 (MaxPooling (None, 20, 75, 128) 0 2D) conv2d_6 (Conv2D) (None, 18, 73, 128) 147584
conv2d_7 (Conv2D) (None, 16, 71, 128) 147584 max_pooling2d_2 (MaxPooling (None, 8, 35, 128) 0 2D) conv2d_8 (Conv2D) (None, 6, 33, 128) 147584
conv2d_9 (Conv2D) (None, 4, 31, 128) 147584 max_pooling2d_3 (MaxPooling (None, 2, 15, 128) 0 2D) flatten (Flatten) (None, 3840) 0 dense (Dense) (None, 384) 1474944 alpha_dropout (AlphaDropout (None, 384) 0
)
dense_1 (Dense) (None, 512) 197120 alpha_dropout_1 (AlphaDropo (None, 512) 0 ut) dense_2 (Dense) (None, 216) 110808 reshape (Reshape) (None, 6, 36) 0 = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = Total params: 2946136 Trainable params: 2946136 Non - trainable params: 0Copy the codeIt is well known that networks can be optimized by increasing the network layers (depth, for example, from ResNet (He et al.) from RESnet18 to RESnet200), or by increasing the width, For example, WideResNet (Zagoruyko & Komodakis, 2016) and Mo-Bilenets (Howard et al., 2017) can expand the width (#channels) of the network, Also, a larger input image size (resolution) can help improve accuracy. — Zhihu elder brother
The original model of the code used by Alexnet is reused, but the calculation is complicated, there are many layers, too many parameters, and the model file is also large. After practicing for a while, I found that the convergence is super slow. Later, the current model was obtained through a certain trade-off combined with application scenarios.
I have to say tF2 is much better than TF1! Apply the Model.fit method to begin alchemy!
I added two callbacks to model.fit, one to save breakpoints and the other to save historical data.
def train_cnn(self) :
x_train, y_train = self.get_train_data()
cp_path = './cp.h5'
print(np.any(np.isnan(y_train)))
print(y_train[0])
save_chec_points = tf.keras.callbacks.ModelCheckpoint(
filepath=cp_path, save_weights_only=False, save_best_only=True)
try:
model = tf.keras.models.load_model(self.model_save_dir)
except Exception as e:
model = self.generate_model()
try:
model.load_weights(cp_path)
except Exception as e:
pass
filename = 'log.csv'
history_logger = tf.keras.callbacks.CSVLogger(
filename, separator=",", append=True)
history = model.fit(x_train, y_train, batch_size=24,
epochs=200, validation_split=0.1, callbacks=[save_chec_points, history_logger],
shuffle=True)
model.save(self.model_save_dir)
plt.plot(history.history['accuracy'], label='accuracy')
plt.plot(history.history['val_accuracy'], label='val_accuracy')
plt.xlabel('Epoch')
plt.ylabel('Accuracy')
plt.ylim([0.1])
plt.legend(loc='lower right')
plt.show()
Copy the codeCallback adds two functions to the callback to store checkpoint and epoch data.
Find a long time to turn on the computer alchemy, practice 200 epoch, draw the following image (forget to draw loss, nevermind) :
It can be seen that after training 200 epochs, the accuracy of the model has been about 0.9. Then I added another 50 epochs and stabilized at 0.94.
100 pictures of this artificial retarded it can probably understand 94, this performance can! What bicycle is needed as a product of no tuning.
2.2.2 Tread pit records
When you’re writing code, you’re going to have to step on holes.
First, she stepped on a sinkhole. Every time she trained, the loss would increase rapidly in the first epoch until she became NaN… What’s going on here? Obviously the loss function and the activation function seem to be correct!
Took a long time to find!
It turned out that my model code was modified from the official document tutorial, and the last step in it was finished by activating the function softmax, while the training part was modified from someone else’s TF1 code. But finding out that categorical_crossentropy is 0 is needed that the output should not be (1,216) but 0 is 0 0
0 add model. Add (layers. new ((self.max_captcha,self.char_set_len))) 0) 0 make it a (0, 6,36) shape
Then, it was found that the loss would become NaN in a random epoch in the middle of the process, and the historical records were clueless. According to some websites to check whether the input and label are NaN, found that is not relevant.
It is found that when the optimal algorithm is Adam and the loss function is categorical_crossentropy, the irrational NaN change will not occur. Changing the optimal algorithm to SGD or other methods will not occur.
Later, other verification code recognition models were trained, and it was found that with the BatchNormalization layer, convergence was faster, and the lost and accuracy changes of each epoch were more stable.
Then you put in the tensor (N,80,300,1), output the tensor (N,6,36), and then you match the output tensor according to onehot order.
3. Summary
Although it was just a Toy Project, I still put a lot of effort into it, from environment matching to code writing to alchemy. It was the first successful Project of Tensorflow2. Along the way, I am very grateful for my GTX1660 silent pay!
This should be my last technical article in 2021, end scatter flower!