<p>I have been working on character detection + recognition problem for industrial application. From my experience using only deep CNN and dense layer to predict character class is not the best approach to solve this problem. There are good research papers for scene text recognition problem, one common approach to design character recognition problem is to have ---</p>
<ol>
<li>any deep CNN model like VGG, ResNet or EfficientNet to extract the image feature.</li>
<li>Then add some RNN layers on top of CNN backbone to get character sequence from the extracted features. This would be a great plus if you want to predict variable length of character.</li>
<li>After getting character sequence from RNN layers, the next step is to decode this character sequence. For this you can either use <a href="https://www.cs.toronto.edu/%7Egraves/icml_2006.pdf" rel="nofollow noreferrer">CTC based method</a> or <a href="https://arxiv.org/abs/1706.03762" rel="nofollow noreferrer">attention mechanism</a>. Both of these methods have their own pros and cons. CTC based methods are fast but performance is bit poor, on the other hand, attention based models give good results but they are very slow. So the selection of the method depends on your requirement.</li>
</ol>
<p>Below image from very famous text recognition paper <a href="https://arxiv.org/pdf/1507.05717v1.pdf" rel="nofollow noreferrer">CRNN</a> gives general idea about above steps.
[<img src="https://i.stack.imgur.com/oXNzR.png" alt="Steps for text recognition from very famous research paper: CRNN ][2]" /></p>
<p>For training the model, @Hemerson has given good suggestions. Try to build and train this type of model with multiple stages and I am sure you will get better results:)</p>
<p>Best regards!</p>
Overfitting - huge difference between training and validation accuracy
<p>I have a dataset of 180k images for which I try to recognize the characters on the images (License plate recognition). All of these license plates contain seven characters and 35 characters are possible, so the output vector y is of shape <code>(7, 35)</code>. I therefore onehot-encoded every license plate label.</p>
<p>I applied the bottom of the EfficicentNet-B0 model (<a href="https://keras.io/api/applications/efficientnet/#efficientnetb0-function" rel="nofollow noreferrer">https://keras.io/api/applications/efficientnet/#efficientnetb0-function</a>) together with a customized top, which is divided in 7 branches (because of seven characters per license plate). I used the weights of the imagenet and freezed the bottom layers of <code>efnB0_model</code>:</p>
<pre><code>def create_model(input_shape = (224, 224, 3)):
    input_img = Input(shape=input_shape)
    model = efnB0_model (input_img)
    model = GlobalAveragePooling2D(name='avg_pool')(model)
    model = Dropout(0.2)(model)
    backbone = model

    branches = []
    for i in range(7):
        branches.append(backbone)
        branches[i] = Dense(360, name=&quot;branch_&quot;+str(i)+&quot;_Dense_16000&quot;)(branches[i])
        branches[i] = BatchNormalization()(branches[i])
        branches[i] = Activation(&quot;relu&quot;) (branches[i])
        branches[i] = Dropout(0.2)(branches[i])
              
        branches[i] = Dense(35, activation = &quot;softmax&quot;, name=&quot;branch_&quot;+str(i)+&quot;_output&quot;)(branches[i])
    
    output = Concatenate(axis=1)(branches)
    output = Reshape((7, 35))(output)
    model = Model(input_img, output)

    return model

model.compile(optimizer=&quot;adam&quot;, loss=&quot;categorical_crossentropy&quot;, metrics=[&quot;accuracy&quot;])
</code></pre>
<p>For training and validating the model I only use 10.000 training images and 3.000 validation images due to the big size of my model and the huge number of data which would make my training very, very slow.</p>
<p>I use this DataGenerator to feed batches to my model:</p>
<pre><code>class DataGenerator(Sequence):

    def __init__(self, x_set, y_set, batch_size):
        self.x, self.y = x_set, y_set
        self.batch_size = batch_size

    def __len__(self):
        return math.ceil(len(self.x) / self.batch_size)

    def __getitem__(self, idx):
        batch_x = self.x[idx*self.batch_size : (idx + 1)*self.batch_size]
        batch_x = np.array([resize(imread(file_name), (224, 224)) for file_name in batch_x])
        batch_x = batch_x * 1./255
        batch_y = self.y[idx*self.batch_size : (idx + 1)*self.batch_size]
        batch_y = np.array(batch_y)

        return batch_x, batch_y
</code></pre>
<p>I fit the model using this code:</p>
<pre><code>model.fit_generator(generator=training_generator,
                    validation_data=validation_generator,
                    steps_per_epoch = num_train_samples // 32,
                    validation_steps = num_val_samples // 32,
                    epochs = 10, workers=6, use_multiprocessing=True)
</code></pre>
<p>Now, after several epochs of training, I observed big differences regarding training accuracy and validation accuracy. I think, one reason for that is the small size of data. Which other factors influence this overfitting in my model? Do you think, there is something completely wrong with my code/model? Do you think the model is to big and complex as well or is it maybe due to the preprocessing of the data?</p>
<p>Note: I already experimented with Data Augmentation and tried the model without Transfer Learning. That leads to poor results on training AND validation data. So, is there anything what I could do additionally?</p>
<p><a href="https://i.stack.imgur.com/bF7QG.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/bF7QG.png" alt="accuracy" /></a></p>