<p>You want to <strong>recognize text of a document containing multiple lines</strong>. There are <strong>two ways</strong> to achieve this:</p>

<ol>
<li><p><strong>Segment</strong> the document into <strong>lines</strong> as a <strong>pre-processing</strong> step, then feed each segmented line separately into your neural network.
If you want to go this way, e.g. read the paper [1] from Bunke and Marti. They essentially count the black-white transitions for each scanline and create a histogram out of it. They use the minimums of the histogram to split the document into individual lines. There are some other methods too to segment a document into lines.</p></li>
<li><p><strong>Train</strong> the <strong>neural network</strong> to <strong>implicitly segment</strong> the document into <strong>lines</strong>.
You need to add attention to the neural network, such that it can focus on individual lines.
Bluche has done some great work towards text recognition on document-level. 
See the paper [2] and the website [3].</p></li>
</ol>

<p>[1] Bunke, Marti: The IAM-database: an English sentence database for offline handwriting recognition. Download via Springer</p>

<p>[2] Bluche: Joint Line Segmentation and Transcription for End-to-End Handwritten Paragraph Recognition. Download via <a href="https://arxiv.org/abs/1604.08352" rel="noreferrer">https://arxiv.org/abs/1604.08352</a></p>

<p>[3] Bluche: Scan, Attend and Read. See <a href="http://www.tbluche.com/scan_attend_read.html" rel="noreferrer">http://www.tbluche.com/scan_attend_read.html</a> and look for "Handwriting Recognition with MDLSTM and CTC" and "The Collapse Layer and its Proposed Replacements"</p>
Optical Character Recognition Multiple Line Detection
<p>I'm building an OCR. For that I'm using <code>CNN</code>, <code>RNN</code> and <code>CTC</code> Loss Function.
My input layer gets image and output layer predicts what's written on that image. Labels are converted into integer.</p>

<pre><code>['A', 'B', 'C'] -&gt; A = 0, B = 1, C = 2
</code></pre>

<p>If the image is ABC, training label will be 0,1,2 (Single row vector)</p>

<p>I'm able to accomplish this on single line. For eg. '<code>ABCDE</code>' is written on an image and model works great. But if the image is</p>

<pre><code>'ABC'

'CAB'
</code></pre>

<p>then what should be the training label ? How can I tell the model about next line ? I want to train a model on multiple line.</p>