<p>I've also been working on this super-resolution field and found some promising results but haven't tried yet,
<a href="https://doi.org/10.1016/j.heliyon.2021.e08341" rel="nofollow noreferrer">first paper</a> (license plate base text) they implement the image enhancement first then do the super-resolution in a later stage.
<a href="https://arxiv.org/abs/2106.15368" rel="nofollow noreferrer">second paper</a> and <a href="https://github.com/mjq11302010044/TPGSR" rel="nofollow noreferrer">github</a> in this paper they use text prior to guide the super-resolution network.</p>
Scene Text Image Super-Resolution for OCR
<p>I am working on an OCR system.  A challenge that I'm facing for recognizing the text within <strong>ROI</strong> is due to the <strong>shakiness</strong> or <strong>motion effect</strong> shot or text that is <strong>not focus</strong> due to <strong>angle positions</strong>. Please consider the following demo sample</p>
<p><a href="https://i.stack.imgur.com/R3UwZ.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/R3UwZ.jpg" alt="enter image description here" /></a></p>
<p>If you notice the texts (for ex. the mark as a red), in such cases the OCR system couldn't properly recognize the text. However, this scenario can also come on with no angle shot where the image is too blurry that the OCR system can't recognize or partially recognize the text. Sometimes they are <strong>blurry</strong> or sometimes very <strong>low resolution</strong> or <strong>pixelated</strong>. For example</p>
<p><a href="https://i.stack.imgur.com/dpkKl.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/dpkKl.jpg" alt="enter image description here" /></a></p>
<h2>Methods we've tried</h2>
<p>Firstly we've tried various methods available on SO. But sadly no luck.</p>
<ul>
<li><a href="https://stackoverflow.com/questions/54497882/how-improve-image-quality-to-extract-text-from-image-using-tesseract">How to improve image quality to extract text from image using Tesseract</a></li>
<li><a href="https://stackoverflow.com/questions/52004133/how-to-improve-image-quality">How to improve image quality? [closed]</a></li>
<li><a href="https://stackoverflow.com/questions/32848301/image-quality-improvement-in-opencv">Image quality improvement in Opencv</a></li>
</ul>
<p>Next, we've tried the following three most promising methods as below.</p>
<p><strong>1.TSRN</strong></p>
<p>A recent research work (<a href="https://arxiv.org/abs/2005.03341" rel="nofollow noreferrer">TSRN</a>) mainly focuses on such cases. The main intuitive of it is to introduce <strong>super-resolution</strong> (SR) techniques as pre-processing. This <a href="https://github.com/JasonBoy1/TextZoom" rel="nofollow noreferrer">implementation</a> looks by far the most promising. However, it fails to do magic on our custom dataset (for example the second images above, the blue text). Here are some example from their demonstration:</p>
<p><a href="https://i.stack.imgur.com/N0vc5.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/N0vc5.jpg" alt="enter image description here" /></a></p>
<p><strong>2. Neural Enhance</strong></p>
<p>After looking at its illustration on <a href="https://github.com/alexjc/neural-enhance" rel="nofollow noreferrer">its page</a>, we believed it might work. But sadly it also couldn't address the problem. However, I was a bit confusing even with their showed example because I couldn't reproduce them too. I've raised an <a href="https://github.com/alexjc/neural-enhance/issues/259" rel="nofollow noreferrer">issue on github</a> where I demonstrated this more in detail. Here are some example from their demonstration:</p>
<p><a href="https://i.stack.imgur.com/PDPIL.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/PDPIL.jpg" alt="enter image description here" /></a></p>
<p><strong>3. ISR</strong></p>
<p>The last choice with minimum hope with <a href="https://github.com/idealo/image-super-resolution" rel="nofollow noreferrer">this</a> implementation. No luck either.</p>
<h3>Update 1</h3>
<ul>
<li><p>[Method]: Apart from the above, we also tried some traditional approaches such as <a href="https://docs.opencv.org/master/de/d3c/tutorial_out_of_focus_deblur_filter.html" rel="nofollow noreferrer">Out-of-focus Deblur Filter</a> (Wiener filter and also unsupervised Weiner filter). We also checked the <a href="https://scikit-image.org/docs/dev/auto_examples/filters/plot_deconvolution.html" rel="nofollow noreferrer">Richardson-Lucy</a> method. but no improvement with this approach either.</p>
</li>
<li><p>[Method]: We’ve checked out a GAN based DeBlur solution. <a href="https://github.com/KupynOrest/DeblurGAN" rel="nofollow noreferrer">DeblurGAN</a> I have tried this network. What attracted me was the approach of the <strong>Blind Motion Deblurring</strong> mechanism.</p>
</li>
</ul>
<p>Lastly, from this <a href="https://stackoverflow.com/questions/48674106/deblur-image-with-text-to-be-recognized-by-ocr">discussion</a> we encounter <a href="http://www.fit.vutbr.cz/%7Eihradis/CNN-Deblur/" rel="nofollow noreferrer">this research work</a> which seems really good enough. Didn't try this yet.</p>
<p><a href="https://i.stack.imgur.com/xhozQ.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/xhozQ.jpg" alt="enter image description here" /></a></p>
<h3>Update 2</h3>
<ol>
<li><p>[Method]: <strong>Real-World Super-Resolution via Kernel Estimation and Noise Injection</strong>
Tried this method. Promising. However, didn't work in our case. <a href="https://github.com/jixiaozhong/RealSR" rel="nofollow noreferrer">Code</a>.</p>
</li>
<li><p>[Method]: <strong>Photo Restoration</strong>
Comparative to the above all methods, it performs the best surprisingly in super text resolution for OCR. It greatly removes noise, blurriness, etc., and makes the image much clearer and which enhance model generalization better. <a href="https://github.com/microsoft/Bringing-Old-Photos-Back-to-Life" rel="nofollow noreferrer">Code</a>.</p>
</li>
</ol>
<h2>My Query</h2>
<p>Is there any effective workaround to tackle such cases? Any methods that could improve such <strong>blurry</strong> or <strong>low-resolution</strong> pixels whether the texts are <strong>in front</strong> or <strong>far away</strong> due to the camera angle?</p>