Still in 2019, many scanned documents come into businesses in non-digital format. Text to be extracted from real world documents is often nestled inside rich formatting, such as tabular structures or forms with fill-in-the-blank boxes or underlines whose ink often touches or even strikes through the ink of the text itself. Such ink artifacts can severely interfere with the performance of recognition algorithms or other downstream processing tasks. In this work, we propose DeepErase, a neural preprocessor to erase ink artifacts from text images. We devise a method to programmatically augment text images with real artifacts, and use them to train a segmentation network in an weakly supervised manner. In additional to high segmentation accuracy, we show that our cleansed images achieve a significant boost in downstream recognition accuracy by popular OCR software such as Tesseract 4.0. We test DeepErase on out-of-distribution datasets (NIST SDB) of scanned IRS tax return forms and achieve double-digit improvements in recognition accuracy over baseline for both printed and handwritten text. Despite the digitization of information over the past twenty years, large swaths of industry still rely on paper documents for data entry and ingestion. Optical character recognition (OCR) has thus become a widely adopted tool for automatically transcribing text images to text strings. Modern convolutional neural networks have driven many major advances in the performance of OCR systems, culminating in the large-scale adoption of OCR tools such as Tesseract 4.0, Abbyy Fine Reader, or Microsoft Computer Vision OCR. The relevant text to be extracted from real world documents are often nestled inside of rich formatting such as tabular structures or forms with fill-in-the-blank boxes or underlines. Furthermore, documents with handwriting entries often contain handwritten strokes which do not stay within confines of the boxes or lines in which they belong and can encroach into regions occupied by other text that needs to be transcribed (henceforth such encroachment strokes will be called spurious strokes). When extracting text regions from such richly formatted documents, it is inevitable that such document ink artifacts are present in the cropped image even if the localization is perfect. Such artifacts can severely degrade the performance of recognition algorithms, as shown in Figure 1 . Despite the prevalence of these artifacts in the real world, many document text recognition datasets, including IAM Marti and Bunke [2002] , NIST SDB19 Johnson [2012] , and IFN/ENIT El Abed and Margner [2007] contain only images which are cleanly cropped and are more or less free from artifacts. Even the recently released FUNSD dataset of noisy scanned documents Guillaume Jaume [2019] segment their words free of underlines, boxes, and spurious strokes. Consequently, most results on text recognition have reported their performance on clean test examples Graves and Schmidhuber [2009] , Bluche [2016] , typically in the form of well-aligned, well-spaced text lines, which are not representative of the noisy, marked-up, richly formatted scanned documents encountered in the wild. Little work has been done leveraging deep learning for document artifact removal. In this work, we present DeepErase, which inputs a document text image with ink artifacts and outputs the same image with artifacts erased (Figure 1 ). Training is weakly supervised as we use a simple artifact assembler program to produce dirty images along with their segmentation masks for training. Note that henceforth we may refer to images with artifacts as "dirty". We evaluate the performance of DeepErase by passing the cleansed images into two popular text recognition tools: Tesseract and SimpleHTR. On these recognition engines, DeepErase achieves a 40-60% word accuracy improvement (over the dirty images) on our validation set and a 14% improvement on the NIST SDB2 and SDB6 datasets of scanned IRS documents. We have presented DeepErase, a neural-based approach to removing artifacts from document text images. This task is challenging because it must rely solely on spatial structure (rather than differences in shading since the images are binarized) to do semantic segmentation of a wide variety of artifacts. We present a method to programmatically assemble unlimited realistic-looking text artifact images from real data and use them to train DeepErase in weakly supervised manner. The results on the validation set are excellent, showing good segmentation along with a 40 to 60% boost in recognition accuracy for both printed and handwritten text using common recognition software. On the real-world IRS dataset, DeepErase improves recognition accuracy by about 14% on both printed and handwritten text. The cleansed images on both printed and handwritten examples look visually convincing. Next steps include better modeling the test distribution during the artifact generation process such that the trained model performs better at test time. A Example image results from validation set
Neural-based removal of document ink artifacts (underlines, smudges, etc.) using no manually annotated training data