Multi-lingual mathematical document recognition by InftyReader
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Suzuki, Masakazu
InftyReader is the software developed in Kyushu University to recognize mathematical documents including various formulas of pure and applied mathematics. It uses commercial OCR engines to recognize ordinary text parts. One of the crucial points to keep high accuracy of the recognition is the segmentation of the text area and math expression area to combine commercial OCR and InftyOCR. Recently, we are trying to use the OCR engine of ABBY FineReader to adapt InftyReader to various European languages. In the talk, I will briefly sketch the methods to combine different OCR engines and will give some demonstrations of the current state of our New InftyReader.
 I'm from Kishin University and also I'm the head of Impty Project and also a non-procurement organization, Science Accessibility Team. Impty Project is a research group of mathematical information processing and mainly we are developing ocean software and environment to deal with mathematical document information using interface or converting. And also we can find on this website of our project related papers and some database which is useful for developing systems concerning mathematical documents. And Science Accessibility Net is an organization to help visually impaired people in scientific fields. We develop products and release and maintain the support of the software. And also we are organizing some special events for visually impaired students and some kind of some account. And also we are helping students and professors working on science making documents accessible. Now today I'd like to speak of Impty Leader, a new aspect of Impty Leader. This is Impty Leader is an ocean software to recognize mathematical documents and output the result into various format, XML or including a MathML and Radek and MS or 2007 as the current states. And this software can be used to convert scanned images or papers or PDF files to IML. IML is our original XML format and we have a chat in Impty which can speak aloud the recognize documents. And also the blind people can edit scientific mathematical documents using speech output. And on the other hand, not putting the result into Radek or MassML or world, we can use it to convert them into various formats. And also we are making, we are currently working on a breakout output and easier output from these formats. Now I'm that short for one example of demo. I have here some page of scanned images of geometry textbook. Now let's recognize this page by our software. So it takes some time to recognize it maybe 10 seconds for a page. Now you can see here the recognition result of the document, the page. Here you can see the original image and here the recognition result. And this result can be easily edited by using key operations. This is our editor and chat in Impty is, this is in the editor, which we released. And chat in Impty is a programming software to Impty editor with speech which can speak aloud mass expressions. But I'll skip the demo of chat in Impty. Now let's go back to my presentation. So Impty RIDA has a different flow, consists of several steps to digitalize documents. First task is the layout analysis, automatic segmentation of text area including mass and figures and tables. And then line segmentation. The use task is some difficulty by the existence of mass expression but I'll skip it. And then recognition of ordinary text and separation of mass expressions. This is done simultaneously. And then recognize the extracted mass expressions and output the result into various formats after logical structure analysis of documents. And today, digital concerns the second step, recognition of ordinary text and separation of mass expressions. The current version of Impty RIDA uses two commercial OSHR engines inside. And our Impty project has our original OSHR engine. So text part is recognized simultaneously by using three OSHR engines. And the combination of different OSHR is very efficient to improve revolution rate and explain it later. So today I'd like to explain our new trial to improve our system to recognize different languages. Since the current version of Impty RIDA recognize Japanese documents and English documents. And to some extent, French and Japanese is OK. But for other languages, you will find many mystery conventions. And for Russian document or for Eastern European language with many extended writings, some of them cannot recognize them at all. So now we are trying to include our final user engine inside. We arrived to agreement with AB and currently the implementation is going on. So I'll go to demonstrate it later the current stage application. And to include different OSHR inside is not straightforward. Even if you have very good OSHR engine, it returns. We cannot use it directly. It returns always a strange result for mass media expressions. Since it can use OSHR engine to try to recognize everything as usual text. So we should deal with the result of OSHR. And user commercial OSHR engine currently recognizes very well ordinary text. But if the document includes mass media expressions, even the ordinary text part, a mystery commission of ordinary text part near to the mass expressions increase. So we combine different OSHR to correct each other the commission result of ordinary text part. For example, here you can see original image of some world. This is Japanese address. And here is a revolution result of three OSHR here. And the original space spelling is M I I H A M A I from G U N. And the first OSHR engine returns N and U will load instead of I I and H A M A G U N. The second OSHR recognizes the first part correctly. It returns M I I H A R M A. It returns R N instead of M and A G U N. The third one recognizes M I I H, but it cuts H into Slashan D Ota and A and M A G U N. So from these revolution results, we can get good results in the following way. We first cut all the original image by possible segmentation. It is some kind of over segmentation. And then from different possibility of segmentation, we get M I I H M A G U N. We can also select the better one by using voting method or something like that. This process can be implemented easily using dynamic programming. But the algorithm is simple, but we have to take care of the score of the OSHR. Usually, the score of OSHR is some kind of reliability of character recognition. But usually, at the current stage of research, the score of any OSHR is not reliable. So we have to balance those scores. It is the subject we are managing since we introduced new OSHR finally. Here you can see another problem. You can see here some kind of recognition result of text including mass by Komoshiro Shiyan. Text path is recognized correctly in almost all worlds. But for mass expression, it returns some result. It doesn't reject usually. So we have to throw out these strange expressions. To do that, we find a basic line of text line. We reject the output of Komoshiro Shiyan with the compatibility of size and functional character. In this way, you can throw out this strange result path. It remains some characters which should be sent to mass expression path. In this path, we use basically language information. And I will show some demonstration. This is a new version of E-reader. The appearance is the same as the current version. But the only difference is the language selection part. The current version is the selection of Japanese and English. Here you can select check for example. And select file. I think I have prepared some check sign. I will show some original image here. You can see one page of check paper. This paper contains mass expression but it is not a problem. Let's start the show. This message is not corrected. The implementation is going on. But you can understand that it is working. So now, we can... This is some image here. This path is recognized as image. This is a title. And the content is check text recognized basically very well. But you can observe that here there are some blue characters. Blue character means it is an image reader judged as mass expression. And this isolated character is treated as mass expression at the current stage. Since check word dictionary is not yet included in our current version. So to improve the result we need some dictionary words. In infidery we need not full dictionary of all the words. We need only short dictionary of short words of language less than high. Which is sufficient for our software. So thank you. Thank you for your interesting presentation. Are there some questions? One message. We haven't yet dictionary of different languages. But I need output recognized very many languages. So to adapt to languages we need list of basic word dictionaries. If some of you could send us dictionary of your own language we will be happy. How did you say for the very simple inline math that was maybe just a no baseline shift like A plus B. How do you recognize that that's math instead of just text? We compare the result of infid software. We can manage the reliability of the result by infid system. If we return a green character with sufficient reliability we understand it as math. And for text sometimes there are some characters with function name with F. So it is difficult problem for checking if the A is B and this A is math. But if you have a function the I is text. So we need dictionary or some kind of linguistic information of those. Further questions please. When you are using multiple recognizers do you use some sort of Bayesian voting system or something to decide which answer to accept? Or as you said you use three different OCR systems. How did you decide which was the best answer? I didn't explain the algorithm but we use the dynamic program. Selection of many, from selection of best score path from many possible paths. Further questions?
 I'm from Kishin University and also I'm the head of Impty Project and also a non-procurement organization, Science Accessibility Team.  Impty Project is a research group of mathematical information processing and mainly we are developing ocean software and environment to deal with mathematical document information using interface or converting.  And also we can find on this website of our project related papers and some database which is useful for developing systems concerning mathematical documents.  And Science Accessibility Net is an organization to help visually impaired people in scientific fields.  We develop products and release and maintain the support of the software.  And also we are organizing some special events for visually impaired students and some kind of some account.  And also we are helping students and professors working on science making documents accessible.  Now today I'd like to speak of Impty Leader, a new aspect of Impty Leader.  This is Impty Leader is an ocean software to recognize mathematical documents and output the result into various format, XML or including a MathML and Radek and MS or 2007 as the current states.  And this software can be used to convert scanned images or papers or PDF files to IML.  IML is our original XML format and we have a chat in Impty which can speak aloud the recognize documents.  And also the blind people can edit scientific mathematical documents using speech output.  And on the other hand, not putting the result into Radek or MassML or world, we can use it to convert them into various formats.  And also we are making, we are currently working on a breakout output and easier output from these formats.  Now I'm that short for one example of demo. I have here some page of scanned images of geometry textbook.  Now let's recognize this page by our software.  So it takes some time to recognize it maybe 10 seconds for a page.  Now you can see here the recognition result of the document, the page.  Here you can see the original image and here the recognition result.  And this result can be easily edited by using key operations.  This is our editor and chat in Impty is, this is in the editor, which we released.  And chat in Impty is a programming software to Impty editor with speech which can speak aloud mass expressions.  But I'll skip the demo of chat in Impty.  Now let's go back to my presentation.  So Impty RIDA has a different flow, consists of several steps to digitalize documents.  First task is the layout analysis, automatic segmentation of text area including mass and figures and tables.  And then line segmentation. The use task is some difficulty by the existence of mass expression but I'll skip it.  And then recognition of ordinary text and separation of mass expressions.  This is done simultaneously.  And then recognize the extracted mass expressions and output the result into various formats after logical structure analysis of documents.  And today, digital concerns the second step, recognition of ordinary text and separation of mass expressions.  The current version of Impty RIDA uses two commercial OSHR engines inside.  And our Impty project has our original OSHR engine.  So text part is recognized simultaneously by using three OSHR engines.  And the combination of different OSHR is very efficient to improve revolution rate and explain it later.  So today I'd like to explain our new trial to improve our system to recognize different languages.  Since the current version of Impty RIDA recognize Japanese documents and English documents.  And to some extent, French and Japanese is OK.  But for other languages, you will find many mystery conventions.  And for Russian document or for Eastern European language with many extended writings, some of them cannot recognize them at all.  So now we are trying to include our final user engine inside.  We arrived to agreement with AB and currently the implementation is going on.  So I'll go to demonstrate it later the current stage application.  And to include different OSHR inside is not straightforward.  Even if you have very good OSHR engine, it returns.  We cannot use it directly.  It returns always a strange result for mass media expressions.  Since it can use OSHR engine to try to recognize everything as usual text.  So we should deal with the result of OSHR.  And user commercial OSHR engine currently recognizes very well ordinary text.  But if the document includes mass media expressions, even the ordinary text part, a mystery commission of ordinary text part near to the mass expressions increase.  So we combine different OSHR to correct each other the commission result of ordinary text part.  For example, here you can see original image of some world. This is Japanese address.  And here is a revolution result of three OSHR here.  And the original space spelling is M I I H A M A I from G U N. And the first OSHR engine returns N and U will load instead of I I and H A M A G U N.  The second OSHR recognizes the first part correctly. It returns M I I H A R M A. It returns R N instead of M and A G U N.  The third one recognizes M I I H, but it cuts H into Slashan D Ota and A and M A G U N.  So from these revolution results, we can get good results in the following way. We first cut all the original image by possible segmentation.  It is some kind of over segmentation. And then from different possibility of segmentation, we get M I I H M A G U N.  We can also select the better one by using voting method or something like that. This process can be implemented easily using dynamic programming.  But the algorithm is simple, but we have to take care of the score of the OSHR. Usually, the score of OSHR is some kind of reliability of character recognition.  But usually, at the current stage of research, the score of any OSHR is not reliable.  So we have to balance those scores. It is the subject we are managing since we introduced new OSHR finally.  Here you can see another problem. You can see here some kind of recognition result of text including mass by Komoshiro Shiyan.  Text path is recognized correctly in almost all worlds. But for mass expression, it returns some result. It doesn't reject usually.  So we have to throw out these strange expressions. To do that, we find a basic line of text line.  We reject the output of Komoshiro Shiyan with the compatibility of size and functional character.  In this way, you can throw out this strange result path. It remains some characters which should be sent to mass expression path.  In this path, we use basically language information. And I will show some demonstration.  This is a new version of E-reader. The appearance is the same as the current version.  But the only difference is the language selection part. The current version is the selection of Japanese and English.  Here you can select check for example. And select file. I think I have prepared some check sign.  I will show some original image here. You can see one page of check paper.  This paper contains mass expression but it is not a problem.  Let's start the show.  This message is not corrected. The implementation is going on.  But you can understand that it is working.  So now, we can... This is some image here. This path is recognized as image.  This is a title. And the content is check text recognized basically very well.  But you can observe that here there are some blue characters. Blue character means it is an image reader judged as mass expression.  And this isolated character is treated as mass expression at the current stage.  Since check word dictionary is not yet included in our current version.  So to improve the result we need some dictionary words.  In infidery we need not full dictionary of all the words.  We need only short dictionary of short words of language less than high.  Which is sufficient for our software.  So thank you.  Thank you for your interesting presentation. Are there some questions?  One message. We haven't yet dictionary of different languages.  But I need output recognized very many languages.  So to adapt to languages we need list of basic word dictionaries.  If some of you could send us dictionary of your own language we will be happy.  How did you say for the very simple inline math that was maybe just a no baseline shift like A plus B.  How do you recognize that that's math instead of just text?  We compare the result of infid software.  We can manage the reliability of the result by infid system.  If we return a green character with sufficient reliability we understand it as math.  And for text sometimes there are some characters with function name with F.  So it is difficult problem for checking if the A is B and this A is math.  But if you have a function the I is text.  So we need dictionary or some kind of linguistic information of those.  Further questions please.  When you are using multiple recognizers do you use some sort of Bayesian voting system or something to decide which answer to accept?  Or as you said you use three different OCR systems. How did you decide which was the best answer?  I didn't explain the algorithm but we use the dynamic program.  Selection of many, from selection of best score path from many possible paths.  Further questions?
Multi-lingual_mathematical_document_recognition_by_InftyReader_0_3083.png Multi-lingual_mathematical_document_recognition_by_InftyReader_3084_4871.png Multi-lingual_mathematical_document_recognition_by_InftyReader_4872_5651.png Multi-lingual_mathematical_document_recognition_by_InftyReader_5652_9119.png Multi-lingual_mathematical_document_recognition_by_InftyReader_9120_12071.png Multi-lingual_mathematical_document_recognition_by_InftyReader_12072_17315.png Multi-lingual_mathematical_document_recognition_by_InftyReader_17316_22391.png Multi-lingual_mathematical_document_recognition_by_InftyReader_22392_23651.png Multi-lingual_mathematical_document_recognition_by_InftyReader_23652_24755.png Multi-lingual_mathematical_document_recognition_by_InftyReader_24756_25211.png Multi-lingual_mathematical_document_recognition_by_InftyReader_25212_25499.png Multi-lingual_mathematical_document_recognition_by_InftyReader_25500_26171.png Multi-lingual_mathematical_document_recognition_by_InftyReader_26172_26591.png Multi-lingual_mathematical_document_recognition_by_InftyReader_26592_26651.png Multi-lingual_mathematical_document_recognition_by_InftyReader_26652_26723.png Multi-lingual_mathematical_document_recognition_by_InftyReader_26724_26771.png Multi-lingual_mathematical_document_recognition_by_InftyReader_26772_26831.png Multi-lingual_mathematical_document_recognition_by_InftyReader_26832_26927.png Multi-lingual_mathematical_document_recognition_by_InftyReader_26928_26999.png Multi-lingual_mathematical_document_recognition_by_InftyReader_27000_27071.png Multi-lingual_mathematical_document_recognition_by_InftyReader_27072_27143.png Multi-lingual_mathematical_document_recognition_by_InftyReader_27144_28127.png Multi-lingual_mathematical_document_recognition_by_InftyReader_28128_28175.png Multi-lingual_mathematical_document_recognition_by_InftyReader_28176_28283.png Multi-lingual_mathematical_document_recognition_by_InftyReader_28284_28379.png Multi-lingual_mathematical_document_recognition_by_InftyReader_28380_28535.png Multi-lingual_mathematical_document_recognition_by_InftyReader_28536_28799.png Multi-lingual_mathematical_document_recognition_by_InftyReader_28800_30335.png Multi-lingual_mathematical_document_recognition_by_InftyReader_30336_30455.png Multi-lingual_mathematical_document_recognition_by_InftyReader_30456_30527.png Multi-lingual_mathematical_document_recognition_by_InftyReader_30528_31175.png Multi-lingual_mathematical_document_recognition_by_InftyReader_31176_36838.png