{"action":"created","issue":{"url":"https://api.github.com/repos/fchollet/keras/issues/3436","repository_url":"https://api.github.com/repos/fchollet/keras","labels_url":"https://api.github.com/repos/fchollet/keras/issues/3436/labels{/name}","comments_url":"https://api.github.com/repos/fchollet/keras/issues/3436/comments","events_url":"https://api.github.com/repos/fchollet/keras/issues/3436/events","html_url":"https://github.com/fchollet/keras/pull/3436","id":170321509,"number":3436,"title":"Added support for CTC in both Theano and Tensorflow along with image OCR example.","user":{"login":"mbhenry","id":6217742,"avatar_url":"https://avatars.githubusercontent.com/u/6217742?v=3","gravatar_id":"","url":"https://api.github.com/users/mbhenry","html_url":"https://github.com/mbhenry","followers_url":"https://api.github.com/users/mbhenry/followers","following_url":"https://api.github.com/users/mbhenry/following{/other_user}","gists_url":"https://api.github.com/users/mbhenry/gists{/gist_id}","starred_url":"https://api.github.com/users/mbhenry/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mbhenry/subscriptions","organizations_url":"https://api.github.com/users/mbhenry/orgs","repos_url":"https://api.github.com/users/mbhenry/repos","events_url":"https://api.github.com/users/mbhenry/events{/privacy}","received_events_url":"https://api.github.com/users/mbhenry/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2016-08-10T03:44:35Z","updated_at":"2016-08-16T19:19:53Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/fchollet/keras/pulls/3436","html_url":"https://github.com/fchollet/keras/pull/3436","diff_url":"https://github.com/fchollet/keras/pull/3436.diff","patch_url":"https://github.com/fchollet/keras/pull/3436.patch"},"body":"This commit adds support for training RNNs with Connectionist Temporal Classification (CTC), which is a popular loss function for streams where the temporal or translational alignment between the input data and labels is unknown.  An example would be raw speech spectrograms as input data and phonemes as labels.  Another example is an input image that includes rendered text with an unknown translational location, word/character spacing, or rotation. \r\n\r\nFor Tensorflow, a wrapper was created for the built-in CTC code and put in tensorflow_backend.py.  This wrapper is fairly complex as it has to transform a dense tensor into a sparse tensor. Note that in the bleeding-edge Tensorflow, they moved the location of CTC from contrib to util.  \r\n\r\nFor Theano, an implementation was included courtesy of and used with permission from @shawntan.  Because it was not written for batch processing, its quite a bit slower than Tensorflow - but it does work. \r\n\r\nThis commit includes an example that performs OCR on an image.  The example works with both Theano and Tensorflow. The text-based image is generated using a list of single words (wordlist_mono_clean.txt) and double words (wordlist_bi_clean.txt). I did my best to make sure no profanity ended up in these lists, but apologies in advance if I missed something.  Here is an example output after 40 epochs: \r\n\r\n![e39](https://cloud.githubusercontent.com/assets/6217742/17540970/e2027018-5e6f-11e6-9aaa-10be23d9ed16.png)\r\n\r\nThe text is printed onto a 512 x 64 image using a variety of fonts (note the font list works in Centos 7, but not sure what will happen on other OSes).  This is done on the fly for all training images using generators. A random amount of speckle noise, font, rotation, and translation is applied.  These images are then fed into a network consisting of two convolutional layers, a fully connected layer, two bidirectional recurrent layers, and finally a fully connected layer with 28 outputs (26 letters, space, and CTC blank).  After about 10 epochs it does pretty well with 5 letter words, so harder words are introduced.  After 20 epochs, phrases with spaces are introduced. \r\n\r\nAdditional notes:\r\n-  I have no idea if its learning general text shapes or just remembering fonts. This is more to demonstrate CTC with Keras and actually uses quite a few useful Keras features, including callbacks, functional layers, and dataset generators. \r\n- A wrapper is also included for Tensorflow's constrained dictionary decoding, but is not used in any tests or examples yet. This functionality would be useful for OCR or speech recognition.\r\n- For Theano, the following option is required: \"on_unused_input='ignore'.  This is because the loss function has extra parameters which Keras doesn't support, so a dummy loss function was required to make Keras happy. \r\n\r\nThis is my first Github commit ever so please go easy on me if I mucked something up :)\r\n"},"comment":{"url":"https://api.github.com/repos/fchollet/keras/issues/comments/240208964","html_url":"https://github.com/fchollet/keras/pull/3436#issuecomment-240208964","issue_url":"https://api.github.com/repos/fchollet/keras/issues/3436","id":240208964,"user":{"login":"mbhenry","id":6217742,"avatar_url":"https://avatars.githubusercontent.com/u/6217742?v=3","gravatar_id":"","url":"https://api.github.com/users/mbhenry","html_url":"https://github.com/mbhenry","followers_url":"https://api.github.com/users/mbhenry/followers","following_url":"https://api.github.com/users/mbhenry/following{/other_user}","gists_url":"https://api.github.com/users/mbhenry/gists{/gist_id}","starred_url":"https://api.github.com/users/mbhenry/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mbhenry/subscriptions","organizations_url":"https://api.github.com/users/mbhenry/orgs","repos_url":"https://api.github.com/users/mbhenry/repos","events_url":"https://api.github.com/users/mbhenry/events{/privacy}","received_events_url":"https://api.github.com/users/mbhenry/received_events","type":"User","site_admin":false},"created_at":"2016-08-16T19:19:53Z","updated_at":"2016-08-16T19:19:53Z","body":"Latest commit fixes those unused imports. This has been a learning experience...."}}
{"actor":{"display_login":"mbhenry"}}