{"action":"created","issue":{"url":"https://api.github.com/repos/cltk/cltk/issues/130","repository_url":"https://api.github.com/repos/cltk/cltk","labels_url":"https://api.github.com/repos/cltk/cltk/issues/130/labels{/name}","comments_url":"https://api.github.com/repos/cltk/cltk/issues/130/comments","events_url":"https://api.github.com/repos/cltk/cltk/issues/130/events","html_url":"https://github.com/cltk/cltk/issues/130","id":133771541,"number":130,"title":"Add Levenstein distance enhancements","user":{"login":"kylepjohnson","id":1854968,"avatar_url":"https://avatars.githubusercontent.com/u/1854968?v=3","gravatar_id":"","url":"https://api.github.com/users/kylepjohnson","html_url":"https://github.com/kylepjohnson","followers_url":"https://api.github.com/users/kylepjohnson/followers","following_url":"https://api.github.com/users/kylepjohnson/following{/other_user}","gists_url":"https://api.github.com/users/kylepjohnson/gists{/gist_id}","starred_url":"https://api.github.com/users/kylepjohnson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kylepjohnson/subscriptions","organizations_url":"https://api.github.com/users/kylepjohnson/orgs","repos_url":"https://api.github.com/users/kylepjohnson/repos","events_url":"https://api.github.com/users/kylepjohnson/events{/privacy}","received_events_url":"https://api.github.com/users/kylepjohnson/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/cltk/cltk/labels/easy","name":"easy","color":"009800"},{"url":"https://api.github.com/repos/cltk/cltk/labels/enhancement","name":"enhancement","color":"84b6eb"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2016-02-15T17:21:45Z","updated_at":"2016-02-16T18:11:51Z","closed_at":null,"body":"We have a good implementation now by @lukehollis . Now just a matter of thinking a little more about interface. The following, from [my comment to Luke's PR](https://github.com/cltk/cltk/pull/129), are things for us to think about for next iteration:\r\n\r\n* Define language in class declaration (?)\r\n* Add a function which will process just strings w/o tokenization\r\n* Think about optional lemmatization/stem which would happen after sentence splitting but before fuzzywuzzy.\r\n* Add fuzzywuzzy to `setup.py` as a project dependency; `python-Levenshtein`, too\r\n* Add module to docs\r\n* I'm OK with `reuse` for now; maybe we can come up with another dir or name (no idea what that would be)\r\n* Could we offer a language-agnostic version of this? Would that be useful to people? For example, we don't have a Pali sentence tokenizer, however someone still might want to look for similar sentences in a Pali corpus.\r\n* Write tests for each function"},"comment":{"url":"https://api.github.com/repos/cltk/cltk/issues/comments/184805797","html_url":"https://github.com/cltk/cltk/issues/130#issuecomment-184805797","issue_url":"https://api.github.com/repos/cltk/cltk/issues/130","id":184805797,"user":{"login":"kylepjohnson","id":1854968,"avatar_url":"https://avatars.githubusercontent.com/u/1854968?v=3","gravatar_id":"","url":"https://api.github.com/users/kylepjohnson","html_url":"https://github.com/kylepjohnson","followers_url":"https://api.github.com/users/kylepjohnson/followers","following_url":"https://api.github.com/users/kylepjohnson/following{/other_user}","gists_url":"https://api.github.com/users/kylepjohnson/gists{/gist_id}","starred_url":"https://api.github.com/users/kylepjohnson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kylepjohnson/subscriptions","organizations_url":"https://api.github.com/users/kylepjohnson/orgs","repos_url":"https://api.github.com/users/kylepjohnson/repos","events_url":"https://api.github.com/users/kylepjohnson/events{/privacy}","received_events_url":"https://api.github.com/users/kylepjohnson/received_events","type":"User","site_admin":false},"created_at":"2016-02-16T18:11:51Z","updated_at":"2016-02-16T18:11:51Z","body":"So much here to discuss.\r\n\r\n* The transliteration might have value here (though the one linked to seems to be doing it phonetically from Modern Greek). Hadn't considered this before, but we could definitely catch proper nouns, which folks would find really useful. Clever.\r\n\r\n* @diyclassics wrote some [good code on alliteration](https://github.com/diyclassics/Alliteration-in-Latin-Literature) for an ACA talk he gave a few years ago. Patrick might have been targeting a different use case. I'd be interested to hear his thoughts on the matter.\r\n\r\n* On semantic comparisons: Word2Vec stands out to me for capturing similar words/phrases/sentences. One (maybe crazy) way could be to add the vectors of a sentence, then compare these sums to one another those of all other available sentences of a corpus.\r\n\r\n* On thematics, this is a perfect job for topic modeling (my experience with LDA, [Gensim](https://radimrehurek.com/gensim/models/ldamodel.html) and [scikit](http://scikit-learn.org/0.16/modules/generated/sklearn.lda.LDA.html), and [NMF](http://scikit-learn.org/stable/auto_examples/applications/topics_extraction_with_nmf_lda.html)). Lots of gotchas in here, but the core goal would be several pre-trained models. The different models would have different #s of topics (say, 10, 20, 50, 100); then run text by author, text, and sentence. Tf-idf similarities have a place here, need to learn more though ([some old code](https://github.com/kylepjohnson/ipython/blob/master/text_analytics/Tf-idf%20pairwise%20similarity%2C%20Greek.ipynb)).\r\n\r\nThis is all rather hand-wavy, I know. Just thinking out loud: it'd be a lot of fun to share these with Thomas and Farrell once we have a few more working prototypes."}}