{"action":"created","issue":{"url":"https://api.github.com/repos/opencog/link-grammar/issues/690","repository_url":"https://api.github.com/repos/opencog/link-grammar","labels_url":"https://api.github.com/repos/opencog/link-grammar/issues/690/labels{/name}","comments_url":"https://api.github.com/repos/opencog/link-grammar/issues/690/comments","events_url":"https://api.github.com/repos/opencog/link-grammar/issues/690/events","html_url":"https://github.com/opencog/link-grammar/issues/690","id":303251363,"number":690,"title":"Capitalization is a kind-of pseduo-morphology","user":{"login":"linas","id":94368,"avatar_url":"https://avatars2.githubusercontent.com/u/94368?v=4","gravatar_id":"","url":"https://api.github.com/users/linas","html_url":"https://github.com/linas","followers_url":"https://api.github.com/users/linas/followers","following_url":"https://api.github.com/users/linas/following{/other_user}","gists_url":"https://api.github.com/users/linas/gists{/gist_id}","starred_url":"https://api.github.com/users/linas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/linas/subscriptions","organizations_url":"https://api.github.com/users/linas/orgs","repos_url":"https://api.github.com/users/linas/repos","events_url":"https://api.github.com/users/linas/events{/privacy}","received_events_url":"https://api.github.com/users/linas/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":44,"created_at":"2018-03-07T20:40:49Z","updated_at":"2018-05-06T18:31:10Z","closed_at":null,"author_association":"OWNER","body":"This is a meta-issue, design-change request, to treat capitalization (and possibly other things) as a kind-of pseudo-morphology.  See issue #42 for context.  The general issue is about refomulating tokenization (and related issues) into a collection of rules (that could be encoded in a file). The example is capitalization.\r\n\r\nFor example: capitalization: we have a rule (coded in C) that if a word is at the beginning  of a sentence, then we should search for a lower-case version of it. ... or if the word is after a semicolon, then we should search for a lower-case version of it. ... or if the word is after a quote, then we should search for a lower-case version of it.  (`Lincoln said, \"Four-score and seven...\"`)   There is an obvious solution: design a new file, and place semi-colons, quotes and LEFT-WORD as markers for capitalization.  Maybe this is kind-of-like a \"new kind of affix rule\" ??   All of the other affix rules state \"if there is a certain sequence, then insert whitespace\"  while this new rule is \"if there is a certain sequence, then look for downcase\".\r\n\r\nIn the language learning code, I don't downcase any data in advance. Instead, the system eventually learns that certain words behave the same way, grammatically, whether they are uppercased or not. The system is blind to uppercasing: it just sees two different UTF8 strings that happen to fall into the same grammatical class. \r\n\r\nTo \"solve\" this problem, one can imagine three steps. First, a \"morphological\" analysis: given a certain grammatical class, compare  pairs to strings to see if they have a common substring - for example, if the whole string matches, except for the first letter.  This would imply that some words have a \"morphology\", where the first letter can be either one of two, while the rest of the word is the same.\r\n\r\nThe second step is to realize that there is a meta-morphology-rule, which states that there are many words, all of which have the property that they can begin with either one of two different initial letters.  The correct choice of the initial letter depends on whether the preceding token was a semicolon, a quote, or the left-wall.\r\n\r\nThe third step is to realize that the meta-morphology-rule can be factored into approximately 26 different classes. That is, in principle, there are 52-squared/2=1352 possible sets containing two (initial) letters. Of these, only 26 are seen:  {A, a}, {B, b}, {C, c} ....and one never ever sees {P, Q} or {Z, a}. \r\n\r\nAs long was we write C code, and know in advance that we are dealing with capital letters, then we can use pre-defined POSIX locales for capitalization.   I'm trying to take two or three steps backwards, here.   One is to treat capitalization as a kind of morphology, just like any other kind of morphology. The second is to create morphology classes - the pseudo-morpheme `A` is only substitutable by the pseudo-morpheme `a`.  The third is that all of this should be somehow rule-driven and \"generic\" in some way. \r\n\r\nThe meta-meta-meta issue is that I want to expand the framework beyond just language written as UTF8 strings, but more generally, language with associated intonation, affect, facial expressions, or \"language\" from other domains (biology, spatial relations, etc.)   "},"comment":{"url":"https://api.github.com/repos/opencog/link-grammar/issues/comments/386901719","html_url":"https://github.com/opencog/link-grammar/issues/690#issuecomment-386901719","issue_url":"https://api.github.com/repos/opencog/link-grammar/issues/690","id":386901719,"user":{"login":"linas","id":94368,"avatar_url":"https://avatars2.githubusercontent.com/u/94368?v=4","gravatar_id":"","url":"https://api.github.com/users/linas","html_url":"https://github.com/linas","followers_url":"https://api.github.com/users/linas/followers","following_url":"https://api.github.com/users/linas/following{/other_user}","gists_url":"https://api.github.com/users/linas/gists{/gist_id}","starred_url":"https://api.github.com/users/linas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/linas/subscriptions","organizations_url":"https://api.github.com/users/linas/orgs","repos_url":"https://api.github.com/users/linas/repos","events_url":"https://api.github.com/users/linas/events{/privacy}","received_events_url":"https://api.github.com/users/linas/received_events","type":"User","site_admin":false},"created_at":"2018-05-06T18:31:10Z","updated_at":"2018-05-06T18:31:10Z","author_association":"OWNER","body":"> Do you know the approximate number of words in which each glyph can participate (order of magnitude)?\r\n\r\nNo -- it is almost surely a Zipfian, with a slope of 1.01 -- viz the most common glyph will appear in 10K words, the 10th-most-common will be in 1K words, the 100th-most-common will be in 100 words, sliding downhill from there.  "}}
{"actor":{"display_login":"linas"}}