{"action":"created","issue":{"url":"https://api.github.com/repos/AbeHandler/rookie/issues/154","repository_url":"https://api.github.com/repos/AbeHandler/rookie","labels_url":"https://api.github.com/repos/AbeHandler/rookie/issues/154/labels{/name}","comments_url":"https://api.github.com/repos/AbeHandler/rookie/issues/154/comments","events_url":"https://api.github.com/repos/AbeHandler/rookie/issues/154/events","html_url":"https://github.com/AbeHandler/rookie/issues/154","id":137091375,"number":154,"title":"Snippet logic too slow","user":{"login":"AbeHandler","id":1252925,"avatar_url":"https://avatars.githubusercontent.com/u/1252925?v=3","gravatar_id":"","url":"https://api.github.com/users/AbeHandler","html_url":"https://github.com/AbeHandler","followers_url":"https://api.github.com/users/AbeHandler/followers","following_url":"https://api.github.com/users/AbeHandler/following{/other_user}","gists_url":"https://api.github.com/users/AbeHandler/gists{/gist_id}","starred_url":"https://api.github.com/users/AbeHandler/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AbeHandler/subscriptions","organizations_url":"https://api.github.com/users/AbeHandler/orgs","repos_url":"https://api.github.com/users/AbeHandler/repos","events_url":"https://api.github.com/users/AbeHandler/events{/privacy}","received_events_url":"https://api.github.com/users/AbeHandler/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":14,"created_at":"2016-02-28T20:17:49Z","updated_at":"2016-03-30T14:45:18Z","closed_at":null,"body":"it takes ~10 ms per doc to loop thru all sentences and find the sentences that store q and f. This is taking too long (ex ~3 secs for 300 docs). need to index what ngrams are in what sentences and do a lookup.\r\n\r\nCould you replicate the sparse vector thing from facets? I.e. store sparse matrixes of ngrams by sentence for each doc and fit in memory?\r\n\r\nGawker has 48761 ngrams."},"comment":{"url":"https://api.github.com/repos/AbeHandler/rookie/issues/comments/203467273","html_url":"https://github.com/AbeHandler/rookie/issues/154#issuecomment-203467273","issue_url":"https://api.github.com/repos/AbeHandler/rookie/issues/154","id":203467273,"user":{"login":"brendano","id":1677,"avatar_url":"https://avatars.githubusercontent.com/u/1677?v=3","gravatar_id":"","url":"https://api.github.com/users/brendano","html_url":"https://github.com/brendano","followers_url":"https://api.github.com/users/brendano/followers","following_url":"https://api.github.com/users/brendano/following{/other_user}","gists_url":"https://api.github.com/users/brendano/gists{/gist_id}","starred_url":"https://api.github.com/users/brendano/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brendano/subscriptions","organizations_url":"https://api.github.com/users/brendano/orgs","repos_url":"https://api.github.com/users/brendano/repos","events_url":"https://api.github.com/users/brendano/events{/privacy}","received_events_url":"https://api.github.com/users/brendano/received_events","type":"User","site_admin":false},"created_at":"2016-03-30T14:45:18Z","updated_at":"2016-03-30T14:45:18Z","body":"can postgres (or whatever storage system) be made to store the same\ninformation that the \"@\"lrucache decorator stores, or something close?  i'm\nassuming the lrucache decorator uses pickle or something like that, on the\nmost usable, post-processed, form of the data.  this could just be a binary\nblob within postgres.\n\nbut it could be the case the deserialization time is still high here. i'm\nnot sure.  i guess i was worried about the translation between the format\nin the DB, versus the objects that the application code users, as a reason\nfor the difference between the decorator versus caching in an external\nstorage system.\n"}}