{"action":"created","issue":{"url":"https://api.github.com/repos/chartbeat-labs/textacy/issues/169","repository_url":"https://api.github.com/repos/chartbeat-labs/textacy","labels_url":"https://api.github.com/repos/chartbeat-labs/textacy/issues/169/labels{/name}","comments_url":"https://api.github.com/repos/chartbeat-labs/textacy/issues/169/comments","events_url":"https://api.github.com/repos/chartbeat-labs/textacy/issues/169/events","html_url":"https://github.com/chartbeat-labs/textacy/issues/169","id":300353750,"number":169,"title":"to_bag_of_terms(named_entities=False) doesn't exclude named entities ","user":{"login":"JonathanReeve","id":1843676,"avatar_url":"https://avatars2.githubusercontent.com/u/1843676?v=4","gravatar_id":"","url":"https://api.github.com/users/JonathanReeve","html_url":"https://github.com/JonathanReeve","followers_url":"https://api.github.com/users/JonathanReeve/followers","following_url":"https://api.github.com/users/JonathanReeve/following{/other_user}","gists_url":"https://api.github.com/users/JonathanReeve/gists{/gist_id}","starred_url":"https://api.github.com/users/JonathanReeve/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JonathanReeve/subscriptions","organizations_url":"https://api.github.com/users/JonathanReeve/orgs","repos_url":"https://api.github.com/users/JonathanReeve/repos","events_url":"https://api.github.com/users/JonathanReeve/events{/privacy}","received_events_url":"https://api.github.com/users/JonathanReeve/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-02-26T18:52:31Z","updated_at":"2018-02-27T05:11:57Z","closed_at":null,"author_association":"NONE","body":"I'm testing out Textacy using Jane Austen's novel _Persuasion_. Here's the output of `doc.to_bag_of_terms(named_entities=True, as_strings=True)`: \r\n\r\n{'al haines': 1,\r\n 'persuasion': 9,\r\n '1818': 1,\r\n 'chapter 1': 2,\r\n 'walter elliot': 14,\r\n 'kellynch hall': 26,\r\n 'somersetshire': 5,\r\n\r\nOk great. Now here's the output of `doc.to_bag_of_terms(named_entities=False, as_strings=True)`: \r\n\r\n{'al': 1,\r\n 'haines': 1,\r\n 'persuasion': 8,\r\n 'jane': 1,\r\n 'austen': 1,\r\n '1818': 1,\r\n 'chapter': 24,\r\n '1': 3,\r\n 'sir': 149,\r\n 'walter': 140,\r\n 'elliot': 295,\r\n 'kellynch': 73,\r\n\r\nIt seems it doesn't ignore named entities, it just breaks them into smaller pieces. The names are still there (\"jane,\" \"austen\"). \r\n\r\n## Expected Behavior\r\nThe algorithm should exclude parts of named entities in addition to complete ones. \r\n\r\n## Current Behavior\r\nI think it's excluding named entities, but not excluding their parts. \r\n\r\n## Possible Solution\r\nExclude all words which are parts of named entities, in addition to complete ones. \r\n\r\n## Steps to Reproduce (for bugs)\r\nRun the commands listed above. \r\n\r\n- operating system: Linux\r\n- python version: 3\r\n- `spacy` version: latest\r\n- installed `spacy` models: english (large)\r\n- `textacy` version: latest\r\n"},"comment":{"url":"https://api.github.com/repos/chartbeat-labs/textacy/issues/comments/368749692","html_url":"https://github.com/chartbeat-labs/textacy/issues/169#issuecomment-368749692","issue_url":"https://api.github.com/repos/chartbeat-labs/textacy/issues/169","id":368749692,"user":{"login":"JonathanReeve","id":1843676,"avatar_url":"https://avatars2.githubusercontent.com/u/1843676?v=4","gravatar_id":"","url":"https://api.github.com/users/JonathanReeve","html_url":"https://github.com/JonathanReeve","followers_url":"https://api.github.com/users/JonathanReeve/followers","following_url":"https://api.github.com/users/JonathanReeve/following{/other_user}","gists_url":"https://api.github.com/users/JonathanReeve/gists{/gist_id}","starred_url":"https://api.github.com/users/JonathanReeve/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JonathanReeve/subscriptions","organizations_url":"https://api.github.com/users/JonathanReeve/orgs","repos_url":"https://api.github.com/users/JonathanReeve/repos","events_url":"https://api.github.com/users/JonathanReeve/events{/privacy}","received_events_url":"https://api.github.com/users/JonathanReeve/received_events","type":"User","site_admin":false},"created_at":"2018-02-27T05:11:57Z","updated_at":"2018-02-27T05:11:57Z","author_association":"NONE","body":"Hey thanks for your help on this. It's probably just that I don't understand how this is working, so apologies in advance. \r\n\r\nI'm not quite sure what you mean by ngrams—do you mean ngrams where n>1? I've tried excluding n>1 grams, but I'm still seeing the same issue. Is this the desired behavior? \r\n\r\n```\r\n>>>bag = doc.to_bag_of_terms(named_entities=False, ngrams=(1), as_strings=True)\r\n>>>bag\r\n{'al': 1,\r\n 'haines': 1,\r\n 'persuasion': 8,\r\n 'jane': 1,\r\n 'austen': 1,\r\n '1818': 1,\r\n 'chapter': 24,\r\n '1': 3,\r\n 'sir': 149,\r\n 'walter': 140,\r\n 'elliot': 295,\r\n 'kellynch': 73,\r\n 'hall': 28,\r\n 'somersetshire': 4,\r\n```\r\n\r\nFWIW, I can roll my own unigram solution with something like this: \r\n\r\n```\r\ndef getBagOfWords(text): \r\n    \"\"\" Gets a bag of words representation of a text, \r\n    ignoring named entities, and lemmatizing along the way.\"\"\"\r\n    doc = textacy.Doc(text, lang='en')\r\n    wordDict = {}\r\n    for word in doc: \r\n        if word.ent_type == 0:\r\n            if word in wordDict: \r\n                wordDict[word.lemma_] += 1\r\n            else: \r\n                wordDict[word.lemma_] = 1\r\n    return wordDict\r\n```\r\nBut it'd be great if there were a way to do this with textacy. At the very least, I'd like to be able to get a bag of words that didn't have any named entities or parts of named entities. I'm finding it hard to imagine how it'd be good to exclude \"Jane Austen\" but include that same name, as separate words, \"Jane\" and \"Austen.\" \r\n\r\nAs one example, let's say I'm trying to topic model a novel. The top words in a topic are mostly names (Elizabeth, Wentworth, etc.). If I want a representation of the subjects discussed in the novel that aren't characters (other nouns, verbs, etc.), I'd like to use a bag of words that doesn't contain named entities. "}}
{"actor":{"display_login":"JonathanReeve"}}