{"action":"created","issue":{"url":"https://api.github.com/repos/berkmancenter/mediacloud/issues/346","repository_url":"https://api.github.com/repos/berkmancenter/mediacloud","labels_url":"https://api.github.com/repos/berkmancenter/mediacloud/issues/346/labels{/name}","comments_url":"https://api.github.com/repos/berkmancenter/mediacloud/issues/346/comments","events_url":"https://api.github.com/repos/berkmancenter/mediacloud/issues/346/events","html_url":"https://github.com/berkmancenter/mediacloud/issues/346","id":307694837,"node_id":"MDU6SXNzdWUzMDc2OTQ4Mzc=","number":346,"title":"fix topic story deduping","user":{"login":"hroberts","id":1830735,"node_id":"MDQ6VXNlcjE4MzA3MzU=","avatar_url":"https://avatars2.githubusercontent.com/u/1830735?v=4","gravatar_id":"","url":"https://api.github.com/users/hroberts","html_url":"https://github.com/hroberts","followers_url":"https://api.github.com/users/hroberts/followers","following_url":"https://api.github.com/users/hroberts/following{/other_user}","gists_url":"https://api.github.com/users/hroberts/gists{/gist_id}","starred_url":"https://api.github.com/users/hroberts/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hroberts/subscriptions","organizations_url":"https://api.github.com/users/hroberts/orgs","repos_url":"https://api.github.com/users/hroberts/repos","events_url":"https://api.github.com/users/hroberts/events{/privacy}","received_events_url":"https://api.github.com/users/hroberts/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"hroberts","id":1830735,"node_id":"MDQ6VXNlcjE4MzA3MzU=","avatar_url":"https://avatars2.githubusercontent.com/u/1830735?v=4","gravatar_id":"","url":"https://api.github.com/users/hroberts","html_url":"https://github.com/hroberts","followers_url":"https://api.github.com/users/hroberts/followers","following_url":"https://api.github.com/users/hroberts/following{/other_user}","gists_url":"https://api.github.com/users/hroberts/gists{/gist_id}","starred_url":"https://api.github.com/users/hroberts/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hroberts/subscriptions","organizations_url":"https://api.github.com/users/hroberts/orgs","repos_url":"https://api.github.com/users/hroberts/repos","events_url":"https://api.github.com/users/hroberts/events{/privacy}","received_events_url":"https://api.github.com/users/hroberts/received_events","type":"User","site_admin":false},"assignees":[{"login":"hroberts","id":1830735,"node_id":"MDQ6VXNlcjE4MzA3MzU=","avatar_url":"https://avatars2.githubusercontent.com/u/1830735?v=4","gravatar_id":"","url":"https://api.github.com/users/hroberts","html_url":"https://github.com/hroberts","followers_url":"https://api.github.com/users/hroberts/followers","following_url":"https://api.github.com/users/hroberts/following{/other_user}","gists_url":"https://api.github.com/users/hroberts/gists{/gist_id}","starred_url":"https://api.github.com/users/hroberts/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hroberts/subscriptions","organizations_url":"https://api.github.com/users/hroberts/orgs","repos_url":"https://api.github.com/users/hroberts/repos","events_url":"https://api.github.com/users/hroberts/events{/privacy}","received_events_url":"https://api.github.com/users/hroberts/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/berkmancenter/mediacloud/milestones/11","html_url":"https://github.com/berkmancenter/mediacloud/milestone/11","labels_url":"https://api.github.com/repos/berkmancenter/mediacloud/milestones/11/labels","id":3075115,"node_id":"MDk6TWlsZXN0b25lMzA3NTExNQ==","number":11,"title":"8","description":"","creator":{"login":"rahulbot","id":673178,"node_id":"MDQ6VXNlcjY3MzE3OA==","avatar_url":"https://avatars3.githubusercontent.com/u/673178?v=4","gravatar_id":"","url":"https://api.github.com/users/rahulbot","html_url":"https://github.com/rahulbot","followers_url":"https://api.github.com/users/rahulbot/followers","following_url":"https://api.github.com/users/rahulbot/following{/other_user}","gists_url":"https://api.github.com/users/rahulbot/gists{/gist_id}","starred_url":"https://api.github.com/users/rahulbot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rahulbot/subscriptions","organizations_url":"https://api.github.com/users/rahulbot/orgs","repos_url":"https://api.github.com/users/rahulbot/repos","events_url":"https://api.github.com/users/rahulbot/events{/privacy}","received_events_url":"https://api.github.com/users/rahulbot/received_events","type":"User","site_admin":false},"open_issues":3,"closed_issues":23,"state":"open","created_at":"2018-01-30T14:53:22Z","updated_at":"2018-06-12T15:32:10Z","due_on":null,"closed_at":null},"comments":0,"created_at":"2018-03-22T15:31:11Z","updated_at":"2018-06-12T15:33:52Z","closed_at":null,"author_association":"MEMBER","body":"story deduping is not finding all of the duplicates it should.  for example, after deduping, the following stories remained in the nyt:\r\n\r\n```\r\nOpinion | When the World Is Led by a Child - The New York Times\r\nOp-Ed Columnist: When the World Is Led by a Child\r\nOpinion | When the World Is Led by a Child\r\n```\r\n\r\nI think the problem is that the current deduping code only looks for dups of title parts that appear as a complete title at least once.  In the above example, the substantive title never appears on its own, so the alternatives cannot be matched.\r\n\r\nThis is a pressing issue for the election work, because we need this deduped data for the 2017 twitter topics ASAP.\r\n"},"comment":{"url":"https://api.github.com/repos/berkmancenter/mediacloud/issues/comments/396633904","html_url":"https://github.com/berkmancenter/mediacloud/issues/346#issuecomment-396633904","issue_url":"https://api.github.com/repos/berkmancenter/mediacloud/issues/346","id":396633904,"node_id":"MDEyOklzc3VlQ29tbWVudDM5NjYzMzkwNA==","user":{"login":"rahulbot","id":673178,"node_id":"MDQ6VXNlcjY3MzE3OA==","avatar_url":"https://avatars3.githubusercontent.com/u/673178?v=4","gravatar_id":"","url":"https://api.github.com/users/rahulbot","html_url":"https://github.com/rahulbot","followers_url":"https://api.github.com/users/rahulbot/followers","following_url":"https://api.github.com/users/rahulbot/following{/other_user}","gists_url":"https://api.github.com/users/rahulbot/gists{/gist_id}","starred_url":"https://api.github.com/users/rahulbot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rahulbot/subscriptions","organizations_url":"https://api.github.com/users/rahulbot/orgs","repos_url":"https://api.github.com/users/rahulbot/repos","events_url":"https://api.github.com/users/rahulbot/events{/privacy}","received_events_url":"https://api.github.com/users/rahulbot/received_events","type":"User","site_admin":false},"created_at":"2018-06-12T15:33:52Z","updated_at":"2018-06-12T15:33:52Z","author_association":"COLLABORATOR","body":"Related to #396 "}}
{"actor":{"display_login":"rahulbot"}}