{"action":"created","issue":{"url":"https://api.github.com/repos/uvacw/inca/issues/450","repository_url":"https://api.github.com/repos/uvacw/inca","labels_url":"https://api.github.com/repos/uvacw/inca/issues/450/labels{/name}","comments_url":"https://api.github.com/repos/uvacw/inca/issues/450/comments","events_url":"https://api.github.com/repos/uvacw/inca/issues/450/events","html_url":"https://github.com/uvacw/inca/issues/450","id":378756502,"node_id":"MDU6SXNzdWUzNzg3NTY1MDI=","number":450,"title":"Softcosine: performance issues","user":{"login":"damian0604","id":5979167,"node_id":"MDQ6VXNlcjU5NzkxNjc=","avatar_url":"https://avatars0.githubusercontent.com/u/5979167?v=4","gravatar_id":"","url":"https://api.github.com/users/damian0604","html_url":"https://github.com/damian0604","followers_url":"https://api.github.com/users/damian0604/followers","following_url":"https://api.github.com/users/damian0604/following{/other_user}","gists_url":"https://api.github.com/users/damian0604/gists{/gist_id}","starred_url":"https://api.github.com/users/damian0604/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/damian0604/subscriptions","organizations_url":"https://api.github.com/users/damian0604/orgs","repos_url":"https://api.github.com/users/damian0604/repos","events_url":"https://api.github.com/users/damian0604/events{/privacy}","received_events_url":"https://api.github.com/users/damian0604/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-11-08T14:31:04Z","updated_at":"2018-11-08T14:39:33Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"In the softcosine analysis, it is possible to limit the comparisons to be made with the days_before/days_after parameters. I found that this didn't work in the branch we pushed and I changed some things around (in the branch similarities2) and now it does work (as well as supplying multiple doctypes). However, I found that this method takes MUCH longer than calculating all the comparisons (see below), and then limiting them afterwards. The example below is done with 20 nu.nl articles, of which 10 are published on the 18th of October, and the other 10 are published on the 26th, which means that in theory it should be shorter. \r\n\r\nI think it has to do with creating the similarity matrix, which takes a considerable amount of time. In the days_before/days_after method we create a list of relevant targets for each source. Then it has to go through creating the similarity matrix for each list. Perhaps starting this several times for shorter lists takes longer than starting it once for a longer list? However, I cannot think of another way to do this, any ideas?\r\n\r\nAlso, I found that in the output I have several sources without a target or similarity score, and I can't figure out where this is coming from... \r\n\r\nMarieke\r\n\r\n\r\n```\r\nIn [3]: myinca.analysis.softcosine_similarity.fit('/media/sf_virtualbox_folder/mymodel', 'nu', 'nu', days_before=1, d\r\n   ...: ays_after=1, to_csv=True)\r\nINFO:INCA:The results of the similarity analysis could be inflated when not using the recommended text processing steps (stopword removal, punctuation removal, stemming) beforehand\r\n100%|███████████████████████████████████████████████████████████████████████████████| 20/20 [00:00<00:00, 100.21it/s]\r\n100%|███████████████████████████████████████████████████████████████████████████████| 20/20 [00:00<00:00, 262.44it/s]\r\n2018-11-06 16:20:57.917742\r\n1 out of 19\r\n2 out of 19\r\n3 out of 19\r\n4 out of 19\r\n5 out of 19\r\n6 out of 19\r\n7 out of 19\r\n8 out of 19\r\n9 out of 19\r\n10 out of 19\r\n11 out of 19\r\n12 out of 19\r\n13 out of 19\r\n14 out of 19\r\n15 out of 19\r\n16 out of 19\r\n17 out of 19\r\n18 out of 19\r\n19 out of 19\r\n2018-11-06 16:24:55.709522\r\n\r\nIn [4]: myinca.analysis.softcosine_similarity.fit('/media/sf_virtualbox_folder/mymodel', 'nu', 'nu', to_csv=True)\r\nINFO:INCA:The results of the similarity analysis could be inflated when not using the recommended text processing steps (stopword removal, punctuation removal, stemming) beforehand\r\n100%|███████████████████████████████████████████████████████████████████████████████| 20/20 [00:00<00:00, 266.76it/s]\r\n100%|███████████████████████████████████████████████████████████████████████████████| 20/20 [00:00<00:00, 384.62it/s]\r\n2018-11-06 16:28:50.159596\r\n2018-11-06 16:29:11.789980\r\n```"},"comment":{"url":"https://api.github.com/repos/uvacw/inca/issues/comments/437015741","html_url":"https://github.com/uvacw/inca/issues/450#issuecomment-437015741","issue_url":"https://api.github.com/repos/uvacw/inca/issues/450","id":437015741,"node_id":"MDEyOklzc3VlQ29tbWVudDQzNzAxNTc0MQ==","user":{"login":"damian0604","id":5979167,"node_id":"MDQ6VXNlcjU5NzkxNjc=","avatar_url":"https://avatars0.githubusercontent.com/u/5979167?v=4","gravatar_id":"","url":"https://api.github.com/users/damian0604","html_url":"https://github.com/damian0604","followers_url":"https://api.github.com/users/damian0604/followers","following_url":"https://api.github.com/users/damian0604/following{/other_user}","gists_url":"https://api.github.com/users/damian0604/gists{/gist_id}","starred_url":"https://api.github.com/users/damian0604/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/damian0604/subscriptions","organizations_url":"https://api.github.com/users/damian0604/orgs","repos_url":"https://api.github.com/users/damian0604/repos","events_url":"https://api.github.com/users/damian0604/events{/privacy}","received_events_url":"https://api.github.com/users/damian0604/received_events","type":"User","site_admin":false},"created_at":"2018-11-08T14:39:33Z","updated_at":"2018-11-08T14:39:33Z","author_association":"CONTRIBUTOR","body":"The bottleneck indeed seems to be the estimation of the matrix within the for-loop.\r\n\r\nPossible approaches:\r\n- Actually estimate the whole similiarty matrix for all possible comparisons *even though* we do not need all entries. This would move the estimation of the matrix out of the loop and leave only the query in. This may be less stupid than it sounds, according to https://radimrehurek.com/gensim/similarities/docsim.html#gensim.similarities.docsim.Similarity 1 million documents take only 1 GB of RAM, and one can specify a tmpfile for outsourcing to disk.\r\n\r\n- The same link to the documentation also specifies that it is possible to add and remove documents. So maybe it is an option to have a sliding (time) window add and remove the documents? So basically instead of re-estimating the matrix add new entries and remove earlier ones? \r\n\r\nI'll think about it a bit more..."}}
{"actor":{"display_login":"damian0604"}}