{"action":"created","issue":{"url":"https://api.github.com/repos/TimDettmers/ConvE/issues/9","repository_url":"https://api.github.com/repos/TimDettmers/ConvE","labels_url":"https://api.github.com/repos/TimDettmers/ConvE/issues/9/labels{/name}","comments_url":"https://api.github.com/repos/TimDettmers/ConvE/issues/9/comments","events_url":"https://api.github.com/repos/TimDettmers/ConvE/issues/9/events","html_url":"https://github.com/TimDettmers/ConvE/issues/9","id":305084068,"number":9,"title":"Error in spodernet.preprocessing.batching.DataLoaderSlave.clean_cache","user":{"login":"TristHas","id":15460655,"avatar_url":"https://avatars0.githubusercontent.com/u/15460655?v=4","gravatar_id":"","url":"https://api.github.com/users/TristHas","html_url":"https://github.com/TristHas","followers_url":"https://api.github.com/users/TristHas/followers","following_url":"https://api.github.com/users/TristHas/following{/other_user}","gists_url":"https://api.github.com/users/TristHas/gists{/gist_id}","starred_url":"https://api.github.com/users/TristHas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/TristHas/subscriptions","organizations_url":"https://api.github.com/users/TristHas/orgs","repos_url":"https://api.github.com/users/TristHas/repos","events_url":"https://api.github.com/users/TristHas/events{/privacy}","received_events_url":"https://api.github.com/users/TristHas/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-03-14T09:39:01Z","updated_at":"2018-03-24T11:51:42Z","closed_at":"2018-03-24T11:51:42Z","author_association":"NONE","body":"Hi,\r\nThanks a lot for sharing your code, it's been very helpful for me.\r\nI ran your code with smaller smaller cache_size in the StreamBatcher objects because I had memory issues dealing with larger graphs (the full Wordnet hierarchy).\r\n\r\nI ran into some errors iterating over StreamBatcher objects in my setting.\r\nI checked the code and found that you pop elements from the cache list while iteration over it:\r\n\r\n    def clean_cache(self, current_paths):\r\n        # delete unused cached data\r\n        for i in range(len(self.cache_order)):\r\n            if self.cache_order[i] in current_paths: continue\r\n            path = self.cache_order.pop(i)\r\n            self.current_data.pop(path, None)\r\n            GB_usage = self.determine_cache_size()\r\n            if GB_usage < self.cache_size_GB: break\r\n\r\nSimply inversing the order of the iteration fixed it for me:\r\nfor i in list(range(len(self.cache_order)))[::-1]\r\n\r\nSo the full functions looks like this:\r\n\r\n    def clean_cache(self, current_paths):\r\n        # delete unused cached data\r\n        for i in list(range(len(self.cache_order)))[::-1]:\r\n            if self.cache_order[i] in current_paths: continue\r\n            path = self.cache_order.pop(i)\r\n            self.current_data.pop(path, None)\r\n            GB_usage = self.determine_cache_size()\r\n            if GB_usage < self.cache_size_GB: break\r\n\r\nI hope it helps, thanks again for sharing.\r\n"},"comment":{"url":"https://api.github.com/repos/TimDettmers/ConvE/issues/comments/375878720","html_url":"https://github.com/TimDettmers/ConvE/issues/9#issuecomment-375878720","issue_url":"https://api.github.com/repos/TimDettmers/ConvE/issues/9","id":375878720,"user":{"login":"TimDettmers","id":5260050,"avatar_url":"https://avatars1.githubusercontent.com/u/5260050?v=4","gravatar_id":"","url":"https://api.github.com/users/TimDettmers","html_url":"https://github.com/TimDettmers","followers_url":"https://api.github.com/users/TimDettmers/followers","following_url":"https://api.github.com/users/TimDettmers/following{/other_user}","gists_url":"https://api.github.com/users/TimDettmers/gists{/gist_id}","starred_url":"https://api.github.com/users/TimDettmers/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/TimDettmers/subscriptions","organizations_url":"https://api.github.com/users/TimDettmers/orgs","repos_url":"https://api.github.com/users/TimDettmers/repos","events_url":"https://api.github.com/users/TimDettmers/events{/privacy}","received_events_url":"https://api.github.com/users/TimDettmers/received_events","type":"User","site_admin":false},"created_at":"2018-03-24T11:51:42Z","updated_at":"2018-03-24T11:51:42Z","author_association":"OWNER","body":"Please see: \r\n\r\nhttps://github.com/TimDettmers/spodernet/commit/e818873618699c8d96706143d83eeddf44e8d308\r\nhttps://github.com/TimDettmers/spodernet/commit/aa1534209d62cad2f7ce32a0b72ae319c2c4db43"}}
{"actor":{"display_login":"TimDettmers"}}