{"action":"created","issue":{"url":"https://api.github.com/repos/markfasheh/duperemove/issues/122","repository_url":"https://api.github.com/repos/markfasheh/duperemove","labels_url":"https://api.github.com/repos/markfasheh/duperemove/issues/122/labels{/name}","comments_url":"https://api.github.com/repos/markfasheh/duperemove/issues/122/comments","events_url":"https://api.github.com/repos/markfasheh/duperemove/issues/122/events","html_url":"https://github.com/markfasheh/duperemove/issues/122","id":155597449,"number":122,"title":"find_all_dupes takes >1 day","user":{"login":"zielmicha","id":364357,"avatar_url":"https://avatars.githubusercontent.com/u/364357?v=3","gravatar_id":"","url":"https://api.github.com/users/zielmicha","html_url":"https://github.com/zielmicha","followers_url":"https://api.github.com/users/zielmicha/followers","following_url":"https://api.github.com/users/zielmicha/following{/other_user}","gists_url":"https://api.github.com/users/zielmicha/gists{/gist_id}","starred_url":"https://api.github.com/users/zielmicha/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zielmicha/subscriptions","organizations_url":"https://api.github.com/users/zielmicha/orgs","repos_url":"https://api.github.com/users/zielmicha/repos","events_url":"https://api.github.com/users/zielmicha/events{/privacy}","received_events_url":"https://api.github.com/users/zielmicha/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2016-05-18T20:46:06Z","updated_at":"2016-06-06T11:31:07Z","closed_at":null,"body":"I'm trying to deduplicate about 2 TB of VM images backup (the images are mostly zeros). First hashing phase completes in about 2 hours, but then CPU-bound phase starts and hasn't ended yet.\r\n\"Hashing completed. Calculating duplicate extents - this may take some time.\"\r\nFirst it has halted on 0% and then on about 40% and it's running for >24 hours. I've tried running both master (for few hours) and v0.10.\r\n\r\nIs it expected? \r\n(From quick glance at the code, it looks like walk_dupe_hashes can be quadratic in worst case)."},"comment":{"url":"https://api.github.com/repos/markfasheh/duperemove/issues/comments/223933753","html_url":"https://github.com/markfasheh/duperemove/issues/122#issuecomment-223933753","issue_url":"https://api.github.com/repos/markfasheh/duperemove/issues/122","id":223933753,"user":{"login":"zielmicha","id":364357,"avatar_url":"https://avatars.githubusercontent.com/u/364357?v=3","gravatar_id":"","url":"https://api.github.com/users/zielmicha","html_url":"https://github.com/zielmicha","followers_url":"https://api.github.com/users/zielmicha/followers","following_url":"https://api.github.com/users/zielmicha/following{/other_user}","gists_url":"https://api.github.com/users/zielmicha/gists{/gist_id}","starred_url":"https://api.github.com/users/zielmicha/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zielmicha/subscriptions","organizations_url":"https://api.github.com/users/zielmicha/orgs","repos_url":"https://api.github.com/users/zielmicha/repos","events_url":"https://api.github.com/users/zielmicha/events{/privacy}","received_events_url":"https://api.github.com/users/zielmicha/received_events","type":"User","site_admin":false},"created_at":"2016-06-06T11:31:07Z","updated_at":"2016-06-06T11:31:07Z","body":"I have created a prototype of this algorithm:\r\nhttps://github.com/zielmicha/btrfs-dedup\r\n\r\nUnfortunately it seems I have hit some kernel bug - after slamming kernel with millions of incorrect ioctl(BTRFS_IOC_FILE_EXTENT_SAME) it made one file permanently broken - BTRFS kernel thread hangs when I try to access it using show-shared-extents. I didn't have time to investigate it yet."}}
{"actor":{"display_login":"zielmicha"}}