{"action":"created","issue":{"url":"https://api.github.com/repos/markfasheh/duperemove/issues/161","repository_url":"https://api.github.com/repos/markfasheh/duperemove","labels_url":"https://api.github.com/repos/markfasheh/duperemove/issues/161/labels{/name}","comments_url":"https://api.github.com/repos/markfasheh/duperemove/issues/161/comments","events_url":"https://api.github.com/repos/markfasheh/duperemove/issues/161/events","html_url":"https://github.com/markfasheh/duperemove/pull/161","id":186911310,"number":161,"title":"[RFC] Increase performance of duplicate digest search","user":{"login":"Floyddotnet","id":498914,"avatar_url":"https://avatars.githubusercontent.com/u/498914?v=3","gravatar_id":"","url":"https://api.github.com/users/Floyddotnet","html_url":"https://github.com/Floyddotnet","followers_url":"https://api.github.com/users/Floyddotnet/followers","following_url":"https://api.github.com/users/Floyddotnet/following{/other_user}","gists_url":"https://api.github.com/users/Floyddotnet/gists{/gist_id}","starred_url":"https://api.github.com/users/Floyddotnet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Floyddotnet/subscriptions","organizations_url":"https://api.github.com/users/Floyddotnet/orgs","repos_url":"https://api.github.com/users/Floyddotnet/repos","events_url":"https://api.github.com/users/Floyddotnet/events{/privacy}","received_events_url":"https://api.github.com/users/Floyddotnet/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-11-02T20:29:02Z","updated_at":"2016-11-03T20:11:27Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/markfasheh/duperemove/pulls/161","html_url":"https://github.com/markfasheh/duperemove/pull/161","diff_url":"https://github.com/markfasheh/duperemove/pull/161.diff","patch_url":"https://github.com/markfasheh/duperemove/pull/161.patch"},"body":"This patch increase the performance of duplicate searches by using an index table for digest.\r\nIf we deduplicate very large directorys (30 TB for example) the old implementation of dbfile_load_hashes will take many hours if you didn't have enough memory.\r\n\r\nDetails for a 3,9 TB directory, with 4723 objects:\r\n\r\nold implementation of dbfile_load_hashes took 36593ms\r\nnew  implementation of dbfile_load_hashes took 11ms\r\n\r\nNote:\r\n\r\nIf these changes are encouraged i will do some more work. Currently on the todo list:\r\n1. Increase db version (DB_FILE_MAJOR to 3)\r\n2. Add migration script for older db-files (DB_FILE_MAJOR == 2)\r\n3. Delete orphaned entries in digest-table (refCount == 0)\r\n\r\nImportant:\r\n\r\nI am not a c / c ++ developer. Please reviewe my changes also in regard to errors in the memory management."},"comment":{"url":"https://api.github.com/repos/markfasheh/duperemove/issues/comments/258259786","html_url":"https://github.com/markfasheh/duperemove/pull/161#issuecomment-258259786","issue_url":"https://api.github.com/repos/markfasheh/duperemove/issues/161","id":258259786,"user":{"login":"Floyddotnet","id":498914,"avatar_url":"https://avatars.githubusercontent.com/u/498914?v=3","gravatar_id":"","url":"https://api.github.com/users/Floyddotnet","html_url":"https://github.com/Floyddotnet","followers_url":"https://api.github.com/users/Floyddotnet/followers","following_url":"https://api.github.com/users/Floyddotnet/following{/other_user}","gists_url":"https://api.github.com/users/Floyddotnet/gists{/gist_id}","starred_url":"https://api.github.com/users/Floyddotnet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Floyddotnet/subscriptions","organizations_url":"https://api.github.com/users/Floyddotnet/orgs","repos_url":"https://api.github.com/users/Floyddotnet/repos","events_url":"https://api.github.com/users/Floyddotnet/events{/privacy}","received_events_url":"https://api.github.com/users/Floyddotnet/received_events","type":"User","site_admin":false},"created_at":"2016-11-03T20:11:27Z","updated_at":"2016-11-03T20:11:27Z","body":"Please take a look at me new branche https://github.com/Floyddotnet/duperemove/commits/digest_trigger\r\n\r\nA new version with sqlite triggers"}}
{"actor":{"display_login":"Floyddotnet"}}