{"action":"created","issue":{"url":"https://api.github.com/repos/adrianlopezroche/fdupes/issues/79","repository_url":"https://api.github.com/repos/adrianlopezroche/fdupes","labels_url":"https://api.github.com/repos/adrianlopezroche/fdupes/issues/79/labels{/name}","comments_url":"https://api.github.com/repos/adrianlopezroche/fdupes/issues/79/comments","events_url":"https://api.github.com/repos/adrianlopezroche/fdupes/issues/79/events","html_url":"https://github.com/adrianlopezroche/fdupes/issues/79","id":208112724,"node_id":"MDU6SXNzdWUyMDgxMTI3MjQ=","number":79,"title":"Possible speed improvement: Don't compute hash of whole big file at first","user":{"login":"Harvie","id":276504,"node_id":"MDQ6VXNlcjI3NjUwNA==","avatar_url":"https://avatars3.githubusercontent.com/u/276504?v=4","gravatar_id":"","url":"https://api.github.com/users/Harvie","html_url":"https://github.com/Harvie","followers_url":"https://api.github.com/users/Harvie/followers","following_url":"https://api.github.com/users/Harvie/following{/other_user}","gists_url":"https://api.github.com/users/Harvie/gists{/gist_id}","starred_url":"https://api.github.com/users/Harvie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Harvie/subscriptions","organizations_url":"https://api.github.com/users/Harvie/orgs","repos_url":"https://api.github.com/users/Harvie/repos","events_url":"https://api.github.com/users/Harvie/events{/privacy}","received_events_url":"https://api.github.com/users/Harvie/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2017-02-16T12:51:25Z","updated_at":"2018-07-08T17:40:09Z","closed_at":null,"author_association":"NONE","body":"I am using fdupes on 6TB archive of audio files (usualy ranging from 100MB to 3GB) and it takes around 30 hours to complete. I have idea how processing of such big files can be sped up.\r\n\r\nMake md5 sum only of first 1MB of the file. If file is bigger than 1MB there should be stored flag that the file was truncated. Also size should be stored along with that hash. Then later when there are more files with same truncated hash and same size, hashes of full file should be computed to check if files are really duplicate in full length.\r\n\r\nThis would eliminate computing of hashes of big files if it's not really necessary (=when there is no other file with same size and same 1MB at the beginning).\r\n\r\n\r\nFurther improvements to this approach:\r\n\r\n1.) Truncate file to 1MB only when it's bigger than 3MB. Doesn't make much sense to truncate 1.2MB file to 1MB and then compute the whole hash again.\r\n\r\n2.) Do not truncate file by just taking first 1MB, but rather concatenate 0.3MB from beginning, 0.3MB from middle and 0.3MB from end and then make truncated hash of this. This should allow performance improvement even for cases where you have lots of big different files that have duplicate header or footer. This should be configurable, because i am not sure that seeking in file will not degrade performance.\r\n\r\n3.) Fiddle with values. Bigfile limit (3MB) and truncate length (1MB) should be configurable, so we can experiment and find most effective defaults. Maybe 1MB is too much. Maybe truncating to 1024B for files bigger than 1MB will be enough."},"comment":{"url":"https://api.github.com/repos/adrianlopezroche/fdupes/issues/comments/403303642","html_url":"https://github.com/adrianlopezroche/fdupes/issues/79#issuecomment-403303642","issue_url":"https://api.github.com/repos/adrianlopezroche/fdupes/issues/79","id":403303642,"node_id":"MDEyOklzc3VlQ29tbWVudDQwMzMwMzY0Mg==","user":{"login":"adrianlopezroche","id":6228477,"node_id":"MDQ6VXNlcjYyMjg0Nzc=","avatar_url":"https://avatars1.githubusercontent.com/u/6228477?v=4","gravatar_id":"","url":"https://api.github.com/users/adrianlopezroche","html_url":"https://github.com/adrianlopezroche","followers_url":"https://api.github.com/users/adrianlopezroche/followers","following_url":"https://api.github.com/users/adrianlopezroche/following{/other_user}","gists_url":"https://api.github.com/users/adrianlopezroche/gists{/gist_id}","starred_url":"https://api.github.com/users/adrianlopezroche/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/adrianlopezroche/subscriptions","organizations_url":"https://api.github.com/users/adrianlopezroche/orgs","repos_url":"https://api.github.com/users/adrianlopezroche/repos","events_url":"https://api.github.com/users/adrianlopezroche/events{/privacy}","received_events_url":"https://api.github.com/users/adrianlopezroche/received_events","type":"User","site_admin":false},"created_at":"2018-07-08T17:40:09Z","updated_at":"2018-07-08T17:40:09Z","author_association":"OWNER","body":"It adds complexity for the sake of a feature most people should not be using. I believe in protecting users from themselves."}}
{"actor":{"display_login":"adrianlopezroche"}}