{"action":"created","issue":{"url":"https://api.github.com/repos/abrt/retrace-server/issues/182","repository_url":"https://api.github.com/repos/abrt/retrace-server","labels_url":"https://api.github.com/repos/abrt/retrace-server/issues/182/labels{/name}","comments_url":"https://api.github.com/repos/abrt/retrace-server/issues/182/comments","events_url":"https://api.github.com/repos/abrt/retrace-server/issues/182/events","html_url":"https://github.com/abrt/retrace-server/pull/182","id":308668539,"number":182,"title":"Add dedup_vmcore to RetraceWorker and call from retrace-server-cleanup","user":{"login":"DaveWysochanskiRH","id":17653883,"avatar_url":"https://avatars1.githubusercontent.com/u/17653883?v=4","gravatar_id":"","url":"https://api.github.com/users/DaveWysochanskiRH","html_url":"https://github.com/DaveWysochanskiRH","followers_url":"https://api.github.com/users/DaveWysochanskiRH/followers","following_url":"https://api.github.com/users/DaveWysochanskiRH/following{/other_user}","gists_url":"https://api.github.com/users/DaveWysochanskiRH/gists{/gist_id}","starred_url":"https://api.github.com/users/DaveWysochanskiRH/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DaveWysochanskiRH/subscriptions","organizations_url":"https://api.github.com/users/DaveWysochanskiRH/orgs","repos_url":"https://api.github.com/users/DaveWysochanskiRH/repos","events_url":"https://api.github.com/users/DaveWysochanskiRH/events{/privacy}","received_events_url":"https://api.github.com/users/DaveWysochanskiRH/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-03-26T17:38:30Z","updated_at":"2018-03-26T17:45:51Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/abrt/retrace-server/pulls/182","html_url":"https://github.com/abrt/retrace-server/pull/182","diff_url":"https://github.com/abrt/retrace-server/pull/182.diff","patch_url":"https://github.com/abrt/retrace-server/pull/182.patch"},"body":"Today retrace-server allows the same ftp vmcore to be submitted multiple times.\r\nThis means disk space can be consumed for the same vmcore / multiple tasks.\r\nIn our production system, we have seen where this may take up around 10% of the\r\nspace, or 4TB so it is not insignificant.  To eliminate this overhead,\r\nintroduce dedup_vmcore method to RetraceTask.\r\n\r\nThe dedup_vmcore method takes advantage of the fact we usually have the\r\nmd5sum of a downloaded file prior to processing it.  While this md5sum\r\nis not the same as the md5sum of the vmcore, the ending vmcore file after\r\nall processing should always be the same if the same archive is submitted\r\ntwice (anything else is a pathalogical case we don't care about).  Also\r\nwe can do a simple 'size' check and if the md5sum is the same and the size\r\nis the same we have enough certainty it is ok to deduplicate the vmcores.\r\n\r\nAdd a new section to retrace-server-cleanup where we get a list of tasks\r\nthat have completed (have 'finished_time', and either STATUS_SUCCESS or\r\nSTATUS_FAIL) as well as have a md5sum file.  Then, go through these tasks\r\nbuilding up a dictionary based on each unique md5sum which is the first\r\nportion of the md5sum file (Note that we actually store the output of\r\n'md5sum' command in the 'md5sum' file on the task).  While iteration of\r\nthe tasks with md5sums, if we look in the dictionary and one task is\r\nalready there, call the new dedup_vmcore method.  This method first does\r\na couple checks to make sure these two vmcores are the same and should be\r\ndeduped:\r\n- existence of each file (stat on the file)\r\n- inodes are different\r\n- sizes are the same\r\n- md5sums (of the originally downloaded file) are the same\r\n\r\nIf all of the above checks out, then the dedup_vmcore proceeds with the\r\nnext steps:\r\n- create a temporary hardlink name based on the second vmcore name\r\n- create a hardlink from the second vmcore to the first using the temp name\r\n- remove the second vmcore\r\n- change the hardlink name to the name of the second vmcore that was removed\r\n\r\nIf all of the above works, then we have have safely deduplicated the\r\nvmcore file for two tasks that are equal.  The return value of dedup_vmcore\r\nis the number of bytes saved in the deplication, and in a successful\r\noperation, the return will be non-zero, while in a failure it will be 0.\r\n\r\nIf something goes wrong in the process of deduplicating the vmcores, or\r\neven if we successfully dedup the vmcores, we write a message into the\r\nsecond task's retrace-log so we know what the cleanup job has done.\r\n\r\nSigned-off-by: Dave Wysochanski <dwysocha@redhat.com>"},"comment":{"url":"https://api.github.com/repos/abrt/retrace-server/issues/comments/376252028","html_url":"https://github.com/abrt/retrace-server/pull/182#issuecomment-376252028","issue_url":"https://api.github.com/repos/abrt/retrace-server/issues/182","id":376252028,"user":{"login":"DaveWysochanskiRH","id":17653883,"avatar_url":"https://avatars1.githubusercontent.com/u/17653883?v=4","gravatar_id":"","url":"https://api.github.com/users/DaveWysochanskiRH","html_url":"https://github.com/DaveWysochanskiRH","followers_url":"https://api.github.com/users/DaveWysochanskiRH/followers","following_url":"https://api.github.com/users/DaveWysochanskiRH/following{/other_user}","gists_url":"https://api.github.com/users/DaveWysochanskiRH/gists{/gist_id}","starred_url":"https://api.github.com/users/DaveWysochanskiRH/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DaveWysochanskiRH/subscriptions","organizations_url":"https://api.github.com/users/DaveWysochanskiRH/orgs","repos_url":"https://api.github.com/users/DaveWysochanskiRH/repos","events_url":"https://api.github.com/users/DaveWysochanskiRH/events{/privacy}","received_events_url":"https://api.github.com/users/DaveWysochanskiRH/received_events","type":"User","site_admin":false},"created_at":"2018-03-26T17:45:51Z","updated_at":"2018-03-26T17:45:51Z","author_association":"CONTRIBUTOR","body":"Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1558903\r\nThis new patch addresses all issues in the previous patch.  Changes are:\r\n- move dedup_vmcore from RetraceTask to RetraceWorker where it fits much better\r\n- use start_logging / end_logging and standard log_* functions to write to the retrace_log\r\n- check to make sure the task is truly finished before adding to md5sum array\r\n\r\nPlease review but do not merge it yet as I'm still testing though I don't anticipate the patch will change much unless you find some show-stopper.\r\n\r\nThanks."}}
{"actor":{"display_login":"DaveWysochanskiRH"}}