{"action":"created","issue":{"url":"https://api.github.com/repos/hpc/mpifileutils/issues/103","repository_url":"https://api.github.com/repos/hpc/mpifileutils","labels_url":"https://api.github.com/repos/hpc/mpifileutils/issues/103/labels{/name}","comments_url":"https://api.github.com/repos/hpc/mpifileutils/issues/103/comments","events_url":"https://api.github.com/repos/hpc/mpifileutils/issues/103/events","html_url":"https://github.com/hpc/mpifileutils/pull/103","id":296258482,"number":103,"title":"dcp: rework the data copy routine with libcircle","user":{"login":"KnightKu","id":6672858,"avatar_url":"https://avatars2.githubusercontent.com/u/6672858?v=4","gravatar_id":"","url":"https://api.github.com/users/KnightKu","html_url":"https://github.com/KnightKu","followers_url":"https://api.github.com/users/KnightKu/followers","following_url":"https://api.github.com/users/KnightKu/following{/other_user}","gists_url":"https://api.github.com/users/KnightKu/gists{/gist_id}","starred_url":"https://api.github.com/users/KnightKu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/KnightKu/subscriptions","organizations_url":"https://api.github.com/users/KnightKu/orgs","repos_url":"https://api.github.com/users/KnightKu/repos","events_url":"https://api.github.com/users/KnightKu/events{/privacy}","received_events_url":"https://api.github.com/users/KnightKu/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-02-12T03:34:10Z","updated_at":"2018-02-14T00:40:07Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/hpc/mpifileutils/pulls/103","html_url":"https://github.com/hpc/mpifileutils/pull/103","diff_url":"https://github.com/hpc/mpifileutils/pull/103.diff","patch_url":"https://github.com/hpc/mpifileutils/pull/103.patch"},"body":"Rework the data copy routinue with libcircle, so that we can trace\r\nthe progress timely, like dwalk.\r\n\r\n[2018-02-12T11:06:41] [0] [../../../src/common/mfu_flist_copy.c:1508] Copying data.\r\n2018-02-12 11:06:51: files copied 13.45%..[844/6274]...\r\n2018-02-12 11:07:02: files copied 28.36%..[1779/6274]...\r\n2018-02-12 11:07:12: files copied 46.21%..[2899/6274]...\r\n2018-02-12 11:07:22: files copied 69.51%..[4361/6274]...\r\n2018-02-12 11:07:32: files copied 82.07%..[5149/6274]...\r\n2018-02-12 11:07:41: files copied 100.00%..[6274/6274]...\r\n\r\nSigned-off-by: Gu Zheng <cengku@gmail.com>"},"comment":{"url":"https://api.github.com/repos/hpc/mpifileutils/issues/comments/365455599","html_url":"https://github.com/hpc/mpifileutils/pull/103#issuecomment-365455599","issue_url":"https://api.github.com/repos/hpc/mpifileutils/issues/103","id":365455599,"user":{"login":"adammoody","id":1105742,"avatar_url":"https://avatars1.githubusercontent.com/u/1105742?v=4","gravatar_id":"","url":"https://api.github.com/users/adammoody","html_url":"https://github.com/adammoody","followers_url":"https://api.github.com/users/adammoody/followers","following_url":"https://api.github.com/users/adammoody/following{/other_user}","gists_url":"https://api.github.com/users/adammoody/gists{/gist_id}","starred_url":"https://api.github.com/users/adammoody/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/adammoody/subscriptions","organizations_url":"https://api.github.com/users/adammoody/orgs","repos_url":"https://api.github.com/users/adammoody/repos","events_url":"https://api.github.com/users/adammoody/events{/privacy}","received_events_url":"https://api.github.com/users/adammoody/received_events","type":"User","site_admin":false},"created_at":"2018-02-14T00:40:07Z","updated_at":"2018-02-14T00:40:07Z","author_association":"OWNER","body":"@KnightKu, The main reason we implemented this second dcp tool w/o libcircle is to prevent consecutive regions of files from being randomly assigned to processes.  This has the effect of reducing file byte-range lock contention within lustre.  We found this to be helpful when using a lot of processes to copy a single big file (100GB+).  Adding libcircle in there would cause that problem to come back.\r\n\r\nHaving said that, it would be nice to expose this an option to the user.  We could default to the pre-assigned and static method we have now and allow the user to switch to this libcircle version, or perhaps we could chose dynamically based on the stats we gather from the file list.\r\n\r\nCould you modify this to provide both methods instead of replacing the one that was there?\r\n\r\nAlso, it'd be useful to report bytes copied, too, in case the user is moving a single large file."}}
{"actor":{"display_login":"adammoody"}}