{"action":"created","comment":{"url":"https://api.github.com/repos/cooperative-computing-lab/cctools/pulls/comments/76280183","id":76280183,"diff_hunk":"@@ -0,0 +1,287 @@\n+/*\n+ * Copyright (C) 2013- The University of Notre Dame\n+ * This software is distributed under the GNU General Public License.\n+ * See the file COPYING for details.\n+ */\n+\n+#include <assert.h>\n+#include <dirent.h>\n+#include <errno.h>\n+#include <libgen.h>\n+#include <limits.h>\n+#include <string.h>\n+#include <stdio.h>\n+#include <sys/stat.h>\n+#include <sys/types.h>\n+#include <unistd.h>\n+\n+#include \"copy_tree.h\"\n+#include \"copy_stream.h\"\n+#include \"create_dir.h\"\n+#include \"debug.h\"\n+#include \"path.h\"\n+#include \"xxmalloc.h\"\n+\n+mode_t default_dirmode = S_IRWXU | S_IRGRP | S_IXGRP | S_IROTH | S_IXOTH;\n+\n+int copy_dir_real(const char *source, const char *target);\n+\n+int copy_symlink(const char *source, const char *target) {\n+\tchar linkname[PATH_MAX];\n+\tssize_t r;\n+\n+\tassert(source);\n+\tassert(target);\n+\n+\tif(!access(target, F_OK)) {\n+\t\tLDEBUG(\"%s already exists!\\n\", target);\n+\t\treturn -1;\n+\t}\n+\n+\tr = readlink(source, linkname, sizeof(linkname));\n+\tif(r == -1) {\n+\t\tLDEBUG(\"readlink(`%s`) failed: %s!\\n\", source, strerror(errno));\n+\t\treturn -1;\n+\t}\n+\tlinkname[r] = '\\0';\n+\n+\tif(symlink(linkname, target)) {\n+\t\tLDEBUG(\"symlink(`%s`, `%s`) failed: %s\\n\", linkname, target, strerror(errno));\n+\t\treturn -1;\n+\t}\n+\treturn 0;\n+}\n+\n+/* return 0 on success, return -1 on failure. */\n+int copy_direntry(const char *s, const char *t) {\n+\tstruct stat s_stat;\n+\n+\tassert(s);\n+\tassert(t);\n+\n+\tif(lstat(s, &s_stat)) {\n+\t\tLDEBUG(\"lstat(`%s`): %s\\n\", s, strerror(errno));\n+\t\treturn -1;\n+\t}\n+\n+\tif(S_ISDIR(s_stat.st_mode)) {\n+\t\tif(!create_dir(t, (int)default_dirmode)) return -1;\n+\t\treturn copy_dir_real(s, t);\n+\t} else if(S_ISREG(s_stat.st_mode)) {\n+\t\treturn copy_file_to_file(s, t);\n+\t} else if(S_ISLNK(s_stat.st_mode)) {\n+\t\treturn copy_symlink(s, t);\n+\t} else {\n+\t\tLDEBUG(\"Ignore Copying %s: only dir, regular files, and symlink are supported!\\n\", s);\n+\t\treturn -1;\n+\t}\n+}\n+\n+/* copy_dir_real copies from source to target.\n+ * @param source: a file path.\n+ * @param target: a file path which already exists.\n+ * return 0 on success, return -1 on failure.\n+ */\n+int copy_dir_real(const char *source, const char *target) {\n+\tDIR *dir;\n+\tstruct dirent *entry;\n+\tint rc = 0;\n+\n+\tassert(source);\n+\tassert(target);\n+\n+\tif((dir = opendir(source)) == NULL) {\n+\t\tLDEBUG(\"opendir(`%s`) failed: %s!\\n\", source, strerror(errno));\n+\t\treturn -1;\n+\t}\n+\n+\twhile((entry = readdir(dir)))\n+\t{\n+\t\tchar *s = NULL;\n+\t\tchar *t = NULL;\n+\n+\t\tif(!strcmp(entry->d_name, \".\") || !strcmp(entry->d_name, \"..\"))\n+\t\t\tcontinue;\n+\n+\t\ts = path_concat(source, entry->d_name);\n+\t\tif(!s) {\n+\t\t\trc = -1;\n+\t\t\tgoto finish;\n+\t\t}\n+\n+\t\tt = path_concat(target, entry->d_name);\n+\t\tif(!t) {\n+\t\t\tfree(s);\n+\t\t\trc = -1;\n+\t\t\tgoto finish;\n+\t\t}\n+\n+\t\t/* try-the-best copy: if failed to copy one direntry, move on to the next direntry instead of exiting. */\n+\t\tcopy_direntry(s, t);","path":"dttools/src/copy_tree.c","position":120,"original_position":120,"commit_id":"6d9646f0038db2074f62eb3b9cb3490e6296452b","original_commit_id":"e7d10bf6e5444f7a8d065b458f5bda60d6c12e57","user":{"login":"btovar","id":3081826,"avatar_url":"https://avatars.githubusercontent.com/u/3081826?v=3","gravatar_id":"","url":"https://api.github.com/users/btovar","html_url":"https://github.com/btovar","followers_url":"https://api.github.com/users/btovar/followers","following_url":"https://api.github.com/users/btovar/following{/other_user}","gists_url":"https://api.github.com/users/btovar/gists{/gist_id}","starred_url":"https://api.github.com/users/btovar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/btovar/subscriptions","organizations_url":"https://api.github.com/users/btovar/orgs","repos_url":"https://api.github.com/users/btovar/repos","events_url":"https://api.github.com/users/btovar/events{/privacy}","received_events_url":"https://api.github.com/users/btovar/received_events","type":"User","site_admin":false},"body":"Let's stop at the first failure. The faster we return failure the better.","created_at":"2016-08-25T16:43:55Z","updated_at":"2016-08-25T16:43:55Z","html_url":"https://github.com/cooperative-computing-lab/cctools/pull/1239#discussion_r76280183","pull_request_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/pulls/1239","_links":{"self":{"href":"https://api.github.com/repos/cooperative-computing-lab/cctools/pulls/comments/76280183"},"html":{"href":"https://github.com/cooperative-computing-lab/cctools/pull/1239#discussion_r76280183"},"pull_request":{"href":"https://api.github.com/repos/cooperative-computing-lab/cctools/pulls/1239"}}},"pull_request":{"url":"https://api.github.com/repos/cooperative-computing-lab/cctools/pulls/1239","id":65200231,"html_url":"https://github.com/cooperative-computing-lab/cctools/pull/1239","diff_url":"https://github.com/cooperative-computing-lab/cctools/pull/1239.diff","patch_url":"https://github.com/cooperative-computing-lab/cctools/pull/1239.patch","issue_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/issues/1239","number":1239,"state":"open","locked":false,"title":"add makeflow support for specifying input deps","user":{"login":"hmeng-19","id":5217421,"avatar_url":"https://avatars.githubusercontent.com/u/5217421?v=3","gravatar_id":"","url":"https://api.github.com/users/hmeng-19","html_url":"https://github.com/hmeng-19","followers_url":"https://api.github.com/users/hmeng-19/followers","following_url":"https://api.github.com/users/hmeng-19/following{/other_user}","gists_url":"https://api.github.com/users/hmeng-19/gists{/gist_id}","starred_url":"https://api.github.com/users/hmeng-19/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hmeng-19/subscriptions","organizations_url":"https://api.github.com/users/hmeng-19/orgs","repos_url":"https://api.github.com/users/hmeng-19/repos","events_url":"https://api.github.com/users/hmeng-19/events{/privacy}","received_events_url":"https://api.github.com/users/hmeng-19/received_events","type":"User","site_admin":false},"body":"Problem:\r\nCurrently, Makeflow allows the user to use absolute paths to specify the input dependencies, which affects the portability of the makeflow.\r\n\r\nSolution:\r\nLimit the behavior of a makeflow within the current working directory, and allow the user to specify the source and target of each input dependency for a makeflow through `--mounts <mountfile >` option. Every line of mountfile should be in the format of `<target> <source>`.\r\n\r\nHere is an example of a mountfile:\r\n\r\n    curl /usr/bin/curl\r\n    convert ../bin/convert\r\n    data/file1 /home/bob/input1\r\n\r\nTo execute a makeflow with a mountfile, named mountfile, specifying its input dependencies:\r\n`makeflow --mounts mountfile example.makeflow`\r\nBefore starting parsing the DAG, Makeflow first parses each line of the mountfile, and copies the specified dependency from the location specified by source field to the location specified by the target field.\r\n\r\nTo cleanup a makeflow together with the input dependencies installed through the --mounts option, run the following command:\r\n`makeflow --mounts mountfile -c example.makeflow`\r\nThe input dependencies set up by the --mounts option can only be cleaned up when the makeflow clean mode is set to all. After Makeflow cleans up the outputs and intermediate files, each target specified in the mountfile will be removed.","created_at":"2016-04-04T19:35:57Z","updated_at":"2016-08-25T16:43:55Z","closed_at":null,"merged_at":null,"merge_commit_sha":"bd31ee8a94d4e1705221a4e66304574d1ec9fc12","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/pulls/1239/commits","review_comments_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/pulls/1239/comments","review_comment_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/pulls/comments{/number}","comments_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/issues/1239/comments","statuses_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/statuses/6d9646f0038db2074f62eb3b9cb3490e6296452b","head":{"label":"hmeng-19:makeflow_mountfile","ref":"makeflow_mountfile","sha":"6d9646f0038db2074f62eb3b9cb3490e6296452b","user":{"login":"hmeng-19","id":5217421,"avatar_url":"https://avatars.githubusercontent.com/u/5217421?v=3","gravatar_id":"","url":"https://api.github.com/users/hmeng-19","html_url":"https://github.com/hmeng-19","followers_url":"https://api.github.com/users/hmeng-19/followers","following_url":"https://api.github.com/users/hmeng-19/following{/other_user}","gists_url":"https://api.github.com/users/hmeng-19/gists{/gist_id}","starred_url":"https://api.github.com/users/hmeng-19/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hmeng-19/subscriptions","organizations_url":"https://api.github.com/users/hmeng-19/orgs","repos_url":"https://api.github.com/users/hmeng-19/repos","events_url":"https://api.github.com/users/hmeng-19/events{/privacy}","received_events_url":"https://api.github.com/users/hmeng-19/received_events","type":"User","site_admin":false},"repo":{"id":12068793,"name":"cctools","full_name":"hmeng-19/cctools","owner":{"login":"hmeng-19","id":5217421,"avatar_url":"https://avatars.githubusercontent.com/u/5217421?v=3","gravatar_id":"","url":"https://api.github.com/users/hmeng-19","html_url":"https://github.com/hmeng-19","followers_url":"https://api.github.com/users/hmeng-19/followers","following_url":"https://api.github.com/users/hmeng-19/following{/other_user}","gists_url":"https://api.github.com/users/hmeng-19/gists{/gist_id}","starred_url":"https://api.github.com/users/hmeng-19/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hmeng-19/subscriptions","organizations_url":"https://api.github.com/users/hmeng-19/orgs","repos_url":"https://api.github.com/users/hmeng-19/repos","events_url":"https://api.github.com/users/hmeng-19/events{/privacy}","received_events_url":"https://api.github.com/users/hmeng-19/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/hmeng-19/cctools","description":"The Cooperative Computing Tools (cctools) enable large scale distributed computations to harness hundreds to thousands of machines from clusters, clouds, and grids.","fork":true,"url":"https://api.github.com/repos/hmeng-19/cctools","forks_url":"https://api.github.com/repos/hmeng-19/cctools/forks","keys_url":"https://api.github.com/repos/hmeng-19/cctools/keys{/key_id}","collaborators_url":"https://api.github.com/repos/hmeng-19/cctools/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/hmeng-19/cctools/teams","hooks_url":"https://api.github.com/repos/hmeng-19/cctools/hooks","issue_events_url":"https://api.github.com/repos/hmeng-19/cctools/issues/events{/number}","events_url":"https://api.github.com/repos/hmeng-19/cctools/events","assignees_url":"https://api.github.com/repos/hmeng-19/cctools/assignees{/user}","branches_url":"https://api.github.com/repos/hmeng-19/cctools/branches{/branch}","tags_url":"https://api.github.com/repos/hmeng-19/cctools/tags","blobs_url":"https://api.github.com/repos/hmeng-19/cctools/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/hmeng-19/cctools/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/hmeng-19/cctools/git/refs{/sha}","trees_url":"https://api.github.com/repos/hmeng-19/cctools/git/trees{/sha}","statuses_url":"https://api.github.com/repos/hmeng-19/cctools/statuses/{sha}","languages_url":"https://api.github.com/repos/hmeng-19/cctools/languages","stargazers_url":"https://api.github.com/repos/hmeng-19/cctools/stargazers","contributors_url":"https://api.github.com/repos/hmeng-19/cctools/contributors","subscribers_url":"https://api.github.com/repos/hmeng-19/cctools/subscribers","subscription_url":"https://api.github.com/repos/hmeng-19/cctools/subscription","commits_url":"https://api.github.com/repos/hmeng-19/cctools/commits{/sha}","git_commits_url":"https://api.github.com/repos/hmeng-19/cctools/git/commits{/sha}","comments_url":"https://api.github.com/repos/hmeng-19/cctools/comments{/number}","issue_comment_url":"https://api.github.com/repos/hmeng-19/cctools/issues/comments{/number}","contents_url":"https://api.github.com/repos/hmeng-19/cctools/contents/{+path}","compare_url":"https://api.github.com/repos/hmeng-19/cctools/compare/{base}...{head}","merges_url":"https://api.github.com/repos/hmeng-19/cctools/merges","archive_url":"https://api.github.com/repos/hmeng-19/cctools/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/hmeng-19/cctools/downloads","issues_url":"https://api.github.com/repos/hmeng-19/cctools/issues{/number}","pulls_url":"https://api.github.com/repos/hmeng-19/cctools/pulls{/number}","milestones_url":"https://api.github.com/repos/hmeng-19/cctools/milestones{/number}","notifications_url":"https://api.github.com/repos/hmeng-19/cctools/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/hmeng-19/cctools/labels{/name}","releases_url":"https://api.github.com/repos/hmeng-19/cctools/releases{/id}","deployments_url":"https://api.github.com/repos/hmeng-19/cctools/deployments","created_at":"2013-08-12T23:37:42Z","updated_at":"2015-01-19T02:02:35Z","pushed_at":"2016-08-25T01:25:09Z","git_url":"git://github.com/hmeng-19/cctools.git","ssh_url":"git@github.com:hmeng-19/cctools.git","clone_url":"https://github.com/hmeng-19/cctools.git","svn_url":"https://github.com/hmeng-19/cctools","homepage":"http://www.nd.edu/~ccl/","size":38549,"stargazers_count":0,"watchers_count":0,"language":"C","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"cooperative-computing-lab:master","ref":"master","sha":"a850651bd7f3a9f2a156dd1fc2495dfba7edb9b1","user":{"login":"cooperative-computing-lab","id":2520573,"avatar_url":"https://avatars.githubusercontent.com/u/2520573?v=3","gravatar_id":"","url":"https://api.github.com/users/cooperative-computing-lab","html_url":"https://github.com/cooperative-computing-lab","followers_url":"https://api.github.com/users/cooperative-computing-lab/followers","following_url":"https://api.github.com/users/cooperative-computing-lab/following{/other_user}","gists_url":"https://api.github.com/users/cooperative-computing-lab/gists{/gist_id}","starred_url":"https://api.github.com/users/cooperative-computing-lab/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cooperative-computing-lab/subscriptions","organizations_url":"https://api.github.com/users/cooperative-computing-lab/orgs","repos_url":"https://api.github.com/users/cooperative-computing-lab/repos","events_url":"https://api.github.com/users/cooperative-computing-lab/events{/privacy}","received_events_url":"https://api.github.com/users/cooperative-computing-lab/received_events","type":"Organization","site_admin":false},"repo":{"id":8759937,"name":"cctools","full_name":"cooperative-computing-lab/cctools","owner":{"login":"cooperative-computing-lab","id":2520573,"avatar_url":"https://avatars.githubusercontent.com/u/2520573?v=3","gravatar_id":"","url":"https://api.github.com/users/cooperative-computing-lab","html_url":"https://github.com/cooperative-computing-lab","followers_url":"https://api.github.com/users/cooperative-computing-lab/followers","following_url":"https://api.github.com/users/cooperative-computing-lab/following{/other_user}","gists_url":"https://api.github.com/users/cooperative-computing-lab/gists{/gist_id}","starred_url":"https://api.github.com/users/cooperative-computing-lab/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cooperative-computing-lab/subscriptions","organizations_url":"https://api.github.com/users/cooperative-computing-lab/orgs","repos_url":"https://api.github.com/users/cooperative-computing-lab/repos","events_url":"https://api.github.com/users/cooperative-computing-lab/events{/privacy}","received_events_url":"https://api.github.com/users/cooperative-computing-lab/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/cooperative-computing-lab/cctools","description":"The Cooperative Computing Tools (cctools) enable large scale distributed computations to harness hundreds to thousands of machines from clusters, clouds, and grids.","fork":false,"url":"https://api.github.com/repos/cooperative-computing-lab/cctools","forks_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/forks","keys_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/keys{/key_id}","collaborators_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/teams","hooks_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/hooks","issue_events_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/issues/events{/number}","events_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/events","assignees_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/assignees{/user}","branches_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/branches{/branch}","tags_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/tags","blobs_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/git/refs{/sha}","trees_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/git/trees{/sha}","statuses_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/statuses/{sha}","languages_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/languages","stargazers_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/stargazers","contributors_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/contributors","subscribers_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/subscribers","subscription_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/subscription","commits_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/commits{/sha}","git_commits_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/git/commits{/sha}","comments_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/comments{/number}","issue_comment_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/issues/comments{/number}","contents_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/contents/{+path}","compare_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/compare/{base}...{head}","merges_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/merges","archive_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/downloads","issues_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/issues{/number}","pulls_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/pulls{/number}","milestones_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/milestones{/number}","notifications_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/labels{/name}","releases_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/releases{/id}","deployments_url":"https://api.github.com/repos/cooperative-computing-lab/cctools/deployments","created_at":"2013-03-13T19:39:45Z","updated_at":"2016-08-17T13:01:09Z","pushed_at":"2016-08-25T15:54:20Z","git_url":"git://github.com/cooperative-computing-lab/cctools.git","ssh_url":"git@github.com:cooperative-computing-lab/cctools.git","clone_url":"https://github.com/cooperative-computing-lab/cctools.git","svn_url":"https://github.com/cooperative-computing-lab/cctools","homepage":"http://ccl.cse.nd.edu","size":91293,"stargazers_count":41,"watchers_count":41,"language":"C","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":57,"mirror_url":null,"open_issues_count":54,"forks":57,"open_issues":54,"watchers":41,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/cooperative-computing-lab/cctools/pulls/1239"},"html":{"href":"https://github.com/cooperative-computing-lab/cctools/pull/1239"},"issue":{"href":"https://api.github.com/repos/cooperative-computing-lab/cctools/issues/1239"},"comments":{"href":"https://api.github.com/repos/cooperative-computing-lab/cctools/issues/1239/comments"},"review_comments":{"href":"https://api.github.com/repos/cooperative-computing-lab/cctools/pulls/1239/comments"},"review_comment":{"href":"https://api.github.com/repos/cooperative-computing-lab/cctools/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/cooperative-computing-lab/cctools/pulls/1239/commits"},"statuses":{"href":"https://api.github.com/repos/cooperative-computing-lab/cctools/statuses/6d9646f0038db2074f62eb3b9cb3490e6296452b"}}}}
{"actor":{"display_login":"btovar"}}