{"action":"created","comment":{"url":"https://api.github.com/repos/pachyderm/pachyderm/pulls/comments/219277123","pull_request_review_id":157415774,"id":219277123,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDIxOTI3NzEyMw==","diff_hunk":"@@ -2223,25 +2274,78 @@ func (d *driver) getFile(ctx context.Context, file *pfs.File, offset int64, size\n \tif err != nil {\n \t\treturn nil, err\n \t}\n-\n \tpaths, err := tree.Glob(file.Path)\n \tif err != nil {\n \t\treturn nil, err\n \t}\n+\tvar objects []*pfs.Object\n+\tvar totalSize int64\n+\t// Add directory path/nodes to unique set of paths\n+\t// so we can collect and header/footer objects as we traverse the\n+\t// paths below\n \tfor path, node := range paths {\n-\t\tif node.FileNode == nil {\n-\t\t\tdelete(paths, path)\n+\t\tif node.FileNode != nil {\n+\t\t\tdirPath := path\n+\t\t\tfor dirPath != \"/\" {\n+\t\t\t\tdirPath = filepath.Dir(dirPath)\n+\t\t\t\tif _, ok := paths[dirPath]; ok {\n+\t\t\t\t\tcontinue\n+\t\t\t\t}\n+\t\t\t\tdirNode, err := tree.Get(dirPath)\n+\t\t\t\tif err != nil {\n+\t\t\t\t\treturn nil, err\n+\t\t\t\t}\n+\t\t\t\tpaths[dirPath] = dirNode\n+\t\t\t}\n+\t\t}\n+\t}","path":"src/server/pfs/server/driver.go","position":null,"original_position":149,"commit_id":"6919ab5a3f78875981a0c9282ff73a01ffb8f3f9","original_commit_id":"023ae815c87c10883cbe52dc6db2da483a01c621","user":{"login":"jdoliner","id":43867,"node_id":"MDQ6VXNlcjQzODY3","avatar_url":"https://avatars3.githubusercontent.com/u/43867?v=4","gravatar_id":"","url":"https://api.github.com/users/jdoliner","html_url":"https://github.com/jdoliner","followers_url":"https://api.github.com/users/jdoliner/followers","following_url":"https://api.github.com/users/jdoliner/following{/other_user}","gists_url":"https://api.github.com/users/jdoliner/gists{/gist_id}","starred_url":"https://api.github.com/users/jdoliner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jdoliner/subscriptions","organizations_url":"https://api.github.com/users/jdoliner/orgs","repos_url":"https://api.github.com/users/jdoliner/repos","events_url":"https://api.github.com/users/jdoliner/events{/privacy}","received_events_url":"https://api.github.com/users/jdoliner/received_events","type":"User","site_admin":false},"body":"Looking forward to when we merge this with @brycemcanally's performance changes this is going to need to be implemented as a single pass over the paths, otherwise it's going to be painfully slow. I don't think it should be too hard to refactor this code to be a single pass. We can chat about how to do it in person.","created_at":"2018-09-20T18:47:24Z","updated_at":"2018-09-20T18:47:24Z","html_url":"https://github.com/pachyderm/pachyderm/pull/3064#discussion_r219277123","pull_request_url":"https://api.github.com/repos/pachyderm/pachyderm/pulls/3064","author_association":"MEMBER","_links":{"self":{"href":"https://api.github.com/repos/pachyderm/pachyderm/pulls/comments/219277123"},"html":{"href":"https://github.com/pachyderm/pachyderm/pull/3064#discussion_r219277123"},"pull_request":{"href":"https://api.github.com/repos/pachyderm/pachyderm/pulls/3064"}}},"pull_request":{"url":"https://api.github.com/repos/pachyderm/pachyderm/pulls/3064","id":202713971,"node_id":"MDExOlB1bGxSZXF1ZXN0MjAyNzEzOTcx","html_url":"https://github.com/pachyderm/pachyderm/pull/3064","diff_url":"https://github.com/pachyderm/pachyderm/pull/3064.diff","patch_url":"https://github.com/pachyderm/pachyderm/pull/3064.patch","issue_url":"https://api.github.com/repos/pachyderm/pachyderm/issues/3064","number":3064,"state":"open","locked":false,"title":"Sql pipelines","user":{"login":"sjezewski","id":597953,"node_id":"MDQ6VXNlcjU5Nzk1Mw==","avatar_url":"https://avatars1.githubusercontent.com/u/597953?v=4","gravatar_id":"","url":"https://api.github.com/users/sjezewski","html_url":"https://github.com/sjezewski","followers_url":"https://api.github.com/users/sjezewski/followers","following_url":"https://api.github.com/users/sjezewski/following{/other_user}","gists_url":"https://api.github.com/users/sjezewski/gists{/gist_id}","starred_url":"https://api.github.com/users/sjezewski/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sjezewski/subscriptions","organizations_url":"https://api.github.com/users/sjezewski/orgs","repos_url":"https://api.github.com/users/sjezewski/repos","events_url":"https://api.github.com/users/sjezewski/events{/privacy}","received_events_url":"https://api.github.com/users/sjezewski/received_events","type":"User","site_admin":false},"body":"Adds the following SQL support to putfile split:\r\n\r\n- `Delimiter_SQL` type\r\n- parsing support SQL PGDump data\r\n- pokes SQL support through to `pachctl`\r\n\r\nAlso adds the header/footer abstraction to `put-file` split:\r\n\r\n- so that `pachctl get-file /somedir` will return just the header + footer\r\n- `pachctl get-file /somedir/000000003` will return the header + file contents + footer\r\n- glob files like `pachctl get-file /somedir/*` will return the header + all child file contents + footer\r\n- file sizes are reported to be the size of (header + content + footer)\r\n- dir sizes are reported to be the size of (header + all child contents + footer)\r\n- `header` and `footer` flags on `pachctl put-file` that allow users to overwrite / delete the header/footer explicitly\r\n\r\nThis will come in handy for other formats (CSV,  other sql dump formats) as well.","created_at":"2018-07-19T23:59:18Z","updated_at":"2018-09-20T18:47:24Z","closed_at":null,"merged_at":null,"merge_commit_sha":"500ec0fa5996cd2a6ef818bbb468e15e5f408e55","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/pachyderm/pachyderm/pulls/3064/commits","review_comments_url":"https://api.github.com/repos/pachyderm/pachyderm/pulls/3064/comments","review_comment_url":"https://api.github.com/repos/pachyderm/pachyderm/pulls/comments{/number}","comments_url":"https://api.github.com/repos/pachyderm/pachyderm/issues/3064/comments","statuses_url":"https://api.github.com/repos/pachyderm/pachyderm/statuses/6919ab5a3f78875981a0c9282ff73a01ffb8f3f9","head":{"label":"pachyderm:sql_pipelines","ref":"sql_pipelines","sha":"6919ab5a3f78875981a0c9282ff73a01ffb8f3f9","user":{"login":"pachyderm","id":10432478,"node_id":"MDEyOk9yZ2FuaXphdGlvbjEwNDMyNDc4","avatar_url":"https://avatars3.githubusercontent.com/u/10432478?v=4","gravatar_id":"","url":"https://api.github.com/users/pachyderm","html_url":"https://github.com/pachyderm","followers_url":"https://api.github.com/users/pachyderm/followers","following_url":"https://api.github.com/users/pachyderm/following{/other_user}","gists_url":"https://api.github.com/users/pachyderm/gists{/gist_id}","starred_url":"https://api.github.com/users/pachyderm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pachyderm/subscriptions","organizations_url":"https://api.github.com/users/pachyderm/orgs","repos_url":"https://api.github.com/users/pachyderm/repos","events_url":"https://api.github.com/users/pachyderm/events{/privacy}","received_events_url":"https://api.github.com/users/pachyderm/received_events","type":"Organization","site_admin":false},"repo":{"id":23653453,"node_id":"MDEwOlJlcG9zaXRvcnkyMzY1MzQ1Mw==","name":"pachyderm","full_name":"pachyderm/pachyderm","private":false,"owner":{"login":"pachyderm","id":10432478,"node_id":"MDEyOk9yZ2FuaXphdGlvbjEwNDMyNDc4","avatar_url":"https://avatars3.githubusercontent.com/u/10432478?v=4","gravatar_id":"","url":"https://api.github.com/users/pachyderm","html_url":"https://github.com/pachyderm","followers_url":"https://api.github.com/users/pachyderm/followers","following_url":"https://api.github.com/users/pachyderm/following{/other_user}","gists_url":"https://api.github.com/users/pachyderm/gists{/gist_id}","starred_url":"https://api.github.com/users/pachyderm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pachyderm/subscriptions","organizations_url":"https://api.github.com/users/pachyderm/orgs","repos_url":"https://api.github.com/users/pachyderm/repos","events_url":"https://api.github.com/users/pachyderm/events{/privacy}","received_events_url":"https://api.github.com/users/pachyderm/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/pachyderm/pachyderm","description":"Reproducible Data Science at Scale!","fork":false,"url":"https://api.github.com/repos/pachyderm/pachyderm","forks_url":"https://api.github.com/repos/pachyderm/pachyderm/forks","keys_url":"https://api.github.com/repos/pachyderm/pachyderm/keys{/key_id}","collaborators_url":"https://api.github.com/repos/pachyderm/pachyderm/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/pachyderm/pachyderm/teams","hooks_url":"https://api.github.com/repos/pachyderm/pachyderm/hooks","issue_events_url":"https://api.github.com/repos/pachyderm/pachyderm/issues/events{/number}","events_url":"https://api.github.com/repos/pachyderm/pachyderm/events","assignees_url":"https://api.github.com/repos/pachyderm/pachyderm/assignees{/user}","branches_url":"https://api.github.com/repos/pachyderm/pachyderm/branches{/branch}","tags_url":"https://api.github.com/repos/pachyderm/pachyderm/tags","blobs_url":"https://api.github.com/repos/pachyderm/pachyderm/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/pachyderm/pachyderm/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/pachyderm/pachyderm/git/refs{/sha}","trees_url":"https://api.github.com/repos/pachyderm/pachyderm/git/trees{/sha}","statuses_url":"https://api.github.com/repos/pachyderm/pachyderm/statuses/{sha}","languages_url":"https://api.github.com/repos/pachyderm/pachyderm/languages","stargazers_url":"https://api.github.com/repos/pachyderm/pachyderm/stargazers","contributors_url":"https://api.github.com/repos/pachyderm/pachyderm/contributors","subscribers_url":"https://api.github.com/repos/pachyderm/pachyderm/subscribers","subscription_url":"https://api.github.com/repos/pachyderm/pachyderm/subscription","commits_url":"https://api.github.com/repos/pachyderm/pachyderm/commits{/sha}","git_commits_url":"https://api.github.com/repos/pachyderm/pachyderm/git/commits{/sha}","comments_url":"https://api.github.com/repos/pachyderm/pachyderm/comments{/number}","issue_comment_url":"https://api.github.com/repos/pachyderm/pachyderm/issues/comments{/number}","contents_url":"https://api.github.com/repos/pachyderm/pachyderm/contents/{+path}","compare_url":"https://api.github.com/repos/pachyderm/pachyderm/compare/{base}...{head}","merges_url":"https://api.github.com/repos/pachyderm/pachyderm/merges","archive_url":"https://api.github.com/repos/pachyderm/pachyderm/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/pachyderm/pachyderm/downloads","issues_url":"https://api.github.com/repos/pachyderm/pachyderm/issues{/number}","pulls_url":"https://api.github.com/repos/pachyderm/pachyderm/pulls{/number}","milestones_url":"https://api.github.com/repos/pachyderm/pachyderm/milestones{/number}","notifications_url":"https://api.github.com/repos/pachyderm/pachyderm/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/pachyderm/pachyderm/labels{/name}","releases_url":"https://api.github.com/repos/pachyderm/pachyderm/releases{/id}","deployments_url":"https://api.github.com/repos/pachyderm/pachyderm/deployments","created_at":"2014-09-04T07:50:02Z","updated_at":"2018-09-19T18:48:22Z","pushed_at":"2018-09-20T18:46:47Z","git_url":"git://github.com/pachyderm/pachyderm.git","ssh_url":"git@github.com:pachyderm/pachyderm.git","clone_url":"https://github.com/pachyderm/pachyderm.git","svn_url":"https://github.com/pachyderm/pachyderm","homepage":"http://pachyderm.io","size":118334,"stargazers_count":3063,"watchers_count":3063,"language":"Go","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":295,"mirror_url":null,"archived":false,"open_issues_count":422,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":295,"open_issues":422,"watchers":3063,"default_branch":"master"}},"base":{"label":"pachyderm:master","ref":"master","sha":"a1d5720bd27f39d75927d49d579002144fd4cf6e","user":{"login":"pachyderm","id":10432478,"node_id":"MDEyOk9yZ2FuaXphdGlvbjEwNDMyNDc4","avatar_url":"https://avatars3.githubusercontent.com/u/10432478?v=4","gravatar_id":"","url":"https://api.github.com/users/pachyderm","html_url":"https://github.com/pachyderm","followers_url":"https://api.github.com/users/pachyderm/followers","following_url":"https://api.github.com/users/pachyderm/following{/other_user}","gists_url":"https://api.github.com/users/pachyderm/gists{/gist_id}","starred_url":"https://api.github.com/users/pachyderm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pachyderm/subscriptions","organizations_url":"https://api.github.com/users/pachyderm/orgs","repos_url":"https://api.github.com/users/pachyderm/repos","events_url":"https://api.github.com/users/pachyderm/events{/privacy}","received_events_url":"https://api.github.com/users/pachyderm/received_events","type":"Organization","site_admin":false},"repo":{"id":23653453,"node_id":"MDEwOlJlcG9zaXRvcnkyMzY1MzQ1Mw==","name":"pachyderm","full_name":"pachyderm/pachyderm","private":false,"owner":{"login":"pachyderm","id":10432478,"node_id":"MDEyOk9yZ2FuaXphdGlvbjEwNDMyNDc4","avatar_url":"https://avatars3.githubusercontent.com/u/10432478?v=4","gravatar_id":"","url":"https://api.github.com/users/pachyderm","html_url":"https://github.com/pachyderm","followers_url":"https://api.github.com/users/pachyderm/followers","following_url":"https://api.github.com/users/pachyderm/following{/other_user}","gists_url":"https://api.github.com/users/pachyderm/gists{/gist_id}","starred_url":"https://api.github.com/users/pachyderm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pachyderm/subscriptions","organizations_url":"https://api.github.com/users/pachyderm/orgs","repos_url":"https://api.github.com/users/pachyderm/repos","events_url":"https://api.github.com/users/pachyderm/events{/privacy}","received_events_url":"https://api.github.com/users/pachyderm/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/pachyderm/pachyderm","description":"Reproducible Data Science at Scale!","fork":false,"url":"https://api.github.com/repos/pachyderm/pachyderm","forks_url":"https://api.github.com/repos/pachyderm/pachyderm/forks","keys_url":"https://api.github.com/repos/pachyderm/pachyderm/keys{/key_id}","collaborators_url":"https://api.github.com/repos/pachyderm/pachyderm/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/pachyderm/pachyderm/teams","hooks_url":"https://api.github.com/repos/pachyderm/pachyderm/hooks","issue_events_url":"https://api.github.com/repos/pachyderm/pachyderm/issues/events{/number}","events_url":"https://api.github.com/repos/pachyderm/pachyderm/events","assignees_url":"https://api.github.com/repos/pachyderm/pachyderm/assignees{/user}","branches_url":"https://api.github.com/repos/pachyderm/pachyderm/branches{/branch}","tags_url":"https://api.github.com/repos/pachyderm/pachyderm/tags","blobs_url":"https://api.github.com/repos/pachyderm/pachyderm/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/pachyderm/pachyderm/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/pachyderm/pachyderm/git/refs{/sha}","trees_url":"https://api.github.com/repos/pachyderm/pachyderm/git/trees{/sha}","statuses_url":"https://api.github.com/repos/pachyderm/pachyderm/statuses/{sha}","languages_url":"https://api.github.com/repos/pachyderm/pachyderm/languages","stargazers_url":"https://api.github.com/repos/pachyderm/pachyderm/stargazers","contributors_url":"https://api.github.com/repos/pachyderm/pachyderm/contributors","subscribers_url":"https://api.github.com/repos/pachyderm/pachyderm/subscribers","subscription_url":"https://api.github.com/repos/pachyderm/pachyderm/subscription","commits_url":"https://api.github.com/repos/pachyderm/pachyderm/commits{/sha}","git_commits_url":"https://api.github.com/repos/pachyderm/pachyderm/git/commits{/sha}","comments_url":"https://api.github.com/repos/pachyderm/pachyderm/comments{/number}","issue_comment_url":"https://api.github.com/repos/pachyderm/pachyderm/issues/comments{/number}","contents_url":"https://api.github.com/repos/pachyderm/pachyderm/contents/{+path}","compare_url":"https://api.github.com/repos/pachyderm/pachyderm/compare/{base}...{head}","merges_url":"https://api.github.com/repos/pachyderm/pachyderm/merges","archive_url":"https://api.github.com/repos/pachyderm/pachyderm/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/pachyderm/pachyderm/downloads","issues_url":"https://api.github.com/repos/pachyderm/pachyderm/issues{/number}","pulls_url":"https://api.github.com/repos/pachyderm/pachyderm/pulls{/number}","milestones_url":"https://api.github.com/repos/pachyderm/pachyderm/milestones{/number}","notifications_url":"https://api.github.com/repos/pachyderm/pachyderm/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/pachyderm/pachyderm/labels{/name}","releases_url":"https://api.github.com/repos/pachyderm/pachyderm/releases{/id}","deployments_url":"https://api.github.com/repos/pachyderm/pachyderm/deployments","created_at":"2014-09-04T07:50:02Z","updated_at":"2018-09-19T18:48:22Z","pushed_at":"2018-09-20T18:46:47Z","git_url":"git://github.com/pachyderm/pachyderm.git","ssh_url":"git@github.com:pachyderm/pachyderm.git","clone_url":"https://github.com/pachyderm/pachyderm.git","svn_url":"https://github.com/pachyderm/pachyderm","homepage":"http://pachyderm.io","size":118334,"stargazers_count":3063,"watchers_count":3063,"language":"Go","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":295,"mirror_url":null,"archived":false,"open_issues_count":422,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":295,"open_issues":422,"watchers":3063,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/pachyderm/pachyderm/pulls/3064"},"html":{"href":"https://github.com/pachyderm/pachyderm/pull/3064"},"issue":{"href":"https://api.github.com/repos/pachyderm/pachyderm/issues/3064"},"comments":{"href":"https://api.github.com/repos/pachyderm/pachyderm/issues/3064/comments"},"review_comments":{"href":"https://api.github.com/repos/pachyderm/pachyderm/pulls/3064/comments"},"review_comment":{"href":"https://api.github.com/repos/pachyderm/pachyderm/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/pachyderm/pachyderm/pulls/3064/commits"},"statuses":{"href":"https://api.github.com/repos/pachyderm/pachyderm/statuses/6919ab5a3f78875981a0c9282ff73a01ffb8f3f9"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"jdoliner"}}