{"action":"created","comment":{"url":"https://api.github.com/repos/uwescience/myria/pulls/comments/75037321","id":75037321,"diff_hunk":"@@ -179,30 +179,36 @@ public CSVFileScanFragment(\n   @Override\n   protected TupleBatch fetchNextReady() throws IOException, DbException {\n     long lineNumberBegin = lineNumber;\n+    boolean unfinishedEncapsulatedTokenRow = false;","path":"src/edu/washington/escience/myria/operator/CSVFileScanFragment.java","position":null,"original_position":4,"commit_id":"44a1de15f7a2db4b75ccb2d21a2f4fc1fef0add4","original_commit_id":"44a1de15f7a2db4b75ccb2d21a2f4fc1fef0add4","user":{"login":"jortiz16","id":3239603,"avatar_url":"https://avatars.githubusercontent.com/u/3239603?v=3","gravatar_id":"","url":"https://api.github.com/users/jortiz16","html_url":"https://github.com/jortiz16","followers_url":"https://api.github.com/users/jortiz16/followers","following_url":"https://api.github.com/users/jortiz16/following{/other_user}","gists_url":"https://api.github.com/users/jortiz16/gists{/gist_id}","starred_url":"https://api.github.com/users/jortiz16/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jortiz16/subscriptions","organizations_url":"https://api.github.com/users/jortiz16/orgs","repos_url":"https://api.github.com/users/jortiz16/repos","events_url":"https://api.github.com/users/jortiz16/events{/privacy}","received_events_url":"https://api.github.com/users/jortiz16/received_events","type":"User","site_admin":false},"body":"ok sure!","created_at":"2016-08-16T23:06:02Z","updated_at":"2016-08-16T23:06:02Z","html_url":"https://github.com/uwescience/myria/pull/822#discussion_r75037321","pull_request_url":"https://api.github.com/repos/uwescience/myria/pulls/822","_links":{"self":{"href":"https://api.github.com/repos/uwescience/myria/pulls/comments/75037321"},"html":{"href":"https://github.com/uwescience/myria/pull/822#discussion_r75037321"},"pull_request":{"href":"https://api.github.com/repos/uwescience/myria/pulls/822"}}},"pull_request":{"url":"https://api.github.com/repos/uwescience/myria/pulls/822","id":67316777,"html_url":"https://github.com/uwescience/myria/pull/822","diff_url":"https://github.com/uwescience/myria/pull/822.diff","patch_url":"https://github.com/uwescience/myria/pull/822.patch","issue_url":"https://api.github.com/repos/uwescience/myria/issues/822","number":822,"state":"open","locked":false,"title":"Parallel Ingest","user":{"login":"jortiz16","id":3239603,"avatar_url":"https://avatars.githubusercontent.com/u/3239603?v=3","gravatar_id":"","url":"https://api.github.com/users/jortiz16","html_url":"https://github.com/jortiz16","followers_url":"https://api.github.com/users/jortiz16/followers","following_url":"https://api.github.com/users/jortiz16/following{/other_user}","gists_url":"https://api.github.com/users/jortiz16/gists{/gist_id}","starred_url":"https://api.github.com/users/jortiz16/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jortiz16/subscriptions","organizations_url":"https://api.github.com/users/jortiz16/orgs","repos_url":"https://api.github.com/users/jortiz16/repos","events_url":"https://api.github.com/users/jortiz16/events{/privacy}","received_events_url":"https://api.github.com/users/jortiz16/received_events","type":"User","site_admin":false},"body":"Addresses #798 \r\n\r\nAs a first attempt at ingesting in parallel, I added a new class called ```CSVFileScanFragment``` that is really similar to ```FileScan```. Although, I'm not sure if we can or want to merge the two. \r\n\r\nThe master sends each worker a ```CSVFileScanFragment``` so they can read in parallel. In the constructor, we initialize the amount of bytes each worker should read. During init(), the operator opens a stream which is given by the ```UriSource```. Throughout the parallel read, the ```CSVFileScanFragment``` keeps asking the UriSource for streams based on byte ranges.\r\n\r\nDuring ```fetchNextReady()``` in ```CSVFileScanFragment```, if we run into a partial row, we ignore it if it's at the beginning of the stream (another worker should take care of it). If the partial row is the last row,  we keep asking for more bytes until we get a valid row. Once the last row is found, the parser is closed. \r\n\r\nThe ```UriSource``` class has two new methods, ```getChunkInputStream()``` and ```getFileSize()```.  The ```CSVFileScanFragment``` will always call the ```getChunkInputStream()``` method instead of the usual ```getInputStream()``` method.\r\n\r\n**Other notes:**\r\n**1)** I spent a while trying to get AmazonS3URI class to work for the ```getChunkInputStream()``` and ```getFileSize()``` methods , but it only works with aws sdk 1.10.65. When I tried to upgrade the aws sdk, I ran into issues for other systemtests. \r\n\r\n**2)** Calls to getObject() in the S3 API are expensive (in terms of time). We want to avoid that as much as possible I think. \r\n\r\n**3)** In the test class ```ParallelIngestS3Test```, the second method runs a ```diff``` operation against two types of ingest -- the normal one that goes through the coordinator and the new parallel ingest. If there is an easier way to do the diff please let me know. I had to take the relations and ingest them to one worker, which then ran the diff locally. \r\n\r\n**4)** I did not know how we wanted to expose this to the users. Should we write an encoding ? Or will the plan be built on Raco? I also wasn't quite yet sure how this would fit in with the new refactoring proposal for DataInput.\r\n\r\n**5)** I also ran the parallel ingest on a 3-node cluster on AWS and compared it to the old way of running an ingest in terms of elapsed time. To get the parallel ingest to work, I temporarily wrote an encoding and a parallelIngest method in the Server class. I did not include that in this PR (see issue 4) \r\n\r\ndata : 1.5GB\r\nold ingest: 1minute 15 seconds\r\nparallel ingest: 53 seconds\r\n\r\ndata: 2.8GB\r\nold ingest: 2 minutes 36 seconds\r\nparallel ingest:  1 minute 59 seconds\r\n\r\nThanks. Please let me know if something is not clear . \r\n\r\n","created_at":"2016-04-21T08:44:13Z","updated_at":"2016-08-16T23:06:02Z","closed_at":null,"merged_at":null,"merge_commit_sha":"92c50e1571a7858cbbdc959f6829eb73eb4bf51f","assignee":{"login":"senderista","id":131425,"avatar_url":"https://avatars.githubusercontent.com/u/131425?v=3","gravatar_id":"","url":"https://api.github.com/users/senderista","html_url":"https://github.com/senderista","followers_url":"https://api.github.com/users/senderista/followers","following_url":"https://api.github.com/users/senderista/following{/other_user}","gists_url":"https://api.github.com/users/senderista/gists{/gist_id}","starred_url":"https://api.github.com/users/senderista/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/senderista/subscriptions","organizations_url":"https://api.github.com/users/senderista/orgs","repos_url":"https://api.github.com/users/senderista/repos","events_url":"https://api.github.com/users/senderista/events{/privacy}","received_events_url":"https://api.github.com/users/senderista/received_events","type":"User","site_admin":false},"assignees":[{"login":"senderista","id":131425,"avatar_url":"https://avatars.githubusercontent.com/u/131425?v=3","gravatar_id":"","url":"https://api.github.com/users/senderista","html_url":"https://github.com/senderista","followers_url":"https://api.github.com/users/senderista/followers","following_url":"https://api.github.com/users/senderista/following{/other_user}","gists_url":"https://api.github.com/users/senderista/gists{/gist_id}","starred_url":"https://api.github.com/users/senderista/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/senderista/subscriptions","organizations_url":"https://api.github.com/users/senderista/orgs","repos_url":"https://api.github.com/users/senderista/repos","events_url":"https://api.github.com/users/senderista/events{/privacy}","received_events_url":"https://api.github.com/users/senderista/received_events","type":"User","site_admin":false}],"milestone":null,"commits_url":"https://api.github.com/repos/uwescience/myria/pulls/822/commits","review_comments_url":"https://api.github.com/repos/uwescience/myria/pulls/822/comments","review_comment_url":"https://api.github.com/repos/uwescience/myria/pulls/comments{/number}","comments_url":"https://api.github.com/repos/uwescience/myria/issues/822/comments","statuses_url":"https://api.github.com/repos/uwescience/myria/statuses/44a1de15f7a2db4b75ccb2d21a2f4fc1fef0add4","head":{"label":"uwescience:parallelIngest","ref":"parallelIngest","sha":"44a1de15f7a2db4b75ccb2d21a2f4fc1fef0add4","user":{"login":"uwescience","id":2243113,"avatar_url":"https://avatars.githubusercontent.com/u/2243113?v=3","gravatar_id":"","url":"https://api.github.com/users/uwescience","html_url":"https://github.com/uwescience","followers_url":"https://api.github.com/users/uwescience/followers","following_url":"https://api.github.com/users/uwescience/following{/other_user}","gists_url":"https://api.github.com/users/uwescience/gists{/gist_id}","starred_url":"https://api.github.com/users/uwescience/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/uwescience/subscriptions","organizations_url":"https://api.github.com/users/uwescience/orgs","repos_url":"https://api.github.com/users/uwescience/repos","events_url":"https://api.github.com/users/uwescience/events{/privacy}","received_events_url":"https://api.github.com/users/uwescience/received_events","type":"Organization","site_admin":false},"repo":{"id":5612317,"name":"myria","full_name":"uwescience/myria","owner":{"login":"uwescience","id":2243113,"avatar_url":"https://avatars.githubusercontent.com/u/2243113?v=3","gravatar_id":"","url":"https://api.github.com/users/uwescience","html_url":"https://github.com/uwescience","followers_url":"https://api.github.com/users/uwescience/followers","following_url":"https://api.github.com/users/uwescience/following{/other_user}","gists_url":"https://api.github.com/users/uwescience/gists{/gist_id}","starred_url":"https://api.github.com/users/uwescience/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/uwescience/subscriptions","organizations_url":"https://api.github.com/users/uwescience/orgs","repos_url":"https://api.github.com/users/uwescience/repos","events_url":"https://api.github.com/users/uwescience/events{/privacy}","received_events_url":"https://api.github.com/users/uwescience/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/uwescience/myria","description":"Myria is a scalable Analytics-as-a-Service platform based on relational algebra.","fork":false,"url":"https://api.github.com/repos/uwescience/myria","forks_url":"https://api.github.com/repos/uwescience/myria/forks","keys_url":"https://api.github.com/repos/uwescience/myria/keys{/key_id}","collaborators_url":"https://api.github.com/repos/uwescience/myria/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/uwescience/myria/teams","hooks_url":"https://api.github.com/repos/uwescience/myria/hooks","issue_events_url":"https://api.github.com/repos/uwescience/myria/issues/events{/number}","events_url":"https://api.github.com/repos/uwescience/myria/events","assignees_url":"https://api.github.com/repos/uwescience/myria/assignees{/user}","branches_url":"https://api.github.com/repos/uwescience/myria/branches{/branch}","tags_url":"https://api.github.com/repos/uwescience/myria/tags","blobs_url":"https://api.github.com/repos/uwescience/myria/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/uwescience/myria/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/uwescience/myria/git/refs{/sha}","trees_url":"https://api.github.com/repos/uwescience/myria/git/trees{/sha}","statuses_url":"https://api.github.com/repos/uwescience/myria/statuses/{sha}","languages_url":"https://api.github.com/repos/uwescience/myria/languages","stargazers_url":"https://api.github.com/repos/uwescience/myria/stargazers","contributors_url":"https://api.github.com/repos/uwescience/myria/contributors","subscribers_url":"https://api.github.com/repos/uwescience/myria/subscribers","subscription_url":"https://api.github.com/repos/uwescience/myria/subscription","commits_url":"https://api.github.com/repos/uwescience/myria/commits{/sha}","git_commits_url":"https://api.github.com/repos/uwescience/myria/git/commits{/sha}","comments_url":"https://api.github.com/repos/uwescience/myria/comments{/number}","issue_comment_url":"https://api.github.com/repos/uwescience/myria/issues/comments{/number}","contents_url":"https://api.github.com/repos/uwescience/myria/contents/{+path}","compare_url":"https://api.github.com/repos/uwescience/myria/compare/{base}...{head}","merges_url":"https://api.github.com/repos/uwescience/myria/merges","archive_url":"https://api.github.com/repos/uwescience/myria/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/uwescience/myria/downloads","issues_url":"https://api.github.com/repos/uwescience/myria/issues{/number}","pulls_url":"https://api.github.com/repos/uwescience/myria/pulls{/number}","milestones_url":"https://api.github.com/repos/uwescience/myria/milestones{/number}","notifications_url":"https://api.github.com/repos/uwescience/myria/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/uwescience/myria/labels{/name}","releases_url":"https://api.github.com/repos/uwescience/myria/releases{/id}","deployments_url":"https://api.github.com/repos/uwescience/myria/deployments","created_at":"2012-08-30T08:27:36Z","updated_at":"2016-05-31T02:53:00Z","pushed_at":"2016-08-16T16:47:41Z","git_url":"git://github.com/uwescience/myria.git","ssh_url":"git@github.com:uwescience/myria.git","clone_url":"https://github.com/uwescience/myria.git","svn_url":"https://github.com/uwescience/myria","homepage":"myria.cs.washington.edu","size":77104,"stargazers_count":41,"watchers_count":41,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":23,"mirror_url":null,"open_issues_count":99,"forks":23,"open_issues":99,"watchers":41,"default_branch":"master"}},"base":{"label":"uwescience:master","ref":"master","sha":"50fffb8878bb19ac37851bbc3f625e7e325c1732","user":{"login":"uwescience","id":2243113,"avatar_url":"https://avatars.githubusercontent.com/u/2243113?v=3","gravatar_id":"","url":"https://api.github.com/users/uwescience","html_url":"https://github.com/uwescience","followers_url":"https://api.github.com/users/uwescience/followers","following_url":"https://api.github.com/users/uwescience/following{/other_user}","gists_url":"https://api.github.com/users/uwescience/gists{/gist_id}","starred_url":"https://api.github.com/users/uwescience/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/uwescience/subscriptions","organizations_url":"https://api.github.com/users/uwescience/orgs","repos_url":"https://api.github.com/users/uwescience/repos","events_url":"https://api.github.com/users/uwescience/events{/privacy}","received_events_url":"https://api.github.com/users/uwescience/received_events","type":"Organization","site_admin":false},"repo":{"id":5612317,"name":"myria","full_name":"uwescience/myria","owner":{"login":"uwescience","id":2243113,"avatar_url":"https://avatars.githubusercontent.com/u/2243113?v=3","gravatar_id":"","url":"https://api.github.com/users/uwescience","html_url":"https://github.com/uwescience","followers_url":"https://api.github.com/users/uwescience/followers","following_url":"https://api.github.com/users/uwescience/following{/other_user}","gists_url":"https://api.github.com/users/uwescience/gists{/gist_id}","starred_url":"https://api.github.com/users/uwescience/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/uwescience/subscriptions","organizations_url":"https://api.github.com/users/uwescience/orgs","repos_url":"https://api.github.com/users/uwescience/repos","events_url":"https://api.github.com/users/uwescience/events{/privacy}","received_events_url":"https://api.github.com/users/uwescience/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/uwescience/myria","description":"Myria is a scalable Analytics-as-a-Service platform based on relational algebra.","fork":false,"url":"https://api.github.com/repos/uwescience/myria","forks_url":"https://api.github.com/repos/uwescience/myria/forks","keys_url":"https://api.github.com/repos/uwescience/myria/keys{/key_id}","collaborators_url":"https://api.github.com/repos/uwescience/myria/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/uwescience/myria/teams","hooks_url":"https://api.github.com/repos/uwescience/myria/hooks","issue_events_url":"https://api.github.com/repos/uwescience/myria/issues/events{/number}","events_url":"https://api.github.com/repos/uwescience/myria/events","assignees_url":"https://api.github.com/repos/uwescience/myria/assignees{/user}","branches_url":"https://api.github.com/repos/uwescience/myria/branches{/branch}","tags_url":"https://api.github.com/repos/uwescience/myria/tags","blobs_url":"https://api.github.com/repos/uwescience/myria/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/uwescience/myria/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/uwescience/myria/git/refs{/sha}","trees_url":"https://api.github.com/repos/uwescience/myria/git/trees{/sha}","statuses_url":"https://api.github.com/repos/uwescience/myria/statuses/{sha}","languages_url":"https://api.github.com/repos/uwescience/myria/languages","stargazers_url":"https://api.github.com/repos/uwescience/myria/stargazers","contributors_url":"https://api.github.com/repos/uwescience/myria/contributors","subscribers_url":"https://api.github.com/repos/uwescience/myria/subscribers","subscription_url":"https://api.github.com/repos/uwescience/myria/subscription","commits_url":"https://api.github.com/repos/uwescience/myria/commits{/sha}","git_commits_url":"https://api.github.com/repos/uwescience/myria/git/commits{/sha}","comments_url":"https://api.github.com/repos/uwescience/myria/comments{/number}","issue_comment_url":"https://api.github.com/repos/uwescience/myria/issues/comments{/number}","contents_url":"https://api.github.com/repos/uwescience/myria/contents/{+path}","compare_url":"https://api.github.com/repos/uwescience/myria/compare/{base}...{head}","merges_url":"https://api.github.com/repos/uwescience/myria/merges","archive_url":"https://api.github.com/repos/uwescience/myria/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/uwescience/myria/downloads","issues_url":"https://api.github.com/repos/uwescience/myria/issues{/number}","pulls_url":"https://api.github.com/repos/uwescience/myria/pulls{/number}","milestones_url":"https://api.github.com/repos/uwescience/myria/milestones{/number}","notifications_url":"https://api.github.com/repos/uwescience/myria/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/uwescience/myria/labels{/name}","releases_url":"https://api.github.com/repos/uwescience/myria/releases{/id}","deployments_url":"https://api.github.com/repos/uwescience/myria/deployments","created_at":"2012-08-30T08:27:36Z","updated_at":"2016-05-31T02:53:00Z","pushed_at":"2016-08-16T16:47:41Z","git_url":"git://github.com/uwescience/myria.git","ssh_url":"git@github.com:uwescience/myria.git","clone_url":"https://github.com/uwescience/myria.git","svn_url":"https://github.com/uwescience/myria","homepage":"myria.cs.washington.edu","size":77104,"stargazers_count":41,"watchers_count":41,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":23,"mirror_url":null,"open_issues_count":99,"forks":23,"open_issues":99,"watchers":41,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/uwescience/myria/pulls/822"},"html":{"href":"https://github.com/uwescience/myria/pull/822"},"issue":{"href":"https://api.github.com/repos/uwescience/myria/issues/822"},"comments":{"href":"https://api.github.com/repos/uwescience/myria/issues/822/comments"},"review_comments":{"href":"https://api.github.com/repos/uwescience/myria/pulls/822/comments"},"review_comment":{"href":"https://api.github.com/repos/uwescience/myria/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/uwescience/myria/pulls/822/commits"},"statuses":{"href":"https://api.github.com/repos/uwescience/myria/statuses/44a1de15f7a2db4b75ccb2d21a2f4fc1fef0add4"}}}}
{"actor":{"display_login":"jortiz16"}}