{"action":"created","comment":{"url":"https://api.github.com/repos/uwescience/myria/pulls/comments/75568982","id":75568982,"diff_hunk":"@@ -430,6 +441,23 @@ protected void init(final ImmutableMap<String, Object> execEnvVars) throws DbExc\n                 new BufferedReader(new InputStreamReader(partitionInputStream)),\n                 CSVFormat.newFormat(delimiter).withQuote(quote).withEscape(escape));\n         iterator = parser.iterator();\n+\n+        try {\n+          if (!iterator.hasNext()) {\n+            flagAsIncomplete = true;\n+          }\n+        } catch (Exception e) {\n+          /*\n+           * FIX ME: If we hit an exception for a malformed row (in case of quotes for example), we mark this as the\n+           * last row\n+           */\n+          if (e.getMessage() != null && e.getMessage().contains(truncatedQuoteErrorMessage)) {\n+            onLastRow = true;\n+          } else {\n+            throw e;\n+          }\n+        }\n+\n         for (int i = 0; i < numberOfSkippedLines; i++) {","path":"src/edu/washington/escience/myria/operator/CSVFileScanFragment.java","position":85,"original_position":85,"commit_id":"0c7b6c3fd29acf8639c4c5356d837a052d3a5ff4","original_commit_id":"0c7b6c3fd29acf8639c4c5356d837a052d3a5ff4","user":{"login":"jortiz16","id":3239603,"avatar_url":"https://avatars.githubusercontent.com/u/3239603?v=3","gravatar_id":"","url":"https://api.github.com/users/jortiz16","html_url":"https://github.com/jortiz16","followers_url":"https://api.github.com/users/jortiz16/followers","following_url":"https://api.github.com/users/jortiz16/following{/other_user}","gists_url":"https://api.github.com/users/jortiz16/gists{/gist_id}","starred_url":"https://api.github.com/users/jortiz16/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jortiz16/subscriptions","organizations_url":"https://api.github.com/users/jortiz16/orgs","repos_url":"https://api.github.com/users/jortiz16/repos","events_url":"https://api.github.com/users/jortiz16/events{/privacy}","received_events_url":"https://api.github.com/users/jortiz16/received_events","type":"User","site_admin":false},"body":"Yes, this should be conditioned for the first worker....\r\n\r\nBut headers in subsequent partitions? How likely would that happen given that we try to keep the size of the partitions at a minimum number (believe it was 100MB). I guess a case where there is a very long header?\r\n\r\n","created_at":"2016-08-20T00:49:50Z","updated_at":"2016-08-20T00:49:50Z","html_url":"https://github.com/uwescience/myria/pull/848#discussion_r75568982","pull_request_url":"https://api.github.com/repos/uwescience/myria/pulls/848","_links":{"self":{"href":"https://api.github.com/repos/uwescience/myria/pulls/comments/75568982"},"html":{"href":"https://github.com/uwescience/myria/pull/848#discussion_r75568982"},"pull_request":{"href":"https://api.github.com/repos/uwescience/myria/pulls/848"}}},"pull_request":{"url":"https://api.github.com/repos/uwescience/myria/pulls/848","id":82056174,"html_url":"https://github.com/uwescience/myria/pull/848","diff_url":"https://github.com/uwescience/myria/pull/848.diff","patch_url":"https://github.com/uwescience/myria/pull/848.patch","issue_url":"https://api.github.com/repos/uwescience/myria/issues/848","number":848,"state":"open","locked":false,"title":"parallel ingest bug fix","user":{"login":"jortiz16","id":3239603,"avatar_url":"https://avatars.githubusercontent.com/u/3239603?v=3","gravatar_id":"","url":"https://api.github.com/users/jortiz16","html_url":"https://github.com/jortiz16","followers_url":"https://api.github.com/users/jortiz16/followers","following_url":"https://api.github.com/users/jortiz16/following{/other_user}","gists_url":"https://api.github.com/users/jortiz16/gists{/gist_id}","starred_url":"https://api.github.com/users/jortiz16/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jortiz16/subscriptions","organizations_url":"https://api.github.com/users/jortiz16/orgs","repos_url":"https://api.github.com/users/jortiz16/repos","events_url":"https://api.github.com/users/jortiz16/events{/privacy}","received_events_url":"https://api.github.com/users/jortiz16/received_events","type":"User","site_admin":false},"body":"@parmitam  was our first tester and ran into a bug. I realized I forgot to test the last changes we made for more than 2 workers. \r\n\r\nWith this fix, I tested for 3 and 5 workers by temporarily adding workers to the SystemTestBase class.\r\n\r\nTwo fixes:\r\n\r\n1) I added a global variable ```byteOffsetFromTruncatedRowAtStart``` which keeps track of the number of bytes we read when skipping over the first partial row. This variable becomes important in cases when a worker needs to reinitialize the parser -- we must have the correct offset otherwise our character skipping will be incorrect. Our 2-worker tests did not cover this because worker 1 never has to worry about this offset when reinitializing the parser (it is always going to be 0) and worker 2 is the last worker so it doesn't need to reinitialize the parser for the last row.\r\n\r\n2) This other problem is more subtle. There might be a case when a worker has a partition such as \"this_is_an_incomplete_row\\n\". Our code will read up to the new line ('\\n') but we don't have a way to check if there are any other lines to read after that. So I added a way to check if there is something to read. If there isn't, the row is marked as ```flagAsIncomplete```. Here, we still need to check for the quote try/catch in cases as well.","created_at":"2016-08-19T22:46:13Z","updated_at":"2016-08-20T00:49:50Z","closed_at":null,"merged_at":null,"merge_commit_sha":"37e0ca6b58d02ea6b9004fc0f1a68716cabcc198","assignee":{"login":"senderista","id":131425,"avatar_url":"https://avatars.githubusercontent.com/u/131425?v=3","gravatar_id":"","url":"https://api.github.com/users/senderista","html_url":"https://github.com/senderista","followers_url":"https://api.github.com/users/senderista/followers","following_url":"https://api.github.com/users/senderista/following{/other_user}","gists_url":"https://api.github.com/users/senderista/gists{/gist_id}","starred_url":"https://api.github.com/users/senderista/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/senderista/subscriptions","organizations_url":"https://api.github.com/users/senderista/orgs","repos_url":"https://api.github.com/users/senderista/repos","events_url":"https://api.github.com/users/senderista/events{/privacy}","received_events_url":"https://api.github.com/users/senderista/received_events","type":"User","site_admin":false},"assignees":[{"login":"senderista","id":131425,"avatar_url":"https://avatars.githubusercontent.com/u/131425?v=3","gravatar_id":"","url":"https://api.github.com/users/senderista","html_url":"https://github.com/senderista","followers_url":"https://api.github.com/users/senderista/followers","following_url":"https://api.github.com/users/senderista/following{/other_user}","gists_url":"https://api.github.com/users/senderista/gists{/gist_id}","starred_url":"https://api.github.com/users/senderista/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/senderista/subscriptions","organizations_url":"https://api.github.com/users/senderista/orgs","repos_url":"https://api.github.com/users/senderista/repos","events_url":"https://api.github.com/users/senderista/events{/privacy}","received_events_url":"https://api.github.com/users/senderista/received_events","type":"User","site_admin":false}],"milestone":null,"commits_url":"https://api.github.com/repos/uwescience/myria/pulls/848/commits","review_comments_url":"https://api.github.com/repos/uwescience/myria/pulls/848/comments","review_comment_url":"https://api.github.com/repos/uwescience/myria/pulls/comments{/number}","comments_url":"https://api.github.com/repos/uwescience/myria/issues/848/comments","statuses_url":"https://api.github.com/repos/uwescience/myria/statuses/0c7b6c3fd29acf8639c4c5356d837a052d3a5ff4","head":{"label":"uwescience:parallelIngest-bug","ref":"parallelIngest-bug","sha":"0c7b6c3fd29acf8639c4c5356d837a052d3a5ff4","user":{"login":"uwescience","id":2243113,"avatar_url":"https://avatars.githubusercontent.com/u/2243113?v=3","gravatar_id":"","url":"https://api.github.com/users/uwescience","html_url":"https://github.com/uwescience","followers_url":"https://api.github.com/users/uwescience/followers","following_url":"https://api.github.com/users/uwescience/following{/other_user}","gists_url":"https://api.github.com/users/uwescience/gists{/gist_id}","starred_url":"https://api.github.com/users/uwescience/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/uwescience/subscriptions","organizations_url":"https://api.github.com/users/uwescience/orgs","repos_url":"https://api.github.com/users/uwescience/repos","events_url":"https://api.github.com/users/uwescience/events{/privacy}","received_events_url":"https://api.github.com/users/uwescience/received_events","type":"Organization","site_admin":false},"repo":{"id":5612317,"name":"myria","full_name":"uwescience/myria","owner":{"login":"uwescience","id":2243113,"avatar_url":"https://avatars.githubusercontent.com/u/2243113?v=3","gravatar_id":"","url":"https://api.github.com/users/uwescience","html_url":"https://github.com/uwescience","followers_url":"https://api.github.com/users/uwescience/followers","following_url":"https://api.github.com/users/uwescience/following{/other_user}","gists_url":"https://api.github.com/users/uwescience/gists{/gist_id}","starred_url":"https://api.github.com/users/uwescience/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/uwescience/subscriptions","organizations_url":"https://api.github.com/users/uwescience/orgs","repos_url":"https://api.github.com/users/uwescience/repos","events_url":"https://api.github.com/users/uwescience/events{/privacy}","received_events_url":"https://api.github.com/users/uwescience/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/uwescience/myria","description":"Myria is a scalable Analytics-as-a-Service platform based on relational algebra.","fork":false,"url":"https://api.github.com/repos/uwescience/myria","forks_url":"https://api.github.com/repos/uwescience/myria/forks","keys_url":"https://api.github.com/repos/uwescience/myria/keys{/key_id}","collaborators_url":"https://api.github.com/repos/uwescience/myria/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/uwescience/myria/teams","hooks_url":"https://api.github.com/repos/uwescience/myria/hooks","issue_events_url":"https://api.github.com/repos/uwescience/myria/issues/events{/number}","events_url":"https://api.github.com/repos/uwescience/myria/events","assignees_url":"https://api.github.com/repos/uwescience/myria/assignees{/user}","branches_url":"https://api.github.com/repos/uwescience/myria/branches{/branch}","tags_url":"https://api.github.com/repos/uwescience/myria/tags","blobs_url":"https://api.github.com/repos/uwescience/myria/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/uwescience/myria/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/uwescience/myria/git/refs{/sha}","trees_url":"https://api.github.com/repos/uwescience/myria/git/trees{/sha}","statuses_url":"https://api.github.com/repos/uwescience/myria/statuses/{sha}","languages_url":"https://api.github.com/repos/uwescience/myria/languages","stargazers_url":"https://api.github.com/repos/uwescience/myria/stargazers","contributors_url":"https://api.github.com/repos/uwescience/myria/contributors","subscribers_url":"https://api.github.com/repos/uwescience/myria/subscribers","subscription_url":"https://api.github.com/repos/uwescience/myria/subscription","commits_url":"https://api.github.com/repos/uwescience/myria/commits{/sha}","git_commits_url":"https://api.github.com/repos/uwescience/myria/git/commits{/sha}","comments_url":"https://api.github.com/repos/uwescience/myria/comments{/number}","issue_comment_url":"https://api.github.com/repos/uwescience/myria/issues/comments{/number}","contents_url":"https://api.github.com/repos/uwescience/myria/contents/{+path}","compare_url":"https://api.github.com/repos/uwescience/myria/compare/{base}...{head}","merges_url":"https://api.github.com/repos/uwescience/myria/merges","archive_url":"https://api.github.com/repos/uwescience/myria/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/uwescience/myria/downloads","issues_url":"https://api.github.com/repos/uwescience/myria/issues{/number}","pulls_url":"https://api.github.com/repos/uwescience/myria/pulls{/number}","milestones_url":"https://api.github.com/repos/uwescience/myria/milestones{/number}","notifications_url":"https://api.github.com/repos/uwescience/myria/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/uwescience/myria/labels{/name}","releases_url":"https://api.github.com/repos/uwescience/myria/releases{/id}","deployments_url":"https://api.github.com/repos/uwescience/myria/deployments","created_at":"2012-08-30T08:27:36Z","updated_at":"2016-05-31T02:53:00Z","pushed_at":"2016-08-19T22:46:13Z","git_url":"git://github.com/uwescience/myria.git","ssh_url":"git@github.com:uwescience/myria.git","clone_url":"https://github.com/uwescience/myria.git","svn_url":"https://github.com/uwescience/myria","homepage":"myria.cs.washington.edu","size":77143,"stargazers_count":41,"watchers_count":41,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":23,"mirror_url":null,"open_issues_count":99,"forks":23,"open_issues":99,"watchers":41,"default_branch":"master"}},"base":{"label":"uwescience:master","ref":"master","sha":"2918e59d9c686bf8b9f5296ea691cd24afa7d865","user":{"login":"uwescience","id":2243113,"avatar_url":"https://avatars.githubusercontent.com/u/2243113?v=3","gravatar_id":"","url":"https://api.github.com/users/uwescience","html_url":"https://github.com/uwescience","followers_url":"https://api.github.com/users/uwescience/followers","following_url":"https://api.github.com/users/uwescience/following{/other_user}","gists_url":"https://api.github.com/users/uwescience/gists{/gist_id}","starred_url":"https://api.github.com/users/uwescience/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/uwescience/subscriptions","organizations_url":"https://api.github.com/users/uwescience/orgs","repos_url":"https://api.github.com/users/uwescience/repos","events_url":"https://api.github.com/users/uwescience/events{/privacy}","received_events_url":"https://api.github.com/users/uwescience/received_events","type":"Organization","site_admin":false},"repo":{"id":5612317,"name":"myria","full_name":"uwescience/myria","owner":{"login":"uwescience","id":2243113,"avatar_url":"https://avatars.githubusercontent.com/u/2243113?v=3","gravatar_id":"","url":"https://api.github.com/users/uwescience","html_url":"https://github.com/uwescience","followers_url":"https://api.github.com/users/uwescience/followers","following_url":"https://api.github.com/users/uwescience/following{/other_user}","gists_url":"https://api.github.com/users/uwescience/gists{/gist_id}","starred_url":"https://api.github.com/users/uwescience/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/uwescience/subscriptions","organizations_url":"https://api.github.com/users/uwescience/orgs","repos_url":"https://api.github.com/users/uwescience/repos","events_url":"https://api.github.com/users/uwescience/events{/privacy}","received_events_url":"https://api.github.com/users/uwescience/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/uwescience/myria","description":"Myria is a scalable Analytics-as-a-Service platform based on relational algebra.","fork":false,"url":"https://api.github.com/repos/uwescience/myria","forks_url":"https://api.github.com/repos/uwescience/myria/forks","keys_url":"https://api.github.com/repos/uwescience/myria/keys{/key_id}","collaborators_url":"https://api.github.com/repos/uwescience/myria/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/uwescience/myria/teams","hooks_url":"https://api.github.com/repos/uwescience/myria/hooks","issue_events_url":"https://api.github.com/repos/uwescience/myria/issues/events{/number}","events_url":"https://api.github.com/repos/uwescience/myria/events","assignees_url":"https://api.github.com/repos/uwescience/myria/assignees{/user}","branches_url":"https://api.github.com/repos/uwescience/myria/branches{/branch}","tags_url":"https://api.github.com/repos/uwescience/myria/tags","blobs_url":"https://api.github.com/repos/uwescience/myria/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/uwescience/myria/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/uwescience/myria/git/refs{/sha}","trees_url":"https://api.github.com/repos/uwescience/myria/git/trees{/sha}","statuses_url":"https://api.github.com/repos/uwescience/myria/statuses/{sha}","languages_url":"https://api.github.com/repos/uwescience/myria/languages","stargazers_url":"https://api.github.com/repos/uwescience/myria/stargazers","contributors_url":"https://api.github.com/repos/uwescience/myria/contributors","subscribers_url":"https://api.github.com/repos/uwescience/myria/subscribers","subscription_url":"https://api.github.com/repos/uwescience/myria/subscription","commits_url":"https://api.github.com/repos/uwescience/myria/commits{/sha}","git_commits_url":"https://api.github.com/repos/uwescience/myria/git/commits{/sha}","comments_url":"https://api.github.com/repos/uwescience/myria/comments{/number}","issue_comment_url":"https://api.github.com/repos/uwescience/myria/issues/comments{/number}","contents_url":"https://api.github.com/repos/uwescience/myria/contents/{+path}","compare_url":"https://api.github.com/repos/uwescience/myria/compare/{base}...{head}","merges_url":"https://api.github.com/repos/uwescience/myria/merges","archive_url":"https://api.github.com/repos/uwescience/myria/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/uwescience/myria/downloads","issues_url":"https://api.github.com/repos/uwescience/myria/issues{/number}","pulls_url":"https://api.github.com/repos/uwescience/myria/pulls{/number}","milestones_url":"https://api.github.com/repos/uwescience/myria/milestones{/number}","notifications_url":"https://api.github.com/repos/uwescience/myria/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/uwescience/myria/labels{/name}","releases_url":"https://api.github.com/repos/uwescience/myria/releases{/id}","deployments_url":"https://api.github.com/repos/uwescience/myria/deployments","created_at":"2012-08-30T08:27:36Z","updated_at":"2016-05-31T02:53:00Z","pushed_at":"2016-08-19T22:46:13Z","git_url":"git://github.com/uwescience/myria.git","ssh_url":"git@github.com:uwescience/myria.git","clone_url":"https://github.com/uwescience/myria.git","svn_url":"https://github.com/uwescience/myria","homepage":"myria.cs.washington.edu","size":77143,"stargazers_count":41,"watchers_count":41,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":23,"mirror_url":null,"open_issues_count":99,"forks":23,"open_issues":99,"watchers":41,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/uwescience/myria/pulls/848"},"html":{"href":"https://github.com/uwescience/myria/pull/848"},"issue":{"href":"https://api.github.com/repos/uwescience/myria/issues/848"},"comments":{"href":"https://api.github.com/repos/uwescience/myria/issues/848/comments"},"review_comments":{"href":"https://api.github.com/repos/uwescience/myria/pulls/848/comments"},"review_comment":{"href":"https://api.github.com/repos/uwescience/myria/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/uwescience/myria/pulls/848/commits"},"statuses":{"href":"https://api.github.com/repos/uwescience/myria/statuses/0c7b6c3fd29acf8639c4c5356d837a052d3a5ff4"}}}}
{"actor":{"display_login":"jortiz16"}}