{"action":"created","comment":{"url":"https://api.github.com/repos/uber/hudi/pulls/comments/192330018","pull_request_review_id":125085876,"id":192330018,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDE5MjMzMDAxOA==","diff_hunk":"@@ -128,7 +131,7 @@ private void assertSqlContext() {\n \n     assertSqlContext();\n     JavaPairRDD<HoodieKey, Optional<String>> keyToFileRDD = index\n-        .fetchRecordLocation(hoodieKeys, hoodieTable);\n+        .fetchRecordLocation(hoodieKeys, jsc, hoodieTable.getMetaClient());","path":"hoodie-client/src/main/java/com/uber/hoodie/HoodieReadClient.java","position":20,"original_position":29,"commit_id":"305cc7f19ae74a02ee5a207814d4056e330abcf6","original_commit_id":"c27bf94e0536f4ba3ae7cf0eddffbb2f54c6cd3d","user":{"login":"n3nash","id":2722167,"node_id":"MDQ6VXNlcjI3MjIxNjc=","avatar_url":"https://avatars1.githubusercontent.com/u/2722167?v=4","gravatar_id":"","url":"https://api.github.com/users/n3nash","html_url":"https://github.com/n3nash","followers_url":"https://api.github.com/users/n3nash/followers","following_url":"https://api.github.com/users/n3nash/following{/other_user}","gists_url":"https://api.github.com/users/n3nash/gists{/gist_id}","starred_url":"https://api.github.com/users/n3nash/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/n3nash/subscriptions","organizations_url":"https://api.github.com/users/n3nash/orgs","repos_url":"https://api.github.com/users/n3nash/repos","events_url":"https://api.github.com/users/n3nash/events{/privacy}","received_events_url":"https://api.github.com/users/n3nash/received_events","type":"User","site_admin":false},"body":"I want to remove the SparkContext from Index since we are already passing it to the table and apart from a method in BloomIndex, none of the index actually requires that. I already synced with @bvaradar on this after explaining to him that we are moving index under the table which he was unaware of when he made this comment. ","created_at":"2018-06-01T08:32:36Z","updated_at":"2018-06-01T08:32:36Z","html_url":"https://github.com/uber/hudi/pull/400#discussion_r192330018","pull_request_url":"https://api.github.com/repos/uber/hudi/pulls/400","author_association":"COLLABORATOR","_links":{"self":{"href":"https://api.github.com/repos/uber/hudi/pulls/comments/192330018"},"html":{"href":"https://github.com/uber/hudi/pull/400#discussion_r192330018"},"pull_request":{"href":"https://api.github.com/repos/uber/hudi/pulls/400"}},"in_reply_to_id":191828946},"pull_request":{"url":"https://api.github.com/repos/uber/hudi/pulls/400","id":191298010,"node_id":"MDExOlB1bGxSZXF1ZXN0MTkxMjk4MDEw","html_url":"https://github.com/uber/hudi/pull/400","diff_url":"https://github.com/uber/hudi/pull/400.diff","patch_url":"https://github.com/uber/hudi/pull/400.patch","issue_url":"https://api.github.com/repos/uber/hudi/issues/400","number":400,"state":"open","locked":false,"title":"Adding ability for inserts to be written to log files","user":{"login":"n3nash","id":2722167,"node_id":"MDQ6VXNlcjI3MjIxNjc=","avatar_url":"https://avatars1.githubusercontent.com/u/2722167?v=4","gravatar_id":"","url":"https://api.github.com/users/n3nash","html_url":"https://github.com/n3nash","followers_url":"https://api.github.com/users/n3nash/followers","following_url":"https://api.github.com/users/n3nash/following{/other_user}","gists_url":"https://api.github.com/users/n3nash/gists{/gist_id}","starred_url":"https://api.github.com/users/n3nash/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/n3nash/subscriptions","organizations_url":"https://api.github.com/users/n3nash/orgs","repos_url":"https://api.github.com/users/n3nash/repos","events_url":"https://api.github.com/users/n3nash/events{/privacy}","received_events_url":"https://api.github.com/users/n3nash/received_events","type":"User","site_admin":false},"body":"1. This does not have the changes needed to run async compaction.\r\n2. Pending tests to ensure inserts into log files work after compaction is scheduled etc.\r\n\r\n1. MOR table type supports inserts written to log files (along with inserts written to parquet files). This is determined based on whether the log files are indexable.\r\n2. Static sizing of logfiles -> parquet files compression ratio (TODO : check if the parquetCompressionRatio is the same as this and then use that)\r\n3. Moved HoodieIndex under HoodieTable.\r\n4. Added tests for inserts into log files.\r\n5. TODO - Small file handling is still based on the smallest parquet file in the partition. Open item to implement small file handling by knowing how many inserts vs updates present in log files for a file slice. After this is implement, revisit rollback and tests around that.\r\n\r\n![image1 8](https://user-images.githubusercontent.com/2722167/40014531-aaa114e4-5765-11e8-8988-1db229ea8285.jpeg)\r\n","created_at":"2018-05-29T23:45:19Z","updated_at":"2018-06-01T08:32:36Z","closed_at":null,"merged_at":null,"merge_commit_sha":null,"assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/uber/hudi/pulls/400/commits","review_comments_url":"https://api.github.com/repos/uber/hudi/pulls/400/comments","review_comment_url":"https://api.github.com/repos/uber/hudi/pulls/comments{/number}","comments_url":"https://api.github.com/repos/uber/hudi/issues/400/comments","statuses_url":"https://api.github.com/repos/uber/hudi/statuses/305cc7f19ae74a02ee5a207814d4056e330abcf6","head":{"label":"n3nash:inserts_into_logfiles_only_changes","ref":"inserts_into_logfiles_only_changes","sha":"305cc7f19ae74a02ee5a207814d4056e330abcf6","user":{"login":"n3nash","id":2722167,"node_id":"MDQ6VXNlcjI3MjIxNjc=","avatar_url":"https://avatars1.githubusercontent.com/u/2722167?v=4","gravatar_id":"","url":"https://api.github.com/users/n3nash","html_url":"https://github.com/n3nash","followers_url":"https://api.github.com/users/n3nash/followers","following_url":"https://api.github.com/users/n3nash/following{/other_user}","gists_url":"https://api.github.com/users/n3nash/gists{/gist_id}","starred_url":"https://api.github.com/users/n3nash/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/n3nash/subscriptions","organizations_url":"https://api.github.com/users/n3nash/orgs","repos_url":"https://api.github.com/users/n3nash/repos","events_url":"https://api.github.com/users/n3nash/events{/privacy}","received_events_url":"https://api.github.com/users/n3nash/received_events","type":"User","site_admin":false},"repo":{"id":79266493,"node_id":"MDEwOlJlcG9zaXRvcnk3OTI2NjQ5Mw==","name":"hoodie","full_name":"n3nash/hoodie","owner":{"login":"n3nash","id":2722167,"node_id":"MDQ6VXNlcjI3MjIxNjc=","avatar_url":"https://avatars1.githubusercontent.com/u/2722167?v=4","gravatar_id":"","url":"https://api.github.com/users/n3nash","html_url":"https://github.com/n3nash","followers_url":"https://api.github.com/users/n3nash/followers","following_url":"https://api.github.com/users/n3nash/following{/other_user}","gists_url":"https://api.github.com/users/n3nash/gists{/gist_id}","starred_url":"https://api.github.com/users/n3nash/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/n3nash/subscriptions","organizations_url":"https://api.github.com/users/n3nash/orgs","repos_url":"https://api.github.com/users/n3nash/repos","events_url":"https://api.github.com/users/n3nash/events{/privacy}","received_events_url":"https://api.github.com/users/n3nash/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/n3nash/hoodie","description":"Spark Library for Hadoop Upserts And Incrementals","fork":true,"url":"https://api.github.com/repos/n3nash/hoodie","forks_url":"https://api.github.com/repos/n3nash/hoodie/forks","keys_url":"https://api.github.com/repos/n3nash/hoodie/keys{/key_id}","collaborators_url":"https://api.github.com/repos/n3nash/hoodie/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/n3nash/hoodie/teams","hooks_url":"https://api.github.com/repos/n3nash/hoodie/hooks","issue_events_url":"https://api.github.com/repos/n3nash/hoodie/issues/events{/number}","events_url":"https://api.github.com/repos/n3nash/hoodie/events","assignees_url":"https://api.github.com/repos/n3nash/hoodie/assignees{/user}","branches_url":"https://api.github.com/repos/n3nash/hoodie/branches{/branch}","tags_url":"https://api.github.com/repos/n3nash/hoodie/tags","blobs_url":"https://api.github.com/repos/n3nash/hoodie/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/n3nash/hoodie/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/n3nash/hoodie/git/refs{/sha}","trees_url":"https://api.github.com/repos/n3nash/hoodie/git/trees{/sha}","statuses_url":"https://api.github.com/repos/n3nash/hoodie/statuses/{sha}","languages_url":"https://api.github.com/repos/n3nash/hoodie/languages","stargazers_url":"https://api.github.com/repos/n3nash/hoodie/stargazers","contributors_url":"https://api.github.com/repos/n3nash/hoodie/contributors","subscribers_url":"https://api.github.com/repos/n3nash/hoodie/subscribers","subscription_url":"https://api.github.com/repos/n3nash/hoodie/subscription","commits_url":"https://api.github.com/repos/n3nash/hoodie/commits{/sha}","git_commits_url":"https://api.github.com/repos/n3nash/hoodie/git/commits{/sha}","comments_url":"https://api.github.com/repos/n3nash/hoodie/comments{/number}","issue_comment_url":"https://api.github.com/repos/n3nash/hoodie/issues/comments{/number}","contents_url":"https://api.github.com/repos/n3nash/hoodie/contents/{+path}","compare_url":"https://api.github.com/repos/n3nash/hoodie/compare/{base}...{head}","merges_url":"https://api.github.com/repos/n3nash/hoodie/merges","archive_url":"https://api.github.com/repos/n3nash/hoodie/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/n3nash/hoodie/downloads","issues_url":"https://api.github.com/repos/n3nash/hoodie/issues{/number}","pulls_url":"https://api.github.com/repos/n3nash/hoodie/pulls{/number}","milestones_url":"https://api.github.com/repos/n3nash/hoodie/milestones{/number}","notifications_url":"https://api.github.com/repos/n3nash/hoodie/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/n3nash/hoodie/labels{/name}","releases_url":"https://api.github.com/repos/n3nash/hoodie/releases{/id}","deployments_url":"https://api.github.com/repos/n3nash/hoodie/deployments","created_at":"2017-01-17T20:02:08Z","updated_at":"2018-05-16T18:03:29Z","pushed_at":"2018-05-31T17:45:05Z","git_url":"git://github.com/n3nash/hoodie.git","ssh_url":"git@github.com:n3nash/hoodie.git","clone_url":"https://github.com/n3nash/hoodie.git","svn_url":"https://github.com/n3nash/hoodie","homepage":"https://uber.github.io/hoodie","size":5418,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"uber:master","ref":"master","sha":"a97814462d1b7b2b7a1a880df1e30bc5a8c3544b","user":{"login":"uber","id":538264,"node_id":"MDEyOk9yZ2FuaXphdGlvbjUzODI2NA==","avatar_url":"https://avatars3.githubusercontent.com/u/538264?v=4","gravatar_id":"","url":"https://api.github.com/users/uber","html_url":"https://github.com/uber","followers_url":"https://api.github.com/users/uber/followers","following_url":"https://api.github.com/users/uber/following{/other_user}","gists_url":"https://api.github.com/users/uber/gists{/gist_id}","starred_url":"https://api.github.com/users/uber/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/uber/subscriptions","organizations_url":"https://api.github.com/users/uber/orgs","repos_url":"https://api.github.com/users/uber/repos","events_url":"https://api.github.com/users/uber/events{/privacy}","received_events_url":"https://api.github.com/users/uber/received_events","type":"Organization","site_admin":false},"repo":{"id":76474200,"node_id":"MDEwOlJlcG9zaXRvcnk3NjQ3NDIwMA==","name":"hudi","full_name":"uber/hudi","owner":{"login":"uber","id":538264,"node_id":"MDEyOk9yZ2FuaXphdGlvbjUzODI2NA==","avatar_url":"https://avatars3.githubusercontent.com/u/538264?v=4","gravatar_id":"","url":"https://api.github.com/users/uber","html_url":"https://github.com/uber","followers_url":"https://api.github.com/users/uber/followers","following_url":"https://api.github.com/users/uber/following{/other_user}","gists_url":"https://api.github.com/users/uber/gists{/gist_id}","starred_url":"https://api.github.com/users/uber/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/uber/subscriptions","organizations_url":"https://api.github.com/users/uber/orgs","repos_url":"https://api.github.com/users/uber/repos","events_url":"https://api.github.com/users/uber/events{/privacy}","received_events_url":"https://api.github.com/users/uber/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/uber/hudi","description":"Spark Library for Hadoop Upserts And Incrementals","fork":false,"url":"https://api.github.com/repos/uber/hudi","forks_url":"https://api.github.com/repos/uber/hudi/forks","keys_url":"https://api.github.com/repos/uber/hudi/keys{/key_id}","collaborators_url":"https://api.github.com/repos/uber/hudi/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/uber/hudi/teams","hooks_url":"https://api.github.com/repos/uber/hudi/hooks","issue_events_url":"https://api.github.com/repos/uber/hudi/issues/events{/number}","events_url":"https://api.github.com/repos/uber/hudi/events","assignees_url":"https://api.github.com/repos/uber/hudi/assignees{/user}","branches_url":"https://api.github.com/repos/uber/hudi/branches{/branch}","tags_url":"https://api.github.com/repos/uber/hudi/tags","blobs_url":"https://api.github.com/repos/uber/hudi/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/uber/hudi/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/uber/hudi/git/refs{/sha}","trees_url":"https://api.github.com/repos/uber/hudi/git/trees{/sha}","statuses_url":"https://api.github.com/repos/uber/hudi/statuses/{sha}","languages_url":"https://api.github.com/repos/uber/hudi/languages","stargazers_url":"https://api.github.com/repos/uber/hudi/stargazers","contributors_url":"https://api.github.com/repos/uber/hudi/contributors","subscribers_url":"https://api.github.com/repos/uber/hudi/subscribers","subscription_url":"https://api.github.com/repos/uber/hudi/subscription","commits_url":"https://api.github.com/repos/uber/hudi/commits{/sha}","git_commits_url":"https://api.github.com/repos/uber/hudi/git/commits{/sha}","comments_url":"https://api.github.com/repos/uber/hudi/comments{/number}","issue_comment_url":"https://api.github.com/repos/uber/hudi/issues/comments{/number}","contents_url":"https://api.github.com/repos/uber/hudi/contents/{+path}","compare_url":"https://api.github.com/repos/uber/hudi/compare/{base}...{head}","merges_url":"https://api.github.com/repos/uber/hudi/merges","archive_url":"https://api.github.com/repos/uber/hudi/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/uber/hudi/downloads","issues_url":"https://api.github.com/repos/uber/hudi/issues{/number}","pulls_url":"https://api.github.com/repos/uber/hudi/pulls{/number}","milestones_url":"https://api.github.com/repos/uber/hudi/milestones{/number}","notifications_url":"https://api.github.com/repos/uber/hudi/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/uber/hudi/labels{/name}","releases_url":"https://api.github.com/repos/uber/hudi/releases{/id}","deployments_url":"https://api.github.com/repos/uber/hudi/deployments","created_at":"2016-12-14T15:53:41Z","updated_at":"2018-06-01T04:40:18Z","pushed_at":"2018-05-31T17:45:07Z","git_url":"git://github.com/uber/hudi.git","ssh_url":"git@github.com:uber/hudi.git","clone_url":"https://github.com/uber/hudi.git","svn_url":"https://github.com/uber/hudi","homepage":"https://uber.github.io/hudi","size":5205,"stargazers_count":161,"watchers_count":161,"language":"Java","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":52,"mirror_url":null,"archived":false,"open_issues_count":110,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":52,"open_issues":110,"watchers":161,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/uber/hudi/pulls/400"},"html":{"href":"https://github.com/uber/hudi/pull/400"},"issue":{"href":"https://api.github.com/repos/uber/hudi/issues/400"},"comments":{"href":"https://api.github.com/repos/uber/hudi/issues/400/comments"},"review_comments":{"href":"https://api.github.com/repos/uber/hudi/pulls/400/comments"},"review_comment":{"href":"https://api.github.com/repos/uber/hudi/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/uber/hudi/pulls/400/commits"},"statuses":{"href":"https://api.github.com/repos/uber/hudi/statuses/305cc7f19ae74a02ee5a207814d4056e330abcf6"}},"author_association":"COLLABORATOR"}}
{"actor":{"display_login":"n3nash"}}