{"action":"created","comment":{"url":"https://api.github.com/repos/apache/incubator-gobblin/pulls/comments/176372342","pull_request_review_id":106060282,"id":176372342,"diff_hunk":"@@ -0,0 +1,135 @@\n+/*\n+ * Licensed to the Apache Software Foundation (ASF) under one or more\n+ * contributor license agreements.  See the NOTICE file distributed with\n+ * this work for additional information regarding copyright ownership.\n+ * The ASF licenses this file to You under the Apache License, Version 2.0\n+ * (the \"License\"); you may not use this file except in compliance with\n+ * the License.  You may obtain a copy of the License at\n+ *\n+ *    http://www.apache.org/licenses/LICENSE-2.0\n+ *\n+ * Unless required by applicable law or agreed to in writing, software\n+ * distributed under the License is distributed on an \"AS IS\" BASIS,\n+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n+ * See the License for the specific language governing permissions and\n+ * limitations under the License.\n+ */\n+\n+package org.apache.gobblin.hive.policy;\n+\n+import java.io.IOException;\n+import java.util.List;\n+import java.util.regex.Matcher;\n+import java.util.regex.Pattern;\n+\n+import org.apache.hadoop.fs.Path;\n+import org.slf4j.Logger;\n+import org.slf4j.LoggerFactory;\n+\n+import com.google.common.base.Optional;\n+import com.google.common.base.Strings;\n+import com.google.common.collect.Lists;\n+\n+import org.apache.gobblin.configuration.State;\n+import org.apache.gobblin.hive.HivePartition;\n+import org.apache.gobblin.hive.HiveRegistrationUnit;\n+import org.apache.gobblin.hive.HiveTable;\n+\n+\n+/**\n+ * This is the partition aware implementation of {@link HiveRegistrationPolicyBase}.\n+ * You can specify hive.partition.regex where the first match is the table location and the\n+ * rest match are the partitions.\n+ * The partition names can be specified with the hive.table.partition.keys property.\n+ * The order of the partition names should match the order of regexp matches in the hive.partition.regexp expression.\n+ *\n+ * For example in the case of path: s3://testbucket/myawesomlogs/compacted/dt=20170101/hr=22/\n+ * The hive.partition.regexp would look like: hive.partition.regex=(s3://testbucket/myawesomelogs/compacted/)dt=(.*)/hr=(.*)\n+ * and the hive.table.partition.keys=dt,hr\n+ *\n+ * @author Tamas Nemeth\n+ */\n+\n+public class PartitionAwareHiveRegistrationPolicy extends HiveRegistrationPolicyBase{\n+  private static final Logger LOG = LoggerFactory.getLogger(PartitionAwareHiveRegistrationPolicy.class);\n+\n+  public static final String HIVE_PARTITION_REGEX = \"hive.partition.regex\";\n+  public static final String HIVE_TABLE_PARTITION_KEYS = \"hive.table.partition.keys\";\n+\n+  public PartitionAwareHiveRegistrationPolicy(State props)\n+      throws IOException {\n+    super(props);\n+  }\n+\n+  protected HiveTable getTable(Path path, String dbName, String tableName) throws IOException {\n+    LOG.debug(\"Getting table definition for {}\", tableName);\n+\n+    HiveTable table = super.getTable(path, dbName, tableName);\n+\n+    LOG.debug(\"Getting partition keys for {}\", tableName);\n+    if (!Strings.isNullOrEmpty(this.props.getProp(HIVE_TABLE_PARTITION_KEYS))) {\n+      List<HiveRegistrationUnit.Column> partitionKeyColumns = Lists.<HiveRegistrationUnit.Column> newArrayList();\n+\n+      for (String key : this.props.getPropAsList(HIVE_TABLE_PARTITION_KEYS)) {\n+        LOG.debug(\"Setting partition key {} for table {}\", key, tableName);\n+        partitionKeyColumns.add(new HiveRegistrationUnit.Column(key , \"string\", null));\n+      }\n+\n+      table.setPartitionKeys(partitionKeyColumns);\n+    }\n+\n+    return table;\n+  }\n+\n+\n+  protected Optional<HivePartition> getPartition(Path path, HiveTable table) throws IOException {\n+\n+    if (this.props.contains(HIVE_PARTITION_REGEX)) {\n+      Pattern pattern = Pattern.compile(this.props.getProp(HIVE_PARTITION_REGEX));\n+      List<String> partitionValues = Lists.newArrayList();\n+\n+      Matcher matcher = pattern.matcher(path.toString());\n+      if (matcher.matches() && matcher.groupCount() >=2){\n+        for (int i = 2; i <= matcher.groupCount(); i++) {\n+          partitionValues.add(matcher.group(i));\n+        }","path":"gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/policy/PartitionAwareHiveRegistrationPolicy.java","position":95,"original_position":95,"commit_id":"37ed64eab84566dfb0ad4cd7455ab40d39932cc9","original_commit_id":"e5c775c743f81bb1764e4bfada2620b9ab35ff17","user":{"login":"deepak-batra","id":7312271,"avatar_url":"https://avatars0.githubusercontent.com/u/7312271?v=4","gravatar_id":"","url":"https://api.github.com/users/deepak-batra","html_url":"https://github.com/deepak-batra","followers_url":"https://api.github.com/users/deepak-batra/followers","following_url":"https://api.github.com/users/deepak-batra/following{/other_user}","gists_url":"https://api.github.com/users/deepak-batra/gists{/gist_id}","starred_url":"https://api.github.com/users/deepak-batra/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/deepak-batra/subscriptions","organizations_url":"https://api.github.com/users/deepak-batra/orgs","repos_url":"https://api.github.com/users/deepak-batra/repos","events_url":"https://api.github.com/users/deepak-batra/events{/privacy}","received_events_url":"https://api.github.com/users/deepak-batra/received_events","type":"User","site_admin":false},"body":"Hey Tamas, \r\nAs we're specifying partition keys, shouldn't we iterate on them to make sure all the partition key values are filled. For ex.: \r\nsay \r\n`partition.keys` = dt, hr, min\r\nand the regex is = (some-path)/dt=*/hr=* (something like this). `\r\nAlthough the table will be registered with different partition keys, we would pass lesser number of values to register a partition which would lead an exception from HiveMetastore's thrift API. However, if we do it the other way, then the HivePartitionBuilder would fail quite early as compared to the other fail scenario.","created_at":"2018-03-22T10:34:21Z","updated_at":"2018-03-22T10:34:21Z","html_url":"https://github.com/apache/incubator-gobblin/pull/2302#discussion_r176372342","pull_request_url":"https://api.github.com/repos/apache/incubator-gobblin/pulls/2302","author_association":"NONE","_links":{"self":{"href":"https://api.github.com/repos/apache/incubator-gobblin/pulls/comments/176372342"},"html":{"href":"https://github.com/apache/incubator-gobblin/pull/2302#discussion_r176372342"},"pull_request":{"href":"https://api.github.com/repos/apache/incubator-gobblin/pulls/2302"}}},"pull_request":{"url":"https://api.github.com/repos/apache/incubator-gobblin/pulls/2302","id":174400806,"html_url":"https://github.com/apache/incubator-gobblin/pull/2302","diff_url":"https://github.com/apache/incubator-gobblin/pull/2302.diff","patch_url":"https://github.com/apache/incubator-gobblin/pull/2302.patch","issue_url":"https://api.github.com/repos/apache/incubator-gobblin/issues/2302","number":2302,"state":"open","locked":false,"title":"[GOBBLIN-425] Partitionaware Hive Registration Policy","user":{"login":"treff7es","id":1339772,"avatar_url":"https://avatars0.githubusercontent.com/u/1339772?v=4","gravatar_id":"","url":"https://api.github.com/users/treff7es","html_url":"https://github.com/treff7es","followers_url":"https://api.github.com/users/treff7es/followers","following_url":"https://api.github.com/users/treff7es/following{/other_user}","gists_url":"https://api.github.com/users/treff7es/gists{/gist_id}","starred_url":"https://api.github.com/users/treff7es/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/treff7es/subscriptions","organizations_url":"https://api.github.com/users/treff7es/orgs","repos_url":"https://api.github.com/users/treff7es/repos","events_url":"https://api.github.com/users/treff7es/events{/privacy}","received_events_url":"https://api.github.com/users/treff7es/received_events","type":"User","site_admin":false},"body":"Dear Gobblin maintainers,\r\n\r\nPlease accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!\r\n\r\n\r\n### JIRA\r\n- [ ] My PR addresses the following [Gobblin JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references them in the PR title. For example, \"[GOBBLIN-XXX] My Gobblin PR\"\r\n    - https://issues.apache.org/jira/browse/GOBBLIN-425\r\n\r\n\r\n### Description\r\nThis is the partition aware implementation of {@link HiveRegistrationPolicyBase}.\r\nYou can specify hive.partition.regex where the first match is the table location and the rest match are the partitions.\r\nThe partition names can be specified with the hive.table.partition.keys property.\r\nThe order of the partition names should match the order of regexp matches in the hive.partition.regexp expression.\r\n\r\n For example in the case of path: s3://testbucket/myawesomlogs/compacted/dt=20170101/hr=22/\r\n The hive.partition.regexp would look like: hive.partition.regex=(s3://testbucket/myawesomelogs/compacted/)dt=(.*)/hr=(.*) and the hive.table.partition.keys=dt,hr\r\n\r\n### Tests\r\n- [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason:\r\n\r\n\r\n### Commits\r\n- [ ] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from \"[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)\":\r\n    1. Subject is separated from body by a blank line\r\n    2. Subject is limited to 50 characters\r\n    3. Subject does not end with a period\r\n    4. Subject uses the imperative mood (\"add\", not \"adding\")\r\n    5. Body wraps at 72 characters\r\n    6. Body explains \"what\" and \"why\", not \"how\"\r\n\r\n","created_at":"2018-03-12T15:18:04Z","updated_at":"2018-03-22T10:34:21Z","closed_at":null,"merged_at":null,"merge_commit_sha":"080598acead1efd82c500226ced7e34429885c3f","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/apache/incubator-gobblin/pulls/2302/commits","review_comments_url":"https://api.github.com/repos/apache/incubator-gobblin/pulls/2302/comments","review_comment_url":"https://api.github.com/repos/apache/incubator-gobblin/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/incubator-gobblin/issues/2302/comments","statuses_url":"https://api.github.com/repos/apache/incubator-gobblin/statuses/37ed64eab84566dfb0ad4cd7455ab40d39932cc9","head":{"label":"prezi:PartitionAwareHiveRegistrationPolicy","ref":"PartitionAwareHiveRegistrationPolicy","sha":"37ed64eab84566dfb0ad4cd7455ab40d39932cc9","user":{"login":"prezi","id":1989101,"avatar_url":"https://avatars2.githubusercontent.com/u/1989101?v=4","gravatar_id":"","url":"https://api.github.com/users/prezi","html_url":"https://github.com/prezi","followers_url":"https://api.github.com/users/prezi/followers","following_url":"https://api.github.com/users/prezi/following{/other_user}","gists_url":"https://api.github.com/users/prezi/gists{/gist_id}","starred_url":"https://api.github.com/users/prezi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/prezi/subscriptions","organizations_url":"https://api.github.com/users/prezi/orgs","repos_url":"https://api.github.com/users/prezi/repos","events_url":"https://api.github.com/users/prezi/events{/privacy}","received_events_url":"https://api.github.com/users/prezi/received_events","type":"Organization","site_admin":false},"repo":{"id":77775641,"name":"gobblin","full_name":"prezi/gobblin","owner":{"login":"prezi","id":1989101,"avatar_url":"https://avatars2.githubusercontent.com/u/1989101?v=4","gravatar_id":"","url":"https://api.github.com/users/prezi","html_url":"https://github.com/prezi","followers_url":"https://api.github.com/users/prezi/followers","following_url":"https://api.github.com/users/prezi/following{/other_user}","gists_url":"https://api.github.com/users/prezi/gists{/gist_id}","starred_url":"https://api.github.com/users/prezi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/prezi/subscriptions","organizations_url":"https://api.github.com/users/prezi/orgs","repos_url":"https://api.github.com/users/prezi/repos","events_url":"https://api.github.com/users/prezi/events{/privacy}","received_events_url":"https://api.github.com/users/prezi/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/prezi/gobblin","description":"Universal data ingestion framework for Hadoop.","fork":true,"url":"https://api.github.com/repos/prezi/gobblin","forks_url":"https://api.github.com/repos/prezi/gobblin/forks","keys_url":"https://api.github.com/repos/prezi/gobblin/keys{/key_id}","collaborators_url":"https://api.github.com/repos/prezi/gobblin/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/prezi/gobblin/teams","hooks_url":"https://api.github.com/repos/prezi/gobblin/hooks","issue_events_url":"https://api.github.com/repos/prezi/gobblin/issues/events{/number}","events_url":"https://api.github.com/repos/prezi/gobblin/events","assignees_url":"https://api.github.com/repos/prezi/gobblin/assignees{/user}","branches_url":"https://api.github.com/repos/prezi/gobblin/branches{/branch}","tags_url":"https://api.github.com/repos/prezi/gobblin/tags","blobs_url":"https://api.github.com/repos/prezi/gobblin/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/prezi/gobblin/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/prezi/gobblin/git/refs{/sha}","trees_url":"https://api.github.com/repos/prezi/gobblin/git/trees{/sha}","statuses_url":"https://api.github.com/repos/prezi/gobblin/statuses/{sha}","languages_url":"https://api.github.com/repos/prezi/gobblin/languages","stargazers_url":"https://api.github.com/repos/prezi/gobblin/stargazers","contributors_url":"https://api.github.com/repos/prezi/gobblin/contributors","subscribers_url":"https://api.github.com/repos/prezi/gobblin/subscribers","subscription_url":"https://api.github.com/repos/prezi/gobblin/subscription","commits_url":"https://api.github.com/repos/prezi/gobblin/commits{/sha}","git_commits_url":"https://api.github.com/repos/prezi/gobblin/git/commits{/sha}","comments_url":"https://api.github.com/repos/prezi/gobblin/comments{/number}","issue_comment_url":"https://api.github.com/repos/prezi/gobblin/issues/comments{/number}","contents_url":"https://api.github.com/repos/prezi/gobblin/contents/{+path}","compare_url":"https://api.github.com/repos/prezi/gobblin/compare/{base}...{head}","merges_url":"https://api.github.com/repos/prezi/gobblin/merges","archive_url":"https://api.github.com/repos/prezi/gobblin/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/prezi/gobblin/downloads","issues_url":"https://api.github.com/repos/prezi/gobblin/issues{/number}","pulls_url":"https://api.github.com/repos/prezi/gobblin/pulls{/number}","milestones_url":"https://api.github.com/repos/prezi/gobblin/milestones{/number}","notifications_url":"https://api.github.com/repos/prezi/gobblin/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/prezi/gobblin/labels{/name}","releases_url":"https://api.github.com/repos/prezi/gobblin/releases{/id}","deployments_url":"https://api.github.com/repos/prezi/gobblin/deployments","created_at":"2017-01-01T11:09:44Z","updated_at":"2018-03-12T13:00:28Z","pushed_at":"2018-03-22T10:12:55Z","git_url":"git://github.com/prezi/gobblin.git","ssh_url":"git@github.com:prezi/gobblin.git","clone_url":"https://github.com/prezi/gobblin.git","svn_url":"https://github.com/prezi/gobblin","homepage":"https://github.com/linkedin/gobblin/wiki","size":114868,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":null,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"979ad2a090600495ad3b47de462d39b2f2ab33ea","user":{"login":"apache","id":47359,"avatar_url":"https://avatars0.githubusercontent.com/u/47359?v=4","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":27392543,"name":"incubator-gobblin","full_name":"apache/incubator-gobblin","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars0.githubusercontent.com/u/47359?v=4","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/incubator-gobblin","description":"Gobblin is a distributed big data integration framework (ingestion, replication, compliance, retention)  for batch and streaming systems. Gobblin features integrations with Apache Hadoop, Apache Kafka, Salesforce, S3, MySQL, Google etc.","fork":false,"url":"https://api.github.com/repos/apache/incubator-gobblin","forks_url":"https://api.github.com/repos/apache/incubator-gobblin/forks","keys_url":"https://api.github.com/repos/apache/incubator-gobblin/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/incubator-gobblin/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/incubator-gobblin/teams","hooks_url":"https://api.github.com/repos/apache/incubator-gobblin/hooks","issue_events_url":"https://api.github.com/repos/apache/incubator-gobblin/issues/events{/number}","events_url":"https://api.github.com/repos/apache/incubator-gobblin/events","assignees_url":"https://api.github.com/repos/apache/incubator-gobblin/assignees{/user}","branches_url":"https://api.github.com/repos/apache/incubator-gobblin/branches{/branch}","tags_url":"https://api.github.com/repos/apache/incubator-gobblin/tags","blobs_url":"https://api.github.com/repos/apache/incubator-gobblin/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/incubator-gobblin/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/incubator-gobblin/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/incubator-gobblin/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/incubator-gobblin/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/incubator-gobblin/languages","stargazers_url":"https://api.github.com/repos/apache/incubator-gobblin/stargazers","contributors_url":"https://api.github.com/repos/apache/incubator-gobblin/contributors","subscribers_url":"https://api.github.com/repos/apache/incubator-gobblin/subscribers","subscription_url":"https://api.github.com/repos/apache/incubator-gobblin/subscription","commits_url":"https://api.github.com/repos/apache/incubator-gobblin/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/incubator-gobblin/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/incubator-gobblin/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/incubator-gobblin/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/incubator-gobblin/contents/{+path}","compare_url":"https://api.github.com/repos/apache/incubator-gobblin/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/incubator-gobblin/merges","archive_url":"https://api.github.com/repos/apache/incubator-gobblin/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/incubator-gobblin/downloads","issues_url":"https://api.github.com/repos/apache/incubator-gobblin/issues{/number}","pulls_url":"https://api.github.com/repos/apache/incubator-gobblin/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/incubator-gobblin/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/incubator-gobblin/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/incubator-gobblin/labels{/name}","releases_url":"https://api.github.com/repos/apache/incubator-gobblin/releases{/id}","deployments_url":"https://api.github.com/repos/apache/incubator-gobblin/deployments","created_at":"2014-12-01T18:10:50Z","updated_at":"2018-03-22T00:57:58Z","pushed_at":"2018-03-22T10:12:58Z","git_url":"git://github.com/apache/incubator-gobblin.git","ssh_url":"git@github.com:apache/incubator-gobblin.git","clone_url":"https://github.com/apache/incubator-gobblin.git","svn_url":"https://github.com/apache/incubator-gobblin","homepage":"https://github.com/linkedin/gobblin/wiki","size":122066,"stargazers_count":1246,"watchers_count":1246,"language":"Java","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":491,"mirror_url":null,"archived":false,"open_issues_count":25,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0"},"forks":491,"open_issues":25,"watchers":1246,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/incubator-gobblin/pulls/2302"},"html":{"href":"https://github.com/apache/incubator-gobblin/pull/2302"},"issue":{"href":"https://api.github.com/repos/apache/incubator-gobblin/issues/2302"},"comments":{"href":"https://api.github.com/repos/apache/incubator-gobblin/issues/2302/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/incubator-gobblin/pulls/2302/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/incubator-gobblin/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/incubator-gobblin/pulls/2302/commits"},"statuses":{"href":"https://api.github.com/repos/apache/incubator-gobblin/statuses/37ed64eab84566dfb0ad4cd7455ab40d39932cc9"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"deepak-batra"}}