{"action":"closed","number":351,"pull_request":{"url":"https://api.github.com/repos/bloomberg/chef-bach/pulls/351","id":53772780,"html_url":"https://github.com/bloomberg/chef-bach/pull/351","diff_url":"https://github.com/bloomberg/chef-bach/pull/351.diff","patch_url":"https://github.com/bloomberg/chef-bach/pull/351.patch","issue_url":"https://api.github.com/repos/bloomberg/chef-bach/issues/351","number":351,"state":"closed","locked":false,"title":"enable oozie to get credentials for hcatalog and hbase","user":{"login":"pu239ppy","id":4019043,"avatar_url":"https://avatars.githubusercontent.com/u/4019043?v=3","gravatar_id":"","url":"https://api.github.com/users/pu239ppy","html_url":"https://github.com/pu239ppy","followers_url":"https://api.github.com/users/pu239ppy/followers","following_url":"https://api.github.com/users/pu239ppy/following{/other_user}","gists_url":"https://api.github.com/users/pu239ppy/gists{/gist_id}","starred_url":"https://api.github.com/users/pu239ppy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pu239ppy/subscriptions","organizations_url":"https://api.github.com/users/pu239ppy/orgs","repos_url":"https://api.github.com/users/pu239ppy/repos","events_url":"https://api.github.com/users/pu239ppy/events{/privacy}","received_events_url":"https://api.github.com/users/pu239ppy/received_events","type":"User","site_admin":false},"body":"Oozie hive jobs are currently failing because only default database is being exposed by hive.  In order to see other databases oozie launched hive needs to be able to connect to the metastore.\r\n\r\n**Verification Procedure**\r\n- Database creation\r\ncreate database sasl_meta_test;\r\nuse sasl_meta_test;\r\ncreate table sasl_test_data (id int, data string);\r\ninsert into table sasl_test_data values(1, \"aaa111\");\r\ninsert into table sasl_test_data values(2, \"bbb222\");\r\ninsert into table sasl_test_data values(3, \"ccc333\");\r\n\r\n- Ooozie job configuration\r\n- job.properties\r\n```text\r\nnameNode=hdfs://f-bcpc-vm1.bcpc.example.com:8020\r\njobTracker=f-bcpc-vm2.bcpc.example.com:8032\r\nfs.defaultFS=hdfs://f-bcpc-vm1.bcpc.example.com:8020\r\nmapreduce.jobtracker.address=f-bcpc-vm2.bcpc.example.com:8032\r\noozie.libpath=hdfs://f-bcpc-vm1.bcpc.example.com:8020/user/oozie/share/lib/lib_20150422122443\r\noozie.use.system.libpath=true\r\nqueueName=default\r\nexamplesRoot=oozie-tests\r\noozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/hive\r\noutputDir=hive_output\r\n````\r\n\r\n- workflow.xml\r\n````xml\r\n<workflow-app xmlns=\"uri:oozie:workflow:0.3\" name=\"map-reduce-wf\">\r\n  <credentials>\r\n    <credential name=\"hcat_creds\" type=\"hcat\">\r\n      <property>\r\n        <name>hcat.metastore.uri</name>\r\n        <value>thrift://f-bcpc-vm2.bcpc.example.com:9083</value>\r\n      </property>\r\n      <property>\r\n        <name>hive.metastore.kerberos.principal</name>\r\n        <value>hive/_HOST@BCPC.EXAMPLE.COM</value>\r\n      </property>\r\n      <property>\r\n        <name>hcat.metastore.principal</name>\r\n        <value>hive/_HOST@BCPC.EXAMPLE.COM</value>\r\n      </property>\r\n    </credential>\r\n  </credentials>\r\n  <start to=\"hive-query\"/>\r\n  <action name=\"hive-query\" cred=\"hcat_creds\">\r\n    <hive xmlns=\"uri:oozie:hive-action:0.2\">\r\n     <job-tracker>${jobTracker}</job-tracker>\r\n     <name-node>${nameNode}</name-node>\r\n     <configuration>\r\n          <property>\r\n              <name>oozie.launcher.mapreduce.map.memory.mb</name>\r\n              <value>256</value>\r\n          </property>\r\n          <property>\r\n              <name>oozie.launcher.mapreduce.map.java.opts</name>\r\n              <value>\"-Xmx205m\"</value>\r\n          </property>\r\n          <property>\r\n              <name>oozie.launcher.mapreduce.reduce.memory.mb</name>\r\n              <value>256</value>\r\n          </property>\r\n          <property>\r\n              <name>oozie.launcher.mapreduce.reduce.java.opts</name>\r\n              <value>\"-Xmx205m\"</value>\r\n          </property>\r\n          <property>\r\n              <name>oozie.launcher.yarn.app.mapreduce.am.resource.mb</name>\r\n              <value>256</value>\r\n          </property>\r\n     </configuration>\r\n     <script>hivescript.hql</script>\r\n     <param>OUTPUT=hive_output</param>\r\n    </hive>\r\n   <ok to=\"end\"/>\r\n   <error to=\"fail\"/>\r\n  </action>\r\n  <kill name=\"fail\">\r\n    <message>Oozie hive action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>\r\n  </kill>\r\n  <end name=\"end\"/>\r\n</workflow-app>\r\n````\r\n\r\n- hivescript.hql\r\n````text\r\nuse sasl_meta_test;\r\ninsert overwrite directory '${OUTPUT}' select * from sasl_test_data;\r\n````\r\n\r\n**OUTPUT**\r\n````text\r\nubuntu@bcpc-vm3:~$ hdfs dfs -cat hive_output/000000_0\r\n1aaa111\r\n2bbb222\r\n3ccc333\r\n````\r\n\r\n\r\n**TESTING CAVEATS**\r\n- manually assign the hive principal name to hive/f-bcpc-vm2.bcpc.example.com@BCPC.EXAMPLE.COM otherwise hive thinks it should be hive/bcpc-\r\nvm2.bcpc.example.com@BCPC.EXAMPLE.COM and that does not match the keytab.\r\nThis is described in #361 and only affects the VM environment\r\n\r\n- oozie workflow configuration needed adjustments for smaller memory size on VM cluster (as I was unable to get it to read mapred-site.xml)\r\n````xml\r\n<configuration>\r\n  <property>\r\n      <name>oozie.launcher.mapreduce.map.memory.mb</name>\r\n      <value>256</value>\r\n  </property>\r\n\r\n  <property>\r\n      <name>oozie.launcher.mapreduce.map.java.opts</name>\r\n      <value>\"-Xmx205m\"</value>\r\n  </property>\r\n\r\n  <property>\r\n      <name>oozie.launcher.mapreduce.reduce.memory.mb</name>\r\n      <value>256</value>\r\n  </property>\r\n\r\n  <property>\r\n      <name>oozie.launcher.mapreduce.reduce.java.opts</name>\r\n      <value>\"-Xmx205m\"</value>\r\n  </property>\r\n\r\n  <property>\r\n      <name>oozie.launcher.yarn.app.mapreduce.am.resource.mb</name>\r\n      <value>256</value>\r\n  </property>\r\n</configuration>\r\n\r\n````\r\n\r\n- Two worker VMs with 8GB each with two vcores exposed in node manager.  Probbaly could have just used one and assigned 16 and 4-6 cores","created_at":"2015-12-15T22:35:30Z","updated_at":"2016-01-11T16:13:16Z","closed_at":"2016-01-11T16:13:15Z","merged_at":null,"merge_commit_sha":"f4a16205dc1c31adbebfb92d97e2f1b307b49594","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/bloomberg/chef-bach/pulls/351/commits","review_comments_url":"https://api.github.com/repos/bloomberg/chef-bach/pulls/351/comments","review_comment_url":"https://api.github.com/repos/bloomberg/chef-bach/pulls/comments{/number}","comments_url":"https://api.github.com/repos/bloomberg/chef-bach/issues/351/comments","statuses_url":"https://api.github.com/repos/bloomberg/chef-bach/statuses/df40d2bbb3112c0fd358c2cc0451be7e3a563a37","head":{"label":"pu239ppy:oozie_credentials_classes","ref":"oozie_credentials_classes","sha":"df40d2bbb3112c0fd358c2cc0451be7e3a563a37","user":{"login":"pu239ppy","id":4019043,"avatar_url":"https://avatars.githubusercontent.com/u/4019043?v=3","gravatar_id":"","url":"https://api.github.com/users/pu239ppy","html_url":"https://github.com/pu239ppy","followers_url":"https://api.github.com/users/pu239ppy/followers","following_url":"https://api.github.com/users/pu239ppy/following{/other_user}","gists_url":"https://api.github.com/users/pu239ppy/gists{/gist_id}","starred_url":"https://api.github.com/users/pu239ppy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pu239ppy/subscriptions","organizations_url":"https://api.github.com/users/pu239ppy/orgs","repos_url":"https://api.github.com/users/pu239ppy/repos","events_url":"https://api.github.com/users/pu239ppy/events{/privacy}","received_events_url":"https://api.github.com/users/pu239ppy/received_events","type":"User","site_admin":false},"repo":{"id":36989576,"name":"chef-bach","full_name":"pu239ppy/chef-bach","owner":{"login":"pu239ppy","id":4019043,"avatar_url":"https://avatars.githubusercontent.com/u/4019043?v=3","gravatar_id":"","url":"https://api.github.com/users/pu239ppy","html_url":"https://github.com/pu239ppy","followers_url":"https://api.github.com/users/pu239ppy/followers","following_url":"https://api.github.com/users/pu239ppy/following{/other_user}","gists_url":"https://api.github.com/users/pu239ppy/gists{/gist_id}","starred_url":"https://api.github.com/users/pu239ppy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pu239ppy/subscriptions","organizations_url":"https://api.github.com/users/pu239ppy/orgs","repos_url":"https://api.github.com/users/pu239ppy/repos","events_url":"https://api.github.com/users/pu239ppy/events{/privacy}","received_events_url":"https://api.github.com/users/pu239ppy/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/pu239ppy/chef-bach","description":"Chef recipes for Bloomberg's deployment of Hadoop and related components","fork":true,"url":"https://api.github.com/repos/pu239ppy/chef-bach","forks_url":"https://api.github.com/repos/pu239ppy/chef-bach/forks","keys_url":"https://api.github.com/repos/pu239ppy/chef-bach/keys{/key_id}","collaborators_url":"https://api.github.com/repos/pu239ppy/chef-bach/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/pu239ppy/chef-bach/teams","hooks_url":"https://api.github.com/repos/pu239ppy/chef-bach/hooks","issue_events_url":"https://api.github.com/repos/pu239ppy/chef-bach/issues/events{/number}","events_url":"https://api.github.com/repos/pu239ppy/chef-bach/events","assignees_url":"https://api.github.com/repos/pu239ppy/chef-bach/assignees{/user}","branches_url":"https://api.github.com/repos/pu239ppy/chef-bach/branches{/branch}","tags_url":"https://api.github.com/repos/pu239ppy/chef-bach/tags","blobs_url":"https://api.github.com/repos/pu239ppy/chef-bach/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/pu239ppy/chef-bach/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/pu239ppy/chef-bach/git/refs{/sha}","trees_url":"https://api.github.com/repos/pu239ppy/chef-bach/git/trees{/sha}","statuses_url":"https://api.github.com/repos/pu239ppy/chef-bach/statuses/{sha}","languages_url":"https://api.github.com/repos/pu239ppy/chef-bach/languages","stargazers_url":"https://api.github.com/repos/pu239ppy/chef-bach/stargazers","contributors_url":"https://api.github.com/repos/pu239ppy/chef-bach/contributors","subscribers_url":"https://api.github.com/repos/pu239ppy/chef-bach/subscribers","subscription_url":"https://api.github.com/repos/pu239ppy/chef-bach/subscription","commits_url":"https://api.github.com/repos/pu239ppy/chef-bach/commits{/sha}","git_commits_url":"https://api.github.com/repos/pu239ppy/chef-bach/git/commits{/sha}","comments_url":"https://api.github.com/repos/pu239ppy/chef-bach/comments{/number}","issue_comment_url":"https://api.github.com/repos/pu239ppy/chef-bach/issues/comments{/number}","contents_url":"https://api.github.com/repos/pu239ppy/chef-bach/contents/{+path}","compare_url":"https://api.github.com/repos/pu239ppy/chef-bach/compare/{base}...{head}","merges_url":"https://api.github.com/repos/pu239ppy/chef-bach/merges","archive_url":"https://api.github.com/repos/pu239ppy/chef-bach/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/pu239ppy/chef-bach/downloads","issues_url":"https://api.github.com/repos/pu239ppy/chef-bach/issues{/number}","pulls_url":"https://api.github.com/repos/pu239ppy/chef-bach/pulls{/number}","milestones_url":"https://api.github.com/repos/pu239ppy/chef-bach/milestones{/number}","notifications_url":"https://api.github.com/repos/pu239ppy/chef-bach/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/pu239ppy/chef-bach/labels{/name}","releases_url":"https://api.github.com/repos/pu239ppy/chef-bach/releases{/id}","created_at":"2015-06-06T17:52:02Z","updated_at":"2016-01-11T15:39:36Z","pushed_at":"2016-01-11T15:39:35Z","git_url":"git://github.com/pu239ppy/chef-bach.git","ssh_url":"git@github.com:pu239ppy/chef-bach.git","clone_url":"https://github.com/pu239ppy/chef-bach.git","svn_url":"https://github.com/pu239ppy/chef-bach","homepage":null,"size":2388,"stargazers_count":0,"watchers_count":0,"language":"HTML","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"bloomberg:master","ref":"master","sha":"3574736bcbdbc9f5ec364d64d87cc0cf3cf242b4","user":{"login":"bloomberg","id":1416818,"avatar_url":"https://avatars.githubusercontent.com/u/1416818?v=3","gravatar_id":"","url":"https://api.github.com/users/bloomberg","html_url":"https://github.com/bloomberg","followers_url":"https://api.github.com/users/bloomberg/followers","following_url":"https://api.github.com/users/bloomberg/following{/other_user}","gists_url":"https://api.github.com/users/bloomberg/gists{/gist_id}","starred_url":"https://api.github.com/users/bloomberg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bloomberg/subscriptions","organizations_url":"https://api.github.com/users/bloomberg/orgs","repos_url":"https://api.github.com/users/bloomberg/repos","events_url":"https://api.github.com/users/bloomberg/events{/privacy}","received_events_url":"https://api.github.com/users/bloomberg/received_events","type":"Organization","site_admin":false},"repo":{"id":24933148,"name":"chef-bach","full_name":"bloomberg/chef-bach","owner":{"login":"bloomberg","id":1416818,"avatar_url":"https://avatars.githubusercontent.com/u/1416818?v=3","gravatar_id":"","url":"https://api.github.com/users/bloomberg","html_url":"https://github.com/bloomberg","followers_url":"https://api.github.com/users/bloomberg/followers","following_url":"https://api.github.com/users/bloomberg/following{/other_user}","gists_url":"https://api.github.com/users/bloomberg/gists{/gist_id}","starred_url":"https://api.github.com/users/bloomberg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bloomberg/subscriptions","organizations_url":"https://api.github.com/users/bloomberg/orgs","repos_url":"https://api.github.com/users/bloomberg/repos","events_url":"https://api.github.com/users/bloomberg/events{/privacy}","received_events_url":"https://api.github.com/users/bloomberg/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/bloomberg/chef-bach","description":"Chef recipes for Bloomberg's deployment of Hadoop and related components","fork":false,"url":"https://api.github.com/repos/bloomberg/chef-bach","forks_url":"https://api.github.com/repos/bloomberg/chef-bach/forks","keys_url":"https://api.github.com/repos/bloomberg/chef-bach/keys{/key_id}","collaborators_url":"https://api.github.com/repos/bloomberg/chef-bach/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/bloomberg/chef-bach/teams","hooks_url":"https://api.github.com/repos/bloomberg/chef-bach/hooks","issue_events_url":"https://api.github.com/repos/bloomberg/chef-bach/issues/events{/number}","events_url":"https://api.github.com/repos/bloomberg/chef-bach/events","assignees_url":"https://api.github.com/repos/bloomberg/chef-bach/assignees{/user}","branches_url":"https://api.github.com/repos/bloomberg/chef-bach/branches{/branch}","tags_url":"https://api.github.com/repos/bloomberg/chef-bach/tags","blobs_url":"https://api.github.com/repos/bloomberg/chef-bach/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/bloomberg/chef-bach/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/bloomberg/chef-bach/git/refs{/sha}","trees_url":"https://api.github.com/repos/bloomberg/chef-bach/git/trees{/sha}","statuses_url":"https://api.github.com/repos/bloomberg/chef-bach/statuses/{sha}","languages_url":"https://api.github.com/repos/bloomberg/chef-bach/languages","stargazers_url":"https://api.github.com/repos/bloomberg/chef-bach/stargazers","contributors_url":"https://api.github.com/repos/bloomberg/chef-bach/contributors","subscribers_url":"https://api.github.com/repos/bloomberg/chef-bach/subscribers","subscription_url":"https://api.github.com/repos/bloomberg/chef-bach/subscription","commits_url":"https://api.github.com/repos/bloomberg/chef-bach/commits{/sha}","git_commits_url":"https://api.github.com/repos/bloomberg/chef-bach/git/commits{/sha}","comments_url":"https://api.github.com/repos/bloomberg/chef-bach/comments{/number}","issue_comment_url":"https://api.github.com/repos/bloomberg/chef-bach/issues/comments{/number}","contents_url":"https://api.github.com/repos/bloomberg/chef-bach/contents/{+path}","compare_url":"https://api.github.com/repos/bloomberg/chef-bach/compare/{base}...{head}","merges_url":"https://api.github.com/repos/bloomberg/chef-bach/merges","archive_url":"https://api.github.com/repos/bloomberg/chef-bach/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/bloomberg/chef-bach/downloads","issues_url":"https://api.github.com/repos/bloomberg/chef-bach/issues{/number}","pulls_url":"https://api.github.com/repos/bloomberg/chef-bach/pulls{/number}","milestones_url":"https://api.github.com/repos/bloomberg/chef-bach/milestones{/number}","notifications_url":"https://api.github.com/repos/bloomberg/chef-bach/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/bloomberg/chef-bach/labels{/name}","releases_url":"https://api.github.com/repos/bloomberg/chef-bach/releases{/id}","created_at":"2014-10-08T09:16:26Z","updated_at":"2015-12-23T14:24:39Z","pushed_at":"2016-01-11T15:39:36Z","git_url":"git://github.com/bloomberg/chef-bach.git","ssh_url":"git@github.com:bloomberg/chef-bach.git","clone_url":"https://github.com/bloomberg/chef-bach.git","svn_url":"https://github.com/bloomberg/chef-bach","homepage":null,"size":3051,"stargazers_count":22,"watchers_count":22,"language":"HTML","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":17,"mirror_url":null,"open_issues_count":88,"forks":17,"open_issues":88,"watchers":22,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/bloomberg/chef-bach/pulls/351"},"html":{"href":"https://github.com/bloomberg/chef-bach/pull/351"},"issue":{"href":"https://api.github.com/repos/bloomberg/chef-bach/issues/351"},"comments":{"href":"https://api.github.com/repos/bloomberg/chef-bach/issues/351/comments"},"review_comments":{"href":"https://api.github.com/repos/bloomberg/chef-bach/pulls/351/comments"},"review_comment":{"href":"https://api.github.com/repos/bloomberg/chef-bach/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/bloomberg/chef-bach/pulls/351/commits"},"statuses":{"href":"https://api.github.com/repos/bloomberg/chef-bach/statuses/df40d2bbb3112c0fd358c2cc0451be7e3a563a37"}},"merged":false,"mergeable":true,"mergeable_state":"clean","merged_by":null,"comments":1,"review_comments":0,"commits":1,"additions":19,"deletions":1,"changed_files":2}}