{"action":"created","issue":{"url":"https://api.github.com/repos/linkedin/gobblin/issues/715","repository_url":"https://api.github.com/repos/linkedin/gobblin","labels_url":"https://api.github.com/repos/linkedin/gobblin/issues/715/labels{/name}","comments_url":"https://api.github.com/repos/linkedin/gobblin/issues/715/comments","events_url":"https://api.github.com/repos/linkedin/gobblin/issues/715/events","html_url":"https://github.com/linkedin/gobblin/issues/715","id":134193394,"number":715,"title":"Error when restarting GobblinYarn","user":{"login":"kadaan","id":1904898,"avatar_url":"https://avatars.githubusercontent.com/u/1904898?v=3","gravatar_id":"","url":"https://api.github.com/users/kadaan","html_url":"https://github.com/kadaan","followers_url":"https://api.github.com/users/kadaan/followers","following_url":"https://api.github.com/users/kadaan/following{/other_user}","gists_url":"https://api.github.com/users/kadaan/gists{/gist_id}","starred_url":"https://api.github.com/users/kadaan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kadaan/subscriptions","organizations_url":"https://api.github.com/users/kadaan/orgs","repos_url":"https://api.github.com/users/kadaan/repos","events_url":"https://api.github.com/users/kadaan/events{/privacy}","received_events_url":"https://api.github.com/users/kadaan/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-02-17T06:37:52Z","updated_at":"2016-02-17T06:49:18Z","closed_at":null,"body":"The following error occurs after trying to restart the GobblinYarn driver:\r\n```\r\n2016-02-17 06:27:15 UTC WARN  [main] org.apache.helix.manager.zk.ZKHelixAdmin  495 - Root directory exists.Cleaning the root directory:/GobblinYarn\r\nException in thread \"main\" org.I0Itec.zkclient.exception.ZkException: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Dir\r\nectory not empty for /GobblinYarn/CONTROLLER\r\n        at org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:68)\r\n        at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:685)\r\n        at org.apache.helix.manager.zk.ZkClient.delete(ZkClient.java:348)\r\n        at org.I0Itec.zkclient.ZkClient.deleteRecursive(ZkClient.java:516)\r\n        at org.I0Itec.zkclient.ZkClient.deleteRecursive(ZkClient.java:511)\r\n        at org.apache.helix.manager.zk.ZKHelixAdmin.addCluster(ZKHelixAdmin.java:496)\r\n        at org.apache.helix.tools.ClusterSetup.addCluster(ClusterSetup.java:154)\r\n        at gobblin.yarn.YarnHelixUtils.createGobblinYarnHelixCluster(YarnHelixUtils.java:67)\r\n        at gobblin.yarn.GobblinYarnAppLauncher.launch(GobblinYarnAppLauncher.java:243)\r\n        at gobblin.yarn.GobblinYarnAppLauncher.main(GobblinYarnAppLauncher.java:784)\r\nCaused by: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /GobblinYarn/CONTROLLER\r\n        at org.apache.zookeeper.KeeperException.create(KeeperException.java:125)\r\n        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)\r\n        at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)\r\n        at org.I0Itec.zkclient.ZkConnection.delete(ZkConnection.java:91)\r\n        at org.apache.helix.manager.zk.ZkClient$8.call(ZkClient.java:352)\r\n        at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)\r\n        ... 8 more\r\n```\r\n\r\nThese are the logs of the driver shutting down prior to restart:\r\n```\r\n2016-02-17 06:25:01 UTC INFO  [Thread-4] gobblin.yarn.GobblinYarnAppLauncher  301 - Stopping the GobblinYarnAppLauncher\r\n2016-02-17 06:25:01 UTC INFO  [Thread-4] org.apache.helix.messaging.DefaultMessagingService  84 - Send 1 messages with criteria instanceName=%resourceName=%partitionName=%partitionState=%\r\n2016-02-17 06:25:02 UTC INFO  [LogCopier STOPPING] gobblin.util.ExecutorsUtils  125 - Attempting to shutdown ExecutorService: java.util.concurrent.ScheduledThreadPoolExecutor@73240b61[Shutting down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 1862]\r\n2016-02-17 06:25:02 UTC INFO  [LogCopier STOPPING] gobblin.util.ExecutorsUtils  144 - Successfully shutdown ExecutorService: java.util.concurrent.ScheduledThreadPoolExecutor@73240b61[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1862]\r\n2016-02-17 06:25:02 UTC INFO  [JobExecutionInfoServer STOPPING] gobblin.rest.JobExecutionInfoServer  94 - Stopping the job execution information server\r\nShutting down\r\n2016-02-17 06:25:02 UTC INFO  [AdminWebServer STOPPING] org.eclipse.jetty.server.AbstractConnector  306 - Stopped ServerConnector@35e0c350{HTTP/1.1}{localhost:8280}\r\n2016-02-17 06:25:02 UTC INFO  [Thread-4] gobblin.util.ExecutorsUtils  125 - Attempting to shutdown ExecutorService: java.util.concurrent.Executors$DelegatedScheduledExecutorService@185aaf1f\r\n2016-02-17 06:25:02 UTC INFO  [Thread-4] gobblin.util.ExecutorsUtils  144 - Successfully shutdown ExecutorService: java.util.concurrent.Executors$DelegatedScheduledExecutorService@185aaf1f\r\n2016-02-17 06:25:02 UTC INFO  [Thread-4] org.apache.helix.manager.zk.ZKHelixManager  546 - disconnect ip-169-0-0-1(SPECTATOR) from GobblinYarn\r\n2016-02-17 06:25:02 UTC INFO  [Thread-4] org.apache.helix.messaging.handling.HelixTaskExecutor  679 - Shutting down HelixTaskExecutor\r\n2016-02-17 06:25:02 UTC INFO  [Thread-4] org.apache.helix.messaging.handling.HelixTaskExecutor  443 - Reset HelixTaskExecutor\r\n2016-02-17 06:25:02 UTC INFO  [Thread-4] org.apache.helix.messaging.handling.HelixTaskExecutor  453 - Reset exectuor for msgType: TASK_REPLY, pool: java.util.concurrent.ThreadPoolExecutor@3f197a46[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]\r\n2016-02-17 06:25:02 UTC INFO  [Thread-4] org.apache.helix.messaging.handling.HelixTaskExecutor  397 - Shutting down pool: java.util.concurrent.ThreadPoolExecutor@3f197a46[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]\r\n2016-02-17 06:25:02 UTC INFO  [Thread-4] org.apache.helix.messaging.handling.HelixTaskExecutor  684 - Shutdown HelixTaskExecutor finished\r\n2016-02-17 06:25:02 UTC INFO  [Thread-4] org.apache.helix.manager.zk.ZkClient  130 - Closing zkclient: State:CONNECTED Timeout:30000 sessionid:0xd452eb397b640065 local:/169.0.0.1:51319 remoteserver:ip-138-0-0-1.ec2.internal/138.0.0.2:2181 lastZxid:60129782948 xid:17 sent:140 recv:140 queuedpkts:0 pendingresp:0 queuedevents:0\r\n2016-02-17 06:25:02 UTC INFO  [ZkClient-EventThread-17-zk.server:2181] org.I0Itec.zkclient.ZkEventThread  82 - Terminate ZkClient event thread.\r\n2016-02-17 06:25:02 UTC INFO  [main-EventThread] org.apache.zookeeper.ClientCnxn$EventThread  512 - EventThread shut down\r\n2016-02-17 06:25:02 UTC INFO  [Thread-4] org.apache.zookeeper.ZooKeeper  684 - Session: 0xd452eb397b640065 closed\r\n2016-02-17 06:25:02 UTC INFO  [Thread-4] org.apache.helix.manager.zk.ZkClient  157 - Closed zkclient\r\n2016-02-17 06:25:02 UTC INFO  [Thread-4] org.apache.helix.manager.zk.ZKHelixManager  570 - Cluster manager: ip-169-0-0-1 disconnected\r\n2016-02-17 06:25:02 UTC INFO  [Thread-4] gobblin.yarn.GobblinYarnAppLauncher  722 - Deleting application working directory hdfs://ec2-145-0-0-1.compute-1.amazonaws.com:9000/user/yarn/GobblinYarn/application_1455654714320_0004\r\n```"},"comment":{"url":"https://api.github.com/repos/linkedin/gobblin/issues/comments/185056071","html_url":"https://github.com/linkedin/gobblin/issues/715#issuecomment-185056071","issue_url":"https://api.github.com/repos/linkedin/gobblin/issues/715","id":185056071,"user":{"login":"liyinan926","id":3709137,"avatar_url":"https://avatars.githubusercontent.com/u/3709137?v=3","gravatar_id":"","url":"https://api.github.com/users/liyinan926","html_url":"https://github.com/liyinan926","followers_url":"https://api.github.com/users/liyinan926/followers","following_url":"https://api.github.com/users/liyinan926/following{/other_user}","gists_url":"https://api.github.com/users/liyinan926/gists{/gist_id}","starred_url":"https://api.github.com/users/liyinan926/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/liyinan926/subscriptions","organizations_url":"https://api.github.com/users/liyinan926/orgs","repos_url":"https://api.github.com/users/liyinan926/repos","events_url":"https://api.github.com/users/liyinan926/events{/privacy}","received_events_url":"https://api.github.com/users/liyinan926/received_events","type":"User","site_admin":false},"created_at":"2016-02-17T06:49:18Z","updated_at":"2016-02-17T06:49:18Z","body":"This happens sometimes. For some reason, the `ZKHelixAdmin` silently fails to fully cleanup the ZK structure corresponding to the Helix cluster instance even if it is configured to do the cleanup before everything else starts up. This causes the Yarn application to fail to start. A workaround is to manually delete the root ZK node `/GobblinYarn`."}}