{"action":"created","issue":{"url":"https://api.github.com/repos/HubSpot/Singularity/issues/620","repository_url":"https://api.github.com/repos/HubSpot/Singularity","labels_url":"https://api.github.com/repos/HubSpot/Singularity/issues/620/labels{/name}","comments_url":"https://api.github.com/repos/HubSpot/Singularity/issues/620/comments","events_url":"https://api.github.com/repos/HubSpot/Singularity/issues/620/events","html_url":"https://github.com/HubSpot/Singularity/issues/620","id":95030638,"number":620,"title":"\"Path must start with / character\" leads to Singularity hanging forever","user":{"login":"stevenschlansker","id":129097,"avatar_url":"https://avatars.githubusercontent.com/u/129097?v=3","gravatar_id":"","url":"https://api.github.com/users/stevenschlansker","html_url":"https://github.com/stevenschlansker","followers_url":"https://api.github.com/users/stevenschlansker/followers","following_url":"https://api.github.com/users/stevenschlansker/following{/other_user}","gists_url":"https://api.github.com/users/stevenschlansker/gists{/gist_id}","starred_url":"https://api.github.com/users/stevenschlansker/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stevenschlansker/subscriptions","organizations_url":"https://api.github.com/users/stevenschlansker/orgs","repos_url":"https://api.github.com/users/stevenschlansker/repos","events_url":"https://api.github.com/users/stevenschlansker/events{/privacy}","received_events_url":"https://api.github.com/users/stevenschlansker/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":13,"created_at":"2015-07-14T20:19:48Z","updated_at":"2016-03-18T17:39:27Z","closed_at":"2016-03-18T17:39:27Z","body":"We had a minor production incident caused by Singularity's connection to ZK entering a zombie state.\r\n\r\n```\r\nINFO  [2015-07-14 07:59:13,127] org.eclipse.jetty.server.handler.ContextHandler: Started i.d.j.MutableServletContextHandler@54f4a7f0{/,null,AVAILABLE}\r\nINFO  [2015-07-14 07:59:13,129] io.dropwizard.setup.AdminEnvironment: tasks = \r\n\r\n    POST    /tasks/gc (io.dropwizard.servlets.tasks.GarbageCollectionTask)\r\n\r\nINFO  [2015-07-14 07:59:13,133] org.eclipse.jetty.server.handler.ContextHandler: Started i.d.j.MutableServletContextHandler@408e96d9{/admin,null,AVAILABLE}\r\nINFO  [2015-07-14 07:59:13,152] org.eclipse.jetty.server.ServerConnector: Started SingularityService@12f8b1d8{HTTP/1.1}{0.0.0.0:7099}\r\nINFO  [2015-07-14 07:59:32,309] org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x24e8b9313780002, likely server has closed socket, closing socket connection and attempting reconnect\r\nINFO  [2015-07-14 07:59:32,413] org.apache.curator.framework.state.ConnectionStateManager: State change: SUSPENDED\r\nINFO  [2015-07-14 07:59:32,481] org.apache.zookeeper.ClientCnxn: Opening socket connection to server mesos3-prod-sc.otsql.opentable.com/10.20.16.252:2181. Will not attempt to authenticate using SASL (unknown error)\r\nINFO  [2015-07-14 07:59:32,482] org.apache.zookeeper.ClientCnxn: Socket connection established to mesos3-prod-sc.otsql.opentable.com/10.20.16.252:2181, initiating session\r\nINFO  [2015-07-14 07:59:32,785] org.apache.zookeeper.ClientCnxn: Session establishment complete on server mesos3-prod-sc.otsql.opentable.com/10.20.16.252:2181, sessionid = 0x24e8b9313780002, negotiated timeout = 40000\r\nINFO  [2015-07-14 07:59:32,786] org.apache.curator.framework.state.ConnectionStateManager: State change: RECONNECTED\r\nERROR [2015-07-14 07:59:33,428] org.apache.curator.framework.imps.CuratorFrameworkImpl: Background exception was not retry-able or retry gave up\r\n! java.lang.IllegalArgumentException: Path must start with / character\r\n! at org.apache.curator.utils.PathUtils.validatePath(PathUtils.java:53) ~[singularity-20150708-2.jar:0.4.3-SNAPSHOT]\r\n! at org.apache.curator.utils.ZKPaths.getNodeFromPath(ZKPaths.java:56) ~[singularity-20150708-2.jar:0.4.3-SNAPSHOT]\r\n! at org.apache.curator.framework.recipes.leader.LeaderLatch.checkLeadership(LeaderLatch.java:478) ~[singularity-20150708-2.jar:0.4.3-SNAPSHOT]\r\n! at org.apache.curator.framework.recipes.leader.LeaderLatch.access$500(LeaderLatch.java:60) ~[singularity-20150708-2.jar:0.4.3-SNAPSHOT]\r\n! at org.apache.curator.framework.recipes.leader.LeaderLatch$6.processResult(LeaderLatch.java:535) ~[singularity-20150708-2.jar:0.4.3-SNAPSHOT]\r\n! at org.apache.curator.framework.imps.CuratorFrameworkImpl.sendToBackgroundCallback(CuratorFrameworkImpl.java:686) [singularity-20150708-2.jar:0.4.3-SNAPSHOT]\r\n! at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:485) [singularity-20150708-2.jar:0.4.3-SNAPSHOT]\r\n! at org.apache.curator.framework.imps.GetChildrenBuilderImpl$2.processResult(GetChildrenBuilderImpl.java:166) [singularity-20150708-2.jar:0.4.3-SNAPSHOT]\r\n! at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:587) [singularity-20150708-2.jar:0.4.3-SNAPSHOT]\r\n! at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495) [singularity-20150708-2.jar:0.4.3-SNAPSHOT]\r\nINFO  [2015-07-14 10:30:17,357] org.eclipse.jetty.server.ServerConnector: Stopped SingularityService@12f8b1d8{HTTP/1.1}{0.0.0.0:7099}\r\nINFO  [2015-07-14 10:30:17,358] org.eclipse.jetty.server.handler.ContextHandler: Stopped i.d.j.MutableServletContextHandler@408e96d9{/admin,null,UNAVAILABLE}\r\nINFO  [2015-07-14 10:30:17,359] org.eclipse.jetty.server.handler.ContextHandler: Stopped i.d.j.MutableServletContextHandler@54f4a7f0{/,null,UNAVAILABLE}\r\nINFO  [2015-07-14 10:30:17,378] org.apache.zookeeper.ClientCnxn: EventThread shut down\r\nINFO  [2015-07-14 10:30:17,378] org.apache.zookeeper.ZooKeeper: Session: 0x24e8b9313780002 closed\r\n```\r\n\r\nThe three hour gap is until an administrator was paged and came in to restart manually.\r\n\r\nTwo issues here:\r\n* What is this IAE about the path being invalid?\r\n* Why didn't a ZK exception cause Singularity to abort?\r\n\r\nThis is a slightly patched (in a way that shouldn't possibly be relevant, just the PRs I have that are not merged yet) 0.4.3-SNAPSHOT as of a week ago."},"comment":{"url":"https://api.github.com/repos/HubSpot/Singularity/issues/comments/198465886","html_url":"https://github.com/HubSpot/Singularity/issues/620#issuecomment-198465886","issue_url":"https://api.github.com/repos/HubSpot/Singularity/issues/620","id":198465886,"user":{"login":"stevenschlansker","id":129097,"avatar_url":"https://avatars.githubusercontent.com/u/129097?v=3","gravatar_id":"","url":"https://api.github.com/users/stevenschlansker","html_url":"https://github.com/stevenschlansker","followers_url":"https://api.github.com/users/stevenschlansker/followers","following_url":"https://api.github.com/users/stevenschlansker/following{/other_user}","gists_url":"https://api.github.com/users/stevenschlansker/gists{/gist_id}","starred_url":"https://api.github.com/users/stevenschlansker/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stevenschlansker/subscriptions","organizations_url":"https://api.github.com/users/stevenschlansker/orgs","repos_url":"https://api.github.com/users/stevenschlansker/repos","events_url":"https://api.github.com/users/stevenschlansker/events{/privacy}","received_events_url":"https://api.github.com/users/stevenschlansker/received_events","type":"User","site_admin":false},"created_at":"2016-03-18T17:39:27Z","updated_at":"2016-03-18T17:39:27Z","body":"We haven't seen this in a very long time."}}