{"action":"created","issue":{"url":"https://api.github.com/repos/dCache/dcache/issues/3790","repository_url":"https://api.github.com/repos/dCache/dcache","labels_url":"https://api.github.com/repos/dCache/dcache/issues/3790/labels{/name}","comments_url":"https://api.github.com/repos/dCache/dcache/issues/3790/comments","events_url":"https://api.github.com/repos/dCache/dcache/issues/3790/events","html_url":"https://github.com/dCache/dcache/issues/3790","id":293090227,"number":3790,"title":"[prometheus] ZK connection problem leads to domain disappearing","user":{"login":"paulmillar","id":533081,"avatar_url":"https://avatars0.githubusercontent.com/u/533081?v=4","gravatar_id":"","url":"https://api.github.com/users/paulmillar","html_url":"https://github.com/paulmillar","followers_url":"https://api.github.com/users/paulmillar/followers","following_url":"https://api.github.com/users/paulmillar/following{/other_user}","gists_url":"https://api.github.com/users/paulmillar/gists{/gist_id}","starred_url":"https://api.github.com/users/paulmillar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/paulmillar/subscriptions","organizations_url":"https://api.github.com/users/paulmillar/orgs","repos_url":"https://api.github.com/users/paulmillar/repos","events_url":"https://api.github.com/users/paulmillar/events{/privacy}","received_events_url":"https://api.github.com/users/paulmillar/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-01-31T09:39:36Z","updated_at":"2018-02-08T20:53:24Z","closed_at":null,"author_association":"OWNER","body":"Prometheus [Build 1421](https://ci.dcache.org/view/dCache%20master/job/Deploy%20on%20prometheus/1421/) failed due to the inability to upload data.  The failed uploads where logged as:\r\n```\r\n31 Jan 2018 06:04:05 (webdav-secure-web) [door:webdav-secure-web@dCacheDomain:AAVkC2iLm9g] Internal server error: org.dcache.webdav.WebDavException: Unable to reserve space: no linkgroups configured.\r\n31 Jan 2018 06:04:05 (webdav-secure-web) [door:webdav-secure-web@dCacheDomain:AAVkC2iSz5g] Unable to reserve space: no linkgroups configured.\r\n```\r\n\r\nThis, in turn, was cause by SpaceManager believing there are no pools contributing to the linkgroup:\r\n```\r\n[prometheus] (local) admin > \\s SpaceManager ls link groups\r\nFLAGS   CNT RESVD AVAIL FREE          UPDATED NAME\r\n-orc:no 0   0 +   0 =   0 2018-01-31 10:33:39 default-linkgroup\r\n\r\n[prometheus] (local) admin > \r\n```\r\n\r\nOn prometheus, there are two pools (`pool1` and `pool2`) both in the `pools` domain.  PoolManager does not know about these pools:\r\n```\r\n[prometheus] (local) admin > \\sp psu ls pool\r\n[prometheus] (local) admin > \r\n```\r\n\r\nThe `pools` domain is running:\r\n```\r\n[root@prometheus ~]# dcache status\r\nDOMAIN       STATUS                PID   USER   LOG                              \r\ndCacheDomain running (for 4 hours) 32324 dcache /var/log/dcache/dCacheDomain.log \r\nnfsServer    running (for 4 hours) 32384 dcache /var/log/dcache/nfsServer.log    \r\npools        running (for 4 hours) 32439 dcache /var/log/dcache/pools.log        \r\n[root@prometheus ~]# \r\n```\r\nHowever, the two pools are no available through the admin interface, and topo does not show `pools` domain.\r\n```\r\n[root@prometheus ~]# ssh -p 22224 admin@localhost\r\ndCache (4.1.0-SNAPSHOT)\r\nType \"\\?\" for help.\r\n\r\n[prometheus] (local) admin > \\c pool1\r\n(1) Cell does not exist.\r\n[prometheus] (local) admin > \\c pool2\r\n(1) Cell does not exist.\r\n[prometheus] (local) admin > \\c System@pools\r\n(1) Cell does not exist.\r\n[prometheus] (local) admin > \\s topo ls\r\ndCacheDomain\r\n\r\n[prometheus] (local) admin > \r\n```\r\nThe topo should show the `dCacheDomain`, `pools` and `nfsServer` domains.\r\n\r\nThe dCacheDomain log file shows (edited for brevity):\r\n```\r\n2018-01-31 06:01:05 Launching /bin/java -[...]BootLoader start dCacheDomain\r\n[...]\r\n31 Jan 2018 06:01:23 (System) [] ZooKeeper connection to localhost/127.0.0.1:2181 failed (Connection refused), attempting reconnect.\r\n31 Jan 2018 06:01:23 (System) [] ZooKeeper connection to localhost/0:0:0:0:0:0:0:1:2181 failed (Connection refused), attempting reconnect.\r\n31 Jan 2018 06:01:24 (System) [] ZooKeeper connection to localhost/127.0.0.1:2181 failed (Connection refused), attempting reconnect.\r\n31 Jan 2018 06:01:24 (System) [] ZooKeeper connection to localhost/0:0:0:0:0:0:0:1:2181 failed (Connection refused), attempting reconnect.\r\n[... similar connection errors ...]\r\n31 Jan 2018 06:02:29 (System) [] ZooKeeper connection to localhost/127.0.0.1:2181 failed (Connection refused), attempting reconnect.\r\n31 Jan 2018 06:02:29 (System) [] ZooKeeper connection to localhost/0:0:0:0:0:0:0:1:2181 failed (Connection refused), attempting reconnect.\r\n31 Jan 2018 06:02:30 (lm) [] ZooKeeper connection to localhost/0:0:0:0:0:0:0:1:2181 failed (Connection refused), attempting reconnect.\r\n31 Jan 2018 06:02:30 (lm) [] ZooKeeper connection to localhost/127.0.0.1:2181 failed (Connection refused), attempting reconnect.\r\n31 Jan 2018 06:02:30 (lm) [] Failed to connect to zookeeper, using mode none until connection reestablished\r\n31 Jan 2018 06:02:30 (lm) [] \r\norg.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /dcache\r\n        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) ~[zookeeper-3.4.8.jar:3.4.8--1]\r\n        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[zookeeper-3.4.8.jar:3.4.8--1]\r\n        at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102) ~[zookeeper-3.4.8.jar:3.4.8--1]\r\n        at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1130) ~[zookeeper-3.4.8.jar:3.4.8--1]\r\n        at org.apache.curator.utils.ZKPaths.mkdirs(ZKPaths.java:274) ~[curator-client-2.11.1.jar:na]\r\n        at org.apache.curator.framework.imps.ExistsBuilderImpl$2.call(ExistsBuilderImpl.java:199) ~[curator-framework-2.11.1.jar:na]\r\n        at org.apache.curator.framework.imps.ExistsBuilderImpl$2.call(ExistsBuilderImpl.java:193) ~[curator-framework-2.11.1.jar:na]\r\n        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109) ~[curator-client-2.11.1.jar:na]\r\n        at org.apache.curator.framework.imps.ExistsBuilderImpl.pathInForeground(ExistsBuilderImpl.java:190) ~[curator-framework-2.11.1.jar:na]\r\n        at org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:175) ~[curator-framework-2.11.1.jar:na]\r\n        at org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:32) ~[curator-framework-2.11.1.jar:na]\r\n        at org.apache.curator.framework.imps.CuratorFrameworkImpl.createContainers(CuratorFrameworkImpl.java:194) ~[curator-framework-2.11.1.jar:na]\r\n        at dmg.cells.zookeeper.CellCuratorFramework.createContainers(CellCuratorFramework.java:265) ~[cells-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]\r\n        at org.apache.curator.framework.EnsureContainers.internalEnsure(EnsureContainers.java:61) ~[curator-framework-2.11.1.jar:na]\r\n        at org.apache.curator.framework.EnsureContainers.ensure(EnsureContainers.java:53) ~[curator-framework-2.11.1.jar:na]\r\n        at org.apache.curator.framework.recipes.cache.PathChildrenCache.ensurePath(PathChildrenCache.java:576) ~[curator-recipes-2.11.1.jar:na]\r\n        at org.apache.curator.framework.recipes.cache.PathChildrenCache.refresh(PathChildrenCache.java:490) ~[curator-recipes-2.11.1.jar:na]\r\n        at org.apache.curator.framework.recipes.cache.RefreshOperation.invoke(RefreshOperation.java:35) ~[curator-recipes-2.11.1.jar:na]\r\n        at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:773) ~[curator-recipes-2.11.1.jar:na]\r\n        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_161]\r\n        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_161]\r\n        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_161]\r\n        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_161]\r\n        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_161]\r\n        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_161]\r\n        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]\r\n31 Jan 2018 06:02:33 (System) [] The version of ZooKeeper being used doesn't support Container nodes. CreateMode.PERSISTENT will be used instead.\r\n```\r\n\r\nThese initial `ZooKeeper connection to localhost/127.0.0.1:2181 failed` messages are annoying but (I believe) not the cause of the problem.  They are present when dCache is working correctly and are caused by the `System` cell attempts to connect to ZK before the embedded ZK is started.  This connection blocks for ~1 minute before returning, with the ZK client retrying in the background, as mentioned in the message\r\n\r\n```\r\n31 Jan 2018 06:02:30 (lm) [] Failed to connect to zookeeper, using mode none until connection reestablished\r\n```\r\nLater on, the message `version of ZooKeeper being used` indicates that the client was eventually able to connect to ZK:\r\n\r\n```\r\n31 Jan 2018 06:02:33 (System) [] The version of ZooKeeper being used doesn't support Container nodes. CreateMode.PERSISTENT will be used instead.\r\n```\r\n\r\nThe stack-trace is interesting and new.  This indicates that the PathChildrenCache is no longer maintained; for example, if a domain suffers this problem and a core domain restarts (or starts up) then the satellite would never learn of the core domain appearing, so would never initiate a fresh connection to that domain.\r\n\r\nBoth the `nfsServer` and `pools` domains contain a similar stack-trace:\r\n\r\nFor `pools` domain:\r\n```\r\n2018-01-31 06:01:07 Launching /bin/java -server [...] BootLoader start pools\r\n[...]\r\n31 Jan 2018 06:01:24 (System) [] ZooKeeper connection to localhost/0:0:0:0:0:0:0:1:2181 failed (Connection refused), attempting reconnect.\r\n31 Jan 2018 06:01:24 (System) [] ZooKeeper connection to localhost/127.0.0.1:2181 failed (Connection refused), attempting reconnect.\r\n[...]\r\n31 Jan 2018 06:02:29 (System) [] ZooKeeper connection to localhost/127.0.0.1:2181 failed (Connection refused), attempting reconnect.\r\n31 Jan 2018 06:02:31 (System) [] ZooKeeper connection to localhost/0:0:0:0:0:0:0:1:2181 failed (Connection refused), attempting reconnect.\r\n31 Jan 2018 06:02:31 (lm) [] ZooKeeper connection to localhost/127.0.0.1:2181 failed (Connection refused), attempting reconnect.\r\n31 Jan 2018 06:02:31 (lm) [] ZooKeeper connection to localhost/0:0:0:0:0:0:0:1:2181 failed (Connection refused), attempting reconnect.\r\n31 Jan 2018 06:02:31 (lm) [] \r\norg.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /dcache\r\n        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) ~[zookeeper-3.4.8.jar:3.4.8--1]\r\n        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[zookeeper-3.4.8.jar:3.4.8--1]\r\n        at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102) ~[zookeeper-3.4.8.jar:3.4.8--1]\r\n        at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1130) ~[zookeeper-3.4.8.jar:3.4.8--1]\r\n        at org.apache.curator.utils.ZKPaths.mkdirs(ZKPaths.java:274) ~[curator-client-2.11.1.jar:na]\r\n        at org.apache.curator.framework.imps.ExistsBuilderImpl$2.call(ExistsBuilderImpl.java:199) ~[curator-framework-2.11.1.jar:na]\r\n        at org.apache.curator.framework.imps.ExistsBuilderImpl$2.call(ExistsBuilderImpl.java:193) ~[curator-framework-2.11.1.jar:na]\r\n        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109) ~[curator-client-2.11.1.jar:na]\r\n        at org.apache.curator.framework.imps.ExistsBuilderImpl.pathInForeground(ExistsBuilderImpl.java:190) ~[curator-framework-2.11.1.jar:na]\r\n        at org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:175) ~[curator-framework-2.11.1.jar:na]\r\n        at org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:32) ~[curator-framework-2.11.1.jar:na]\r\n        at org.apache.curator.framework.imps.CuratorFrameworkImpl.createContainers(CuratorFrameworkImpl.java:194) ~[curator-framework-2.11.1.jar:na]\r\n        at dmg.cells.zookeeper.CellCuratorFramework.createContainers(CellCuratorFramework.java:265) ~[cells-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]\r\n        at org.apache.curator.framework.EnsureContainers.internalEnsure(EnsureContainers.java:61) ~[curator-framework-2.11.1.jar:na]\r\n        at org.apache.curator.framework.EnsureContainers.ensure(EnsureContainers.java:53) ~[curator-framework-2.11.1.jar:na]\r\n        at org.apache.curator.framework.recipes.cache.PathChildrenCache.ensurePath(PathChildrenCache.java:576) ~[curator-recipes-2.11.1.jar:na]\r\n        at org.apache.curator.framework.recipes.cache.PathChildrenCache.refresh(PathChildrenCache.java:490) ~[curator-recipes-2.11.1.jar:na]\r\n        at org.apache.curator.framework.recipes.cache.RefreshOperation.invoke(RefreshOperation.java:35) ~[curator-recipes-2.11.1.jar:na]\r\n        at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:773) ~[curator-recipes-2.11.1.jar:na]\r\n        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_161]\r\n        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_161]\r\n        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_161]\r\n        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_161]\r\n        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_161]\r\n        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_161]\r\n        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]\r\n```\r\n\r\nand for `nfsServer` domain:\r\n```\r\n2018-01-28 06:01:02 Launching /bin/java -server [...]BootLoader start nfsServer\r\n[...]\r\n31 Jan 2018 06:01:23 (System) [] ZooKeeper connection to localhost/0:0:0:0:0:0:0:1:2181 failed (Connection refused), attempting reconnect.\r\n31 Jan 2018 06:01:24 (System) [] ZooKeeper connection to localhost/127.0.0.1:2181 failed (Connection refused), attempting reconnect.\r\n[...]\r\n31 Jan 2018 06:02:29 (System) [] ZooKeeper connection to localhost/0:0:0:0:0:0:0:1:2181 failed (Connection refused), attempting reconnect.\r\n31 Jan 2018 06:02:29 (System) [] ZooKeeper connection to localhost/127.0.0.1:2181 failed (Connection refused), attempting reconnect.\r\n31 Jan 2018 06:02:31 (lm) [] ZooKeeper connection to localhost/0:0:0:0:0:0:0:1:2181 failed (Connection refused), attempting reconnect.\r\n31 Jan 2018 06:02:31 (lm) [] ZooKeeper connection to localhost/127.0.0.1:2181 failed (Connection refused), attempting reconnect.\r\n31 Jan 2018 06:02:31 (lm) [] \r\norg.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /dcache\r\n        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) ~[zookeeper-3.4.8.jar:3.4.8--1]\r\n        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[zookeeper-3.4.8.jar:3.4.8--1]\r\n        at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102) ~[zookeeper-3.4.8.jar:3.4.8--1]\r\n        at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1130) ~[zookeeper-3.4.8.jar:3.4.8--1]\r\n        at org.apache.curator.utils.ZKPaths.mkdirs(ZKPaths.java:274) ~[curator-client-2.11.1.jar:na]\r\n        at org.apache.curator.framework.imps.ExistsBuilderImpl$2.call(ExistsBuilderImpl.java:199) ~[curator-framework-2.11.1.jar:na]\r\n        at org.apache.curator.framework.imps.ExistsBuilderImpl$2.call(ExistsBuilderImpl.java:193) ~[curator-framework-2.11.1.jar:na]\r\n        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109) ~[curator-client-2.11.1.jar:na]\r\n        at org.apache.curator.framework.imps.ExistsBuilderImpl.pathInForeground(ExistsBuilderImpl.java:190) ~[curator-framework-2.11.1.jar:na]\r\n        at org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:175) ~[curator-framework-2.11.1.jar:na]\r\n        at org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:32) ~[curator-framework-2.11.1.jar:na]\r\n        at org.apache.curator.framework.imps.CuratorFrameworkImpl.createContainers(CuratorFrameworkImpl.java:194) ~[curator-framework-2.11.1.jar:na]\r\n        at dmg.cells.zookeeper.CellCuratorFramework.createContainers(CellCuratorFramework.java:265) ~[cells-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]\r\n        at org.apache.curator.framework.EnsureContainers.internalEnsure(EnsureContainers.java:61) ~[curator-framework-2.11.1.jar:na]\r\n        at org.apache.curator.framework.EnsureContainers.ensure(EnsureContainers.java:53) ~[curator-framework-2.11.1.jar:na]\r\n        at org.apache.curator.framework.recipes.cache.PathChildrenCache.ensurePath(PathChildrenCache.java:576) ~[curator-recipes-2.11.1.jar:na]\r\n        at org.apache.curator.framework.recipes.cache.PathChildrenCache.refresh(PathChildrenCache.java:490) ~[curator-recipes-2.11.1.jar:na]\r\n        at org.apache.curator.framework.recipes.cache.RefreshOperation.invoke(RefreshOperation.java:35) ~[curator-recipes-2.11.1.jar:na]\r\n        at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:773) ~[curator-recipes-2.11.1.jar:na]\r\n        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_161]\r\n        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_161]\r\n        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_161]\r\n        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_161]\r\n        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_161]\r\n        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_161]\r\n        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]\r\n```\r\n```"},"comment":{"url":"https://api.github.com/repos/dCache/dcache/issues/comments/364244910","html_url":"https://github.com/dCache/dcache/issues/3790#issuecomment-364244910","issue_url":"https://api.github.com/repos/dCache/dcache/issues/3790","id":364244910,"user":{"login":"kofemann","id":575965,"avatar_url":"https://avatars0.githubusercontent.com/u/575965?v=4","gravatar_id":"","url":"https://api.github.com/users/kofemann","html_url":"https://github.com/kofemann","followers_url":"https://api.github.com/users/kofemann/followers","following_url":"https://api.github.com/users/kofemann/following{/other_user}","gists_url":"https://api.github.com/users/kofemann/gists{/gist_id}","starred_url":"https://api.github.com/users/kofemann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kofemann/subscriptions","organizations_url":"https://api.github.com/users/kofemann/orgs","repos_url":"https://api.github.com/users/kofemann/repos","events_url":"https://api.github.com/users/kofemann/events{/privacy}","received_events_url":"https://api.github.com/users/kofemann/received_events","type":"User","site_admin":false},"created_at":"2018-02-08T20:53:23Z","updated_at":"2018-02-08T20:53:23Z","author_association":"OWNER","body":"It will be useful next time to make a thread dump to see is it the missing\nextra executor trigger this issue.\n\n-kofemann\n/** caffeinated mutations of the core personality */\n\nOn Wed, Jan 31, 2018 at 10:39 AM, Paul Millar <notifications@github.com>\nwrote:\n\n> Prometheus Build 1421\n> <https://ci.dcache.org/view/dCache%20master/job/Deploy%20on%20prometheus/1421/>\n> failed due to the inability to upload data. The failed uploads where logged\n> as:\n>\n> 31 Jan 2018 06:04:05 (webdav-secure-web) [door:webdav-secure-web@dCacheDomain:AAVkC2iLm9g] Internal server error: org.dcache.webdav.WebDavException: Unable to reserve space: no linkgroups configured.\n> 31 Jan 2018 06:04:05 (webdav-secure-web) [door:webdav-secure-web@dCacheDomain:AAVkC2iSz5g] Unable to reserve space: no linkgroups configured.\n>\n> This, in turn, was cause by SpaceManager believing there are no pools\n> contributing to the linkgroup:\n>\n> [prometheus] (local) admin > \\s SpaceManager ls link groups\n> FLAGS   CNT RESVD AVAIL FREE          UPDATED NAME\n> -orc:no 0   0 +   0 =   0 2018-01-31 10:33:39 default-linkgroup\n>\n> [prometheus] (local) admin >\n>\n> On prometheus, there are two pools (pool1 and pool2) both in the pools\n> domain. PoolManager does not know about these pools:\n>\n> [prometheus] (local) admin > \\sp psu ls pool\n> [prometheus] (local) admin >\n>\n> The pools domain is running:\n>\n> [root@prometheus ~]# dcache status\n> DOMAIN       STATUS                PID   USER   LOG\n> dCacheDomain running (for 4 hours) 32324 dcache /var/log/dcache/dCacheDomain.log\n> nfsServer    running (for 4 hours) 32384 dcache /var/log/dcache/nfsServer.log\n> pools        running (for 4 hours) 32439 dcache /var/log/dcache/pools.log\n> [root@prometheus ~]#\n>\n> However, the two pools are no available through the admin interface, and\n> topo does not show pools domain.\n>\n> [root@prometheus ~]# ssh -p 22224 admin@localhost\n> dCache (4.1.0-SNAPSHOT)\n> Type \"\\?\" for help.\n>\n> [prometheus] (local) admin > \\c pool1\n> (1) Cell does not exist.\n> [prometheus] (local) admin > \\c pool2\n> (1) Cell does not exist.\n> [prometheus] (local) admin > \\c System@pools\n> (1) Cell does not exist.\n> [prometheus] (local) admin > \\s topo ls\n> dCacheDomain\n>\n> [prometheus] (local) admin >\n>\n> The topo should show the dCacheDomain, pools and nfsServer domains.\n>\n> The dCacheDomain log file shows (edited for brevity):\n>\n> 2018-01-31 06:01:05 Launching /bin/java -[...]BootLoader start dCacheDomain\n> [...]\n> 31 Jan 2018 06:01:23 (System) [] ZooKeeper connection to localhost/127.0.0.1:2181 failed (Connection refused), attempting reconnect.\n> 31 Jan 2018 06:01:23 (System) [] ZooKeeper connection to localhost/0:0:0:0:0:0:0:1:2181 failed (Connection refused), attempting reconnect.\n> 31 Jan 2018 06:01:24 (System) [] ZooKeeper connection to localhost/127.0.0.1:2181 failed (Connection refused), attempting reconnect.\n> 31 Jan 2018 06:01:24 (System) [] ZooKeeper connection to localhost/0:0:0:0:0:0:0:1:2181 failed (Connection refused), attempting reconnect.\n> [... similar connection errors ...]\n> 31 Jan 2018 06:02:29 (System) [] ZooKeeper connection to localhost/127.0.0.1:2181 failed (Connection refused), attempting reconnect.\n> 31 Jan 2018 06:02:29 (System) [] ZooKeeper connection to localhost/0:0:0:0:0:0:0:1:2181 failed (Connection refused), attempting reconnect.\n> 31 Jan 2018 06:02:30 (lm) [] ZooKeeper connection to localhost/0:0:0:0:0:0:0:1:2181 failed (Connection refused), attempting reconnect.\n> 31 Jan 2018 06:02:30 (lm) [] ZooKeeper connection to localhost/127.0.0.1:2181 failed (Connection refused), attempting reconnect.\n> 31 Jan 2018 06:02:30 (lm) [] Failed to connect to zookeeper, using mode none until connection reestablished\n> 31 Jan 2018 06:02:30 (lm) []\n> org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /dcache\n>         at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) ~[zookeeper-3.4.8.jar:3.4.8--1]\n>         at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[zookeeper-3.4.8.jar:3.4.8--1]\n>         at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102) ~[zookeeper-3.4.8.jar:3.4.8--1]\n>         at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1130) ~[zookeeper-3.4.8.jar:3.4.8--1]\n>         at org.apache.curator.utils.ZKPaths.mkdirs(ZKPaths.java:274) ~[curator-client-2.11.1.jar:na]\n>         at org.apache.curator.framework.imps.ExistsBuilderImpl$2.call(ExistsBuilderImpl.java:199) ~[curator-framework-2.11.1.jar:na]\n>         at org.apache.curator.framework.imps.ExistsBuilderImpl$2.call(ExistsBuilderImpl.java:193) ~[curator-framework-2.11.1.jar:na]\n>         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109) ~[curator-client-2.11.1.jar:na]\n>         at org.apache.curator.framework.imps.ExistsBuilderImpl.pathInForeground(ExistsBuilderImpl.java:190) ~[curator-framework-2.11.1.jar:na]\n>         at org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:175) ~[curator-framework-2.11.1.jar:na]\n>         at org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:32) ~[curator-framework-2.11.1.jar:na]\n>         at org.apache.curator.framework.imps.CuratorFrameworkImpl.createContainers(CuratorFrameworkImpl.java:194) ~[curator-framework-2.11.1.jar:na]\n>         at dmg.cells.zookeeper.CellCuratorFramework.createContainers(CellCuratorFramework.java:265) ~[cells-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]\n>         at org.apache.curator.framework.EnsureContainers.internalEnsure(EnsureContainers.java:61) ~[curator-framework-2.11.1.jar:na]\n>         at org.apache.curator.framework.EnsureContainers.ensure(EnsureContainers.java:53) ~[curator-framework-2.11.1.jar:na]\n>         at org.apache.curator.framework.recipes.cache.PathChildrenCache.ensurePath(PathChildrenCache.java:576) ~[curator-recipes-2.11.1.jar:na]\n>         at org.apache.curator.framework.recipes.cache.PathChildrenCache.refresh(PathChildrenCache.java:490) ~[curator-recipes-2.11.1.jar:na]\n>         at org.apache.curator.framework.recipes.cache.RefreshOperation.invoke(RefreshOperation.java:35) ~[curator-recipes-2.11.1.jar:na]\n>         at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:773) ~[curator-recipes-2.11.1.jar:na]\n>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_161]\n>         at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_161]\n>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_161]\n>         at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_161]\n>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_161]\n>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_161]\n>         at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]\n> 31 Jan 2018 06:02:33 (System) [] The version of ZooKeeper being used doesn't support Container nodes. CreateMode.PERSISTENT will be used instead.\n>\n> These initial ZooKeeper connection to localhost/127.0.0.1:2181 failed\n> messages are annoying but (I believe) not the cause of the problem. They\n> are present when dCache is working correctly and are caused by the System\n> cell attempts to connect to ZK before the embedded ZK is started. This\n> connection blocks for ~1 minute before returning, with the ZK client\n> retrying in the background, as mentioned in the message\n>\n> 31 Jan 2018 06:02:30 (lm) [] Failed to connect to zookeeper, using mode none until connection reestablished\n>\n> Later on, the message version of ZooKeeper being used indicates that the\n> client was eventually able to connect to ZK:\n>\n> 31 Jan 2018 06:02:33 (System) [] The version of ZooKeeper being used doesn't support Container nodes. CreateMode.PERSISTENT will be used instead.\n>\n> The stack-trace is interesting and new. This indicates that the\n> PathChildrenCache is no longer maintained; for example, if a domain suffers\n> this problem and a core domain restarts (or starts up) then the satellite\n> would never learn of the core domain appearing, so would never initiate a\n> fresh connection to that domain.\n>\n> Both the nfsServer and pools domains contain a similar stack-trace:\n>\n> For pools domain:\n>\n> 2018-01-31 06:01:07 Launching /bin/java -server [...] BootLoader start pools\n> [...]\n> 31 Jan 2018 06:01:24 (System) [] ZooKeeper connection to localhost/0:0:0:0:0:0:0:1:2181 failed (Connection refused), attempting reconnect.\n> 31 Jan 2018 06:01:24 (System) [] ZooKeeper connection to localhost/127.0.0.1:2181 failed (Connection refused), attempting reconnect.\n> [...]\n> 31 Jan 2018 06:02:29 (System) [] ZooKeeper connection to localhost/127.0.0.1:2181 failed (Connection refused), attempting reconnect.\n> 31 Jan 2018 06:02:31 (System) [] ZooKeeper connection to localhost/0:0:0:0:0:0:0:1:2181 failed (Connection refused), attempting reconnect.\n> 31 Jan 2018 06:02:31 (lm) [] ZooKeeper connection to localhost/127.0.0.1:2181 failed (Connection refused), attempting reconnect.\n> 31 Jan 2018 06:02:31 (lm) [] ZooKeeper connection to localhost/0:0:0:0:0:0:0:1:2181 failed (Connection refused), attempting reconnect.\n> 31 Jan 2018 06:02:31 (lm) []\n> org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /dcache\n>         at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) ~[zookeeper-3.4.8.jar:3.4.8--1]\n>         at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[zookeeper-3.4.8.jar:3.4.8--1]\n>         at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102) ~[zookeeper-3.4.8.jar:3.4.8--1]\n>         at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1130) ~[zookeeper-3.4.8.jar:3.4.8--1]\n>         at org.apache.curator.utils.ZKPaths.mkdirs(ZKPaths.java:274) ~[curator-client-2.11.1.jar:na]\n>         at org.apache.curator.framework.imps.ExistsBuilderImpl$2.call(ExistsBuilderImpl.java:199) ~[curator-framework-2.11.1.jar:na]\n>         at org.apache.curator.framework.imps.ExistsBuilderImpl$2.call(ExistsBuilderImpl.java:193) ~[curator-framework-2.11.1.jar:na]\n>         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109) ~[curator-client-2.11.1.jar:na]\n>         at org.apache.curator.framework.imps.ExistsBuilderImpl.pathInForeground(ExistsBuilderImpl.java:190) ~[curator-framework-2.11.1.jar:na]\n>         at org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:175) ~[curator-framework-2.11.1.jar:na]\n>         at org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:32) ~[curator-framework-2.11.1.jar:na]\n>         at org.apache.curator.framework.imps.CuratorFrameworkImpl.createContainers(CuratorFrameworkImpl.java:194) ~[curator-framework-2.11.1.jar:na]\n>         at dmg.cells.zookeeper.CellCuratorFramework.createContainers(CellCuratorFramework.java:265) ~[cells-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]\n>         at org.apache.curator.framework.EnsureContainers.internalEnsure(EnsureContainers.java:61) ~[curator-framework-2.11.1.jar:na]\n>         at org.apache.curator.framework.EnsureContainers.ensure(EnsureContainers.java:53) ~[curator-framework-2.11.1.jar:na]\n>         at org.apache.curator.framework.recipes.cache.PathChildrenCache.ensurePath(PathChildrenCache.java:576) ~[curator-recipes-2.11.1.jar:na]\n>         at org.apache.curator.framework.recipes.cache.PathChildrenCache.refresh(PathChildrenCache.java:490) ~[curator-recipes-2.11.1.jar:na]\n>         at org.apache.curator.framework.recipes.cache.RefreshOperation.invoke(RefreshOperation.java:35) ~[curator-recipes-2.11.1.jar:na]\n>         at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:773) ~[curator-recipes-2.11.1.jar:na]\n>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_161]\n>         at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_161]\n>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_161]\n>         at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_161]\n>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_161]\n>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_161]\n>         at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]\n>\n> and for nfsServer domain:\n>\n> 2018-01-28 06:01:02 Launching /bin/java -server [...]BootLoader start nfsServer\n> [...]\n> 31 Jan 2018 06:01:23 (System) [] ZooKeeper connection to localhost/0:0:0:0:0:0:0:1:2181 failed (Connection refused), attempting reconnect.\n> 31 Jan 2018 06:01:24 (System) [] ZooKeeper connection to localhost/127.0.0.1:2181 failed (Connection refused), attempting reconnect.\n> [...]\n> 31 Jan 2018 06:02:29 (System) [] ZooKeeper connection to localhost/0:0:0:0:0:0:0:1:2181 failed (Connection refused), attempting reconnect.\n> 31 Jan 2018 06:02:29 (System) [] ZooKeeper connection to localhost/127.0.0.1:2181 failed (Connection refused), attempting reconnect.\n> 31 Jan 2018 06:02:31 (lm) [] ZooKeeper connection to localhost/0:0:0:0:0:0:0:1:2181 failed (Connection refused), attempting reconnect.\n> 31 Jan 2018 06:02:31 (lm) [] ZooKeeper connection to localhost/127.0.0.1:2181 failed (Connection refused), attempting reconnect.\n> 31 Jan 2018 06:02:31 (lm) []\n> org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /dcache\n>         at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) ~[zookeeper-3.4.8.jar:3.4.8--1]\n>         at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[zookeeper-3.4.8.jar:3.4.8--1]\n>         at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102) ~[zookeeper-3.4.8.jar:3.4.8--1]\n>         at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1130) ~[zookeeper-3.4.8.jar:3.4.8--1]\n>         at org.apache.curator.utils.ZKPaths.mkdirs(ZKPaths.java:274) ~[curator-client-2.11.1.jar:na]\n>         at org.apache.curator.framework.imps.ExistsBuilderImpl$2.call(ExistsBuilderImpl.java:199) ~[curator-framework-2.11.1.jar:na]\n>         at org.apache.curator.framework.imps.ExistsBuilderImpl$2.call(ExistsBuilderImpl.java:193) ~[curator-framework-2.11.1.jar:na]\n>         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109) ~[curator-client-2.11.1.jar:na]\n>         at org.apache.curator.framework.imps.ExistsBuilderImpl.pathInForeground(ExistsBuilderImpl.java:190) ~[curator-framework-2.11.1.jar:na]\n>         at org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:175) ~[curator-framework-2.11.1.jar:na]\n>         at org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:32) ~[curator-framework-2.11.1.jar:na]\n>         at org.apache.curator.framework.imps.CuratorFrameworkImpl.createContainers(CuratorFrameworkImpl.java:194) ~[curator-framework-2.11.1.jar:na]\n>         at dmg.cells.zookeeper.CellCuratorFramework.createContainers(CellCuratorFramework.java:265) ~[cells-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]\n>         at org.apache.curator.framework.EnsureContainers.internalEnsure(EnsureContainers.java:61) ~[curator-framework-2.11.1.jar:na]\n>         at org.apache.curator.framework.EnsureContainers.ensure(EnsureContainers.java:53) ~[curator-framework-2.11.1.jar:na]\n>         at org.apache.curator.framework.recipes.cache.PathChildrenCache.ensurePath(PathChildrenCache.java:576) ~[curator-recipes-2.11.1.jar:na]\n>         at org.apache.curator.framework.recipes.cache.PathChildrenCache.refresh(PathChildrenCache.java:490) ~[curator-recipes-2.11.1.jar:na]\n>         at org.apache.curator.framework.recipes.cache.RefreshOperation.invoke(RefreshOperation.java:35) ~[curator-recipes-2.11.1.jar:na]\n>         at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:773) ~[curator-recipes-2.11.1.jar:na]\n>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_161]\n>         at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_161]\n>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_161]\n>         at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_161]\n>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_161]\n>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_161]\n>         at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]\n>\n> —\n> You are receiving this because you are subscribed to this thread.\n> Reply to this email directly, view it on GitHub\n> <https://github.com/dCache/dcache/issues/3790>, or mute the thread\n> <https://github.com/notifications/unsubscribe-auth/AAjJ3ep5jEcT6kbeeVMVzQ-PbWk0dP_Iks5tQDTYgaJpZM4RzrSf>\n> .\n>\n"}}
{"actor":{"display_login":"kofemann"}}