{"action":"created","issue":{"url":"https://api.github.com/repos/dCache/dcache/issues/2550","repository_url":"https://api.github.com/repos/dCache/dcache","labels_url":"https://api.github.com/repos/dCache/dcache/issues/2550/labels{/name}","comments_url":"https://api.github.com/repos/dCache/dcache/issues/2550/comments","events_url":"https://api.github.com/repos/dCache/dcache/issues/2550/events","html_url":"https://github.com/dCache/dcache/issues/2550","id":159707785,"number":2550,"title":"dcache nfs41: Internal failure during space management","user":{"login":"calestyo","id":919933,"avatar_url":"https://avatars.githubusercontent.com/u/919933?v=3","gravatar_id":"","url":"https://api.github.com/users/calestyo","html_url":"https://github.com/calestyo","followers_url":"https://api.github.com/users/calestyo/followers","following_url":"https://api.github.com/users/calestyo/following{/other_user}","gists_url":"https://api.github.com/users/calestyo/gists{/gist_id}","starred_url":"https://api.github.com/users/calestyo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/calestyo/subscriptions","organizations_url":"https://api.github.com/users/calestyo/orgs","repos_url":"https://api.github.com/users/calestyo/repos","events_url":"https://api.github.com/users/calestyo/events{/privacy}","received_events_url":"https://api.github.com/users/calestyo/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/dCache/dcache/labels/bug","name":"bug","color":"fc2929"}],"state":"open","locked":false,"assignee":{"login":"kofemann","id":575965,"avatar_url":"https://avatars.githubusercontent.com/u/575965?v=3","gravatar_id":"","url":"https://api.github.com/users/kofemann","html_url":"https://github.com/kofemann","followers_url":"https://api.github.com/users/kofemann/followers","following_url":"https://api.github.com/users/kofemann/following{/other_user}","gists_url":"https://api.github.com/users/kofemann/gists{/gist_id}","starred_url":"https://api.github.com/users/kofemann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kofemann/subscriptions","organizations_url":"https://api.github.com/users/kofemann/orgs","repos_url":"https://api.github.com/users/kofemann/repos","events_url":"https://api.github.com/users/kofemann/events{/privacy}","received_events_url":"https://api.github.com/users/kofemann/received_events","type":"User","site_admin":false},"milestone":null,"comments":11,"created_at":"2016-06-10T19:40:34Z","updated_at":"2016-06-22T06:58:09Z","closed_at":null,"body":"Hi.\r\n\r\ndCache 2.15.3-1\r\n\r\nI noted the following by accident (haven't tried to reproduce it yet).\r\nWe must upload these rucio chimera dumps for ATLAS, in some dir.\r\nApparently the dump dir (on which I've had set WriteToken) got empty and deleted, and subsequently auto-recreated by my script (but without WriteToken set).\r\n\r\nSo first any uploads failed because of this (which is of course expected), but then I set it, and then some error condition came up that only resolved by unmounting pnfs and restarting the nfs door:\r\n\r\nUploads failed with:\r\n```\r\n/pnfs/lrz-muenchen.de/data/atlas/dq2/atlascalibdisk/dumps# echo \"foo\" > bar\r\n-bash: echo: write error: Remote I/O error\r\n\r\n```\r\n\r\nThe nfs door's log said:\r\n```\r\n2016-06-10 15:02:03+02:00 (nfs_lcg-lrz-dcache) [/0:0:0:0:0:0:0:1:849 0000080000000000D2C8BD0130 door:nfs_lcg-lrz-dcache:AAU07CUoUlg 0000268136173D044507B39BEA8436BA3DAB /0:0:0:0:0:0:0:1:849] No write pools configured outside of a linkgroup.\r\n2016-06-10 15:02:03+02:00 (nfs_lcg-lrz-dcache) [/0:0:0:0:0:0:0:1:849 0000080000000000D2C8BD0130] failed to create IO adapter: NFS4ERR_NOSPC : No write pools configured outside of a linkgroup.\r\n2016-06-10 15:02:03+02:00 (nfs_lcg-lrz-dcache) [/0:0:0:0:0:0:0:1:849 0000080000000000D2C8BD0130 door:nfs_lcg-lrz-dcache:AAU07CUocZg 0000268136173D044507B39BEA8436BA3DAB /0:0:0:0:0:0:0:1:849] No write pools configured outside of a linkgroup.\r\n2016-06-10 15:02:03+02:00 (nfs_lcg-lrz-dcache) [/0:0:0:0:0:0:0:1:849 0000080000000000D2C8BD0130] failed to create IO adapter: NFS4ERR_NOSPC : No write pools configured outside of a linkgroup.\r\n2016-06-10 15:02:03+02:00 (nfs_lcg-lrz-dcache) [/0:0:0:0:0:0:0:1:849 0000080000000000D2C8BD0130 door:nfs_lcg-lrz-dcache:AAU07CUoiQg 0000268136173D044507B39BEA8436BA3DAB /0:0:0:0:0:0:0:1:849] No write pools configured outside of a linkgroup.\r\n2016-06-10 15:02:03+02:00 (nfs_lcg-lrz-dcache) [/0:0:0:0:0:0:0:1:849 0000080000000000D2C8BD0130] failed to create IO adapter: NFS4ERR_NOSPC : No write pools configured outside of a linkgroup.\r\n2016-06-10 15:08:01+02:00 (nfs_lcg-lrz-dcache) [] Unhandled exception:\r\norg.dcache.chimera.InvalidArgumentChimeraException: Not a file.\r\n\tat org.dcache.chimera.FsSqlDriver.remove(FsSqlDriver.java:280) ~[chimera-2.15.3.jar:2.15.3]\r\n\tat org.dcache.chimera.JdbcFs.lambda$remove$12(JdbcFs.java:498) ~[chimera-2.15.3.jar:2.15.3]\r\n\tat org.dcache.chimera.JdbcFs.inTransaction(JdbcFs.java:200) ~[chimera-2.15.3.jar:2.15.3]\r\n\tat org.dcache.chimera.JdbcFs.remove(JdbcFs.java:497) ~[chimera-2.15.3.jar:2.15.3]\r\n\tat org.dcache.chimera.DCacheAwareJdbcFs.remove(DCacheAwareJdbcFs.java:181) ~[dcache-chimera-2.15.3.jar:2.15.3]\r\n\tat org.dcache.chimera.nfsv41.door.ChimeraVfs.remove(ChimeraVfs.java:203) ~[dcache-nfs-2.15.3.jar:2.15.3]\r\n\tat org.dcache.nfs.vfs.VfsCache.remove(VfsCache.java:102) ~[nfs4j-core-0.11.2.jar:0.11.2]\r\n\tat org.dcache.nfs.vfs.PseudoFs.remove(PseudoFs.java:264) ~[nfs4j-core-0.11.2.jar:0.11.2]\r\n\tat org.dcache.nfs.v4.OperationREMOVE.process(OperationREMOVE.java:62) ~[nfs4j-core-0.11.2.jar:0.11.2]\r\n\tat org.dcache.chimera.nfsv41.door.proxy.ProxyIoMdsOpFactory$1.lambda$process$17(ProxyIoMdsOpFactory.java:53) ~[dcache-nfs-2.15.3.jar:2.15.3]\r\n\tat java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_72-internal]\r\n\tat javax.security.auth.Subject.doAs(Subject.java:360) ~[na:1.8.0_72-internal]\r\n\tat org.dcache.chimera.nfsv41.door.proxy.ProxyIoMdsOpFactory$1.process(ProxyIoMdsOpFactory.java:50) ~[dcache-nfs-2.15.3.jar:2.15.3]\r\n\tat org.dcache.nfs.v4.NFSServerV41.NFSPROC4_COMPOUND_4(NFSServerV41.java:152) ~[nfs4j-core-0.11.2.jar:0.11.2]\r\n\tat org.dcache.nfs.v4.xdr.nfs4_prot_NFS4_PROGRAM_ServerStub.dispatchOncRpcCall(nfs4_prot_NFS4_PROGRAM_ServerStub.java:48) [nfs4j-core-0.11.2.jar:0.11.2]\r\n\tat org.dcache.xdr.RpcDispatcher$1.run(RpcDispatcher.java:82) [oncrpc4j-core-2.4.2.jar:na]\r\n\tat org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:591) [grizzly-framework-2.3.23.jar:2.3.23]\r\n\tat org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571) [grizzly-framework-2.3.23.jar:2.3.23]\r\n\tat java.lang.Thread.run(Thread.java:745) [na:1.8.0_72-internal]\r\n2016-06-10 15:12:24+02:00 (nfs_lcg-lrz-dcache) [/0:0:0:0:0:0:0:1:849 0000080000000000D2CF7C0130] failed to create IO adapter: NFS4ERR_LAYOUTTRYLATER : Internal failure during space management\r\n2016-06-10 15:14:37+02:00 (nfs_lcg-lrz-dcache) [/0:0:0:0:0:0:0:1:849 0000080000000000D2D0640130] failed to create IO adapter: NFS4ERR_LAYOUTTRYLATER : Internal failure during space management\r\n2016-06-10 21:31:44+02:00 (nfs_lcg-lrz-dcache) [/0:0:0:0:0:0:0:1:849 0000080000000000D38B880130] failed to create IO adapter: NFS4ERR_LAYOUTTRYLATER : Internal failure during space management\r\n2016-06-10 21:33:30+02:00 (nfs_lcg-lrz-dcache) [] Bad client: op: ILLEGAL<57> : NFS4ERR_STALE_CLIENTID : bad client id.\r\n2016-06-10 21:33:53+02:00 (RoutingMgr) [] Unable to notify [>RoutingMgr@nfs] about delivery failure of message sent to [>RoutingMgr@local]: No route for RoutingMgr@nfs in nfs.\r\n2016-06-10 21:33:53+02:00 (c-core-AAUvm9S8bKA) [] Sleeping 29 seconds\r\n2016-06-10 21:33:54 Process terminated\r\n\r\n2016-06-10 21:33:54 Launching /usr/bin/java -server -Xmx4096M -XX:MaxDirectMemorySize=512m -Dsun.net.inetaddr.ttl=1800 -Dorg.globus.tcp.port.range=60000,64999 -Dorg.dcache.dcap.port=0 -Dorg.dcache.net.tcp.portrange=65000:65499 -Dorg.globus.jglobus.delegation.cache.lifetime=30000 -Dorg.globus.jglobus.crl.cache.lifetime=60000 -Djava.security.krb5.realm= -Djava.security.krb5.kdc= -Djavax.security.auth.useSubjectCredsOnly=false -Djava.security.auth.login.config=/etc/dcache/jgss.conf -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/dcache/nfs-oom.hprof -XX:+UseCompressedOops -javaagent:/usr/share/dcache/classes/aspectjweaver-1.8.8.jar -Djava.awt.headless=true -DwantLog4jSetup=n -Ddcache.home=/usr/share/dcache -Ddcache.paths.defaults=/usr/share/dcache/defaults org.dcache.boot.BootLoader start nfs\r\n\r\n```\r\n\r\nThis is from the beginning where no WriteToken was set over the error, until restart.\r\nAfter restart it worked again. Nothing else was changed in between.\r\n\r\nCheers,\r\nChris."},"comment":{"url":"https://api.github.com/repos/dCache/dcache/issues/comments/227659514","html_url":"https://github.com/dCache/dcache/issues/2550#issuecomment-227659514","issue_url":"https://api.github.com/repos/dCache/dcache/issues/2550","id":227659514,"user":{"login":"kofemann","id":575965,"avatar_url":"https://avatars.githubusercontent.com/u/575965?v=3","gravatar_id":"","url":"https://api.github.com/users/kofemann","html_url":"https://github.com/kofemann","followers_url":"https://api.github.com/users/kofemann/followers","following_url":"https://api.github.com/users/kofemann/following{/other_user}","gists_url":"https://api.github.com/users/kofemann/gists{/gist_id}","starred_url":"https://api.github.com/users/kofemann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kofemann/subscriptions","organizations_url":"https://api.github.com/users/kofemann/orgs","repos_url":"https://api.github.com/users/kofemann/repos","events_url":"https://api.github.com/users/kofemann/events{/privacy}","received_events_url":"https://api.github.com/users/kofemann/received_events","type":"User","site_admin":false},"created_at":"2016-06-22T06:58:09Z","updated_at":"2016-06-22T06:58:09Z","body":"Space manager doesn't start the mover, it only selects a pool and if it already did so can return it second time instead of \"Internal failure during space management\"."}}
{"actor":{"display_login":"kofemann"}}