{"action":"created","issue":{"url":"https://api.github.com/repos/xrootd/xrootd/issues/750","repository_url":"https://api.github.com/repos/xrootd/xrootd","labels_url":"https://api.github.com/repos/xrootd/xrootd/issues/750/labels{/name}","comments_url":"https://api.github.com/repos/xrootd/xrootd/issues/750/comments","events_url":"https://api.github.com/repos/xrootd/xrootd/issues/750/events","html_url":"https://github.com/xrootd/xrootd/issues/750","id":333736906,"node_id":"MDU6SXNzdWUzMzM3MzY5MDY=","number":750,"title":"XrdHttp fails to refresh CRLs","user":{"login":"olifre","id":166759,"node_id":"MDQ6VXNlcjE2Njc1OQ==","avatar_url":"https://avatars0.githubusercontent.com/u/166759?v=4","gravatar_id":"","url":"https://api.github.com/users/olifre","html_url":"https://github.com/olifre","followers_url":"https://api.github.com/users/olifre/followers","following_url":"https://api.github.com/users/olifre/following{/other_user}","gists_url":"https://api.github.com/users/olifre/gists{/gist_id}","starred_url":"https://api.github.com/users/olifre/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/olifre/subscriptions","organizations_url":"https://api.github.com/users/olifre/orgs","repos_url":"https://api.github.com/users/olifre/repos","events_url":"https://api.github.com/users/olifre/events{/privacy}","received_events_url":"https://api.github.com/users/olifre/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-06-19T16:00:15Z","updated_at":"2018-12-10T20:05:11Z","closed_at":null,"author_association":"NONE","body":"Using xrootd 4.8.3, I observe the following after a few days:\r\n```\r\n139990299645696:error:80066405:lib(128):func(102):reason(1029):sslutils.c:1915:\r\n139990299645696:error:80066411:lib(128):func(102):reason(1041):sslutils.c:2110:: CRL has expired [subject=/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=ddmadmin/CN=531497/CN=Robot: ATLAS Data Management,issuer=/DC=ch/DC=cern/CN=CERN Grid Certification Authority]\r\n139990299645696:error:14089086:SSL routines:ssl3_get_client_certificate:certificate verify failed:s3_srvr.c:3327:\r\n```\r\nall over in our logs. \r\n\r\nChecking:\r\n```\r\n# ls -lart b4278411.r0 c2a48ab6.r0 5168735f.r0 4339b4bc.r0\r\n-rw-r--r--. 1 root root  1535 19. Jun 12:47 c2a48ab6.r0\r\n-rw-r--r--. 1 root root  1535 19. Jun 12:47 b4278411.r0\r\n-rw-r--r--. 1 root root 33292 19. Jun 12:47 5168735f.r0\r\n-rw-r--r--. 1 root root 33292 19. Jun 12:47 4339b4bc.r0\r\n```\r\nshows they are up-to-date (those are the CRLs for CERN-GridCA and CERN-Root-2, we run fetchcrl regulartly). \r\n\r\nChecking xrootd with `strace`, I find:\r\n```\r\n[pid 13040] futex(0x610f18, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>\r\n[pid 10153] <... accept4 resumed> {sa_family=AF_INET6, sin6_port=htons(51392), inet_pton(AF_INET6, \"::ffff:128.142.132.207\", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28], SOCK_CLOEXEC) = 25\r\n[pid 10153] setsockopt(25, SOL_SOCKET, SO_LINGER, {onoff=1, linger=3}, 8) = 0\r\n[pid 10153] setsockopt(25, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0\r\n[pid 10153] setsockopt(25, SOL_TCP, TCP_NODELAY, [1], 4) = 0\r\n[pid 10153] futex(0x7ffc4e2a27b0, FUTEX_WAKE_PRIVATE, 1) = 1\r\n[pid  3022] <... futex resumed> )       = 0\r\n[pid 10153] poll([{fd=25, events=POLLIN|POLLRDNORM}], 1, 30000 <unfinished ...>\r\n[pid  3022] futex(0x610f18, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>\r\n[pid 13039] <... futex resumed> )       = 0\r\n[pid 10153] <... poll resumed> )        = 1 ([{fd=25, revents=POLLIN|POLLRDNORM}])\r\n[pid  3022] <... futex resumed> )       = 1\r\n[pid 13039] futex(0x610f38, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>\r\n[pid 10153] recvfrom(25,  <unfinished ...>\r\n[pid  3022] futex(0x610f38, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>\r\n[pid 13039] <... futex resumed> )       = -1 EAGAIN (Resource temporarily unavailable)\r\n[pid 10153] <... recvfrom resumed> \"\\26\\3\\1\\1\\24\\1\\0\\1\\20\\3\\3[)'\\305\\261\\351\\17<\\225C\\232\\311\\27V\\240\\221q\\t\\377\\26\\275\"..., 44, MSG_PEEK, NULL, NULL) = 44\r\n[pid  3022] <... futex resumed> )       = 0\r\n[pid 13039] futex(0x610f38, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>\r\n[pid 10153] poll([{fd=25, events=POLLIN|POLLRDNORM}], 1, 10000 <unfinished ...>\r\n[pid  3022] futex(0x7ffc4e2a27b0, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>\r\n[pid 13039] <... futex resumed> )       = 0\r\n[pid 10153] <... poll resumed> )        = 1 ([{fd=25, revents=POLLIN|POLLRDNORM}])\r\n[pid 13039] accept4(17,  <unfinished ...>\r\n[pid 10153] recvfrom(25, \"\\26\\3\\1\\1\\24\\1\\0\\1\\20\\3\\3[)'\\305\\261\", 16, MSG_PEEK, NULL, NULL) = 16\r\n[pid 10153] epoll_ctl(7, EPOLL_CTL_ADD, 25, {0, {u32=738201208, u64=139990902247032}}) = 0\r\n[pid 10153] setsockopt(25, SOL_SOCKET, SO_RCVTIMEO, \"\\n\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\", 16) = 0\r\n[pid 10153] setsockopt(25, SOL_SOCKET, SO_SNDTIMEO, \"\\n\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\", 16) = 0\r\n[pid 10153] read(25, \"\\26\\3\\1\\1\\24\", 5) = 5\r\n[pid 10153] read(25, \"\\1\\0\\1\\20\\3\\3[)'\\305\\261\\351\\17<\\225C\\232\\311\\27V\\240\\221q\\t\\377\\26\\275\\232G\\260\\300\\276\"..., 276) = 276\r\n[pid 10153] write(25, \"\\26\\3\\3\\0:\\2\\0\\0006\\3\\3\\24\\275\\332\\341L+Tw:\\2216\\231\\1\\352~\\213\\37S\\v\\274.\"..., 2216) = 2216\r\n[pid 10153] read(25, \"\\26\\3\\3\\0378\", 5) = 5\r\n[pid 10153] read(25, \"\\v\\0\\0374\\0\\0371\\0\\3L0\\202\\3H0\\202\\2\\261\\240\\3\\2\\1\\2\\2\\1\\0000\\r\\6\\t*\\206\"..., 7992) = 7992\r\n[pid 10153] stat(\"/etc/grid-security/certificates/c2a48ab6.r1\", 0x7f523588a1f0) = -1 ENOENT (No such file or directory)\r\n[pid 10153] stat(\"/etc/localtime\", {st_mode=S_IFREG|0644, st_size=2309, ...}) = 0\r\n[pid 10153] stat(\"/etc/grid-security/certificates/c2a48ab6.r1\", 0x7f523588a1f0) = -1 ENOENT (No such file or directory)\r\n[pid 10153] open(\"/etc/grid-security/certificates/e187c0c8.signing_policy\", O_RDONLY) = -1 ENOENT (No such file or directory)\r\n[pid 10153] open(\"/etc/grid-security/certificates/e187c0c8.namespaces\", O_RDONLY) = -1 ENOENT (No such file or directory)\r\n[pid 10153] open(\"/etc/grid-security/certificates/f27afd4e.signing_policy\", O_RDONLY) = -1 ENOENT (No such file or directory)\r\n[pid 10153] open(\"/etc/grid-security/certificates/f27afd4e.namespaces\", O_RDONLY) = -1 ENOENT (No such file or directory)\r\n[pid 10153] open(\"/etc/grid-security/certificates/4ad425d0.signing_policy\", O_RDONLY) = -1 ENOENT (No such file or directory)\r\n[pid 10153] open(\"/etc/grid-security/certificates/4ad425d0.namespaces\", O_RDONLY) = -1 ENOENT (No such file or directory)\r\n[pid 10153] open(\"/etc/grid-security/certificates/7768cc9b.signing_policy\", O_RDONLY) = -1 ENOENT (No such file or directory)\r\n[pid 10153] open(\"/etc/grid-security/certificates/7768cc9b.namespaces\", O_RDONLY) = -1 ENOENT (No such file or directory)\r\n[pid 10153] open(\"/etc/grid-security/certificates/5168735f.signing_policy\", O_RDONLY) = 26\r\n[pid 10153] ioctl(26, TCGETS, 0x7f5235889900) = -1 ENOTTY (Inappropriate ioctl for device)\r\n[pid 10153] read(26, \"# @(#)$Id: 4339b4bc.signing_poli\"..., 8192) = 269\r\n[pid 10153] read(26, \"\", 8192)          = 0\r\n[pid 10153] ioctl(26, TCGETS, 0x7f5235889910) = -1 ENOTTY (Inappropriate ioctl for device)\r\n[pid 10153] close(26)                   = 0\r\n[pid 10153] open(\"/etc/grid-security/certificates/5168735f.namespaces\", O_RDONLY) = -1 ENOENT (No such file or directory)\r\n[pid 10153] open(\"/etc/grid-security/certificates/c2a48ab6.signing_policy\", O_RDONLY) = 26\r\n[pid 10153] ioctl(26, TCGETS, 0x7f5235889900) = -1 ENOTTY (Inappropriate ioctl for device)\r\n[pid 10153] read(26, \"# @(#)$Id: b4278411.signing_poli\"..., 8192) = 362\r\n[pid 10153] read(26, \"\", 8192)          = 0\r\n[pid 10153] ioctl(26, TCGETS, 0x7f5235889910) = -1 ENOTTY (Inappropriate ioctl for device)\r\n[pid 10153] close(26)                   = 0\r\n[pid 10153] open(\"/etc/grid-security/certificates/c2a48ab6.namespaces\", O_RDONLY) = 26\r\n[pid 10153] ioctl(26, TCGETS, 0x7f5235889900) = -1 ENOTTY (Inappropriate ioctl for device)\r\n[pid 10153] read(26, \"################################\"..., 8192) = 774\r\n[pid 10153] read(26, \"\", 8192)          = 0\r\n[pid 10153] ioctl(26, TCGETS, 0x7f5235889910) = -1 ENOTTY (Inappropriate ioctl for device)\r\n[pid 10153] close(26)                   = 0\r\n[pid 10153] stat(\"/etc/localtime\", {st_mode=S_IFREG|0644, st_size=2309, ...}) = 0\r\n[pid 10153] stat(\"/etc/grid-security/certificates/5168735f.r1\", 0x7f523588a1f0) = -1 ENOENT (No such file or directory)\r\n[pid 10153] write(25, \"\\25\\3\\3\\0\\2\\2-\", 7) = 7\r\n[pid 10153] write(2, \"139991062198016:error:80066405:l\"..., 80) = 80\r\n[pid 10153] write(2, \"139991062198016:error:80066411:l\"..., 261) = 261\r\n[pid 10153] write(2, \"139991062198016:error:14089086:S\"..., 114) = 114\r\n```\r\n\r\nAny ideas what could be causing that? \r\n\r\nRestarting the xrootd service immediately gets rid of that issue. "},"comment":{"url":"https://api.github.com/repos/xrootd/xrootd/issues/comments/445953602","html_url":"https://github.com/xrootd/xrootd/issues/750#issuecomment-445953602","issue_url":"https://api.github.com/repos/xrootd/xrootd/issues/750","id":445953602,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0NTk1MzYwMg==","user":{"login":"olifre","id":166759,"node_id":"MDQ6VXNlcjE2Njc1OQ==","avatar_url":"https://avatars0.githubusercontent.com/u/166759?v=4","gravatar_id":"","url":"https://api.github.com/users/olifre","html_url":"https://github.com/olifre","followers_url":"https://api.github.com/users/olifre/followers","following_url":"https://api.github.com/users/olifre/following{/other_user}","gists_url":"https://api.github.com/users/olifre/gists{/gist_id}","starred_url":"https://api.github.com/users/olifre/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/olifre/subscriptions","organizations_url":"https://api.github.com/users/olifre/orgs","repos_url":"https://api.github.com/users/olifre/repos","events_url":"https://api.github.com/users/olifre/events{/privacy}","received_events_url":"https://api.github.com/users/olifre/received_events","type":"User","site_admin":false},"created_at":"2018-12-10T20:05:11Z","updated_at":"2018-12-10T20:05:11Z","author_association":"NONE","body":"@bbockelm Sadly, testing without `libXrdHttpVOMS.so` in the production system is not really an option. But indeed, your feeling that this plugin may cause the issue sounds likely and would explain why most others who use Lcmaps have not seen it. \r\n\r\n\r\n\r\n> The next version of `libXrdLcmaps.so` can likely be a drop-in replacement. Since that will be released in January (and you'll be getting a new test setup around then), can we put the ticket on hold until then?\r\n\r\nThat's a very good idea! Yes, of course, no problem with that. Daily restarts work very well as a workaround for now. \r\nI dearly hope we can manage to get the test setup early next year, since we also have to commission a new building (with everything, networking, servers etc). 2019 wll be a fun year! :wink:"}}
{"actor":{"display_login":"olifre"}}