{"action":"created","issue":{"url":"https://api.github.com/repos/hashicorp/consul-template/issues/597","repository_url":"https://api.github.com/repos/hashicorp/consul-template","labels_url":"https://api.github.com/repos/hashicorp/consul-template/issues/597/labels{/name}","comments_url":"https://api.github.com/repos/hashicorp/consul-template/issues/597/comments","events_url":"https://api.github.com/repos/hashicorp/consul-template/issues/597/events","html_url":"https://github.com/hashicorp/consul-template/issues/597","id":144179753,"number":597,"title":"Consul Template getting stuck","user":{"login":"jippi","id":22841,"avatar_url":"https://avatars.githubusercontent.com/u/22841?v=3","gravatar_id":"","url":"https://api.github.com/users/jippi","html_url":"https://github.com/jippi","followers_url":"https://api.github.com/users/jippi/followers","following_url":"https://api.github.com/users/jippi/following{/other_user}","gists_url":"https://api.github.com/users/jippi/gists{/gist_id}","starred_url":"https://api.github.com/users/jippi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jippi/subscriptions","organizations_url":"https://api.github.com/users/jippi/orgs","repos_url":"https://api.github.com/users/jippi/repos","events_url":"https://api.github.com/users/jippi/events{/privacy}","received_events_url":"https://api.github.com/users/jippi/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-03-29T07:23:38Z","updated_at":"2016-03-29T09:13:36Z","closed_at":null,"body":"Hi!\r\n\r\nThis may or may not be relevant to https://github.com/hashicorp/vault/issues/1257 \r\n\r\nThe configuration files are 100% the same: https://dl.dropboxusercontent.com/u/27514/hashicorp-debug.tar\r\n\r\nFor some reason, consul-template seem to get \"stuck\" once in awhile, seem to happen after I started to use Vault secrets in my template(s).\r\n\r\nThere are no errors in Consul, CT or Vault that seem to reveal anything interesting\r\n\r\nMy use case:\r\n\r\nVault secret(s) using a default `generic` backend and a `mysql` backend - those will be written to nginx and other config files as `fastcgi` ENVs for our php-fpm backend(s).\r\n\r\nThe last output from CT:\r\n\r\n```\r\nnginx: the configuration file /etc/nginx/nginx.conf syntax is ok\r\nnginx: configuration file /etc/nginx/nginx.conf test is successful\r\nReloading nginx configuration: nginx.\r\n```\r\n\r\nThe last output from Vault\r\n\r\n```\r\n2016/03/28 13:01:02 [INFO] expire: revoked 'mysql/creds/frontend_main/9c752080-3b94-a944-05da-532d57bda31b'\r\n2016/03/28 14:04:49 [INFO] expire: revoked 'mysql/creds/frontend_test/53aaed2c-233e-1e8f-e486-90e5bb0c756f'\r\n2016/03/28 14:04:49 [INFO] expire: revoked 'mysql/creds/frontend_main/128afe3a-99df-9bca-d820-f6eb7e84bcc1'\r\n2016/03/28 14:15:49 [INFO] expire: revoked 'mysql/creds/frontend_test/186dcce0-8632-2e02-1f5d-c80cc63c54a9'\r\n2016/03/28 14:15:49 [INFO] expire: revoked 'mysql/creds/frontend_main/e81baa1c-cb9d-5cfd-6085-1e5ea67d3a21'\r\n2016/03/28 14:24:48 [INFO] expire: revoked 'mysql/creds/frontend_test/68a99e50-093c-0866-422a-21daee32cc18'\r\n2016/03/28 14:24:48 [INFO] expire: revoked 'mysql/creds/frontend_main/8bdd259d-6f0f-bced-e639-b1af36dc44fb'\r\n2016/03/28 14:58:48 [INFO] expire: revoked 'mysql/creds/frontend_test/9fd7fc45-9b72-52fe-f88a-b6e59189f3f4'\r\n2016/03/28 14:58:48 [INFO] expire: revoked 'mysql/creds/frontend_main/78958691-845f-9872-d281-58d96ee73a1e'\r\n```\r\n\r\nThere doesn't seem to be any file-handle craziness going on as the original Vault ticket, things just \"freeze\" up.\r\n\r\nCurrently the processes have been running for ~21 hour, and the mysql secret lease got a lease time configured as follow:\r\n\r\n```\r\n-> vault read mysql/config/lease\r\nKey      \tValue\r\nlease    \t1h0m0s\r\nlease_max\t24h0m0s\r\n```\r\n\r\nwhich, afaik should cause CT to refresh any leases every ~12h (half of the max-lifetime) - but that doesn't seem to happen - this morning when I got in, the app used some revoked credentials and thus the app failed completely\r\n\r\nThe configuration file in question got the following `stat` output\r\n\r\n```\r\n-> stat /etc/nginx/sites-enabled/20-frontend\r\n  File: `/etc/nginx/sites-enabled/20-frontend'\r\n  Size: 11215     \tBlocks: 24         IO Block: 4096   regular file\r\nDevice: 901h/2305d\tInode: 3675505     Links: 1\r\nAccess: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)\r\nAccess: 2016-03-28 14:06:57.653368995 +0000\r\nModify: 2016-03-28 14:06:57.613368599 +0000\r\nChange: 2016-03-28 14:06:57.625368718 +0000\r\n Birth: -\r\n```\r\n\r\nWhen trying to `strace` some Vault threads, I noticed this repeated in 4-5 of the threads running - not sure if its relevant or not\r\n\r\n![image](https://cloud.githubusercontent.com/assets/22841/14100642/3d9873d6-f58f-11e5-8483-8b80c6976e6b.png)\r\n\r\nBoth Vault and Consul seem to be up and responding to any kind of API requests I throw at it as well\r\n\r\nOne thing worth mentioning, mysql seem to have restarted, it has an uptime of `10441 seconds` this morning - could it be a case where Vault / CT does not handle that case very well? \r\n\r\nOr shouldn't CT at least time out or give some kind of error that it failed to talk to Vault within due time?\r\n\r\nRestarting `Vault` does not seem to do anything for the problem\r\nRestarting CT did seem to do the trick though (after restarting Vault)\r\n\r\nWorth mentioning everything (Consul, CT and Vault) are running on the same server, which is firewalled, but not on loopback "},"comment":{"url":"https://api.github.com/repos/hashicorp/consul-template/issues/comments/202793320","html_url":"https://github.com/hashicorp/consul-template/issues/597#issuecomment-202793320","issue_url":"https://api.github.com/repos/hashicorp/consul-template/issues/597","id":202793320,"user":{"login":"sethvargo","id":408570,"avatar_url":"https://avatars.githubusercontent.com/u/408570?v=3","gravatar_id":"","url":"https://api.github.com/users/sethvargo","html_url":"https://github.com/sethvargo","followers_url":"https://api.github.com/users/sethvargo/followers","following_url":"https://api.github.com/users/sethvargo/following{/other_user}","gists_url":"https://api.github.com/users/sethvargo/gists{/gist_id}","starred_url":"https://api.github.com/users/sethvargo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sethvargo/subscriptions","organizations_url":"https://api.github.com/users/sethvargo/orgs","repos_url":"https://api.github.com/users/sethvargo/repos","events_url":"https://api.github.com/users/sethvargo/events{/privacy}","received_events_url":"https://api.github.com/users/sethvargo/received_events","type":"User","site_admin":false},"created_at":"2016-03-29T09:13:35Z","updated_at":"2016-03-29T09:13:35Z","body":"Hi @jippi,\r\n\r\nThank you for opening an issue. Can you share the Consul Template debug logs when this happens? Please share the template as well.\r\n\r\nOne thing I want to note:\r\n\r\n> which, afaik should cause CT to refresh any leases every ~12h (half of the max-lifetime)\r\n\r\nThis is untrue. Consul Template renews at lease_duration/2.0, not lease_max/2.0, so it's going to renew every 30m. \r\n\r\nAre you doing any SIGHUP or reloading of the CT process? Can you show me the `lsof -p $CONSUL_TEMPLATE_PID` when this happens please?\r\n\r\n> Shouldn't CT at least time out or give some kind of error that it failed to talk to Vault within due time?\r\n\r\nThis goes against the fundamental design of Consul Template. CT is designed to be highly fault tolerant. You can look in the logs and see warnings for connection failures, but CT is designed to run in distributed systems where failure is inevitable. It will continuously retry until it can establish the connection to Vault/Consul."}}