{"action":"created","issue":{"url":"https://api.github.com/repos/petems/petems-hiera_vault/issues/25","repository_url":"https://api.github.com/repos/petems/petems-hiera_vault","labels_url":"https://api.github.com/repos/petems/petems-hiera_vault/issues/25/labels{/name}","comments_url":"https://api.github.com/repos/petems/petems-hiera_vault/issues/25/comments","events_url":"https://api.github.com/repos/petems/petems-hiera_vault/issues/25/events","html_url":"https://github.com/petems/petems-hiera_vault/issues/25","id":395317440,"node_id":"MDU6SXNzdWUzOTUzMTc0NDA=","number":25,"title":"Lots of CLOSE_WAITS to Vault","user":{"login":"joaquin386","id":14437027,"node_id":"MDQ6VXNlcjE0NDM3MDI3","avatar_url":"https://avatars0.githubusercontent.com/u/14437027?v=4","gravatar_id":"","url":"https://api.github.com/users/joaquin386","html_url":"https://github.com/joaquin386","followers_url":"https://api.github.com/users/joaquin386/followers","following_url":"https://api.github.com/users/joaquin386/following{/other_user}","gists_url":"https://api.github.com/users/joaquin386/gists{/gist_id}","starred_url":"https://api.github.com/users/joaquin386/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/joaquin386/subscriptions","organizations_url":"https://api.github.com/users/joaquin386/orgs","repos_url":"https://api.github.com/users/joaquin386/repos","events_url":"https://api.github.com/users/joaquin386/events{/privacy}","received_events_url":"https://api.github.com/users/joaquin386/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2019-01-02T17:13:36Z","updated_at":"2019-01-04T18:42:27Z","closed_at":null,"author_association":"NONE","body":"Hi,\r\n\r\nVersion: \r\n```\r\nPuppet: 6.1\r\nVault: 1.0.1\r\nGem Vault: 0.12.0\r\nHiera_vault: Latest repo version\r\n```\r\n\r\nInstall the hiera_vault and everything working fine. BUT there is a huge number of CLOSE_WAITS. Seems the function doesn't close the connection once it finish. Noticing as well that after removing hiera.yaml all ESTABLISH move to CLOSE_WAIT but the connection still not close.\r\n\r\nBefore removing hiera.yaml (after removingthe difference is that all connections are CLOSE_WAIT)\r\n10.100.10.101-> Puppetserver\r\n10.100.2.157 -> Vault\r\n```\r\nnetstat -an | grep 10.100.2.157\r\ntcp        0      0 10.100.10.101:39110     10.100.2.157:8200       ESTABLISHED\r\ntcp       54      0 10.100.10.101:37442     10.100.2.157:8200       CLOSE_WAIT \r\ntcp       54      0 10.100.10.101:38504     10.100.2.157:8200       CLOSE_WAIT \r\ntcp       54      0 10.100.10.101:38530     10.100.2.157:8200       CLOSE_WAIT \r\ntcp       54      0 10.100.10.101:38758     10.100.2.157:8200       CLOSE_WAIT \r\ntcp       54      0 10.100.10.101:38000     10.100.2.157:8200       CLOSE_WAIT \r\ntcp       54      0 10.100.10.101:38506     10.100.2.157:8200       CLOSE_WAIT \r\ntcp       54      0 10.100.10.101:38008     10.100.2.157:8200       CLOSE_WAIT \r\ntcp        0      0 10.100.10.101:39212     10.100.2.157:8200       ESTABLISHED\r\ntcp        0      0 10.100.10.101:39076     10.100.2.157:8200       ESTABLISHED\r\ntcp       54      0 10.100.10.101:37426     10.100.2.157:8200       CLOSE_WAIT \r\ntcp       54      0 10.100.10.101:38006     10.100.2.157:8200       CLOSE_WAIT \r\ntcp       54      0 10.100.10.101:37638     10.100.2.157:8200       CLOSE_WAIT \r\ntcp        0      0 10.100.10.101:39072     10.100.2.157:8200       ESTABLISHED\r\ntcp        0      0 10.100.10.101:39236     10.100.2.157:8200       ESTABLISHED\r\ntcp        0      0 10.100.10.101:39234     10.100.2.157:8200       ESTABLISHED\r\ntcp        0      0 10.100.10.101:39250     10.100.2.157:8200       ESTABLISHED\r\ntcp       54      0 10.100.10.101:37888     10.100.2.157:8200       CLOSE_WAIT \r\ntcp        0      0 10.100.10.101:39078     10.100.2.157:8200       ESTABLISHED\r\ntcp       54      0 10.100.10.101:38772     10.100.2.157:8200       CLOSE_WAIT \r\ntcp        0      0 10.100.10.101:39232     10.100.2.157:8200       ESTABLISHED\r\ntcp       54      0 10.100.10.101:38502     10.100.2.157:8200       CLOSE_WAIT \r\ntcp       54      0 10.100.10.101:38792     10.100.2.157:8200       CLOSE_WAIT \r\ntcp       54      0 10.100.10.101:37440     10.100.2.157:8200       CLOSE_WAIT \r\ntcp       54      0 10.100.10.101:37886     10.100.2.157:8200       CLOSE_WAIT \r\ntcp       54      0 10.100.10.101:37094     10.100.2.157:8200       CLOSE_WAIT \r\ntcp        0      0 10.100.10.101:39238     10.100.2.157:8200       ESTABLISHED\r\ntcp       54      0 10.100.10.101:37822     10.100.2.157:8200       CLOSE_WAIT \r\ntcp        0      0 10.100.10.101:39240     10.100.2.157:8200       ESTABLISHED\r\ntcp       54      0 10.100.10.101:37940     10.100.2.157:8200       CLOSE_WAIT \r\ntcp        0      0 10.100.10.101:39230     10.100.2.157:8200       ESTABLISHED\r\ntcp       54      0 10.100.10.101:37928     10.100.2.157:8200       CLOSE_WAIT \r\ntcp        0      0 10.100.10.101:39074     10.100.2.157:8200       ESTABLISHED\r\ntcp       54      0 10.100.10.101:37116     10.100.2.157:8200       CLOSE_WAIT \r\ntcp       54      0 10.100.10.101:37460     10.100.2.157:8200       CLOSE_WAIT \r\ntcp       54      0 10.100.10.101:38764     10.100.2.157:8200       CLOSE_WAIT \r\ntcp       54      0 10.100.10.101:37648     10.100.2.157:8200       CLOSE_WAIT \r\ntcp        0      0 10.100.10.101:39242     10.100.2.157:8200       ESTABLISHED\r\ntcp       54      0 10.100.10.101:37270     10.100.2.157:8200       CLOSE_WAIT \r\ntcp       54      0 10.100.10.101:38498     10.100.2.157:8200       CLOSE_WAIT \r\ntcp       54      0 10.100.10.101:38766     10.100.2.157:8200       CLOSE_WAIT \r\ntcp       54      0 10.100.10.101:37932     10.100.2.157:8200       CLOSE_WAIT \r\ntcp       54      0 10.100.10.101:38624     10.100.2.157:8200       CLOSE_WAIT \r\ntcp       54      0 10.100.10.101:38642     10.100.2.157:8200       CLOSE_WAIT \r\ntcp       54      0 10.100.10.101:37448     10.100.2.157:8200       CLOSE_WAIT \r\ntcp       54      0 10.100.10.101:38500     10.100.2.157:8200       CLOSE_WAIT \r\ntcp        0      0 10.100.10.101:39070     10.100.2.157:8200       ESTABLISHED\r\ntcp       54      0 10.100.10.101:37452     10.100.2.157:8200       CLOSE_WAIT \r\ntcp       54      0 10.100.10.101:38018     10.100.2.157:8200       CLOSE_WAIT \r\ntcp        0      0 10.100.10.101:39224     10.100.2.157:8200       ESTABLISHED\r\ntcp       54      0 10.100.10.101:37620     10.100.2.157:8200       CLOSE_WAIT \r\ntcp       54      0 10.100.10.101:37884     10.100.2.157:8200       CLOSE_WAIT \r\ntcp       54      0 10.100.10.101:38528     10.100.2.157:8200       CLOSE_WAIT \r\ntcp       54      0 10.100.10.101:38524     10.100.2.157:8200       CLOSE_WAIT \r\ntcp        0      0 10.100.10.101:39246     10.100.2.157:8200       ESTABLISHED\r\ntcp       54      0 10.100.10.101:37130     10.100.2.157:8200       CLOSE_WAIT \r\ntcp       54      0 10.100.10.101:37936     10.100.2.157:8200       CLOSE_WAIT \r\ntcp        0      0 10.100.10.101:39098     10.100.2.157:8200       ESTABLISHED\r\ntcp       54      0 10.100.10.101:38770     10.100.2.157:8200       CLOSE_WAIT \r\ntcp       54      0 10.100.10.101:38768     10.100.2.157:8200       CLOSE_WAIT \r\ntcp       54      0 10.100.10.101:37838     10.100.2.157:8200       CLOSE_WAIT \r\ntcp       54      0 10.100.10.101:38014     10.100.2.157:8200       CLOSE_WAIT \r\ntcp       54      0 10.100.10.101:38756     10.100.2.157:8200       CLOSE_WAIT \r\ntcp        0      0 10.100.10.101:39208     10.100.2.157:8200       ESTABLISHED\r\ntcp       54      0 10.100.10.101:37332     10.100.2.157:8200       CLOSE_WAIT \r\ntcp       54      0 10.100.10.101:37938     10.100.2.157:8200       CLOSE_WAIT \r\ntcp       54      0 10.100.10.101:37852     10.100.2.157:8200       CLOSE_WAIT \r\n```\r\nAnd here hiera\r\n```\r\n---\r\nversion: 5\r\nhierarchy:\r\n  - name: \"Hiera-vault lookup\"\r\n    lookup_key: hiera_vault\r\n    options:\r\n      confine_to_keys:\r\n        - '^vault_.*'\r\n        - '^.*_password$'\r\n        - '^password.*'\r\n      ssl_verify: true\r\n      ssl_ca_cert: /etc/pki/tls/cert.pem\r\n      address: https://vault.server.com:8200\r\n      token: blablabla\r\n      default_field: value\r\n      mounts:\r\n        secret:\r\n          - puppet/%{::trusted.certname}\r\n```\r\n\r\nCiting the Red Hat docs:\r\n```\r\nCLOSE_WAIT - Indicates that the server has received the first FIN signal from the client and the connection is in the process of being closed. This means the socket is waiting for the application to execute close(). A socket can be in CLOSE_WAIT state indefinitely until the application closes it. Faulty scenarios would be like a file descriptor leak: server not executing close() on sockets leading to pile up of CLOSE_WAIT sockets.\r\n```"},"comment":{"url":"https://api.github.com/repos/petems/petems-hiera_vault/issues/comments/451531224","html_url":"https://github.com/petems/petems-hiera_vault/issues/25#issuecomment-451531224","issue_url":"https://api.github.com/repos/petems/petems-hiera_vault/issues/25","id":451531224,"node_id":"MDEyOklzc3VlQ29tbWVudDQ1MTUzMTIyNA==","user":{"login":"joaquin386","id":14437027,"node_id":"MDQ6VXNlcjE0NDM3MDI3","avatar_url":"https://avatars0.githubusercontent.com/u/14437027?v=4","gravatar_id":"","url":"https://api.github.com/users/joaquin386","html_url":"https://github.com/joaquin386","followers_url":"https://api.github.com/users/joaquin386/followers","following_url":"https://api.github.com/users/joaquin386/following{/other_user}","gists_url":"https://api.github.com/users/joaquin386/gists{/gist_id}","starred_url":"https://api.github.com/users/joaquin386/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/joaquin386/subscriptions","organizations_url":"https://api.github.com/users/joaquin386/orgs","repos_url":"https://api.github.com/users/joaquin386/repos","events_url":"https://api.github.com/users/joaquin386/events{/privacy}","received_events_url":"https://api.github.com/users/joaquin386/received_events","type":"User","site_admin":false},"created_at":"2019-01-04T18:42:27Z","updated_at":"2019-01-04T18:42:27Z","author_association":"NONE","body":"```\r\n# netstat -anp | grep 10.100.2.15[7-9] | grep CLOSE | wc -l\r\n136\r\n# systemctl restart puppetserver\r\n# netstat -anp | grep 10.100.2.15[7-9] | grep CLOSE | wc -l\r\n0\r\n```\r\nRestarted puppetserver. Now CLOSE are 0. I have a three cluster vault node."}}
{"actor":{"display_login":"joaquin386"}}