{"action":"created","issue":{"url":"https://api.github.com/repos/tagomoris/fluent-plugin-secure-forward/issues/42","repository_url":"https://api.github.com/repos/tagomoris/fluent-plugin-secure-forward","labels_url":"https://api.github.com/repos/tagomoris/fluent-plugin-secure-forward/issues/42/labels{/name}","comments_url":"https://api.github.com/repos/tagomoris/fluent-plugin-secure-forward/issues/42/comments","events_url":"https://api.github.com/repos/tagomoris/fluent-plugin-secure-forward/issues/42/events","html_url":"https://github.com/tagomoris/fluent-plugin-secure-forward/issues/42","id":166673653,"number":42,"title":"Multiple destinations with out_copy - cert verification fails","user":{"login":"l-53","id":20565334,"avatar_url":"https://avatars.githubusercontent.com/u/20565334?v=3","gravatar_id":"","url":"https://api.github.com/users/l-53","html_url":"https://github.com/l-53","followers_url":"https://api.github.com/users/l-53/followers","following_url":"https://api.github.com/users/l-53/following{/other_user}","gists_url":"https://api.github.com/users/l-53/gists{/gist_id}","starred_url":"https://api.github.com/users/l-53/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/l-53/subscriptions","organizations_url":"https://api.github.com/users/l-53/orgs","repos_url":"https://api.github.com/users/l-53/repos","events_url":"https://api.github.com/users/l-53/events{/privacy}","received_events_url":"https://api.github.com/users/l-53/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2016-07-20T20:25:22Z","updated_at":"2016-07-29T23:12:58Z","closed_at":null,"body":"I couldn't find any previous posts about the issue I'm experiencing, so I'm hoping to find out if it's a bug or a PEBKAC situation.\r\n\r\nI'm attempting to securely forward logs in the following manner:\r\n1. Logs hit internal server \"syslog01\" using in_syslog and get tagged\r\n2. Logs are forwarded from \"syslog01\" to \"fluentd01\" using out_secure\r\n3. On \"fluentd01\" I have a match statement using out_copy, with <store> statements for the following destinations:\r\n  a. One copy is being stored locally for debug purposes until everything works as intended\r\n  b. One copy is being forwarded to \"splunkfwd01\" via out_secure\r\n  c. One copy is supposed to be forwarded to \"efk01\" via out_secure\r\n\r\nThe issue that I'm running into is that if I have both of the out_secure destinations in the config file at step 3, only the first one is able to establish the SSL connection. The second one errors out by failing the SSL verification.\r\n\r\nIf I comment out one of the 2 out_secure destinations and forward the logs only to one destination at a time (either \"splunkfwd01\" or \"efk01\") the logs are forwarded successfully. This tells me my certs/shared secret/passphrase are accurate for either combination of fluentd01->splunkfwd01 or fluentd01->efk01. I am using a separate cert/key pair for each connection. \r\n\r\n`2016-07-20 15:44:12 -0400 [warn]: failed to establish SSL connection error_class=OpenSSL::SSL::SSLError error=#<OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=error: certificate verify failed> host=\"10.10.10.54\" address=\"10.10.10.54\" port=24285`\r\n\r\n(I tried using a different port for \"efk01\", 24285, as a troubleshooting step)\r\n\r\nTo sum it up, the behavior I'm seeing is that for a given out_copy match I can only use one out_secure store at a time. \r\n\r\nAm I just missing something blatantly obvious? \r\n\r\nEdit: \"fluentd01\" is running fluentd 0.12.20, while \"splunkfwd01 and \"efk01\" are running fluentd 0.12.26. All servers are running 'fluent-plugin-secure-forward' version '0.4.2'.\r\nEdit2: The certificates were generated using 'secure-forward-ca-generate' from this plugin. They work fine with a single connection from either fluentd01->splunkfwd01, or for a single connection from fluentd01->efk01. The issue only occurs when I try to forward traffic to both splunkfwd01 and efk01 under the same match statement with out_copy. "},"comment":{"url":"https://api.github.com/repos/tagomoris/fluent-plugin-secure-forward/issues/comments/236316191","html_url":"https://github.com/tagomoris/fluent-plugin-secure-forward/issues/42#issuecomment-236316191","issue_url":"https://api.github.com/repos/tagomoris/fluent-plugin-secure-forward/issues/42","id":236316191,"user":{"login":"l-53","id":20565334,"avatar_url":"https://avatars.githubusercontent.com/u/20565334?v=3","gravatar_id":"","url":"https://api.github.com/users/l-53","html_url":"https://github.com/l-53","followers_url":"https://api.github.com/users/l-53/followers","following_url":"https://api.github.com/users/l-53/following{/other_user}","gists_url":"https://api.github.com/users/l-53/gists{/gist_id}","starred_url":"https://api.github.com/users/l-53/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/l-53/subscriptions","organizations_url":"https://api.github.com/users/l-53/orgs","repos_url":"https://api.github.com/users/l-53/repos","events_url":"https://api.github.com/users/l-53/events{/privacy}","received_events_url":"https://api.github.com/users/l-53/received_events","type":"User","site_admin":false},"created_at":"2016-07-29T23:12:58Z","updated_at":"2016-07-29T23:12:58Z","body":"No luck with the new certificates either. \r\n\r\nHere are my current destinations from fluentd01:\r\n1. 10.10.10.52 - influxdb01 (using your cacert1 cert/key/psk/passphrase)\r\n2. 10.10.10.54 - efk01 (using your cacert2 cert/key/psk/passphrase)\r\n\r\nSo far it seems that every time I reload td-agent, the ssl session is established to either influxdb01 or efk01, but only one server at a time while the other one fails with the same certificate verification error. \r\n\r\nIf I comment out the config for influxdb01 I have no issues connecting to efk01. \r\nIf I comment out the config for efk01 I have no issues connecting to influxdb01. \r\n\r\nThis tells me I didn't typo/mismatch the cert/psk/passphrases anywhere. The only notable differences I can spot between your test and my environment is that you tested on the same host rather than across servers, and your config uses strict verification. \r\n\r\nI've attached my relevant sanitized config. \r\n\r\n[efk01_sanitized.txt](https://github.com/tagomoris/fluent-plugin-secure-forward/files/391553/efk01_sanitized.txt)\r\n[fluentd01_sanitized.txt](https://github.com/tagomoris/fluent-plugin-secure-forward/files/391554/fluentd01_sanitized.txt)\r\n[influxdb01_sanitized.txt](https://github.com/tagomoris/fluent-plugin-secure-forward/files/391555/influxdb01_sanitized.txt)"}}
{"actor":{"display_login":"l-53"}}