{"action":"created","issue":{"url":"https://api.github.com/repos/splunk/splunk-connect-for-kubernetes/issues/50","repository_url":"https://api.github.com/repos/splunk/splunk-connect-for-kubernetes","labels_url":"https://api.github.com/repos/splunk/splunk-connect-for-kubernetes/issues/50/labels{/name}","comments_url":"https://api.github.com/repos/splunk/splunk-connect-for-kubernetes/issues/50/comments","events_url":"https://api.github.com/repos/splunk/splunk-connect-for-kubernetes/issues/50/events","html_url":"https://github.com/splunk/splunk-connect-for-kubernetes/issues/50","id":383580645,"node_id":"MDU6SXNzdWUzODM1ODA2NDU=","number":50,"title":"splunk-heapster cannot connect to splunk-fluentd-heapster","user":{"login":"voki","id":6574785,"node_id":"MDQ6VXNlcjY1NzQ3ODU=","avatar_url":"https://avatars0.githubusercontent.com/u/6574785?v=4","gravatar_id":"","url":"https://api.github.com/users/voki","html_url":"https://github.com/voki","followers_url":"https://api.github.com/users/voki/followers","following_url":"https://api.github.com/users/voki/following{/other_user}","gists_url":"https://api.github.com/users/voki/gists{/gist_id}","starred_url":"https://api.github.com/users/voki/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/voki/subscriptions","organizations_url":"https://api.github.com/users/voki/orgs","repos_url":"https://api.github.com/users/voki/repos","events_url":"https://api.github.com/users/voki/events{/privacy}","received_events_url":"https://api.github.com/users/voki/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-11-22T15:26:52Z","updated_at":"2018-11-22T18:16:29Z","closed_at":null,"author_association":"NONE","body":"Hi,\r\n\r\nI am getting error in metrics pod, looks like one container is unable to connect to other inside same pod.\r\nI have installed connector from Helm chart, version 1.0.1.\r\n\r\nsplunk-heapser container logs:\r\nI1122 11:53:58.124170       1 heapster.go:78] /heapster --source=kubernetes --sink=statsd:udp://127.0.0.1:9001\r\nI1122 11:53:58.124207       1 heapster.go:79] Heapster version v1.5.1\r\nI1122 11:53:58.124437       1 configs.go:61] Using Kubernetes client with master \"https://10.96.0.1:443\" and version v1\r\nI1122 11:53:58.124449       1 configs.go:62] Using kubelet port 10255\r\nI1122 11:53:58.140256       1 driver.go:104] statsd metrics sink using configuration : {host:127.0.0.1:9001 prefix: numMetricsPerMsg:5 protocolType:etsystatsd renameLabels:map[] allowedLabels:map[] customizeLabel:0x15f2560}\r\nI1122 11:53:58.140281       1 driver.go:104] statsd metrics sink using configuration : {host:127.0.0.1:9001 prefix: numMetricsPerMsg:5 protocolType:etsystatsd renameLabels:map[] allowedLabels:map[] customizeLabel:0x15f2560}\r\nI1122 11:53:58.140301       1 heapster.go:202] Starting with StatsD Sink\r\nI1122 11:53:58.140305       1 heapster.go:202] Starting with Metric Sink\r\nI1122 11:53:58.328875       1 heapster.go:112] Starting heapster on port 8082\r\nE1122 11:54:06.526301       1 driver.go:159] **statsd metrics sink - failed to send some metrics : write udp 127.0.0.1:38104->127.0.0.1:9001: write: connection refused**\r\n\r\nsplunk-fluentd-heapster container logs, looks like it is listening:\r\n2018-11-22 11:54:02 +0000 [info]: starting fluentd-1.2.0 pid=8 ruby=\"2.5.1\"\r\n2018-11-22 11:54:02 +0000 [info]: spawn command to main:  cmdline=[\"/usr/local/bin/ruby\", \"-Eascii-8bit:ascii-8bit\", \"/usr/local/bundle/bin/fluentd\", \"-c\", \"/fluentd/etc/fluent.conf\", \"--under-supervisor\"]\r\n2018-11-22 11:54:06 +0000 [info]: gem 'fluent-plugin-concat' version '2.2.2'\r\n2018-11-22 11:54:06 +0000 [info]: gem 'fluent-plugin-jq' version '0.5.1'\r\n2018-11-22 11:54:06 +0000 [info]: gem 'fluent-plugin-prometheus' version '1.0.1'\r\n2018-11-22 11:54:06 +0000 [info]: gem 'fluent-plugin-splunk-hec' version '1.0.1'\r\n2018-11-22 11:54:06 +0000 [info]: gem 'fluent-plugin-systemd' version '0.3.1'\r\n2018-11-22 11:54:06 +0000 [info]: gem 'fluentd' version '1.2.0'\r\n2018-11-22 11:54:06 +0000 [info]: adding match pattern=\"raw.metrics.udp\" type=\"jq\"\r\n2018-11-22 11:54:06 +0000 [info]: adding match pattern=\"metrics.udp\" type=\"splunk_hec\"\r\n2018-11-22 11:54:07 +0000 [info]: adding source type=\"udp\"\r\n2018-11-22 11:54:07 +0000 [info]: #0 starting fluentd worker pid=16 ppid=8 worker=0\r\n2018-11-22 11:54:07 +0000 [info]: #0 listening udp socket bind=\"0.0.0.0\" port=9001\r\n2018-11-22 11:54:07 +0000 [info]: #0 fluentd worker is now running worker=0\r\n\r\nAny idea what might be wrong?\r\n\r\nThanks."},"comment":{"url":"https://api.github.com/repos/splunk/splunk-connect-for-kubernetes/issues/comments/441098591","html_url":"https://github.com/splunk/splunk-connect-for-kubernetes/issues/50#issuecomment-441098591","issue_url":"https://api.github.com/repos/splunk/splunk-connect-for-kubernetes/issues/50","id":441098591,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0MTA5ODU5MQ==","user":{"login":"matthewmodestino","id":31549209,"node_id":"MDQ6VXNlcjMxNTQ5MjA5","avatar_url":"https://avatars3.githubusercontent.com/u/31549209?v=4","gravatar_id":"","url":"https://api.github.com/users/matthewmodestino","html_url":"https://github.com/matthewmodestino","followers_url":"https://api.github.com/users/matthewmodestino/followers","following_url":"https://api.github.com/users/matthewmodestino/following{/other_user}","gists_url":"https://api.github.com/users/matthewmodestino/gists{/gist_id}","starred_url":"https://api.github.com/users/matthewmodestino/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/matthewmodestino/subscriptions","organizations_url":"https://api.github.com/users/matthewmodestino/orgs","repos_url":"https://api.github.com/users/matthewmodestino/repos","events_url":"https://api.github.com/users/matthewmodestino/events{/privacy}","received_events_url":"https://api.github.com/users/matthewmodestino/received_events","type":"User","site_admin":false},"created_at":"2018-11-22T18:16:29Z","updated_at":"2018-11-22T18:16:29Z","author_association":"COLLABORATOR","body":"Hi voki!\r\n\r\nThe metrics pod contains two containers, one for heapster, the other for fluentd. \r\n\r\nLooks like the heapster container cant reach fluent on 9001?"}}
{"actor":{"display_login":"matthewmodestino"}}