{"action":"created","issue":{"url":"https://api.github.com/repos/rackerlabs/blueflood/issues/765","repository_url":"https://api.github.com/repos/rackerlabs/blueflood","labels_url":"https://api.github.com/repos/rackerlabs/blueflood/issues/765/labels{/name}","comments_url":"https://api.github.com/repos/rackerlabs/blueflood/issues/765/comments","events_url":"https://api.github.com/repos/rackerlabs/blueflood/issues/765/events","html_url":"https://github.com/rackerlabs/blueflood/issues/765","id":195289803,"number":765,"title":"graphite-api-blueflood-finder container timing out (empty response)","user":{"login":"beerdedfellow","id":13444979,"avatar_url":"https://avatars.githubusercontent.com/u/13444979?v=3","gravatar_id":"","url":"https://api.github.com/users/beerdedfellow","html_url":"https://github.com/beerdedfellow","followers_url":"https://api.github.com/users/beerdedfellow/followers","following_url":"https://api.github.com/users/beerdedfellow/following{/other_user}","gists_url":"https://api.github.com/users/beerdedfellow/gists{/gist_id}","starred_url":"https://api.github.com/users/beerdedfellow/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/beerdedfellow/subscriptions","organizations_url":"https://api.github.com/users/beerdedfellow/orgs","repos_url":"https://api.github.com/users/beerdedfellow/repos","events_url":"https://api.github.com/users/beerdedfellow/events{/privacy}","received_events_url":"https://api.github.com/users/beerdedfellow/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-12-13T15:52:12Z","updated_at":"2016-12-13T20:50:44Z","closed_at":null,"body":"Hello Team,\r\n\r\nHave a very specific issue here, and was told you may be able to help! For context, I am working on statsd integration with Rackspace Metrics:\r\n\r\nhttps://support.rackspace.com/how-to/integrate-statsd-with-rackspace-metrics/\r\n\r\nI'm having no problem with this integration, and am successfully pumping statsd statistics into metrics intake. I can also confirm I can manually pull this data out via 'curl'. However, at this time we don't support visualization of this data through our \"intelligence\" product, and so the recommended method for visualization is grafana. I was given by the following article by the product manager to accomplish this:\r\n\r\nhttps://github.com/rackerlabs/blueflood/wiki/Grafana-3.0-with-Blueflood-datasource\r\n\r\nThe second step is to spin up a blueflood-finder+graphite-api container in order to use Rackspace metrics as the back end. It's within this container that I am having the issue:\r\n\r\nhttps://github.com/rackerlabs/docker-graphite-api-blueflood-finder\r\n\r\nAs previously mentioned, from within the container I can retrieve the metrics information, authenticating with the variables in the container environment:\r\n\r\n```\r\nroot@730791559fcc:/# env\r\nRAX_USERNAME=<scrubbed>\r\nHOSTNAME=730791559fcc\r\nGPG_KEY=<scrubbed>\r\nTENANT_ID=<scrubbed>\r\nPYTHON_VERSION=2.7.12\r\nPATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\r\nBLUEFLOOD_QUERY_URL=https://global.metrics.api.rackspacecloud.com\r\nPWD=/\r\nLANG=C.UTF-8\r\nPYTHON_PIP_VERSION=9.0.1\r\nSHLVL=1\r\nHOME=/root\r\nRAX_APIKEY=<scrubbed>\r\nGRAFANA_URLS=<scrubbed>\r\n_=/usr/bin/env\r\n\r\nroot@730791559fcc:/# curl -sX GET -H \"X-Auth-Token: $API_TOKEN\" -H \"Content-Type: application/json\" \"$BLUEFLOOD_QUERY_URL/v2.0/<scrubbed>/metrics/search?query=*\" |python -m json.tool\r\n[\r\n    {\r\n        \"metric\": \"statsd.bad_lines_seen\"\r\n    },\r\n    {\r\n        \"metric\": \"sample.gauge\"\r\n    },\r\n    {\r\n        \"metric\": \"statsd.packets_received\"\r\n    },\r\n    {\r\n        \"metric\": \"statsd.timestamp_lag\"\r\n    },\r\n    {\r\n        \"metric\": \"statsd.metrics_received\"\r\n    }\r\n]\r\n```\r\n\r\nHowever, when querying the graphite-api it times out with the following:\r\n```\r\nroot@730791559fcc:/# netstat -pntl\r\nActive Internet connections (only servers)\r\nProto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name\r\ntcp        0      0 0.0.0.0:8888            0.0.0.0:*               LISTEN      30/python2\r\nroot@730791559fcc:/# curl -vv \"http://localhost:8888/metrics/find?query=*\"\r\n* Hostname was NOT found in DNS cache\r\n*   Trying ::1...\r\n* connect to ::1 port 8888 failed: Connection refused\r\n*   Trying 127.0.0.1...\r\n* Connected to localhost (127.0.0.1) port 8888 (#0)\r\n> GET /metrics/find?query=* HTTP/1.1\r\n> User-Agent: curl/7.38.0\r\n> Host: localhost:8888\r\n> Accept: */*\r\n>\r\n* Empty reply from server\r\n* Connection #0 to host localhost left intact\r\ncurl: (52) Empty reply from server\r\n```\r\n\r\nI've enabled debug logging within the app, and captured the following during the request:\r\n```\r\nroot@730791559fcc:/# tail -fn0 /var/log/blueflood-graphite-finder.log\r\n2016-12-12 23:43:04 DEBUG    blueflood_finder TenantBluefloodFinder.query: *\r\n2016-12-12 23:43:04 INFO     blueflood_finder BluefloodClient.find_nodes_from_bf: <FindQuery: * from * until *>\r\n2016-12-12 23:43:36 DEBUG    blueflood_finder Blueflood Finder v31\r\n2016-12-12 23:43:36 DEBUG    blueflood_finder BF enum thread started:\r\n2016-12-12 23:43:36 DEBUG    blueflood_finder BF finder submetrics enabled: False\r\n```\r\n\r\ngunicorn only logs the following, with nothing in the access log:\r\n```\r\nroot@730791559fcc:/# tail /var/log/gunicorn-error.log\r\n[2016-12-12 23:31:31 +0000] [30] [CRITICAL] WORKER TIMEOUT (pid:88)\r\n[2016-12-12 23:31:31 +0000] [94] [INFO] Booting worker with pid: 94\r\n```\r\n\r\nPlease let me know if there is any other information I can provide, and thank you in advance for your help!"},"comment":{"url":"https://api.github.com/repos/rackerlabs/blueflood/issues/comments/266857981","html_url":"https://github.com/rackerlabs/blueflood/issues/765#issuecomment-266857981","issue_url":"https://api.github.com/repos/rackerlabs/blueflood/issues/765","id":266857981,"user":{"login":"beerdedfellow","id":13444979,"avatar_url":"https://avatars.githubusercontent.com/u/13444979?v=3","gravatar_id":"","url":"https://api.github.com/users/beerdedfellow","html_url":"https://github.com/beerdedfellow","followers_url":"https://api.github.com/users/beerdedfellow/followers","following_url":"https://api.github.com/users/beerdedfellow/following{/other_user}","gists_url":"https://api.github.com/users/beerdedfellow/gists{/gist_id}","starred_url":"https://api.github.com/users/beerdedfellow/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/beerdedfellow/subscriptions","organizations_url":"https://api.github.com/users/beerdedfellow/orgs","repos_url":"https://api.github.com/users/beerdedfellow/repos","events_url":"https://api.github.com/users/beerdedfellow/events{/privacy}","received_events_url":"https://api.github.com/users/beerdedfellow/received_events","type":"User","site_admin":false},"created_at":"2016-12-13T20:50:44Z","updated_at":"2016-12-13T20:50:44Z","body":"Thanks Shinta!\n\n \n\nI’ll hop into the channel to make troubleshooting efforts easier. \n\n \n\n--\n\nRyan Hutchison, RHCE/CCNA\n\nEnterprise Support Architect\n\nRackspace Hosting\n\nDirect: (210) 312-4892\n\nMobile: (210) 452-4349\n\n \n\n \n\nFrom: Shinta Smith <notifications@github.com>\nReply-To: rackerlabs/blueflood <reply@reply.github.com>\nDate: Tuesday, December 13, 2016 at 2:25 PM\nTo: rackerlabs/blueflood <blueflood@noreply.github.com>\nCc: Ryan Hutchison <ryan.hutchison@RACKSPACE.COM>, Mention <mention@noreply.github.com>\nSubject: Re: [rackerlabs/blueflood] graphite-api-blueflood-finder container timing out (empty response) (#765)\n\n \n\n@beerdedfellow let me look into this a bit more, maybe test it on my environment. I will get back to you. In the meantime, if you have access to internal irc, we are at #metrics, if you want to ping one of us.\n\n—\nYou are receiving this because you were mentioned.\nReply to this email directly, view it on GitHub, or mute the thread.\n\n"}}
{"actor":{"display_login":"beerdedfellow"}}