{"action":"created","issue":{"url":"https://api.github.com/repos/graphite-ng/carbon-relay-ng/issues/104","repository_url":"https://api.github.com/repos/graphite-ng/carbon-relay-ng","labels_url":"https://api.github.com/repos/graphite-ng/carbon-relay-ng/issues/104/labels{/name}","comments_url":"https://api.github.com/repos/graphite-ng/carbon-relay-ng/issues/104/comments","events_url":"https://api.github.com/repos/graphite-ng/carbon-relay-ng/issues/104/events","html_url":"https://github.com/graphite-ng/carbon-relay-ng/issues/104","id":123380369,"number":104,"title":"Understanding Slow Connections","user":{"login":"maxwax","id":4734173,"avatar_url":"https://avatars.githubusercontent.com/u/4734173?v=3","gravatar_id":"","url":"https://api.github.com/users/maxwax","html_url":"https://github.com/maxwax","followers_url":"https://api.github.com/users/maxwax/followers","following_url":"https://api.github.com/users/maxwax/following{/other_user}","gists_url":"https://api.github.com/users/maxwax/gists{/gist_id}","starred_url":"https://api.github.com/users/maxwax/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/maxwax/subscriptions","organizations_url":"https://api.github.com/users/maxwax/orgs","repos_url":"https://api.github.com/users/maxwax/repos","events_url":"https://api.github.com/users/maxwax/events{/privacy}","received_events_url":"https://api.github.com/users/maxwax/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2015-12-21T23:21:29Z","updated_at":"2016-07-01T00:59:47Z","closed_at":null,"body":"I'm building a Graphite cluster to ingest > 700,000 metrics per minute.  It's working fairly well, but right now I'm seeing dropped metrics via 'action_is_drop.reason_is_slow_conn' stats.\r\n\r\n**What in carbon-relay-ng's perspective is *slow*? And how can I investigate further?**\r\n\r\n```\r\nreplicator -> sharder (node a, localhost) -> carbon-cache -> whisper\r\n           -> sharder (node b, ethernet) -> carbon-cache -> whisper\r\n```\r\n\r\nIn my architecture, on Node A, a 'replicator' carbon-relay-ng process receives from 500+ hosts and does nothing but forward to another carbon-relay-ng on Node A (via localhost) and an identical carbon-relay-ng on node B (via gigabit ethernet). Secondary carbon-relay-ng processes, 'sharders', then distribute the metrics to 16 carbon-cache processes.\r\n\r\nI see 50k metrics dropped on the replicator-to-sharder localhost a-to-a connection and 100k metrics dropped on the replicator-to-sharder ethernet a-to-b connection.\r\n\r\nI don't suspect network performance to be an issue, especially considering NodeA to NodeA is on the localhost device.\r\n\r\nIf I turn off replication from Node A to Node B, I still see issues with the metrics being relayed between the replicator carbon-relay-ng and the sharder carbon-relay-ng on Node A.\r\n\r\nThanks!\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/graphite-ng/carbon-relay-ng/issues/comments/229828996","html_url":"https://github.com/graphite-ng/carbon-relay-ng/issues/104#issuecomment-229828996","issue_url":"https://api.github.com/repos/graphite-ng/carbon-relay-ng/issues/104","id":229828996,"user":{"login":"Dieterbe","id":20774,"avatar_url":"https://avatars.githubusercontent.com/u/20774?v=3","gravatar_id":"","url":"https://api.github.com/users/Dieterbe","html_url":"https://github.com/Dieterbe","followers_url":"https://api.github.com/users/Dieterbe/followers","following_url":"https://api.github.com/users/Dieterbe/following{/other_user}","gists_url":"https://api.github.com/users/Dieterbe/gists{/gist_id}","starred_url":"https://api.github.com/users/Dieterbe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Dieterbe/subscriptions","organizations_url":"https://api.github.com/users/Dieterbe/orgs","repos_url":"https://api.github.com/users/Dieterbe/repos","events_url":"https://api.github.com/users/Dieterbe/events{/privacy}","received_events_url":"https://api.github.com/users/Dieterbe/received_events","type":"User","site_admin":false},"created_at":"2016-07-01T00:59:47Z","updated_at":"2016-07-01T00:59:47Z","body":"sorry, not much i can do to help right now (super busy with work). see my explanation above. perhaps something's wrong with the code. maybe somebody else can have a closer look"}}
{"actor":{"display_login":"Dieterbe"}}