{"action":"created","issue":{"url":"https://api.github.com/repos/lomik/go-carbon/issues/63","repository_url":"https://api.github.com/repos/lomik/go-carbon","labels_url":"https://api.github.com/repos/lomik/go-carbon/issues/63/labels{/name}","comments_url":"https://api.github.com/repos/lomik/go-carbon/issues/63/comments","events_url":"https://api.github.com/repos/lomik/go-carbon/issues/63/events","html_url":"https://github.com/lomik/go-carbon/issues/63","id":164049036,"number":63,"title":"Datapoints (for new metric) not flushed to disk, but kept in cache memory","user":{"login":"rtoma","id":2914051,"avatar_url":"https://avatars.githubusercontent.com/u/2914051?v=3","gravatar_id":"","url":"https://api.github.com/users/rtoma","html_url":"https://github.com/rtoma","followers_url":"https://api.github.com/users/rtoma/followers","following_url":"https://api.github.com/users/rtoma/following{/other_user}","gists_url":"https://api.github.com/users/rtoma/gists{/gist_id}","starred_url":"https://api.github.com/users/rtoma/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rtoma/subscriptions","organizations_url":"https://api.github.com/users/rtoma/orgs","repos_url":"https://api.github.com/users/rtoma/repos","events_url":"https://api.github.com/users/rtoma/events{/privacy}","received_events_url":"https://api.github.com/users/rtoma/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-07-06T10:49:37Z","updated_at":"2016-07-09T11:32:39Z","closed_at":null,"body":"Curious case.\r\n\r\nOur setup:\r\n- 2 nodes\r\n- node1 runs go-carbon\r\n- node2 runs 12 carbon-cache daemons\r\n- carbon-c-relay on both nodes sending each line received to go-carbon on node1 and one of the carbons on node2\r\n\r\nExpected: so each node should have the same whisperfiles\r\n\r\nIn reality: for 99.99% of all metrics its true, but we've found an edge case.\r\n\r\nWhat happens:\r\n- User sends a new metric with sporadic datapoints\r\n- Points have been flushed to disk by carbon-cache and whisper fils have been created\r\n- Go-carbon hasn't flushed them yet, so no whisper files exist\r\n\r\nIt seems go-carbon is holding on to the datapoints in cache memory.\r\nCarbonlink query on go-carbon returns this:\r\n```\r\n[(1467718603, 0.0),\r\n(1467720962, 5.0),\r\n(1467722489, 5.0),\r\n(1467724363, 5.0),\r\n(1467726788, 5.0),\r\n(1467727826, 5.0),\r\n(1467790086, 2106.0)]\r\n```\r\nThat 1st timestamp is 22 hours ago.\r\n\r\n\r\nUsing this script to query:\r\n```\r\n#!/usr/bin/env python\r\nimport sys\r\nsys.path.append('/opt/graphite/webapp')\r\n\r\nimport os\r\nos.environ.setdefault(\"DJANGO_SETTINGS_MODULE\", \"graphite.settings\")\r\nfrom graphite import settings\r\nsettings.LOG_DIR = '.'\r\ndel settings.ADMIN_MEDIA_PREFIX\r\ndel settings.CACHE_BACKEND\r\n\r\nfrom graphite.render import datalib as dl\r\n\r\ndef query(metric):\r\n  hosts = []\r\n  for e in settings.CARBONLINK_HOSTS:\r\n    host, port, instance = e.split(':')\r\n    hosts.append((host, int(port), instance))\r\n\r\n  p = dl.CarbonLinkPool(hosts, 1.0)\r\n  print p.query(metric)\r\n\r\nfor m in sys.argv[1:]:\r\n  query(m)\r\n```"},"comment":{"url":"https://api.github.com/repos/lomik/go-carbon/issues/comments/231529882","html_url":"https://github.com/lomik/go-carbon/issues/63#issuecomment-231529882","issue_url":"https://api.github.com/repos/lomik/go-carbon/issues/63","id":231529882,"user":{"login":"AlexAkulov","id":7137047,"avatar_url":"https://avatars.githubusercontent.com/u/7137047?v=3","gravatar_id":"","url":"https://api.github.com/users/AlexAkulov","html_url":"https://github.com/AlexAkulov","followers_url":"https://api.github.com/users/AlexAkulov/followers","following_url":"https://api.github.com/users/AlexAkulov/following{/other_user}","gists_url":"https://api.github.com/users/AlexAkulov/gists{/gist_id}","starred_url":"https://api.github.com/users/AlexAkulov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AlexAkulov/subscriptions","organizations_url":"https://api.github.com/users/AlexAkulov/orgs","repos_url":"https://api.github.com/users/AlexAkulov/repos","events_url":"https://api.github.com/users/AlexAkulov/events{/privacy}","received_events_url":"https://api.github.com/users/AlexAkulov/received_events","type":"User","site_admin":false},"created_at":"2016-07-09T11:32:39Z","updated_at":"2016-07-09T11:32:39Z","body":"Use this settings\r\n```\r\n[cache]\r\nwrite-strategy = \"sorted\"\r\n```"}}
{"actor":{"display_login":"AlexAkulov"}}