{"action":"created","issue":{"url":"https://api.github.com/repos/openspending/openspending/issues/1347","repository_url":"https://api.github.com/repos/openspending/openspending","labels_url":"https://api.github.com/repos/openspending/openspending/issues/1347/labels{/name}","comments_url":"https://api.github.com/repos/openspending/openspending/issues/1347/comments","events_url":"https://api.github.com/repos/openspending/openspending/issues/1347/events","html_url":"https://github.com/openspending/openspending/issues/1347","id":307579075,"number":1347,"title":"Redis persistence error intermittently  breaks os-api","user":{"login":"brew","id":3993,"avatar_url":"https://avatars1.githubusercontent.com/u/3993?v=4","gravatar_id":"","url":"https://api.github.com/users/brew","html_url":"https://github.com/brew","followers_url":"https://api.github.com/users/brew/followers","following_url":"https://api.github.com/users/brew/following{/other_user}","gists_url":"https://api.github.com/users/brew/gists{/gist_id}","starred_url":"https://api.github.com/users/brew/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brew/subscriptions","organizations_url":"https://api.github.com/users/brew/orgs","repos_url":"https://api.github.com/users/brew/repos","events_url":"https://api.github.com/users/brew/events{/privacy}","received_events_url":"https://api.github.com/users/brew/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-03-22T10:13:01Z","updated_at":"2018-03-28T08:11:28Z","closed_at":null,"author_association":"OWNER","body":"Redis is used as a key/value cache for API results by os-api (using the [os-api-cache](https://github.com/openspending/os-api-cache/) library).\r\n\r\nPeriodically, Redis is unable to persist RDB snapshots to disk and os-api is unable to server requests.\r\n\r\nWe can temporarily fix the issue by restarting the redis container.\r\n\r\nBy default, Redis will stop working if background saving of snapshots fail. This can be configured to allow Redis to continue working with:\r\n\r\n> $ redis-cli\r\n> config set stop-writes-on-bgsave-error no\r\n\r\nAlthough this doesn't address the root cause.\r\n\r\nhttps://sentry.io/open-knowledge/openspending-production/issues/435780505/\r\n\r\n```\r\nResponseError: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.\r\n(10 additional frame(s) were not displayed)\r\n...\r\n  File \"os_api/cache.py\", line 34, in return_cached\r\n    response = cache.get_from_cache(package_id, o.query, o.path)\r\n  File \"os_api_cache/os_cache.py\", line 19, in get_from_cache\r\n    return super(OSCache, self).get(context, params)\r\n  File \"os_api_cache/base_cache.py\", line 14, in get\r\n    pickled = self._get(context, params)\r\n  File \"os_api_cache/redis_cache.py\", line 29, in _get\r\n    self.redis_connection.expire(key, self.timeout())\r\n\r\nResponseError: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.\r\n```"},"comment":{"url":"https://api.github.com/repos/openspending/openspending/issues/comments/376799958","html_url":"https://github.com/openspending/openspending/issues/1347#issuecomment-376799958","issue_url":"https://api.github.com/repos/openspending/openspending/issues/1347","id":376799958,"user":{"login":"brew","id":3993,"avatar_url":"https://avatars1.githubusercontent.com/u/3993?v=4","gravatar_id":"","url":"https://api.github.com/users/brew","html_url":"https://github.com/brew","followers_url":"https://api.github.com/users/brew/followers","following_url":"https://api.github.com/users/brew/following{/other_user}","gists_url":"https://api.github.com/users/brew/gists{/gist_id}","starred_url":"https://api.github.com/users/brew/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brew/subscriptions","organizations_url":"https://api.github.com/users/brew/orgs","repos_url":"https://api.github.com/users/brew/repos","events_url":"https://api.github.com/users/brew/events{/privacy}","received_events_url":"https://api.github.com/users/brew/received_events","type":"User","site_admin":false},"created_at":"2018-03-28T08:11:28Z","updated_at":"2018-03-28T08:11:28Z","author_association":"OWNER","body":"We get this warning when rebooting the Redis server: \r\n\r\n`WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.`\r\n\r\nI'll look into having the Dockerfile set this flag."}}
{"actor":{"display_login":"brew"}}