{"action":"created","issue":{"url":"https://api.github.com/repos/basho/riak_kv/issues/1556","repository_url":"https://api.github.com/repos/basho/riak_kv","labels_url":"https://api.github.com/repos/basho/riak_kv/issues/1556/labels{/name}","comments_url":"https://api.github.com/repos/basho/riak_kv/issues/1556/comments","events_url":"https://api.github.com/repos/basho/riak_kv/issues/1556/events","html_url":"https://github.com/basho/riak_kv/issues/1556","id":193112138,"number":1556,"title":"riak_kv_http_cache hangs under load [JIRA: RIAK-3074]","user":{"login":"nerophon","id":4341829,"avatar_url":"https://avatars.githubusercontent.com/u/4341829?v=3","gravatar_id":"","url":"https://api.github.com/users/nerophon","html_url":"https://github.com/nerophon","followers_url":"https://api.github.com/users/nerophon/followers","following_url":"https://api.github.com/users/nerophon/following{/other_user}","gists_url":"https://api.github.com/users/nerophon/gists{/gist_id}","starred_url":"https://api.github.com/users/nerophon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nerophon/subscriptions","organizations_url":"https://api.github.com/users/nerophon/orgs","repos_url":"https://api.github.com/users/nerophon/repos","events_url":"https://api.github.com/users/nerophon/events{/privacy}","received_events_url":"https://api.github.com/users/nerophon/received_events","type":"User","site_admin":false},"labels":[{"id":234112,"url":"https://api.github.com/repos/basho/riak_kv/labels/Bug","name":"Bug","color":"FF0000","default":false},{"id":175391677,"url":"https://api.github.com/repos/basho/riak_kv/labels/JIRA:%20To%20Do","name":"JIRA: To Do","color":"ededed","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-12-02T13:07:11Z","updated_at":"2016-12-06T21:09:30Z","closed_at":null,"body":"RiakKV stats can become stuck under high load.\r\n\r\nPolling persistently fails as follows:\r\n\r\n```\r\n<html><head><title>500 Internal Server Error</title></head><body><h1>Internal Server Error</h1>The server encountered an error while processing this request:<br><pre>{error, \r\n{exit, \r\n{timeout,{gen_server,call,[riak_kv_http_cache,get_stats]}}, \r\n[{gen_server,call,2,[{file,\"gen_server.erl\"},{line,180}]}, \r\n{riak_kv_wm_stats,produce_body,2, \r\n[{file,\"src/riak_kv_wm_stats.erl\"},{line,77}]}, \r\n{webmachine_resource,resource_call,3, \r\n[{file,\"src/webmachine_resource.erl\"},{line,186}]}, \r\n{webmachine_resource,do,3, \r\n[{file,\"src/webmachine_resource.erl\"},{line,142}]}, \r\n{webmachine_decision_core,resource_call,1, \r\n[{file,\"src/webmachine_decision_core.erl\"},{line,48}]}, \r\n{webmachine_decision_core,decision,1, \r\n[{file,\"src/webmachine_decision_core.erl\"},{line,558}]}, \r\n{webmachine_decision_core,handle_request,2, \r\n[{file,\"src/webmachine_decision_core.erl\"},{line,33}]}, \r\n{webmachine_mochiweb,loop,2, \r\n[{file,\"src/webmachine_mochiweb.erl\"},{line,74}]}]}}</pre><P><HR><ADDRESS>mochiweb+webmachine web server</ADDRESS></body></html>\r\n```\r\n\r\nThis issue has been reproduced by a customer in RiakKV `2.0.7` and by the client services team in RiakKV `2.2.0`. The customer sees it under high production load. We also see it under high test load, specifically a combination of:\r\n\r\n+ upgrade to `2.2.0`\r\n+ AAE tree rebuilds and/or Solr re-indexing\r\n+ disk schedulers (accidentally) set to cfq\r\n\r\nDuring this test we noticed that responses throughout the cluster were slow and measured constrained disk I/O. We consider these elements potentially causal, but not consequential.\r\n\r\nThere is a workaround to \"unstick\" stats as follows:\r\n\r\n```\r\nexit(erlang:whereis(riak_kv_http_cache), kill).\r\n```\r\n\r\nThis has been used successfully by us and by the customer. It solves the problem temporarily, but continuing high load causes the issue to recur. At the customer, the frequency of recurrence was five times per day on average.\r\n"},"comment":{"url":"https://api.github.com/repos/basho/riak_kv/issues/comments/265272982","html_url":"https://github.com/basho/riak_kv/issues/1556#issuecomment-265272982","issue_url":"https://api.github.com/repos/basho/riak_kv/issues/1556","id":265272982,"user":{"login":"Basho-JIRA","id":9125383,"avatar_url":"https://avatars.githubusercontent.com/u/9125383?v=3","gravatar_id":"","url":"https://api.github.com/users/Basho-JIRA","html_url":"https://github.com/Basho-JIRA","followers_url":"https://api.github.com/users/Basho-JIRA/followers","following_url":"https://api.github.com/users/Basho-JIRA/following{/other_user}","gists_url":"https://api.github.com/users/Basho-JIRA/gists{/gist_id}","starred_url":"https://api.github.com/users/Basho-JIRA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Basho-JIRA/subscriptions","organizations_url":"https://api.github.com/users/Basho-JIRA/orgs","repos_url":"https://api.github.com/users/Basho-JIRA/repos","events_url":"https://api.github.com/users/Basho-JIRA/events{/privacy}","received_events_url":"https://api.github.com/users/Basho-JIRA/received_events","type":"User","site_admin":false},"created_at":"2016-12-06T21:09:29Z","updated_at":"2016-12-06T21:09:29Z","body":"This kind of timeout error is generally caused trying to get vnode statuses, which calls out to each vnode to get a few data points:\r\n\r\n1) per_key_epoch information (Counter State for the vnode)\r\n2) Backend status\r\n3) Vnode ID\r\n\r\nOriginally, we thought this was just to get the backend status/eleveldb-specific stats that could have been done a different way, but it ends up there's more than just backend status that comes out of this call. The problem is it's very likely that this will time out in a heavily loaded system, and we don't handle the timeout case in a reasonable way when we cal riak_kv_status:vnode_status/0, which then causes the cascade of failures to end up where you see the error above. We should handle a timeout on vnode status more gracefully and somehow return _other_ stats, just not the vnode stats. \n\n***[posted via JIRA by Douglas Rohrer]***"}}
{"actor":{"display_login":"Basho-JIRA"}}