{"action":"closed","issue":{"url":"https://api.github.com/repos/basho/riak_kv/issues/452","repository_url":"https://api.github.com/repos/basho/riak_kv","labels_url":"https://api.github.com/repos/basho/riak_kv/issues/452/labels{/name}","comments_url":"https://api.github.com/repos/basho/riak_kv/issues/452/comments","events_url":"https://api.github.com/repos/basho/riak_kv/issues/452/events","html_url":"https://github.com/basho/riak_kv/issues/452","id":9297789,"number":452,"title":"Negative clock deskew can cause stats to fail","user":{"login":"angrycub","id":464492,"avatar_url":"https://avatars.githubusercontent.com/u/464492?v=3","gravatar_id":"","url":"https://api.github.com/users/angrycub","html_url":"https://github.com/angrycub","followers_url":"https://api.github.com/users/angrycub/followers","following_url":"https://api.github.com/users/angrycub/following{/other_user}","gists_url":"https://api.github.com/users/angrycub/gists{/gist_id}","starred_url":"https://api.github.com/users/angrycub/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/angrycub/subscriptions","organizations_url":"https://api.github.com/users/angrycub/orgs","repos_url":"https://api.github.com/users/angrycub/repos","events_url":"https://api.github.com/users/angrycub/events{/privacy}","received_events_url":"https://api.github.com/users/angrycub/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/basho/riak_kv/labels/Bug","name":"Bug","color":"FF0000"},{"url":"https://api.github.com/repos/basho/riak_kv/labels/Kanban-Backlog","name":"Kanban-Backlog","color":"FFFFFF"},{"url":"https://api.github.com/repos/basho/riak_kv/labels/Kanban-Cabal-Core","name":"Kanban-Cabal-Core","color":"000000"}],"state":"closed","locked":false,"assignee":{"login":"russelldb","id":56321,"avatar_url":"https://avatars.githubusercontent.com/u/56321?v=3","gravatar_id":"","url":"https://api.github.com/users/russelldb","html_url":"https://github.com/russelldb","followers_url":"https://api.github.com/users/russelldb/followers","following_url":"https://api.github.com/users/russelldb/following{/other_user}","gists_url":"https://api.github.com/users/russelldb/gists{/gist_id}","starred_url":"https://api.github.com/users/russelldb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/russelldb/subscriptions","organizations_url":"https://api.github.com/users/russelldb/orgs","repos_url":"https://api.github.com/users/russelldb/repos","events_url":"https://api.github.com/users/russelldb/events{/privacy}","received_events_url":"https://api.github.com/users/russelldb/received_events","type":"User","site_admin":false},"assignees":[{"login":"russelldb","id":56321,"avatar_url":"https://avatars.githubusercontent.com/u/56321?v=3","gravatar_id":"","url":"https://api.github.com/users/russelldb","html_url":"https://github.com/russelldb","followers_url":"https://api.github.com/users/russelldb/followers","following_url":"https://api.github.com/users/russelldb/following{/other_user}","gists_url":"https://api.github.com/users/russelldb/gists{/gist_id}","starred_url":"https://api.github.com/users/russelldb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/russelldb/subscriptions","organizations_url":"https://api.github.com/users/russelldb/orgs","repos_url":"https://api.github.com/users/russelldb/repos","events_url":"https://api.github.com/users/russelldb/events{/privacy}","received_events_url":"https://api.github.com/users/russelldb/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/basho/riak_kv/milestones/8","html_url":"https://github.com/basho/riak_kv/milestones/2.1","labels_url":"https://api.github.com/repos/basho/riak_kv/milestones/8/labels","id":605250,"number":8,"title":"2.1","description":"Riak 2.1","creator":{"login":"borshop","id":6749250,"avatar_url":"https://avatars.githubusercontent.com/u/6749250?v=3","gravatar_id":"","url":"https://api.github.com/users/borshop","html_url":"https://github.com/borshop","followers_url":"https://api.github.com/users/borshop/followers","following_url":"https://api.github.com/users/borshop/following{/other_user}","gists_url":"https://api.github.com/users/borshop/gists{/gist_id}","starred_url":"https://api.github.com/users/borshop/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/borshop/subscriptions","organizations_url":"https://api.github.com/users/borshop/orgs","repos_url":"https://api.github.com/users/borshop/repos","events_url":"https://api.github.com/users/borshop/events{/privacy}","received_events_url":"https://api.github.com/users/borshop/received_events","type":"User","site_admin":false},"open_issues":82,"closed_issues":11,"state":"open","created_at":"2014-03-21T17:42:11Z","updated_at":"2016-10-06T15:58:58Z","due_on":"2014-09-26T07:00:00Z","closed_at":null},"comments":4,"created_at":"2012-12-14T21:17:20Z","updated_at":"2016-10-06T15:58:58Z","closed_at":"2016-10-06T15:58:58Z","body":"Looking at `riak_kv_put_fsm:add_timing`, it uses `os:timestamp()` to provide time.  If NTP decides to set the clock backward to deskew, then `timer:now_diff(StageEnd, StageStart)` could return a negative. Which will cause `/stats` to return **500 - Internal Server Error**s until the negative falls out of the histogram window. \r\n\r\nSuggestion is to not capture negative timing statistics\r\n\r\nSteps to reproduce:\r\n - From `riak attach` run:\r\n\r\n ```\r\nfolsom_metrics:notify_existing_metric({riak_kv,node_put_fsm_time},-9999,histogram).`\r\n```\r\n\r\n - From a browser make a call to /stats\r\n\r\nResult:\r\n* Calls to /stats fail with a 500\r\n* The following output is logged on a call to /stats\r\n\r\n```\r\n15:44:13.936 [error] Error in process <0.5758.0> on node 'riak@127.0.0.1' with exit value: {badarith,[{bear,scan_values,2,[]},{bear,get_statistics,1,[]},{lists,keysort,2,[]}]}\r\n15:44:13.941 [error] webmachine error: path=\"/stats\"\r\n{error,{error,function_clause,[{proplists,delete,[disk,{error,{badarith,[{bear,scan_values,2,[]},{bear,get_statistics,1,[]},{lists,keysort,2,[]}]}}],[{file,\"proplists.erl\"},{line,362}]},{riak_kv_wm_stats,get_stats,0,[{file,\"src/riak_kv_wm_stats.erl\"},{line,88}]},{riak_kv_wm_stats,produce_body,2,[{file,\"src/riak_kv_wm_stats.erl\"},{line,77}]},{webmachine_resource,resource_call,3,[{file,\"src/webmachine_resource.erl\"},{line,169}]},{webmachine_resource,do,3,[{file,\"src/webmachine_resource.erl\"},{line,128}]},{webmachine_decision_core,resource_call,1,[{file,\"src/webmachine_decision_core.erl\"},{line,48}]},{webmachine_decision_core,decision,1,[{file,\"src/webmachine_decision_core.erl\"},{line,532}]},{webmachine_decision_core,handle_request,2,[{file,\"src/webmachine_decision_core.erl\"},{line,33}]}]}}\r\n```\r\n"}}
{"actor":{"display_login":"bsparrow435"}}