{"action":"created","issue":{"url":"https://api.github.com/repos/EventStore/EventStore/issues/919","repository_url":"https://api.github.com/repos/EventStore/EventStore","labels_url":"https://api.github.com/repos/EventStore/EventStore/issues/919/labels{/name}","comments_url":"https://api.github.com/repos/EventStore/EventStore/issues/919/comments","events_url":"https://api.github.com/repos/EventStore/EventStore/issues/919/events","html_url":"https://github.com/EventStore/EventStore/issues/919","id":154656779,"number":919,"title":"cluster crash after 2 days","user":{"login":"suddenelfilio","id":6460448,"avatar_url":"https://avatars.githubusercontent.com/u/6460448?v=3","gravatar_id":"","url":"https://api.github.com/users/suddenelfilio","html_url":"https://github.com/suddenelfilio","followers_url":"https://api.github.com/users/suddenelfilio/followers","following_url":"https://api.github.com/users/suddenelfilio/following{/other_user}","gists_url":"https://api.github.com/users/suddenelfilio/gists{/gist_id}","starred_url":"https://api.github.com/users/suddenelfilio/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/suddenelfilio/subscriptions","organizations_url":"https://api.github.com/users/suddenelfilio/orgs","repos_url":"https://api.github.com/users/suddenelfilio/repos","events_url":"https://api.github.com/users/suddenelfilio/events{/privacy}","received_events_url":"https://api.github.com/users/suddenelfilio/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":15,"created_at":"2016-05-13T08:13:20Z","updated_at":"2016-05-13T12:06:36Z","closed_at":null,"body":"We are running a cluster on commodity servers on 6 servers each with 2 nodes and mostly default settings. After 2 days we keep getting issues with performance counters.\r\n\r\nat **first** we had: \r\n\r\n> System.InvalidOperationException: The Counter layout for the Category specified is invalid, a counter of the type:  AverageCount64, AverageTimer32, CounterMultiTimer, CounterMultiTimerInverse, CounterMultiTimer100Ns, CounterMultiTimer100NsInverse, RawFraction, or SampleFraction has to be immediately followed by any of the base counter types: AverageBase, CounterMultiBase, RawBase or SampleBase.\r\n   at System.Diagnostics.CategorySample.GetCounterDefinitionSample(String counter)\r\n   at System.Diagnostics.PerformanceCounter.NextSample()\r\n   at EventStore.Core.Services.Monitoring.Utils.PerfCounterHelper.GetGcStats() in c:\\projects\\eventstore\\src\\EventStore.Core\\Services\\Monitoring\\Utils\\PerfCounterHelper.cs:line 119\r\n   at EventStore.Core.Services.Monitoring.SystemStatsHelper.GetSystemStats() in c:\\projects\\eventstore\\src\\EventStore.Core\\Services\\Monitoring\\SystemStatsHelper.cs:line 66\r\n   at EventStore.Core.Services.Monitoring.MonitoringService.CollectStats() in c:\\projects\\eventstore\\src\\EventStore.Core\\Services\\Monitoring\\MonitoringService.cs:line 135\r\n\r\nWe solved this with lodctr \\R\r\n\r\nNow we have:\r\n\r\n> System.InvalidOperationException: Could not Read Category Index: 5908.\r\n>    at System.Diagnostics.CategorySample..ctor(Byte[] data, CategoryEntry entry, PerformanceCounterLib library)\r\n>    at System.Diagnostics.PerformanceCounterLib.GetCategorySample(String machine, String category)\r\n>    at System.Diagnostics.PerformanceCounter.NextSample()\r\n>    at EventStore.Core.Services.Monitoring.Utils.PerfCounterHelper.GetGcStats() in c:\\projects\\eventstore\\src\\EventStore.Core\\Services\\Monitoring\\Utils\\PerfCounterHelper.cs:line 119\r\n>    at EventStore.Core.Services.Monitoring.SystemStatsHelper.GetSystemStats() in c:\\projects\\eventstore\\src\\EventStore.Core\\Services\\Monitoring\\SystemStatsHelper.cs:line 66\r\n>    at EventStore.Core.Services.Monitoring.MonitoringService.CollectStats() in \r\nc:\\projects\\eventstore\\src\\EventStore.Core\\Services\\Monitoring\\MonitoringService.cs:line 135 \r\n\r\nas these errors are thrown consequently the reaction time of the nodes slows down and eventualy the node dies with message:\r\n\r\n> 0904:014 2016.05.13 06:40:36.551 ERROR GossipServiceBase   ] Time difference between us and [172.30.76.18:1112] is too great! UTC now: 2016-05-13 06:40:36.551, peer's time stamp: 2016-05-13 06:39:05.929.\r\n> [PID:10904:296 2016.05.13 06:41:32.676 ERROR QueuedHandlerThreadP] ---!!! VERY SLOW QUEUE MSG [Worker #2]: SendOverHttp - 7500ms. Q: 651/1369.\r\n> [PID:10904:014 2016.05.13 06:41:39.613 ERROR GossipServiceBase   ] Time difference between us and [172.30.78.12:1112] is too great! UTC now: 2016-05-13 06:41:39.613, peer's time stamp: 2016-05-13 06:40:31.294.\r\n> [PID:10904:160 2016.05.13 06:41:40.832 ERROR QueuedHandlerThreadP] ---!!! VERY SLOW QUEUE MSG [Worker #5]: SendOverHttp - 8578ms. Q: 994/1329.\r\n> [PID:10904:014 2016.05.13 06:41:42.769 ERROR GossipServiceBase   ] Time difference between us and [172.30.76.17:2112] is too great! UTC now: 2016-05-13 06:41:42.769, peer's time stamp: 2016-05-13 06:40:40.346.\r\n> [PID:10904:014 2016.05.13 06:41:44.144 ERROR GossipServiceBase   ] Time difference between us and [172.30.78.4:2112] is too great! UTC now: 2016-05-13 06:41:44.144, peer's time stamp: 2016-05-13 06:40:29.914.\r\n> [PID:10904:014 2016.05.13 06:41:44.160 ERROR GossipServiceBase   ] Time difference between us and [172.30.78.4:2112] is too great! UTC now: 2016-05-13 06:41:44.160, peer's time stamp: 2016-05-13 06:40:37.976.\r\n> [PID:10904:014 2016.05.13 06:41:44.363 ERROR GossipServiceBase   ] Time difference between us and [172.30.76.17:2112] is too great! UTC now: 2016-05-13 06:41:44.363, peer's time stamp: 2016-05-13 06:40:27.206.\r\n> [PID:10904:014 2016.05.13 06:41:44.879 ERROR GossipServiceBase   ] Time difference between us and [172.30.78.12:1112] is too great! UTC now: 2016-05-13 06:41:44.879, peer's time stamp: 2016-05-13 06:40:43.420.\r\n> [PID:10904:014 2016.05.13 06:41:45.207 ERROR GossipServiceBase   ] Time difference between us and [172.30.76.17:1112] is too great! UTC now: 2016-05-13 06:41:45.207, peer's time stamp: 2016-05-13 06:40:26.439.\r\n> [PID:10904:014 2016.05.13 06:41:45.394 ERROR GossipServiceBase   ] Time difference between us and [172.30.76.17:3112] is too great! UTC now: 2016-05-13 06:41:45.394, peer's time stamp: 2016-05-13 06:40:35.611.\r\n> [PID:10904:014 2016.05.13 06:41:45.613 ERROR GossipServiceBase   ] Time difference between us and [172.30.78.13:2112] is too great! UTC now: 2016-05-13 06:41:45.613, peer's time stamp: 2016-05-13 06:40:44.861.\r\n> [PID:10904:014 2016.05.13 06:41:48.097 ERROR GossipServiceBase   ] Time difference between us and [172.30.76.17:3112] is too great! UTC now: 2016-05-13 06:41:48.097, peer's time stamp: 2016-05-13 06:40:31.549.\r\n> [PID:10904:014 2016.05.13 06:41:48.754 ERROR GossipServiceBase   ] Time difference between us and [172.30.76.17:2112] is too great! UTC now: 2016-05-13 06:41:48.754, peer's time stamp: 2016-05-13 06:40:43.377.\r\n> [PID:10904:014 2016.05.13 06:41:51.097 ERROR GossipServiceBase   ] Time difference between us and [172.30.78.13:1112] is too great! UTC now: 2016-05-13 06:41:51.097, peer's time stamp: 2016-05-13 06:40:47.283.\r\n> [PID:10904:014 2016.05.13 06:42:08.738 ERROR GossipServiceBase   ] Time difference between us and [172.30.78.4:1112] is too great! UTC now: 2016-05-13 06:42:08.738, peer's time stamp: 2016-05-13 06:40:39.335.\r\n> [PID:10904:014 2016.05.13 06:42:08.925 ERROR GossipServiceBase   ] Time difference between us and [172.30.78.13:2112] is too great! UTC now: 2016-05-13 06:42:08.925, peer's time stamp: 2016-05-13 06:41:08.251.\r\n> [PID:10904:276 2016.05.13 06:42:17.628 ERROR MonitoringService   ] Error while collecting stats\r\n\r\nWe are currently at the point to abandon eventstore as we don't have a lot of time to concern us about these issues, but since I really like this product I want to give it another try.\r\n\r\nany advise?"},"comment":{"url":"https://api.github.com/repos/EventStore/EventStore/issues/comments/219025361","html_url":"https://github.com/EventStore/EventStore/issues/919#issuecomment-219025361","issue_url":"https://api.github.com/repos/EventStore/EventStore/issues/919","id":219025361,"user":{"login":"suddenelfilio","id":6460448,"avatar_url":"https://avatars.githubusercontent.com/u/6460448?v=3","gravatar_id":"","url":"https://api.github.com/users/suddenelfilio","html_url":"https://github.com/suddenelfilio","followers_url":"https://api.github.com/users/suddenelfilio/followers","following_url":"https://api.github.com/users/suddenelfilio/following{/other_user}","gists_url":"https://api.github.com/users/suddenelfilio/gists{/gist_id}","starred_url":"https://api.github.com/users/suddenelfilio/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/suddenelfilio/subscriptions","organizations_url":"https://api.github.com/users/suddenelfilio/orgs","repos_url":"https://api.github.com/users/suddenelfilio/repos","events_url":"https://api.github.com/users/suddenelfilio/events{/privacy}","received_events_url":"https://api.github.com/users/suddenelfilio/received_events","type":"User","site_admin":false},"created_at":"2016-05-13T12:06:36Z","updated_at":"2016-05-13T12:06:36Z","body":"Yes I was under the impression each node needs it data stored locally how else would it be resilient when the node with data goes down?\r\n\r\nI did not do anything in particular for synchronization with ntp? would this apply: http://www.sysadminlab.net/windows/configuring-ntp-on-windows-server-2012"}}