{"action":"created","issue":{"url":"https://api.github.com/repos/EventStore/EventStore/issues/919","repository_url":"https://api.github.com/repos/EventStore/EventStore","labels_url":"https://api.github.com/repos/EventStore/EventStore/issues/919/labels{/name}","comments_url":"https://api.github.com/repos/EventStore/EventStore/issues/919/comments","events_url":"https://api.github.com/repos/EventStore/EventStore/issues/919/events","html_url":"https://github.com/EventStore/EventStore/issues/919","id":154656779,"number":919,"title":"cluster crash after 2 days","user":{"login":"suddenelfilio","id":6460448,"avatar_url":"https://avatars.githubusercontent.com/u/6460448?v=3","gravatar_id":"","url":"https://api.github.com/users/suddenelfilio","html_url":"https://github.com/suddenelfilio","followers_url":"https://api.github.com/users/suddenelfilio/followers","following_url":"https://api.github.com/users/suddenelfilio/following{/other_user}","gists_url":"https://api.github.com/users/suddenelfilio/gists{/gist_id}","starred_url":"https://api.github.com/users/suddenelfilio/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/suddenelfilio/subscriptions","organizations_url":"https://api.github.com/users/suddenelfilio/orgs","repos_url":"https://api.github.com/users/suddenelfilio/repos","events_url":"https://api.github.com/users/suddenelfilio/events{/privacy}","received_events_url":"https://api.github.com/users/suddenelfilio/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":10,"created_at":"2016-05-13T08:13:20Z","updated_at":"2016-05-13T11:35:11Z","closed_at":null,"body":"We are running a cluster on commodity servers on 6 servers each with 2 nodes and mostly default settings. After 2 days we keep getting issues with performance counters.\r\n\r\nat **first** we had: \r\n\r\n> System.InvalidOperationException: The Counter layout for the Category specified is invalid, a counter of the type:  AverageCount64, AverageTimer32, CounterMultiTimer, CounterMultiTimerInverse, CounterMultiTimer100Ns, CounterMultiTimer100NsInverse, RawFraction, or SampleFraction has to be immediately followed by any of the base counter types: AverageBase, CounterMultiBase, RawBase or SampleBase.\r\n   at System.Diagnostics.CategorySample.GetCounterDefinitionSample(String counter)\r\n   at System.Diagnostics.PerformanceCounter.NextSample()\r\n   at EventStore.Core.Services.Monitoring.Utils.PerfCounterHelper.GetGcStats() in c:\\projects\\eventstore\\src\\EventStore.Core\\Services\\Monitoring\\Utils\\PerfCounterHelper.cs:line 119\r\n   at EventStore.Core.Services.Monitoring.SystemStatsHelper.GetSystemStats() in c:\\projects\\eventstore\\src\\EventStore.Core\\Services\\Monitoring\\SystemStatsHelper.cs:line 66\r\n   at EventStore.Core.Services.Monitoring.MonitoringService.CollectStats() in c:\\projects\\eventstore\\src\\EventStore.Core\\Services\\Monitoring\\MonitoringService.cs:line 135\r\n\r\nWe solved this with lodctr \\R\r\n\r\nNow we have:\r\n\r\n> System.InvalidOperationException: Could not Read Category Index: 5908.\r\n>    at System.Diagnostics.CategorySample..ctor(Byte[] data, CategoryEntry entry, PerformanceCounterLib library)\r\n>    at System.Diagnostics.PerformanceCounterLib.GetCategorySample(String machine, String category)\r\n>    at System.Diagnostics.PerformanceCounter.NextSample()\r\n>    at EventStore.Core.Services.Monitoring.Utils.PerfCounterHelper.GetGcStats() in c:\\projects\\eventstore\\src\\EventStore.Core\\Services\\Monitoring\\Utils\\PerfCounterHelper.cs:line 119\r\n>    at EventStore.Core.Services.Monitoring.SystemStatsHelper.GetSystemStats() in c:\\projects\\eventstore\\src\\EventStore.Core\\Services\\Monitoring\\SystemStatsHelper.cs:line 66\r\n>    at EventStore.Core.Services.Monitoring.MonitoringService.CollectStats() in \r\nc:\\projects\\eventstore\\src\\EventStore.Core\\Services\\Monitoring\\MonitoringService.cs:line 135 \r\n\r\nas these errors are thrown consequently the reaction time of the nodes slows down and eventualy the node dies with message:\r\n\r\n> 0904:014 2016.05.13 06:40:36.551 ERROR GossipServiceBase   ] Time difference between us and [172.30.76.18:1112] is too great! UTC now: 2016-05-13 06:40:36.551, peer's time stamp: 2016-05-13 06:39:05.929.\r\n> [PID:10904:296 2016.05.13 06:41:32.676 ERROR QueuedHandlerThreadP] ---!!! VERY SLOW QUEUE MSG [Worker #2]: SendOverHttp - 7500ms. Q: 651/1369.\r\n> [PID:10904:014 2016.05.13 06:41:39.613 ERROR GossipServiceBase   ] Time difference between us and [172.30.78.12:1112] is too great! UTC now: 2016-05-13 06:41:39.613, peer's time stamp: 2016-05-13 06:40:31.294.\r\n> [PID:10904:160 2016.05.13 06:41:40.832 ERROR QueuedHandlerThreadP] ---!!! VERY SLOW QUEUE MSG [Worker #5]: SendOverHttp - 8578ms. Q: 994/1329.\r\n> [PID:10904:014 2016.05.13 06:41:42.769 ERROR GossipServiceBase   ] Time difference between us and [172.30.76.17:2112] is too great! UTC now: 2016-05-13 06:41:42.769, peer's time stamp: 2016-05-13 06:40:40.346.\r\n> [PID:10904:014 2016.05.13 06:41:44.144 ERROR GossipServiceBase   ] Time difference between us and [172.30.78.4:2112] is too great! UTC now: 2016-05-13 06:41:44.144, peer's time stamp: 2016-05-13 06:40:29.914.\r\n> [PID:10904:014 2016.05.13 06:41:44.160 ERROR GossipServiceBase   ] Time difference between us and [172.30.78.4:2112] is too great! UTC now: 2016-05-13 06:41:44.160, peer's time stamp: 2016-05-13 06:40:37.976.\r\n> [PID:10904:014 2016.05.13 06:41:44.363 ERROR GossipServiceBase   ] Time difference between us and [172.30.76.17:2112] is too great! UTC now: 2016-05-13 06:41:44.363, peer's time stamp: 2016-05-13 06:40:27.206.\r\n> [PID:10904:014 2016.05.13 06:41:44.879 ERROR GossipServiceBase   ] Time difference between us and [172.30.78.12:1112] is too great! UTC now: 2016-05-13 06:41:44.879, peer's time stamp: 2016-05-13 06:40:43.420.\r\n> [PID:10904:014 2016.05.13 06:41:45.207 ERROR GossipServiceBase   ] Time difference between us and [172.30.76.17:1112] is too great! UTC now: 2016-05-13 06:41:45.207, peer's time stamp: 2016-05-13 06:40:26.439.\r\n> [PID:10904:014 2016.05.13 06:41:45.394 ERROR GossipServiceBase   ] Time difference between us and [172.30.76.17:3112] is too great! UTC now: 2016-05-13 06:41:45.394, peer's time stamp: 2016-05-13 06:40:35.611.\r\n> [PID:10904:014 2016.05.13 06:41:45.613 ERROR GossipServiceBase   ] Time difference between us and [172.30.78.13:2112] is too great! UTC now: 2016-05-13 06:41:45.613, peer's time stamp: 2016-05-13 06:40:44.861.\r\n> [PID:10904:014 2016.05.13 06:41:48.097 ERROR GossipServiceBase   ] Time difference between us and [172.30.76.17:3112] is too great! UTC now: 2016-05-13 06:41:48.097, peer's time stamp: 2016-05-13 06:40:31.549.\r\n> [PID:10904:014 2016.05.13 06:41:48.754 ERROR GossipServiceBase   ] Time difference between us and [172.30.76.17:2112] is too great! UTC now: 2016-05-13 06:41:48.754, peer's time stamp: 2016-05-13 06:40:43.377.\r\n> [PID:10904:014 2016.05.13 06:41:51.097 ERROR GossipServiceBase   ] Time difference between us and [172.30.78.13:1112] is too great! UTC now: 2016-05-13 06:41:51.097, peer's time stamp: 2016-05-13 06:40:47.283.\r\n> [PID:10904:014 2016.05.13 06:42:08.738 ERROR GossipServiceBase   ] Time difference between us and [172.30.78.4:1112] is too great! UTC now: 2016-05-13 06:42:08.738, peer's time stamp: 2016-05-13 06:40:39.335.\r\n> [PID:10904:014 2016.05.13 06:42:08.925 ERROR GossipServiceBase   ] Time difference between us and [172.30.78.13:2112] is too great! UTC now: 2016-05-13 06:42:08.925, peer's time stamp: 2016-05-13 06:41:08.251.\r\n> [PID:10904:276 2016.05.13 06:42:17.628 ERROR MonitoringService   ] Error while collecting stats\r\n\r\nWe are currently at the point to abandon eventstore as we don't have a lot of time to concern us about these issues, but since I really like this product I want to give it another try.\r\n\r\nany advise?"},"comment":{"url":"https://api.github.com/repos/EventStore/EventStore/issues/comments/219019426","html_url":"https://github.com/EventStore/EventStore/issues/919#issuecomment-219019426","issue_url":"https://api.github.com/repos/EventStore/EventStore/issues/919","id":219019426,"user":{"login":"gregoryyoung","id":381274,"avatar_url":"https://avatars.githubusercontent.com/u/381274?v=3","gravatar_id":"","url":"https://api.github.com/users/gregoryyoung","html_url":"https://github.com/gregoryyoung","followers_url":"https://api.github.com/users/gregoryyoung/followers","following_url":"https://api.github.com/users/gregoryyoung/following{/other_user}","gists_url":"https://api.github.com/users/gregoryyoung/gists{/gist_id}","starred_url":"https://api.github.com/users/gregoryyoung/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gregoryyoung/subscriptions","organizations_url":"https://api.github.com/users/gregoryyoung/orgs","repos_url":"https://api.github.com/users/gregoryyoung/repos","events_url":"https://api.github.com/users/gregoryyoung/events{/privacy}","received_events_url":"https://api.github.com/users/gregoryyoung/received_events","type":"User","site_admin":false},"created_at":"2016-05-13T11:35:11Z","updated_at":"2016-05-13T11:35:11Z","body":"Why is one node 5GB of logs? This sounds like some misconfiguration or?"}}