{"action":"closed","issue":{"url":"https://api.github.com/repos/EventStore/EventStore/issues/903","repository_url":"https://api.github.com/repos/EventStore/EventStore","labels_url":"https://api.github.com/repos/EventStore/EventStore/issues/903/labels{/name}","comments_url":"https://api.github.com/repos/EventStore/EventStore/issues/903/comments","events_url":"https://api.github.com/repos/EventStore/EventStore/issues/903/events","html_url":"https://github.com/EventStore/EventStore/issues/903","id":151269056,"number":903,"title":"Need to be more robust on server stress","user":{"login":"xiangwu007","id":18064248,"avatar_url":"https://avatars.githubusercontent.com/u/18064248?v=3","gravatar_id":"","url":"https://api.github.com/users/xiangwu007","html_url":"https://github.com/xiangwu007","followers_url":"https://api.github.com/users/xiangwu007/followers","following_url":"https://api.github.com/users/xiangwu007/following{/other_user}","gists_url":"https://api.github.com/users/xiangwu007/gists{/gist_id}","starred_url":"https://api.github.com/users/xiangwu007/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xiangwu007/subscriptions","organizations_url":"https://api.github.com/users/xiangwu007/orgs","repos_url":"https://api.github.com/users/xiangwu007/repos","events_url":"https://api.github.com/users/xiangwu007/events{/privacy}","received_events_url":"https://api.github.com/users/xiangwu007/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/EventStore/EventStore/labels/question","name":"question","color":"cc317c"}],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-04-27T00:18:03Z","updated_at":"2016-05-04T12:15:16Z","closed_at":"2016-05-04T12:15:16Z","body":"The EventStore application just can't handle the server stress even on a one node system, and reported TCP error, while the network indeed was fine. \r\n\r\nThe problem happened when I was installing .net frame 4.6, which generated some load on disk/memory and CPU. \r\n\r\n[PID:02688:018 2016.04.25 13:47:14.128 TRACE QueuedHandlerMRES   ] SLOW QUEUE MSG [MainQueue]: PurgeTimedOutRequests - 374ms. Q: 4/6.\r\n[PID:02688:017 2016.04.25 13:47:17.706 TRACE TcpConnectionManager] Closing connection 'external-normal' [127.0.0.1:59532, L127.0.0.1:1113, {6a77a319-6a1c-4fea-80a1-a7a1cab5572f}] cleanly. Reason: HEARTBEAT TIMEOUT at msgNum 317437\r\n[PID:02688:017 2016.04.25 13:47:18.441 INFO  TcpConnection       ] ES TcpConnection closed [13:47:18.426: N127.0.0.1:59532, L127.0.0.1:1113, {6a77a319-6a1c-4fea-80a1-a7a1cab5572f}]:Received bytes: 6983640, Sent bytes: 6983699\r\n[PID:02688:017 2016.04.25 13:47:18.488 INFO  TcpConnection       ] ES TcpConnection closed [13:47:18.488: N127.0.0.1:59532, L127.0.0.1:1113, {6a77a319-6a1c-4fea-80a1-a7a1cab5572f}]:Send calls: 317438, callbacks: 317438\r\n[PID:02688:017 2016.04.25 13:47:18.488 INFO  TcpConnection       ] ES TcpConnection closed [13:47:18.488: N127.0.0.1:59532, L127.0.0.1:1113, {6a77a319-6a1c-4fea-80a1-a7a1cab5572f}]:Receive calls: 317439, callbacks: 317438\r\n[PID:02688:017 2016.04.25 13:47:18.911 INFO  TcpConnection       ] ES TcpConnection closed [13:47:18.911: N127.0.0.1:59532, L127.0.0.1:1113, {6a77a319-6a1c-4fea-80a1-a7a1cab5572f}]:Close reason: [Success] HEARTBEAT TIMEOUT at msgNum 317437\r\n[PID:02688:017 2016.04.25 13:47:19.630 INFO  TcpConnectionManager] Connection 'external-normal' [127.0.0.1:59532, {6a77a319-6a1c-4fea-80a1-a7a1cab5572f}] closed: Success.\r\n[PID:02688:022 2016.04.25 13:47:19.645 DEBUG PersistentSubscripti] Lost connection from 127.0.0.1:59532\r\n[PID:02688:018 2016.04.25 13:47:21.786 TRACE InMemoryBus         ] SLOW BUS MSG [MainBus]: WriteEvents - 109ms. Handler: RequestManagementService.\r\n[PID:02688:018 2016.04.25 13:47:21.786 TRACE QueuedHandlerMRES   ] SLOW QUEUE MSG [MainQueue]: WriteEvents - 109ms. Q: 0/1.\r\n[PID:02688:018 2016.04.25 13:47:21.958 TRACE QueuedHandlerMRES   ] SLOW QUEUE MSG [MainQueue]: WritePrepares - 125ms. Q: 0/0.\r\n[PID:02688:032 2016.04.25 13:47:23.223 INFO  TcpService          ] External TCP connection accepted: [Normal, 127.0.0.1:63735, L127.0.0.1:1113, {b0554f6f-a257-4fe3-9e10-41e2fdfcc5ea}].\r\n[PID:02688:015 2016.04.25 13:47:24.161 TRACE QueuedHandlerMRES   ] SLOW QUEUE MSG [StorageWriterQueue]: WritePrepares - 2203ms. Q: 0/0.\r\n[PID:02688:017 2016.04.25 13:47:25.661 TRACE TcpConnectionManager] Closing connection 'external-normal' [127.0.0.1:63735, L127.0.0.1:1113, {b0554f6f-a257-4fe3-9e10-41e2fdfcc5ea}] cleanly. Reason: HEARTBEAT TIMEOUT at msgNum 1\r\n[PID:02688:017 2016.04.25 13:47:25.661 INFO  TcpConnection       ] ES TcpConnection closed [13:47:25.661: N127.0.0.1:63735, L127.0.0.1:1113, {b0554f6f-a257-4fe3-9e10-41e2fdfcc5ea}]:Received bytes: 48, Sent bytes: 22\r\n[PID:02688:017 2016.04.25 13:47:25.661 INFO  TcpConnection       ] ES TcpConnection closed [13:47:25.661: N127.0.0.1:63735, L127.0.0.1:1113, {b0554f6f-a257-4fe3-9e10-41e2fdfcc5ea}]:Send calls: 1, callbacks: 1\r\n[PID:02688:017 2016.04.25 13:47:25.661 INFO  TcpConnection       ] ES TcpConnection closed [13:47:25.661: N127.0.0.1:63735, L127.0.0.1:1113, {b0554f6f-a257-4fe3-9e10-41e2fdfcc5ea}]:Receive calls: 3, callbacks: 2\r\n[PID:02688:017 2016.04.25 13:47:25.661 INFO  TcpConnection       ] ES TcpConnection closed [13:47:25.661: N127.0.0.1:63735, L127.0.0.1:1113, {b0554f6f-a257-4fe3-9e10-41e2fdfcc5ea}]:Close reason: [Success] HEARTBEAT TIMEOUT at msgNum 1\r\n[PID:02688:017 2016.04.25 13:47:25.661 INFO  TcpConnectionManager] Connection 'external-normal' [127.0.0.1:63735, {b0554f6f-a257-4fe3-9e10-41e2fdfcc5ea}] closed: Success.\r\n[PID:02688:006 2016.04.25 13:47:25.661 DEBUG PersistentSubscripti] Lost connection from 127.0.0.1:63735\r\n[PID:02688:032 2016.04.25 13:47:37.474 INFO  ProtobufExtensions  ] Deserialization to EventStore.Core.Messages.TcpClientMessageDto+TransactionCommit failed\r\nProtoBuf.ProtoException: Invalid field in source data: 0\r\n   at ProtoBuf.ProtoReader.ReadFieldHeader() in c:\\Dev\\protobuf-net\\protobuf-net\\ProtoReader.cs:line 642\r\n   at proto_8(Object , ProtoReader )\r\n   at ProtoBuf.Meta.TypeModel.DeserializeCore(ProtoReader reader, Type type, Object value, Boolean noAutoCreate) in c:\\Dev\\protobuf-net\\protobuf-net\\Meta\\TypeModel.cs:line 679\r\n   at ProtoBuf.Meta.TypeModel.Deserialize(Stream source, Object value, Type type, SerializationContext context) in c:\\Dev\\protobuf-net\\protobuf-net\\Meta\\TypeModel.cs:line 581\r\n   at ProtoBuf.Serializer.Deserialize[T](Stream source) in c:\\Dev\\protobuf-net\\protobuf-net\\Serializer.cs:line 77\r\n   at EventStore.Core.Services.Transport.Tcp.ProtobufExtensions.Deserialize[T](ArraySegment`1 data) in c:\\projects\\eventstore\\src\\EventStore.Core\\Services\\Transport\\Tcp\\ProtobufExtensions.cs:line 58\r\n\r\nDetail please check the log.\r\n[2016-04-25.zip](https://github.com/EventStore/EventStore/files/237719/2016-04-25.zip)\r\n\r\n\r\n"}}