{"action":"closed","issue":{"url":"https://api.github.com/repos/StackExchange/StackExchange.Redis/issues/676","repository_url":"https://api.github.com/repos/StackExchange/StackExchange.Redis","labels_url":"https://api.github.com/repos/StackExchange/StackExchange.Redis/issues/676/labels{/name}","comments_url":"https://api.github.com/repos/StackExchange/StackExchange.Redis/issues/676/comments","events_url":"https://api.github.com/repos/StackExchange/StackExchange.Redis/issues/676/events","html_url":"https://github.com/StackExchange/StackExchange.Redis/issues/676","id":248141014,"node_id":"MDU6SXNzdWUyNDgxNDEwMTQ=","number":676,"title":"RedisDatabase.HashGetAllAsync task never comletes","user":{"login":"rohithahw","id":24279454,"node_id":"MDQ6VXNlcjI0Mjc5NDU0","avatar_url":"https://avatars2.githubusercontent.com/u/24279454?v=4","gravatar_id":"","url":"https://api.github.com/users/rohithahw","html_url":"https://github.com/rohithahw","followers_url":"https://api.github.com/users/rohithahw/followers","following_url":"https://api.github.com/users/rohithahw/following{/other_user}","gists_url":"https://api.github.com/users/rohithahw/gists{/gist_id}","starred_url":"https://api.github.com/users/rohithahw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rohithahw/subscriptions","organizations_url":"https://api.github.com/users/rohithahw/orgs","repos_url":"https://api.github.com/users/rohithahw/repos","events_url":"https://api.github.com/users/rohithahw/events{/privacy}","received_events_url":"https://api.github.com/users/rohithahw/received_events","type":"User","site_admin":false},"labels":[{"id":318341625,"node_id":"MDU6TGFiZWwzMTgzNDE2MjU=","url":"https://api.github.com/repos/StackExchange/StackExchange.Redis/labels/needs-info","name":"needs-info","color":"5319e7","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":12,"created_at":"2017-08-04T23:44:08Z","updated_at":"2018-06-28T21:52:16Z","closed_at":"2018-06-28T21:52:16Z","author_association":"NONE","body":"We ran into an issue where RedisDatabase.HashGetAllAsync got blocked indefinitely. This happened on one of our compute nodes and once the machine got into this state. more and more messages got into this state. We had to reboot the machine to recover from the failures.\r\n\r\nLooking at the memory dump we found that all messages corresponding to blocked task were in a single message queue and it had accumulated more than 190KK messages while all the other message queues were empty.\r\n\r\nHere are some of the info from the dump:\r\n\r\n0:000> !do 00000208fac5d6b8 \r\nThe version of SOS does not match the version of CLR you are debugging.  Please\r\nload the matching version of SOS for the version of CLR you are debugging.\r\nCLR Version: 4.7.2101.1\r\nSOS Version: 4.0.30319.36366\r\nName:        StackExchange.Redis.PhysicalBridge\r\nMethodTable: 00007ffeae9bed90\r\nEEClass:     00007ffeae9cc038\r\nSize:        152(0x98) bytes\r\nFile:        E:\\approot\\StackExchange.Redis.StrongName.dll\r\nFields:\r\n              MT    Field   Offset                 Type VT     Attr            Value Name\r\n00007fff0a156938  4000126        8        System.String  0 instance 00000208fac5d850 Name\r\n00007fff0a159278  4000127       68         System.Int32  1 instance                0 inWriteQueue\r\n00007ffeae9b8be0  400012b       10 ...CompletionManager  0 instance 00000208fac5d8a0 completionManager\r\n00007fff0a138500  400012c       18       System.Int64[]  0 instance 00000208fac5d750 profileLog\r\n00007ffeaeaf21f8  400012d       20 ...edis.MessageQueue  0 instance 00000208fac5d7b8 queue\r\n00007fff0a159278  400012e       6c         System.Int32  1 instance                0 activeWriters\r\n00007fff0a159278  400012f       70         System.Int32  1 instance                0 beating\r\n00007fff0a159278  4000130       74         System.Int32  1 instance                1 failConnectCount\r\n00007fff0a161f18  4000131       88       System.Boolean  1 instance                0 isDisposed\r\n00007fff0a15f780  4000132       48         System.Int64  1 instance 0 nonPreferredEndpointCount\r\n00007fff0a15f780  4000133       50         System.Int64  1 instance 10821073 operationCount\r\n00007fff0a15f780  4000134       58         System.Int64  1 instance 5 socketCount\r\n00007ffeaeaf0f38  4000135       28 ...hysicalConnection  0 instance 0000000000000000 physical\r\n00007fff0a15f780  4000136       60         System.Int64  1 instance 10821073 profileLastLog\r\n00007fff0a159278  4000137       78         System.Int32  1 instance           223892 profileLogIndex\r\n00007fff0a161f18  4000138       89       System.Boolean  1 instance                1 reportNextFailure\r\n00007fff0a161f18  4000139       8a       System.Boolean  1 instance                1 reconfigureNextFailure\r\n00007fff0a159278  400013a       7c         System.Int32  1 instance                2 state\r\n00007fff0a156b68  400013b       30     System.Exception  0 instance 0000000000000000 <LastException>k__BackingField\r\n00007ffeae9b0c00  400013c       80         System.Int32  1 instance                1 <ConnectionType>k__BackingField\r\n00007ffeae9b07a8  400013d       38 ...ectionMultiplexer  0 instance 00000208fac5ce98 <Multiplexer>k__BackingField\r\n00007ffeae9b6f90  400013e       40 ...is.ServerEndPoint  0 instance 00000208fac5d588 <ServerEndPoint>k__BackingField\r\n00007fff0a159278  400013f       84         System.Int32  1 instance        215620250 connectStartTicks\r\n00007ffeae9b93f8  400012a       c0 ...nge.Redis.Message  0   static 000002097ac1ad18 ReusableAskingCommand\r\n0:000> !DumpObj /d 00000208fac5d7b8\r\nThe version of SOS does not match the version of CLR you are debugging.  Please\r\nload the matching version of SOS for the version of CLR you are debugging.\r\nCLR Version: 4.7.2101.1\r\nSOS Version: 4.0.30319.36366\r\nName:        StackExchange.Redis.MessageQueue\r\nMethodTable: 00007ffeaeaf21f8\r\nEEClass:     00007ffeaeaedfd0\r\nSize:        32(0x20) bytes\r\nFile:        E:\\approot\\StackExchange.Redis.StrongName.dll\r\nFields:\r\n              MT    Field   Offset                 Type VT     Attr            Value Name\r\n00007ffeaeaf2288  4000117        8 ...edis.StrongName]]  0 instance 00000208fac5d7d8 regular\r\n00007ffeaeaf2288  4000118       10 ...edis.StrongName]]  0 instance 00000208fac5d808 high\r\n0:000> !DumpObj /d 00000208fac5d808\r\nThe version of SOS does not match the version of CLR you are debugging.  Please\r\nload the matching version of SOS for the version of CLR you are debugging.\r\nCLR Version: 4.7.2101.1\r\nSOS Version: 4.0.30319.36366\r\nName:        System.Collections.Generic.Queue`1[[StackExchange.Redis.Message, StackExchange.Redis.StrongName]]\r\nMethodTable: 00007ffeaeaf2288\r\nEEClass:     00007ffeae9c46b8\r\nSize:        48(0x30) bytes\r\nFile:        D:\\Windows\\Microsoft.Net\\assembly\\GAC_MSIL\\System\\v4.0_4.0.0.0__b77a5c561934e089\\System.dll\r\nFields:\r\n              MT    Field   Offset                 Type VT     Attr            Value Name\r\n00007fff0a0e8730  400238e        8     System.__Canon[]  0 instance 00000208fac5db60 _array\r\n00007fff0a159278  400238f       18         System.Int32  1 instance                0 _head\r\n00007fff0a159278  4002390       1c         System.Int32  1 instance                1 _tail\r\n00007fff0a159278  4002391       20         System.Int32  1 instance                1 _size\r\n00007fff0a159278  4002392       24         System.Int32  1 instance             7191 _version\r\n00007fff0a156f18  4002393       10        System.Object  0 instance 0000000000000000 _syncRoot\r\n00007fff0a0e8730  4002398        8     System.__Canon[]  0   static dynamic statics NYI                 _emptyArray\r\n0:000> !DumpObj /d 00000208fac5db60\r\nThe version of SOS does not match the version of CLR you are debugging.  Please\r\nload the matching version of SOS for the version of CLR you are debugging.\r\nCLR Version: 4.7.2101.1\r\nSOS Version: 4.0.30319.36366\r\nName:        StackExchange.Redis.Message[]\r\nMethodTable: 00007ffeaeaf2558\r\nEEClass:     00007fff09b1aa00\r\nSize:        56(0x38) bytes\r\nArray:       Rank 1, Number of elements 4, Type CLASS (Print Array)\r\nFields:\r\nNone\r\n0:000> !DumpObj /d 00000208fac5d7d8\r\nThe version of SOS does not match the version of CLR you are debugging.  Please\r\nload the matching version of SOS for the version of CLR you are debugging.\r\nCLR Version: 4.7.2101.1\r\nSOS Version: 4.0.30319.36366\r\nName:        System.Collections.Generic.Queue`1[[StackExchange.Redis.Message, StackExchange.Redis.StrongName]]\r\nMethodTable: 00007ffeaeaf2288\r\nEEClass:     00007ffeae9c46b8\r\nSize:        48(0x30) bytes\r\nFile:        D:\\Windows\\Microsoft.Net\\assembly\\GAC_MSIL\\System\\v4.0_4.0.0.0__b77a5c561934e089\\System.dll\r\nFields:\r\n              MT    Field   Offset                 Type VT     Attr            Value Name\r\n00007fff0a0e8730  400238e        8     System.__Canon[]  0 instance 0000020abb97a0e8 _array\r\n00007fff0a159278  400238f       18         System.Int32  1 instance                0 _head\r\n00007fff0a159278  4002390       1c         System.Int32  1 instance           190812 _tail\r\n00007fff0a159278  4002391       20         System.Int32  1 instance           190812 _size\r\n00007fff0a159278  4002392       24         System.Int32  1 instance         21825980 _version\r\n00007fff0a156f18  4002393       10        System.Object  0 instance 0000000000000000 _syncRoot\r\n00007fff0a0e8730  4002398        8     System.__Canon[]  0   static dynamic statics NYI                 _emptyArray\r\n\r\nLet me know if you need any more data from the dump to narrow down the issue.\r\n .Net version : 4.7.2101.1\r\n StackExchange.Redis.StrongName.dll version: 1.1.608"}}
{"actor":{"display_login":"mgravell"}}