{"action":"created","issue":{"url":"https://api.github.com/repos/Azure/iotedge/issues/16","repository_url":"https://api.github.com/repos/Azure/iotedge","labels_url":"https://api.github.com/repos/Azure/iotedge/issues/16/labels{/name}","comments_url":"https://api.github.com/repos/Azure/iotedge/issues/16/comments","events_url":"https://api.github.com/repos/Azure/iotedge/issues/16/events","html_url":"https://github.com/Azure/iotedge/issues/16","id":339875563,"node_id":"MDU6SXNzdWUzMzk4NzU1NjM=","number":16,"title":"'Entering unreachable state' - all edgehub connections timeout after","user":{"login":"chriswaters78","id":28921630,"node_id":"MDQ6VXNlcjI4OTIxNjMw","avatar_url":"https://avatars0.githubusercontent.com/u/28921630?v=4","gravatar_id":"","url":"https://api.github.com/users/chriswaters78","html_url":"https://github.com/chriswaters78","followers_url":"https://api.github.com/users/chriswaters78/followers","following_url":"https://api.github.com/users/chriswaters78/following{/other_user}","gists_url":"https://api.github.com/users/chriswaters78/gists{/gist_id}","starred_url":"https://api.github.com/users/chriswaters78/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chriswaters78/subscriptions","organizations_url":"https://api.github.com/users/chriswaters78/orgs","repos_url":"https://api.github.com/users/chriswaters78/repos","events_url":"https://api.github.com/users/chriswaters78/events{/privacy}","received_events_url":"https://api.github.com/users/chriswaters78/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2018-07-10T14:45:11Z","updated_at":"2018-08-06T20:01:32Z","closed_at":"2018-08-06T20:01:32Z","author_association":"NONE","body":"I have been testing the new GA release. This is on Windows 10 and Linux containers.\r\n\r\nThe module I had deployed stayed up for about 5 days. This module is a simple polling module originally based on the sample temperature module and since upgraded to be compatible with the GA release.\r\n\r\nAfter 5 days my module started throwing connection time out errors.\r\n\r\nHere are the docker logs for my custom module when it stopped working, it sends a reading every 5 seconds:\r\n\r\n```\r\n\t07/05/2018 01:10:13 - sending reading\r\n\t07/05/2018 01:10:18 - sending reading\r\n\t07/05/2018 01:10:23 - sending reading\r\n[ERROR] Unexpected Exception Operation timeout expired.\r\n\tSystem.TimeoutException: Operation timeout expired.\r\n   at Microsoft.Azure.Devices.Client.InternalClient.<>c.<ApplyTimeout>b__62_2(Task t)\r\n   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)\r\n   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at MockModbusModuleV2.Program.<SendSimulationData>d__18.MoveNext() in /app/Program.cs:line 213\r\n\t07/05/2018 01:14:28 - sending reading\r\n[ERROR] Unexpected Exception Operation timeout expired.\r\n\tSystem.TimeoutException: Operation timeout expired.\r\n   at Microsoft.Azure.Devices.Client.InternalClient.<>c.<ApplyTimeout>b__62_2(Task t)\r\n   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)\r\n   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at MockModbusModuleV2.Program.<SendSimulationData>d__18.MoveNext() in /app/Program.cs:line 213\r\n```\r\n\r\nThis is then repeated indefinitely. The line of code it is hanging on is:\r\n\r\n`\r\n                    await moduleClient.SendEventAsync(\"modbusOutput\", message);\r\n`\r\n\r\nHere are the logs from the edgeHub leading up to the issue. The EdgeHubConnectionException entries at the start occurred throughout the whole 5 day period and seem to be associated with the Token refresh but the module was working throughout that time. The 'Entering unreachable state' final log entry had not occurred before and this is when the module stopping working:\r\n\r\n```\r\n2018-07-05 00:13:09.994 +00:00 [INF] - Existing token not found for officepc1/ipswicha-mockmodbus. Getting new token from the client...\r\n2018-07-05 00:13:09.994 +00:00 [WRN] - Closing connection for device: officepc1/ipswicha-mockmodbus, Microsoft.Azure.Devices.Edge.Hub.Core.EdgeHubConnectionException: Connection closed for device officepc1/ipswicha-mockmodbus., \r\n2018-07-05 00:13:09.994 +00:00 [INF] - Disposing MessagingServiceClient for device Id officepc1/ipswicha-mockmodbus because of exception - Microsoft.Azure.Devices.Edge.Hub.Core.EdgeHubConnectionException: Connection closed for device officepc1/ipswicha-mockmodbus.\r\n2018-07-05 00:13:09.995 +00:00 [INF] - Closing device proxy for device Id officepc1/ipswicha-mockmodbus\r\n2018-07-05 00:13:09.995 +00:00 [INF] - Device connection removed for device officepc1/ipswicha-mockmodbus\r\n2018-07-05 00:13:11.013 +00:00 [INF] - Obtained new token for client officepc1/ipswicha-mockmodbus that expires in 00:59:59.9864079\r\n2018-07-05 00:13:11.013 +00:00 [INF] - New cloud connection created for device officepc1/ipswicha-mockmodbus\r\n2018-07-05 00:13:11.013 +00:00 [INF] - Successfully generated identity for clientId officepc1/ipswicha-mockmodbus and username desktop-ce8aaih/officepc1/ipswicha-mockmodbus/?api-version=2018-06-30&DeviceClientType=Microsoft.Azure.Devices.Client%2F1.17.0%20%28.NET%20Core%204.6.0.0%3B%20Linux%204.9.87-linuxkit-aufs%20%231%20SMP%20Wed%20Mar%2014%2015%3A12%3A16%20UTC%202018%3B%20X64%29\r\n2018-07-05 00:13:11.013 +00:00 [INF] - ClientAuthenticated, officepc1/ipswicha-mockmodbus, 7e30919b\r\n2018-07-05 00:13:11.014 +00:00 [INF] - New device connection for device officepc1/ipswicha-mockmodbus\r\n2018-07-05 00:13:11.014 +00:00 [INF] - Bind device proxy for device officepc1/ipswicha-mockmodbus\r\n2018-07-05 00:13:11.014 +00:00 [INF] - Binding message channel for device Id officepc1/ipswicha-mockmodbus\r\n2018-07-05 00:13:11.014 +00:00 [INF] - Set subscriptions from session state for officepc1/ipswicha-mockmodbus\r\n2018-07-05 00:13:32.005 +00:00 [INF] - Existing token not found for officepc1/plymouth-mockmodbus. Getting new token from the client...\r\n2018-07-05 00:13:32.005 +00:00 [WRN] - Closing connection for device: officepc1/plymouth-mockmodbus, Microsoft.Azure.Devices.Edge.Hub.Core.EdgeHubConnectionException: Connection closed for device officepc1/plymouth-mockmodbus., \r\n2018-07-05 00:13:32.005 +00:00 [INF] - Disposing MessagingServiceClient for device Id officepc1/plymouth-mockmodbus because of exception - Microsoft.Azure.Devices.Edge.Hub.Core.EdgeHubConnectionException: Connection closed for device officepc1/plymouth-mockmodbus.\r\n2018-07-05 00:13:32.006 +00:00 [INF] - Closing device proxy for device Id officepc1/plymouth-mockmodbus\r\n2018-07-05 00:13:32.006 +00:00 [INF] - Device connection removed for device officepc1/plymouth-mockmodbus\r\n2018-07-05 00:13:32.855 +00:00 [INF] - Obtained new token for client officepc1/plymouth-mockmodbus that expires in 01:00:00.1448300\r\n2018-07-05 00:13:32.855 +00:00 [INF] - New cloud connection created for device officepc1/plymouth-mockmodbus\r\n2018-07-05 00:13:32.855 +00:00 [INF] - Successfully generated identity for clientId officepc1/plymouth-mockmodbus and username desktop-ce8aaih/officepc1/plymouth-mockmodbus/?api-version=2018-06-30&DeviceClientType=Microsoft.Azure.Devices.Client%2F1.17.0%20%28.NET%20Core%204.6.0.0%3B%20Linux%204.9.87-linuxkit-aufs%20%231%20SMP%20Wed%20Mar%2014%2015%3A12%3A16%20UTC%202018%3B%20X64%29\r\n2018-07-05 00:13:32.855 +00:00 [INF] - ClientAuthenticated, officepc1/plymouth-mockmodbus, 35b842f5\r\n2018-07-05 00:13:32.855 +00:00 [INF] - New device connection for device officepc1/plymouth-mockmodbus\r\n2018-07-05 00:13:32.856 +00:00 [INF] - Bind device proxy for device officepc1/plymouth-mockmodbus\r\n2018-07-05 00:13:32.856 +00:00 [INF] - Binding message channel for device Id officepc1/plymouth-mockmodbus\r\n2018-07-05 00:13:32.857 +00:00 [INF] - Set subscriptions from session state for officepc1/plymouth-mockmodbus\r\n2018-07-05 00:26:13.381 +00:00 [INF] - Started task to cleanup processed and stale messages for endpoint officepc1/crumlin-mockmodbus/input\r\n2018-07-05 00:26:13.381 +00:00 [INF] - Cleaned up 0 messages from queue for endpoint officepc1/crumlin-mockmodbus/input and 0 messages from message store.\r\n2018-07-05 00:26:43.385 +00:00 [INF] - Started task to cleanup processed and stale messages for endpoint officepc1/ipswicha-mockmodbus/input\r\n2018-07-05 00:26:43.385 +00:00 [INF] - Cleaned up 0 messages from queue for endpoint officepc1/ipswicha-mockmodbus/input and 0 messages from message store.\r\n2018-07-05 00:27:13.390 +00:00 [INF] - Started task to cleanup processed and stale messages for endpoint officepc1/stowmarketa-mockmodbus/input\r\n2018-07-05 00:27:13.390 +00:00 [INF] - Cleaned up 0 messages from queue for endpoint officepc1/stowmarketa-mockmodbus/input and 0 messages from message store.\r\n2018-07-05 00:27:43.391 +00:00 [INF] - Started task to cleanup processed and stale messages for endpoint iothub\r\n2018-07-05 00:27:43.737 +00:00 [INF] - Cleaned up 1970 messages from queue for endpoint iothub and 1970 messages from message store.\r\n2018-07-05 00:28:13.726 +00:00 [INF] - Started task to cleanup processed and stale messages for endpoint officepc1/plymouth-mockmodbus/input\r\n2018-07-05 00:28:13.726 +00:00 [INF] - Cleaned up 0 messages from queue for endpoint officepc1/plymouth-mockmodbus/input and 0 messages from message store.\r\n2018-07-05 00:28:43.798 +00:00 [INF] - Started task to cleanup processed and stale messages for endpoint officepc1/ipswichb-mockmodbus/input\r\n2018-07-05 00:28:43.798 +00:00 [INF] - Cleaned up 0 messages from queue for endpoint officepc1/ipswichb-mockmodbus/input and 0 messages from message store.\r\n2018-07-05 00:59:13.806 +00:00 [INF] - Started task to cleanup processed and stale messages for endpoint officepc1/crumlin-mockmodbus/input\r\n2018-07-05 00:59:13.806 +00:00 [INF] - Cleaned up 0 messages from queue for endpoint officepc1/crumlin-mockmodbus/input and 0 messages from message store.\r\n2018-07-05 00:59:43.813 +00:00 [INF] - Started task to cleanup processed and stale messages for endpoint officepc1/ipswicha-mockmodbus/input\r\n2018-07-05 00:59:43.813 +00:00 [INF] - Cleaned up 0 messages from queue for endpoint officepc1/ipswicha-mockmodbus/input and 0 messages from message store.\r\n2018-07-05 01:00:13.821 +00:00 [INF] - Started task to cleanup processed and stale messages for endpoint officepc1/stowmarketa-mockmodbus/input\r\n2018-07-05 01:00:13.822 +00:00 [INF] - Cleaned up 0 messages from queue for endpoint officepc1/stowmarketa-mockmodbus/input and 0 messages from message store.\r\n2018-07-05 01:00:43.820 +00:00 [INF] - Started task to cleanup processed and stale messages for endpoint iothub\r\n2018-07-05 01:00:44.156 +00:00 [INF] - Cleaned up 1975 messages from queue for endpoint iothub and 1975 messages from message store.\r\n2018-07-05 01:01:14.164 +00:00 [INF] - Started task to cleanup processed and stale messages for endpoint officepc1/plymouth-mockmodbus/input\r\n2018-07-05 01:01:14.164 +00:00 [INF] - Cleaned up 0 messages from queue for endpoint officepc1/plymouth-mockmodbus/input and 0 messages from message store.\r\n2018-07-05 01:01:44.171 +00:00 [INF] - Started task to cleanup processed and stale messages for endpoint officepc1/ipswichb-mockmodbus/input\r\n2018-07-05 01:01:44.171 +00:00 [INF] - Cleaned up 0 messages from queue for endpoint officepc1/ipswichb-mockmodbus/input and 0 messages from message store.\r\n2018-07-05 01:10:21.993 +00:00 [INF] - Existing token not found for officepc1/ipswichb-mockmodbus. Getting new token from the client...\r\n2018-07-05 01:10:21.993 +00:00 [WRN] - Closing connection for device: officepc1/ipswichb-mockmodbus, Microsoft.Azure.Devices.Edge.Hub.Core.EdgeHubConnectionException: Connection closed for device officepc1/ipswichb-mockmodbus., \r\n2018-07-05 01:10:21.993 +00:00 [INF] - Disposing MessagingServiceClient for device Id officepc1/ipswichb-mockmodbus because of exception - Microsoft.Azure.Devices.Edge.Hub.Core.EdgeHubConnectionException: Connection closed for device officepc1/ipswichb-mockmodbus.\r\n2018-07-05 01:10:21.994 +00:00 [INF] - Closing device proxy for device Id officepc1/ipswichb-mockmodbus\r\n2018-07-05 01:10:21.995 +00:00 [INF] - Device connection removed for device officepc1/ipswichb-mockmodbus\r\n2018-07-05 01:10:42.002 +00:00 [INF] - Existing token not found for officepc1/crumlin-mockmodbus. Getting new token from the client...\r\n2018-07-05 01:10:42.002 +00:00 [WRN] - Closing connection for device: officepc1/crumlin-mockmodbus, Microsoft.Azure.Devices.Edge.Hub.Core.EdgeHubConnectionException: Connection closed for device officepc1/crumlin-mockmodbus., \r\n2018-07-05 01:10:42.003 +00:00 [INF] - Disposing MessagingServiceClient for device Id officepc1/crumlin-mockmodbus because of exception - Microsoft.Azure.Devices.Edge.Hub.Core.EdgeHubConnectionException: Connection closed for device officepc1/crumlin-mockmodbus.\r\n2018-07-05 01:10:42.003 +00:00 [INF] - Closing device proxy for device Id officepc1/crumlin-mockmodbus\r\n2018-07-05 01:10:42.003 +00:00 [INF] - Device connection removed for device officepc1/crumlin-mockmodbus\r\n2018-07-05 01:10:47.004 +00:00 [INF] - Existing token not found for officepc1/stowmarketa-mockmodbus. Getting new token from the client...\r\n2018-07-05 01:10:47.005 +00:00 [WRN] - Closing connection for device: officepc1/stowmarketa-mockmodbus, Microsoft.Azure.Devices.Edge.Hub.Core.EdgeHubConnectionException: Connection closed for device officepc1/stowmarketa-mockmodbus., \r\n2018-07-05 01:10:47.005 +00:00 [INF] - Disposing MessagingServiceClient for device Id officepc1/stowmarketa-mockmodbus because of exception - Microsoft.Azure.Devices.Edge.Hub.Core.EdgeHubConnectionException: Connection closed for device officepc1/stowmarketa-mockmodbus.\r\n2018-07-05 01:10:47.006 +00:00 [INF] - Closing device proxy for device Id officepc1/stowmarketa-mockmodbus\r\n2018-07-05 01:10:47.006 +00:00 [INF] - Device connection removed for device officepc1/stowmarketa-mockmodbus\r\n2018-07-05 01:11:11.000 +00:00 [INF] - Existing token not found for officepc1/ipswicha-mockmodbus. Getting new token from the client...\r\n2018-07-05 01:11:11.000 +00:00 [WRN] - Closing connection for device: officepc1/ipswicha-mockmodbus, Microsoft.Azure.Devices.Edge.Hub.Core.EdgeHubConnectionException: Connection closed for device officepc1/ipswicha-mockmodbus., \r\n2018-07-05 01:11:11.001 +00:00 [INF] - Disposing MessagingServiceClient for device Id officepc1/ipswicha-mockmodbus because of exception - Microsoft.Azure.Devices.Edge.Hub.Core.EdgeHubConnectionException: Connection closed for device officepc1/ipswicha-mockmodbus.\r\n2018-07-05 01:11:11.001 +00:00 [INF] - Closing device proxy for device Id officepc1/ipswicha-mockmodbus\r\n2018-07-05 01:11:11.001 +00:00 [INF] - Device connection removed for device officepc1/ipswicha-mockmodbus\r\n2018-07-05 01:11:32.992 +00:00 [INF] - Existing token not found for officepc1/plymouth-mockmodbus. Getting new token from the client...\r\n2018-07-05 01:11:32.992 +00:00 [WRN] - Closing connection for device: officepc1/plymouth-mockmodbus, Microsoft.Azure.Devices.Edge.Hub.Core.EdgeHubConnectionException: Connection closed for device officepc1/plymouth-mockmodbus., \r\n2018-07-05 01:11:32.992 +00:00 [INF] - Disposing MessagingServiceClient for device Id officepc1/plymouth-mockmodbus because of exception - Microsoft.Azure.Devices.Edge.Hub.Core.EdgeHubConnectionException: Connection closed for device officepc1/plymouth-mockmodbus.\r\n2018-07-05 01:11:32.992 +00:00 [INF] - Closing device proxy for device Id officepc1/plymouth-mockmodbus\r\n2018-07-05 01:11:32.993 +00:00 [INF] - Device connection removed for device officepc1/plymouth-mockmodbus\r\n2018-07-05 01:22:33.342 +00:00 [INF] - Exiting connected state\r\n2018-07-05 01:22:33.342 +00:00 [INF] - Entering unreachable state\r\n2018-07-05 01:23:33.354 +00:00 [INF] - Entering disconnected state\r\n```\r\n\r\nThere are also various errors in the edgeAgent around this time:\r\n\r\n```\r\n2018-07-04 23:48:38.300 +00:00 [INF] - Starting compaction of stores\r\n2018-07-04 23:48:38.300 +00:00 [INF] - Starting compaction of store deploymentConfig\r\n2018-07-04 23:48:38.301 +00:00 [INF] - Starting compaction of store default\r\n2018-07-04 23:48:38.301 +00:00 [INF] - Starting compaction of store moduleState\r\n2018-07-05 00:20:45.014 +00:00 [ERR] - Error backing up edge agent config to backup.json\r\nSystem.Threading.Tasks.TaskCanceledException: The operation was canceled. ---> System.IO.IOException: Unable to read data from the transport connection: Operation canceled. ---> System.Net.Sockets.SocketException: Operation canceled\r\n   --- End of inner exception stack trace ---\r\n   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error)\r\n   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)\r\n   at System.Net.Http.HttpConnection.FillAsync()\r\n   at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)\r\n   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)\r\n   --- End of inner exception stack trace ---\r\n   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)\r\n   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)\r\n   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)\r\n   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)\r\n   at Microsoft.Azure.Devices.Edge.Util.Edged.GeneratedCode.HttpWorkloadClient.EncryptAsync(String api_version, String name, String genid, EncryptRequest payload, CancellationToken cancellationToken) in /opt/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/generatedCode/HttpWorkloadClient.cs:line 218\r\n   at Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClient.Execute[T](Func`1 func, String operation)\r\n   at Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClient.EncryptAsync(String initializationVector, String plainText) in /opt/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/WorkloadClient.cs:line 57\r\n   at Microsoft.Azure.Devices.Edge.Agent.Core.ConfigSources.FileBackupConfigSource.BackupDeploymentConfig(DeploymentConfigInfo deploymentConfigInfo) in /opt/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/configsources/FileBackupConfigSource.cs:line 68\r\n2018-07-05 00:22:30.084 +00:00 [ERR] - Error backing up edge agent config to backup.json\r\nSystem.Threading.Tasks.TaskCanceledException: The operation was canceled. ---> System.IO.IOException: Unable to read data from the transport connection: Operation canceled. ---> System.Net.Sockets.SocketException: Operation canceled\r\n   --- End of inner exception stack trace ---\r\n   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error)\r\n   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)\r\n   at System.Net.Http.HttpConnection.FillAsync()\r\n   at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)\r\n   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)\r\n   --- End of inner exception stack trace ---\r\n   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)\r\n   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)\r\n   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)\r\n   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)\r\n   at Microsoft.Azure.Devices.Edge.Util.Edged.GeneratedCode.HttpWorkloadClient.EncryptAsync(String api_version, String name, String genid, EncryptRequest payload, CancellationToken cancellationToken) in /opt/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/generatedCode/HttpWorkloadClient.cs:line 218\r\n   at Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClient.Execute[T](Func`1 func, String operation)\r\n   at Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClient.EncryptAsync(String initializationVector, String plainText) in /opt/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/WorkloadClient.cs:line 57\r\n   at Microsoft.Azure.Devices.Edge.Agent.Core.ConfigSources.FileBackupConfigSource.BackupDeploymentConfig(DeploymentConfigInfo deploymentConfigInfo) in /opt/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/configsources/FileBackupConfigSource.cs:line 68\r\n2018-07-05 00:22:30.096 +00:00 [WRN] - Reconcile failed because of the an exception\r\nSystem.Threading.Tasks.TaskCanceledException: The operation was canceled. ---> System.IO.IOException: Unable to read data from the transport connection: Operation canceled. ---> System.Net.Sockets.SocketException: Operation canceled\r\n   --- End of inner exception stack trace ---\r\n   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error)\r\n   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)\r\n   at System.Net.Http.HttpConnection.FillAsync()\r\n   at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)\r\n   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)\r\n   --- End of inner exception stack trace ---\r\n   at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.ReconcileAsync(CancellationToken token) in /opt/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/Agent.cs:line 190\r\n2018-07-05 00:22:30.302 +00:00 [INF] - Updated reported properties\r\n```\r\n\r\nThese exceptions are then repeated indefinitely.\r\n\r\nThings I have since tried to do to get the module working again:\r\n\r\n- restart the iotedgeservice\r\n- restart the individual modules (fails with 'Hyper error' see #15)\r\n- restart docker\r\n- restart the host machine\r\n\r\nThe module still cannot connect to the edgeHub after trying all these, the new exception is:\r\n\r\n```\r\nIoT Hub module Init called\r\nUnhandled Exception: System.AggregateException: One or more errors occurred. (Operation timeout expired.) ---> System.TimeoutException: Operation timeout expired.\r\n   at Microsoft.Azure.Devices.Client.InternalClient.<>c.<ApplyTimeout>b__62_2(Task t)\r\n   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)\r\n   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at MockModbusModuleV2.Program.<Init>d__15.MoveNext() in /app/Program.cs:line 106\r\n   --- End of inner exception stack trace ---\r\n   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)\r\n   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)\r\n   at MockModbusModuleV2.Program.Main(String[] args) in /app/Program.cs:line 72\r\n```\r\n\r\nAnd the line it is hanging on now is this:\r\n\r\n`            await ioTHubModuleClient.OpenAsync();\r\n`"},"comment":{"url":"https://api.github.com/repos/Azure/iotedge/issues/comments/410835101","html_url":"https://github.com/Azure/iotedge/issues/16#issuecomment-410835101","issue_url":"https://api.github.com/repos/Azure/iotedge/issues/16","id":410835101,"node_id":"MDEyOklzc3VlQ29tbWVudDQxMDgzNTEwMQ==","user":{"login":"damonbarry","id":9042469,"node_id":"MDQ6VXNlcjkwNDI0Njk=","avatar_url":"https://avatars0.githubusercontent.com/u/9042469?v=4","gravatar_id":"","url":"https://api.github.com/users/damonbarry","html_url":"https://github.com/damonbarry","followers_url":"https://api.github.com/users/damonbarry/followers","following_url":"https://api.github.com/users/damonbarry/following{/other_user}","gists_url":"https://api.github.com/users/damonbarry/gists{/gist_id}","starred_url":"https://api.github.com/users/damonbarry/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/damonbarry/subscriptions","organizations_url":"https://api.github.com/users/damonbarry/orgs","repos_url":"https://api.github.com/users/damonbarry/repos","events_url":"https://api.github.com/users/damonbarry/events{/privacy}","received_events_url":"https://api.github.com/users/damonbarry/received_events","type":"User","site_admin":false},"created_at":"2018-08-06T20:01:32Z","updated_at":"2018-08-06T20:01:32Z","author_association":"MEMBER","body":"Alright thanks for following up. We'll close this issue for now."}}
{"actor":{"display_login":"damonbarry"}}