{"action":"created","issue":{"url":"https://api.github.com/repos/emqtt/emqttd/issues/1357","repository_url":"https://api.github.com/repos/emqtt/emqttd","labels_url":"https://api.github.com/repos/emqtt/emqttd/issues/1357/labels{/name}","comments_url":"https://api.github.com/repos/emqtt/emqttd/issues/1357/comments","events_url":"https://api.github.com/repos/emqtt/emqttd/issues/1357/events","html_url":"https://github.com/emqtt/emqttd/issues/1357","id":277324052,"number":1357,"title":"Connection Id 462860614 closing with reason: tcp_closed","user":{"login":"venks-warrior","id":29368745,"avatar_url":"https://avatars2.githubusercontent.com/u/29368745?v=4","gravatar_id":"","url":"https://api.github.com/users/venks-warrior","html_url":"https://github.com/venks-warrior","followers_url":"https://api.github.com/users/venks-warrior/followers","following_url":"https://api.github.com/users/venks-warrior/following{/other_user}","gists_url":"https://api.github.com/users/venks-warrior/gists{/gist_id}","starred_url":"https://api.github.com/users/venks-warrior/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/venks-warrior/subscriptions","organizations_url":"https://api.github.com/users/venks-warrior/orgs","repos_url":"https://api.github.com/users/venks-warrior/repos","events_url":"https://api.github.com/users/venks-warrior/events{/privacy}","received_events_url":"https://api.github.com/users/venks-warrior/received_events","type":"User","site_admin":false},"labels":[{"id":22935817,"url":"https://api.github.com/repos/emqtt/emqttd/labels/BUG","name":"BUG","color":"e10c02","default":false}],"state":"open","locked":false,"assignee":{"login":"HJianBo","id":13825269,"avatar_url":"https://avatars1.githubusercontent.com/u/13825269?v=4","gravatar_id":"","url":"https://api.github.com/users/HJianBo","html_url":"https://github.com/HJianBo","followers_url":"https://api.github.com/users/HJianBo/followers","following_url":"https://api.github.com/users/HJianBo/following{/other_user}","gists_url":"https://api.github.com/users/HJianBo/gists{/gist_id}","starred_url":"https://api.github.com/users/HJianBo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/HJianBo/subscriptions","organizations_url":"https://api.github.com/users/HJianBo/orgs","repos_url":"https://api.github.com/users/HJianBo/repos","events_url":"https://api.github.com/users/HJianBo/events{/privacy}","received_events_url":"https://api.github.com/users/HJianBo/received_events","type":"User","site_admin":false},"assignees":[{"login":"HJianBo","id":13825269,"avatar_url":"https://avatars1.githubusercontent.com/u/13825269?v=4","gravatar_id":"","url":"https://api.github.com/users/HJianBo","html_url":"https://github.com/HJianBo","followers_url":"https://api.github.com/users/HJianBo/followers","following_url":"https://api.github.com/users/HJianBo/following{/other_user}","gists_url":"https://api.github.com/users/HJianBo/gists{/gist_id}","starred_url":"https://api.github.com/users/HJianBo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/HJianBo/subscriptions","organizations_url":"https://api.github.com/users/HJianBo/orgs","repos_url":"https://api.github.com/users/HJianBo/repos","events_url":"https://api.github.com/users/HJianBo/events{/privacy}","received_events_url":"https://api.github.com/users/HJianBo/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/emqtt/emqttd/milestones/110","html_url":"https://github.com/emqtt/emqttd/milestone/110","labels_url":"https://api.github.com/repos/emqtt/emqttd/milestones/110/labels","id":2840084,"number":110,"title":"3.0-beta.1","description":"","creator":{"login":"emqplus","id":11869,"avatar_url":"https://avatars0.githubusercontent.com/u/11869?v=4","gravatar_id":"","url":"https://api.github.com/users/emqplus","html_url":"https://github.com/emqplus","followers_url":"https://api.github.com/users/emqplus/followers","following_url":"https://api.github.com/users/emqplus/following{/other_user}","gists_url":"https://api.github.com/users/emqplus/gists{/gist_id}","starred_url":"https://api.github.com/users/emqplus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/emqplus/subscriptions","organizations_url":"https://api.github.com/users/emqplus/orgs","repos_url":"https://api.github.com/users/emqplus/repos","events_url":"https://api.github.com/users/emqplus/events{/privacy}","received_events_url":"https://api.github.com/users/emqplus/received_events","type":"User","site_admin":false},"open_issues":42,"closed_issues":8,"state":"open","created_at":"2017-10-16T12:44:57Z","updated_at":"2018-03-05T11:49:59Z","due_on":null,"closed_at":null},"comments":3,"created_at":"2017-11-28T10:03:06Z","updated_at":"2018-04-06T17:49:06Z","closed_at":null,"author_association":"NONE","body":"#### Environment\r\n\r\n- OS: Linux\r\n- Erlang/OTP: OTP\r\n- EMQ: 2.3\r\n\r\n#### Description\r\n\r\nHi , \r\n\r\nI am using Spark job to read events from EMQTT brokers.I see that, client connection is lost regularly with following errors in emqtt broker.\r\n\r\n09:52:22.497 [debug] Client(*.*.*.*:20594): Terminated for {shutdown,closed}\r\n09:52:23.368 [error] Connection Id 462860614 closing with reason: tcp_closed\r\n09:52:23.368 [error] gen_server <0.1889.0> terminated with reason: tcp_closed\r\n09:52:23.369 [error] CRASH REPORT Process <0.1889.0> with 0 neighbours exited with reason: tcp_closed in gen_server:handle_common_reply/8 line 726\r\n09:52:25.756 [error] Connection Id 462860789 closing with reason: tcp_closed\r\n09:52:25.756 [error] gen_server <0.1890.0> terminated with reason: tcp_closed\r\n09:52:25.756 [error] CRASH REPORT Process <0.1890.0> with 0 neighbours exited with reason: tcp_closed in gen_server:handle_common_reply/8 line 726\r\n09:52:26.072 [debug] Client(10.200.8.176:5549): Terminated for {shutdown,closed}\r\n09:52:26.369 [error] Connection Id 462860817 closing with reason: tcp_closed\r\n09:52:26.369 [error] gen_server <0.1892.0> terminated with reason: tcp_closed\r\n09:52:26.369 [error] CRASH REPORT Process <0.1892.0> with 0 neighbours exited with reason: tcp_closed in gen_server:handle_common_reply/8 line 726\r\n09:52:26.756 [error] Connection Id 462860818 closing with reason: tcp_closed\r\n09:52:26.756 [error] gen_server <0.1893.0> terminated with reason: tcp_closed\r\n09:52:26.756 [error] CRASH REPORT Process <0.1893.0> with 0 neighbours exited with reason: tcp_closed in gen_server:handle_common_reply/8 line 726\r\n09:52:46.369 [error] Connection Id 462862625 closing with reason: tcp_closed\r\n09:52:46.369 [error] gen_server <0.1902.0> terminated with reason: tcp_closed\r\n09:52:46.369 [error] CRASH REPORT Process <0.1902.0> with 0 neighbours exited with reason: tcp_closed in gen_server:handle_common_reply/8 line 726\r\n09:52:46.756 [error] Connection Id 462862692 closing with reason: tcp_closed\r\n09:52:46.756 [error] gen_server <0.1903.0> terminated with reason: tcp_closed\r\n09:52:46.756 [error] CRASH REPORT Process <0.1903.0> with 0 neighbours exited with reason: tcp_closed in gen_server:handle_common_reply/8 line 726\r\n09:52:46.756 [error] Connection Id 462862693 closing with reason: tcp_closed\r\n09:52:46.756 [error] gen_server <0.1904.0> terminated with reason: tcp_closed\r\n09:52:46.756 [error] CRASH REPORT Process <0.1904.0> with 0 neighbours exited with reason: tcp_closed in gen_server:handle_common_reply/8 line 726\r\n09:52:52.497 [debug] Client(*.*.*.*:20614): Terminated for {shutdown,closed}\r\n09:52:56.071 [debug] Client(10.200.8.176:5567): Terminated for {shutdown,closed}\r\n\r\n\r\n[error] Supervisor 'esockd_connection_sup - <0.1346.0>' had child connection started with mochiweb_http:start_link({emq_dashboard,handle_request,[{state,\"/usr/lib/emqttd/lib/emq_dashboard-2.3/priv/www\",#Fun<emq_das..>}]}) at <0.1524.0> exit with reason {json_encode,{bad_term,{error,nodedown}}} in context connection_crashed\r\n10:31:00.697 [debug] Mod:emqttd_rest_api, Fun:nodes, Args:['GET',[]]\r\n10:31:00.699 [debug] Mod:emqttd_rest_api, Fun:stat, Args:['GET',[],\"emq@10.200.6.68\"]\r\n10:31:00.701 [debug] Mod:emqttd_rest_api, Fun:broker, Args:['GET',[],\"emq@10.200.6.68\"]\r\n10:31:07.701 [error] Execute API 'management/nodes/emq@10.200.6.68' Error: {function_clause,[{proplists,get_value,[version,{error,nodedown},undefined],[{file,\"proplists.erl\"},{line,215}]},{emqttd_rest_api,format_broker,1,[{file,\"src/emqttd_rest_api.erl\"},{line,305}]},{emqttd_rest_api,broker,3,[{file,\"src/emqttd_rest_api.erl\"},{line,267}]},{emqttd_http,'-dispatcher/1-fun-6-',3,[{file,\"src/emqttd_http.erl\"},{line,101}]},{mochiweb_http,headers,5,[{file,\"src/mochiweb_http.erl\"},{line,75}]}]}\r\n10:31:07.702 [error] Supervisor 'esockd_connection_sup - <0.1346.0>' had child connection started with mochiweb_http:start_link({emq_dashboard,handle_request,[{state,\"/usr/lib/emqttd/lib/emq_dashboard-2.3/priv/www\",#Fun<emq_das..>}]}) at <0.1533.0> exit with reason {json_encode,{bad_term,{error,nodedown}}} in context connection_crashed\r\n\r\n\r\nI have enabled debug logs.But nothing else seems to be wrong.Also, i am producing my events from java paho client and while producing no issues are seen.\r\n\r\n\r\n\r\n\r\n\r\n*A description of the issue*\r\n\r\nPlease help me solve with this.\r\n\r\nFollowing are the exception logs in my spark streaming job.\r\n\r\n17/11/28 12:52:59 WARN ReceiverSupervisorImpl: Restarting receiver with delay 2000 ms: Connection lost \r\nConnection lost (32109) - java.io.EOFException\r\n\tat org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:146)\r\n\tat java.lang.Thread.run(Thread.java:745)\r\nCaused by: java.io.EOFException\r\n\tat java.io.DataInputStream.readByte(DataInputStream.java:267)\r\n\tat org.eclipse.paho.client.mqttv3.internal.wire.MqttInputStream.readMqttWireMessage(MqttInputStream.java:65)\r\n\tat org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:107)\r\n\t... 1 more\r\n17/11/28 12:52:59 INFO ReceiverSupervisorImpl: Stopping receiver with message: Restarting receiver with delay 2000ms: Connection lost : Connection lost (32109) - java.io.EOFException\r\n17/11/28 12:52:59 INFO ReceiverSupervisorImpl: Called receiver onStop\r\n17/11/28 12:52:59 INFO ReceiverSupervisorImpl: Deregistering receiver 0\r\n17/11/28 12:52:59 INFO ReceiverSupervisorImpl: Stopped receiver 0\r\n17/11/28 12:53:01 INFO ReceiverSupervisorImpl: Starting receiver again\r\n17/11/28 12:53:01 INFO ReceiverSupervisorImpl: Starting receiver 0\r\n17/11/28 12:53:01 INFO ReceiverSupervisorImpl: Called receiver 0 onStart\r\n17/11/28 12:53:01 INFO ReceiverSupervisorImpl: Receiver started again\r\n17/11/28 12:53:02 INFO MemoryStore: Block input-0-1511853781800 stored as bytes in memory (estimated size 956.0 B, free 912.0 MB)\r\n17/11/28 12:53:02 WARN RandomBlockReplicationPolicy: Expecting 1 replicas with only 0 peer/s.\r\n17/11/28 12:53:02 WARN BlockManager: Block input-0-1511853781800 replicated to only 0 peer(s) instead of 1 peers\r\n17/11/28 12:53:02 INFO BlockGenerator: Pushed block input-0-1511853781800\r\n17/11/28 12:53:21 WARN ReceiverSupervisorImpl: Restarting receiver with delay 2000 ms: Connection lost \r\nConnection lost (32109) - java.io.EOFException\r\n\tat org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:146)\r\n\tat java.lang.Thread.run(Thread.java:745)\r\nCaused by: java.io.EOFException\r\n\tat java.io.DataInputStream.readByte(DataInputStream.java:267)\r\n\tat org.eclipse.paho.client.mqttv3.internal.wire.MqttInputStream.readMqttWireMessage(MqttInputStream.java:65)\r\n\tat org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(CommsReceiver.java:107)\r\n\t... 1 more\r\n17/11/28 12:53:21 INFO ReceiverSupervisorImpl: Stopping receiver with message: Restarting receiver with delay 2000ms: Connection lost : Connection lost (32109) - java.io.EOFException\r\n17/11/28 12:53:21 INFO ReceiverSupervisorImpl: Called receiver onStop\r\n17/11/28 12:53:21 INFO ReceiverSupervisorImpl: Deregistering receiver 0\r\n17/11/28 12:53:21 INFO ReceiverSupervisorImpl: Stopped receiver 0\r\n17/11/28 12:53:23 INFO ReceiverSupervisorImpl: Starting receiver again\r\n17/11/28 12:53:23 INFO ReceiverSupervisorImpl: Starting receiver 0\r\n17/11/28 12:53:23 INFO ReceiverSupervisorImpl: Called receiver 0 onStart\r\n17/11/28 12:53:23 INFO ReceiverSupervisorImpl: Receiver started again\r\n17/11/28 12:53:23 INFO MemoryStore: Block input-0-1511853803600 stored as bytes in memory (estimated size 969.0 B, free 912.0 MB)\r\n17/11/28 12:53:23 WARN RandomBlockReplicationPolicy: Expecting 1 replicas with only 0 peer/s.\r\n17/11/28 12:53:23 WARN BlockManager: Block input-0-1511853803600 replicated to only 0 peer(s) instead of 1 peers\r\n17/11/28 12:53:23 INFO BlockGenerator: Pushed block input-0-1511853803600\r\n17/11/28 12:53:52 WARN ReceiverSupervisorImpl: Restarting receiver with delay 2000 ms: Connection lost \r\n\r\n\r\nI am using apache bahir library for reading through MQTT stream\r\n"},"comment":{"url":"https://api.github.com/repos/emqtt/emqttd/issues/comments/379327502","html_url":"https://github.com/emqtt/emqttd/issues/1357#issuecomment-379327502","issue_url":"https://api.github.com/repos/emqtt/emqttd/issues/1357","id":379327502,"user":{"login":"bradleyd","id":151280,"avatar_url":"https://avatars0.githubusercontent.com/u/151280?v=4","gravatar_id":"","url":"https://api.github.com/users/bradleyd","html_url":"https://github.com/bradleyd","followers_url":"https://api.github.com/users/bradleyd/followers","following_url":"https://api.github.com/users/bradleyd/following{/other_user}","gists_url":"https://api.github.com/users/bradleyd/gists{/gist_id}","starred_url":"https://api.github.com/users/bradleyd/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bradleyd/subscriptions","organizations_url":"https://api.github.com/users/bradleyd/orgs","repos_url":"https://api.github.com/users/bradleyd/repos","events_url":"https://api.github.com/users/bradleyd/events{/privacy}","received_events_url":"https://api.github.com/users/bradleyd/received_events","type":"User","site_admin":false},"created_at":"2018-04-06T17:49:06Z","updated_at":"2018-04-06T17:49:06Z","author_association":"NONE","body":"I killed my local epmd and restarted.  The errors went away if because the node from earlier sessions was gone.  Hope that helps @venks-warrior "}}
{"actor":{"display_login":"bradleyd"}}