{"action":"created","issue":{"url":"https://api.github.com/repos/Netflix/dynomite/issues/321","repository_url":"https://api.github.com/repos/Netflix/dynomite","labels_url":"https://api.github.com/repos/Netflix/dynomite/issues/321/labels{/name}","comments_url":"https://api.github.com/repos/Netflix/dynomite/issues/321/comments","events_url":"https://api.github.com/repos/Netflix/dynomite/issues/321/events","html_url":"https://github.com/Netflix/dynomite/issues/321","id":171700882,"number":321,"title":"Segfault on client write","user":{"login":"bp8017","id":7372444,"avatar_url":"https://avatars.githubusercontent.com/u/7372444?v=3","gravatar_id":"","url":"https://api.github.com/users/bp8017","html_url":"https://github.com/bp8017","followers_url":"https://api.github.com/users/bp8017/followers","following_url":"https://api.github.com/users/bp8017/following{/other_user}","gists_url":"https://api.github.com/users/bp8017/gists{/gist_id}","starred_url":"https://api.github.com/users/bp8017/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bp8017/subscriptions","organizations_url":"https://api.github.com/users/bp8017/orgs","repos_url":"https://api.github.com/users/bp8017/repos","events_url":"https://api.github.com/users/bp8017/events{/privacy}","received_events_url":"https://api.github.com/users/bp8017/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2016-08-17T16:22:59Z","updated_at":"2016-08-17T16:40:50Z","closed_at":null,"body":"I have built dynomite on a fresh Ubuntu 14.04 EC2 instance and am getting segfaults when writing to one of the dynomite nodes in the replication cluster.\r\n\r\nScenario:\r\n2 dynomite nodes, 1 in us-west-2 and 1 in us-east-1\r\n1 redis node in us-west-2 and 1 redis node in us-east-1\r\n\r\nWhen starting dynomite node in west first, if I write to it, I get a segfault, but the data is replicated to east. When starting dynomite node in east first, if I write to west dynomite node, there is no crash and data is replicated as expected.\r\n\r\nRelevant configs and logs:\r\n```\r\nroot@ip-XXXXX:~/redis/src# cat /root/dynomite-east.yml \r\ndyn_o_mite:\r\n  datacenter: us-east-1\r\n  rack: rack1\r\n  dyn_listen: 0.0.0.0:8101\r\n  dyn_seeds:\r\n  - XXXXX:8101:rack2:dc2:5468952198\r\n  listen: 0.0.0.0:8102\r\n  servers:\r\n  - dynomite-test-east.XXXXX.0001.use1.cache.amazonaws.com:6379:1\r\n  tokens: '5468952198'\r\n  secure_server_option: datacenter\r\n  pem_key_file: /root/dynomite/conf/dynomite-test.pem\r\n  data_store: 0\r\n\r\nroot@ip-XXXXX:~/dynomite/src# cat /root/dynomite-west.yml \r\ndyn_o_mite:\r\n  datacenter: us-west-2\r\n  rack: rack2\r\n  dyn_listen: 0.0.0.0:8101\r\n  dyn_seeds:\r\n  - XXXXX:8101:rack1:dc1:5468952198\r\n  listen: 0.0.0.0:8102\r\n  servers:\r\n  - dynomite-test-west.XXXXX.0001.usw2.cache.amazonaws.com:6379:1\r\n  tokens: '5468952198'\r\n  secure_server_option: datacenter\r\n  pem_key_file: /root/dynomite/conf/dynomite-test.pem\r\n  data_store: 0\r\n\r\nCRASH LOG:\r\n[2016-08-17 16:19:40.366] dn_assert:336 assert 'ep > 0' failed @ (dyn_epoll.c, 199)\r\n[2016-08-17 16:19:40.366] proxy_accept:220 accepted CLIENT 10 on PROXY 6 from '127.0.0.1:55315'\r\n[2016-08-17 16:19:40.367] dn_assert:336 assert 'ep > 0' failed @ (dyn_epoll.c, 249)\r\n[2016-08-17 16:19:40.367] _msg_get:286 alloc_msg_count: 1 caller: req_get conn: CLIENT sd: 10\r\n[2016-08-17 16:19:40.367] dn_assert:336 assert 'ep > 0' failed @ (dyn_epoll.c, 137)\r\n[2016-08-17 16:19:40.367] _msg_get:286 alloc_msg_count: 2 caller: req_forward_remote_dc conn: CLIENT sd: 10\r\n[2016-08-17 16:19:40.367] dn_assert:336 assert 'ep > 0' failed @ (dyn_epoll.c, 137)\r\n[2016-08-17 16:19:40.367] dn_assert:336 assert 'ep > 0' failed @ (dyn_epoll.c, 168)\r\n[2016-08-17 16:19:40.367] dn_assert:336 assert 'ep > 0' failed @ (dyn_epoll.c, 249)\r\n[2016-08-17 16:19:40.367] dn_assert:336 assert 'ep > 0' failed @ (dyn_epoll.c, 168)\r\n[2016-08-17 16:19:40.367] dn_assert:336 assert 'ep > 0' failed @ (dyn_epoll.c, 168)\r\n[2016-08-17 16:19:40.367] dn_assert:336 assert 'ep > 0' failed @ (dyn_epoll.c, 249)\r\n[2016-08-17 16:19:40.368] _msg_get:286 alloc_msg_count: 3 caller: rsp_get conn: SERVER sd: 5\r\n[2016-08-17 16:19:40.368] dn_assert:336 assert 'ep > 0' failed @ (dyn_epoll.c, 137)\r\n[2016-08-17 16:19:40.368] dn_assert:336 assert 'ep > 0' failed @ (dyn_epoll.c, 249)\r\n[2016-08-17 16:19:40.368] dn_assert:336 assert 'ep > 0' failed @ (dyn_epoll.c, 168)\r\n[2016-08-17 16:19:40.368] dn_assert:336 assert 'ep > 0' failed @ (dyn_epoll.c, 249)\r\n[2016-08-17 16:19:40.453] dn_stacktrace:320 [0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x10330) [0x7fcbd165f330]\r\n??:0\r\n[2016-08-17 16:19:40.455] dn_stacktrace:320 [1] ./dynomite(dmsg_get+0x25) [0x414045]\r\n/root/dynomite/src/dyn_dnode_msg.c:707\r\n[2016-08-17 16:19:40.457] dn_stacktrace:320 [2] ./dynomite() [0x414d9f]\r\n/root/dynomite/src/dyn_dnode_msg.c:50\r\n[2016-08-17 16:19:40.459] dn_stacktrace:320 [3] ./dynomite(dyn_parse_rsp+0x40) [0x415fc0]\r\n/root/dynomite/src/dyn_dnode_msg.c:558 (discriminator 3)\r\n[2016-08-17 16:19:40.461] dn_stacktrace:320 [4] ./dynomite(msg_recv+0x1fe) [0x41fa8e]\r\n/root/dynomite/src/dyn_message.c:874\r\n[2016-08-17 16:19:40.463] dn_stacktrace:320 [5] ./dynomite(core_core+0xe7) [0x40db77]\r\n/root/dynomite/src/dyn_core.c:271\r\n[2016-08-17 16:19:40.465] dn_stacktrace:320 [6] ./dynomite(event_wait+0x164) [0x43be84]\r\n/root/dynomite/src/event/dyn_epoll.c:278\r\n[2016-08-17 16:19:40.468] dn_stacktrace:320 [7] ./dynomite(core_loop+0x9d) [0x40e89d]\r\n/root/dynomite/src/dyn_core.c:539\r\n[2016-08-17 16:19:40.469] dn_stacktrace:320 [8] ./dynomite(main+0x718) [0x40c328]\r\n/root/dynomite/src/dynomite.c:601\r\n[2016-08-17 16:19:40.472] dn_stacktrace:320 [9] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7fcbd0fa5f45]\r\n??:0\r\n[2016-08-17 16:19:40.473] dn_stacktrace:320 [10] ./dynomite() [0x40c783]\r\n??:?\r\n[2016-08-17 16:19:40.475] signal_handler:127 signal 11 (SIGSEGV) received, core dumping\r\nSegmentation fault (core dumped)\r\n\r\nLOG FROM PEER WITH SUCCESSFUL REPLICATION:\r\n[2016-08-17 16:19:38.016] dnode_accept:168 Accepting client connection from XXXXX/55059 on sd 14\r\n[2016-08-17 16:19:38.016] dn_assert:336 assert 'ep > 0' failed @ (dyn_epoll.c, 199)\r\n[2016-08-17 16:19:38.016] dnode_accept:213 dyn: accepted LOCAL_PEER_CLIENT 14 on PEER_PROXY 7 from 'XXXXX:55059'\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/Netflix/dynomite/issues/comments/240471399","html_url":"https://github.com/Netflix/dynomite/issues/321#issuecomment-240471399","issue_url":"https://api.github.com/repos/Netflix/dynomite/issues/321","id":240471399,"user":{"login":"ipapapa","id":4562887,"avatar_url":"https://avatars.githubusercontent.com/u/4562887?v=3","gravatar_id":"","url":"https://api.github.com/users/ipapapa","html_url":"https://github.com/ipapapa","followers_url":"https://api.github.com/users/ipapapa/followers","following_url":"https://api.github.com/users/ipapapa/following{/other_user}","gists_url":"https://api.github.com/users/ipapapa/gists{/gist_id}","starred_url":"https://api.github.com/users/ipapapa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ipapapa/subscriptions","organizations_url":"https://api.github.com/users/ipapapa/orgs","repos_url":"https://api.github.com/users/ipapapa/repos","events_url":"https://api.github.com/users/ipapapa/events{/privacy}","received_events_url":"https://api.github.com/users/ipapapa/received_events","type":"User","site_admin":false},"created_at":"2016-08-17T16:40:50Z","updated_at":"2016-08-17T16:40:50Z","body":"@bp8017 try master as @shailesh33 proposed, and if you can send us the logs for the dev branch that would be useful."}}
{"actor":{"display_login":"ipapapa"}}