{"action":"opened","issue":{"url":"https://api.github.com/repos/indutny/bud/issues/93","repository_url":"https://api.github.com/repos/indutny/bud","labels_url":"https://api.github.com/repos/indutny/bud/issues/93/labels{/name}","comments_url":"https://api.github.com/repos/indutny/bud/issues/93/comments","events_url":"https://api.github.com/repos/indutny/bud/issues/93/events","html_url":"https://github.com/indutny/bud/issues/93","id":187726093,"number":93,"title":"Lots of segfaults","user":{"login":"phillipp","id":766631,"avatar_url":"https://avatars.githubusercontent.com/u/766631?v=3","gravatar_id":"","url":"https://api.github.com/users/phillipp","html_url":"https://github.com/phillipp","followers_url":"https://api.github.com/users/phillipp/followers","following_url":"https://api.github.com/users/phillipp/following{/other_user}","gists_url":"https://api.github.com/users/phillipp/gists{/gist_id}","starred_url":"https://api.github.com/users/phillipp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/phillipp/subscriptions","organizations_url":"https://api.github.com/users/phillipp/orgs","repos_url":"https://api.github.com/users/phillipp/repos","events_url":"https://api.github.com/users/phillipp/events{/privacy}","received_events_url":"https://api.github.com/users/phillipp/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-11-07T14:32:14Z","updated_at":"2016-11-07T14:32:14Z","closed_at":null,"body":"Hi,\r\nyesterday it has come to our attention that bud generates a lot of segfaults/general protection faults and traps on our production systems. For example this is the \"segfault list\" of one of these servers:\r\n\r\n```\r\nNov  7 08:18:21 zeus kernel: [630367.473383] bud[27545]: segfault at 168 ip 00000000005c92ed sp 00007fff1e836410 error 4 in bud[400000+2b3000]\r\nNov  7 08:36:07 zeus kernel: [631433.612205] bud[6319]: segfault at 168 ip 00000000005c92ed sp 00007ffd149a8f80 error 4 in bud[400000+2b3000]\r\nNov  7 08:36:08 zeus kernel: [631433.876243] bud[27585]: segfault at 168 ip 00000000005c92ed sp 00007fff2d05ea80 error 4 in bud[400000+2b3000]\r\nNov  7 08:44:24 zeus kernel: [631930.330693] bud[11727]: segfault at 168 ip 00000000005c92ed sp 00007fff1f4114b0 error 4 in bud[400000+2b3000]\r\nNov  7 09:23:56 zeus kernel: [634302.114321] bud[32231]: segfault at ffffffff ip 00000000005c05c7 sp 00007ffdd6f1a1a0 error 6 in bud[400000+2b3000]\r\nNov  7 10:53:28 zeus kernel: [639674.106113] bud[6388]: segfault at 168 ip 00000000005c92ed sp 00007ffc34b44840 error 4 in bud[400000+2b3000]\r\nNov  7 10:53:29 zeus kernel: [639674.970723] bud[11675]: segfault at 168 ip 00000000005c92ed sp 00007ffec993c500 error 4 in bud[400000+2b3000]\r\nNov  7 11:25:40 zeus kernel: [641605.626789] bud[32186]: segfault at 168 ip 00000000005c92ed sp 00007ffcb23bb890 error 4 in bud[400000+2b3000]\r\nNov  7 12:11:50 zeus kernel: [644375.364522] bud[32229]: segfault at 168 ip 00000000005c92ed sp 00007ffffdf848f0 error 4 in bud[400000+2b3000]\r\nNov  7 12:11:50 zeus kernel: [644375.420023] bud[9447]: segfault at 168 ip 00000000005c92ed sp 00007fff77a8c470 error 4 in bud[400000+2b3000]\r\nNov  7 15:15:56 zeus kernel: [655420.863052] bud[19106]: segfault at 168 ip 00000000005c92ed sp 00007ffd7a1e3bb0 error 4 in bud[400000+2b3000]\r\n```\r\n\r\nWe have multiple servers with the same pattern. After a restart of the master everything is back to normal. When I grep the logs for one of these workers that dies I get the following logs (non-debug right now):\r\n\r\n```\r\nNov  7 11:54:36 hera bud[4482]: spawned bud worker<26364>\r\nNov  7 11:54:36 hera bud[26364]: Worker updated ticket key for context: 0\r\nNov  7 12:01:15 hera bud[26364]: Worker updated ticket key for context: 0\r\nNov  7 12:05:51 hera bud[26364]: client 0x15d4060 on backend closed because: uv_shutdown(client) cb returned -107 (socket is not connected)\r\nNov  7 12:10:07 hera bud[26364]: client 0x1555e30 on backend closed because: uv_shutdown(client) cb returned -107 (socket is not connected)\r\nNov  7 12:43:20 hera bud[4482]: bud worker<26364> died, signal: 11\r\n```\r\n\r\nI've activated the debug logs and core dumps to get to the bottom of this. From the older syslogs it looks like this problem exists for quite some time but was unnoticed. It only has come to our attention becuase the segfaults spiked yesterday and connections were dropped.\r\n\r\nI think we should investigate this thoroughly, because the segfaults might be a hint of a potential vulnerability."}}
{"actor":{"display_login":"phillipp"}}