{"action":"created","issue":{"url":"https://api.github.com/repos/nodejs/post-mortem/issues/18","repository_url":"https://api.github.com/repos/nodejs/post-mortem","labels_url":"https://api.github.com/repos/nodejs/post-mortem/issues/18/labels{/name}","comments_url":"https://api.github.com/repos/nodejs/post-mortem/issues/18/comments","events_url":"https://api.github.com/repos/nodejs/post-mortem/issues/18/events","html_url":"https://github.com/nodejs/post-mortem/issues/18","id":132683366,"number":18,"title":"Responsibly aborting in node","user":{"login":"Raynos","id":479538,"avatar_url":"https://avatars.githubusercontent.com/u/479538?v=3","gravatar_id":"","url":"https://api.github.com/users/Raynos","html_url":"https://github.com/Raynos","followers_url":"https://api.github.com/users/Raynos/followers","following_url":"https://api.github.com/users/Raynos/following{/other_user}","gists_url":"https://api.github.com/users/Raynos/gists{/gist_id}","starred_url":"https://api.github.com/users/Raynos/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Raynos/subscriptions","organizations_url":"https://api.github.com/users/Raynos/orgs","repos_url":"https://api.github.com/users/Raynos/repos","events_url":"https://api.github.com/users/Raynos/events{/privacy}","received_events_url":"https://api.github.com/users/Raynos/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":9,"created_at":"2016-02-10T12:15:30Z","updated_at":"2016-02-10T21:53:30Z","closed_at":null,"body":"Based on a conversation with @benjamingr ( https://github.com/groundwater/nodejs-symposiums/pull/5#issuecomment-182313354 ).\r\n\r\nWe recommend aborting and core dumps as the debugging best practice. However we don't document how to do so responsibly to ensure uptime and availability.\r\n\r\nPeople object to exiting and restarting as its a potential DDOS attack vector, I've brushed this off in the past by saying \"it's a bug. Just get paged and fix this bug\", however I no longer think this can be ignored for a few reasons\r\n\r\n - I've seen accidental internal DDOS due to cascading failures.\r\n - I've seen how even 10 minutes of outage can lose millions in revenue.\r\n - I've even seen accidental DDOS simply due to an embedded gossip library having a corruptive bug.\r\n - I've seen a fleet of servers being destroyed due to a single user uuid and round robin load balancing \r\n\r\nI 100% believe a core dump is the only way to debug that 1 in a 100 edgecase. We all speak positively about how amazing post mortem debugging is but there is little sharing around how to do so responsibly.\r\n\r\nHere is a straw man suggestion and I would love feedback:\r\n\r\n - a programmer error is a bug and needs human attention to verify any undefined state. We recommend setting up alerts or email notifications and verifying the impact of the programmer error.\r\n - running an available service is important, you cannot just abort without consequence. You need a technique to avoid repeated aborting.\r\n\r\nWe could implement --abort-on-uncaught-exception-once=filepath flag.\r\n\r\n - it would check the file on filepath, if it exists, read the value of the file as Unix time stamp (second granularity). \r\n - if the time is less then 24hr in the past then instead of aborting create an error and pass it to process.on(uncaughtException)\r\n - if the file does not exist or time is greater then 24hr in the past then write the current Unix time stamp (second granularity) to the file then continue to abort.\r\n\r\nThis functionality could be implemented outside of node core. If it is we have to clearly in the node core documentation, instruct people how to use --abort-on-uncaught-exception safely.\r\n\r\nBecause the process will not abort for 24hr the application developer must use process.on(uncaughtException) to avoid exiting and to be able to degrade to partial availability.\r\n\r\nI would love to hear some feedback on how to best responsibly use post mortem debugging techniques"},"comment":{"url":"https://api.github.com/repos/nodejs/post-mortem/issues/comments/182597834","html_url":"https://github.com/nodejs/post-mortem/issues/18#issuecomment-182597834","issue_url":"https://api.github.com/repos/nodejs/post-mortem/issues/18","id":182597834,"user":{"login":"Raynos","id":479538,"avatar_url":"https://avatars.githubusercontent.com/u/479538?v=3","gravatar_id":"","url":"https://api.github.com/users/Raynos","html_url":"https://github.com/Raynos","followers_url":"https://api.github.com/users/Raynos/followers","following_url":"https://api.github.com/users/Raynos/following{/other_user}","gists_url":"https://api.github.com/users/Raynos/gists{/gist_id}","starred_url":"https://api.github.com/users/Raynos/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Raynos/subscriptions","organizations_url":"https://api.github.com/users/Raynos/orgs","repos_url":"https://api.github.com/users/Raynos/repos","events_url":"https://api.github.com/users/Raynos/events{/privacy}","received_events_url":"https://api.github.com/users/Raynos/received_events","type":"User","site_admin":false},"created_at":"2016-02-10T21:53:30Z","updated_at":"2016-02-10T21:53:30Z","body":"@bnoordhuis \r\n\r\nSounds like a good technical solution.\r\n\r\nDo you think this is something worth recommending ? Should we recommend to never continue on uncaughts ? What is a responsible default approach to recommend for production ? \r\n\r\nAlternatively we can also recommend \"here are some possible options; figure it out yourself!\""}}