{"action":"created","issue":{"url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/5551","repository_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j","labels_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/5551/labels{/name}","comments_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/5551/comments","events_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/5551/events","html_url":"https://github.com/deeplearning4j/deeplearning4j/pull/5551","id":331033091,"node_id":"MDExOlB1bGxSZXF1ZXN0MTkzODY1NzM1","number":5551,"title":"DL4J: Add OOM Crash Dump Reporting","user":{"login":"AlexDBlack","id":2360237,"node_id":"MDQ6VXNlcjIzNjAyMzc=","avatar_url":"https://avatars0.githubusercontent.com/u/2360237?v=4","gravatar_id":"","url":"https://api.github.com/users/AlexDBlack","html_url":"https://github.com/AlexDBlack","followers_url":"https://api.github.com/users/AlexDBlack/followers","following_url":"https://api.github.com/users/AlexDBlack/following{/other_user}","gists_url":"https://api.github.com/users/AlexDBlack/gists{/gist_id}","starred_url":"https://api.github.com/users/AlexDBlack/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AlexDBlack/subscriptions","organizations_url":"https://api.github.com/users/AlexDBlack/orgs","repos_url":"https://api.github.com/users/AlexDBlack/repos","events_url":"https://api.github.com/users/AlexDBlack/events{/privacy}","received_events_url":"https://api.github.com/users/AlexDBlack/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-06-11T03:15:37Z","updated_at":"2018-06-18T01:02:58Z","closed_at":null,"author_association":"MEMBER","pull_request":{"url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/pulls/5551","html_url":"https://github.com/deeplearning4j/deeplearning4j/pull/5551","diff_url":"https://github.com/deeplearning4j/deeplearning4j/pull/5551.diff","patch_url":"https://github.com/deeplearning4j/deeplearning4j/pull/5551.patch"},"body":"Fixes: https://github.com/deeplearning4j/deeplearning4j/issues/5535\r\n\r\nThe basic idea: when DL4J (MLN and CG specifically) OOMs during training or inference, write an info dump to aid in understanding why, debugging and improving memory config and/or network config.\r\n\r\n\r\nOutput (real-world ResNet50 OOM)\r\nhttps://gist.github.com/AlexDBlack/c605a01dd20300e753dadd7296df6252\r\n\r\nLogged on OOM:\r\n(Note this will by default be written to the current working directory (path here is for unit test))\r\n```\r\no.d.u.CrashReportingUtil - >>> Out of Memory Exception Detected. Memory crash dump written to: C:\\Users\\Alex\\AppData\\Local\\Temp\\junit7109276185443170604\\junit1948375717567978708\\dl4j-memory-crash-dump-1528686384749_1.txt\r\no.d.u.CrashReportingUtil - Memory crash dump reporting can be disabled with CrashUtil.crashDumpsEnabled(false) or using system property -Dorg.deeplearning4j.crash.reporting.enabled=false\r\no.d.u.CrashReportingUtil - Memory crash dump reporting output location CrashUtil.crashDumpOutputDirectory(File) or using system property -Dorg.deeplearning4j.crash.reporting.directory=<path>\r\n```\r\n\r\nRemaining improvements I want to make - but I think I will merge it now, and do a second pass at a later date.\r\n1. Where appropriate, add memory use info for async iterator (maybe unnecessary with (3))\r\n2. Add basic information/explanation to crash dump; add link to DL4J memory page\r\n3. Add info about workspaces used in other threads\r\n"},"comment":{"url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/comments/397919272","html_url":"https://github.com/deeplearning4j/deeplearning4j/pull/5551#issuecomment-397919272","issue_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/5551","id":397919272,"node_id":"MDEyOklzc3VlQ29tbWVudDM5NzkxOTI3Mg==","user":{"login":"AlexDBlack","id":2360237,"node_id":"MDQ6VXNlcjIzNjAyMzc=","avatar_url":"https://avatars0.githubusercontent.com/u/2360237?v=4","gravatar_id":"","url":"https://api.github.com/users/AlexDBlack","html_url":"https://github.com/AlexDBlack","followers_url":"https://api.github.com/users/AlexDBlack/followers","following_url":"https://api.github.com/users/AlexDBlack/following{/other_user}","gists_url":"https://api.github.com/users/AlexDBlack/gists{/gist_id}","starred_url":"https://api.github.com/users/AlexDBlack/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AlexDBlack/subscriptions","organizations_url":"https://api.github.com/users/AlexDBlack/orgs","repos_url":"https://api.github.com/users/AlexDBlack/repos","events_url":"https://api.github.com/users/AlexDBlack/events{/privacy}","received_events_url":"https://api.github.com/users/AlexDBlack/received_events","type":"User","site_admin":false},"created_at":"2018-06-18T01:02:58Z","updated_at":"2018-06-18T01:02:58Z","author_association":"MEMBER","body":"@raver119 Added ```.memoryInfo(int minibatch, InputType inputType)``` convenience methods to make the same info available without a crash.\r\n\r\nThis should be good to do."}}
{"actor":{"display_login":"AlexDBlack"}}