{"action":"created","issue":{"url":"https://api.github.com/repos/artsy/eigen/issues/1246","repository_url":"https://api.github.com/repos/artsy/eigen","labels_url":"https://api.github.com/repos/artsy/eigen/issues/1246/labels{/name}","comments_url":"https://api.github.com/repos/artsy/eigen/issues/1246/comments","events_url":"https://api.github.com/repos/artsy/eigen/issues/1246/events","html_url":"https://github.com/artsy/eigen/issues/1246","id":139906019,"number":1246,"title":"dyld crash at launch","user":{"login":"alloy","id":2320,"avatar_url":"https://avatars.githubusercontent.com/u/2320?v=3","gravatar_id":"","url":"https://api.github.com/users/alloy","html_url":"https://github.com/alloy","followers_url":"https://api.github.com/users/alloy/followers","following_url":"https://api.github.com/users/alloy/following{/other_user}","gists_url":"https://api.github.com/users/alloy/gists{/gist_id}","starred_url":"https://api.github.com/users/alloy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alloy/subscriptions","organizations_url":"https://api.github.com/users/alloy/orgs","repos_url":"https://api.github.com/users/alloy/repos","events_url":"https://api.github.com/users/alloy/events{/privacy}","received_events_url":"https://api.github.com/users/alloy/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/artsy/eigen/labels/Crash","name":"Crash","color":"ededed"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":42,"created_at":"2016-03-10T14:39:42Z","updated_at":"2016-04-27T16:43:37Z","closed_at":null,"body":"* A colleague ran into a situation where the app would crash on launch. Important to note is that she was _also_ seeing it with other apps, such as LinkedIn.app.\r\n* A restart of her device ‘fixed’ the issue.\r\n* These crashes were not being reported to HockeyApp, meaning it was definitely happening very early in the process.\r\n* Extracting the crash reports directly from the device revealed a dyld error.\r\n* Checking iTunesConnect for crash reports revealed that we had ~80 similar crash reports.\r\n\r\n## TL;DR\r\n\r\nThis appears to happen when the device is out of memory, a restart of the device ‘fixes’ it. Presumably [this neat trick](http://www.iphonehacks.com/2016/02/instantly-free-up-ram-on-iphone-to-improve-performance.html) does so as well.\r\n\r\n![](http://www.datumcorp.com/f/2013/09/fullsize-have-you-tried-turning-it-off-and-on-again-23878.jpg)\r\n\r\n## Info about the errors\r\n\r\n```\r\nDyld Error Message:\r\nDyld Message: Library not loaded: @rpath/CocoaLumberjack.framework/CocoaLumberjack\r\n  Referenced from: /var/mobile/Containers/Bundle/Application/D693664B-F6D0-43CB-A839-358E15CAF9E6/Artsy.app/Artsy\r\n  Reason: no suitable image found.  Did find:\r\n\t/private/var/mobile/Containers/Bundle/Application/D693664B-F6D0-43CB-A839-358E15CAF9E6/Artsy.app/Frameworks/CocoaLumberjack.framework/CocoaLumberjack:\r\n\tmremap_encrypted() => -1, errno=12 for /private/var/mobile/Containers/Bundle/Application/D693664B-F6D0-43CB-A839-358E15CAF9E6/Artsy.app/Frameworks/CocoaLumberjack.framework/CocoaLumberjack\r\n\r\n\t/private/var/mobile/Containers/Bundle/Application/D693664B-F6D0-43CB-A839-358E15CAF9E6/Artsy.app/Frameworks/CocoaLumberjack.framework/CocoaLumberjack:\r\n\tmremap_encrypted() => -1, errno=12 for /private/var/mobile/Containers/Bundle/Application/D693664B-F6D0-43CB-A839-358E15CAF9E6/Artsy.app/Frameworks/CocoaLumberjack.framework/CocoaLumberjack\r\n\r\n\t/private/var/mobile/Containers/Bundle/Application/D693664B-F6D0-43CB-A839-358E15CAF9E6/Artsy.app/Frameworks/C\r\n  Dyld Version: 370.1\r\n```\r\n\r\n* The error message and codes are always the same.\r\n* The error is not specific to a framework, it happens with various frameworks.\r\n* It appears to have started from iOS 9.1 and up (but that could very well be iOS 9).\r\n\r\n## Known affected versions\r\n\r\nThese are the various versions for which we have crash reports.\r\n\r\n### iOS versions\r\n\r\n```\r\n~/D/dyld crash » ack -h 'OS Version:' **/*.crash | sort | uniq\r\nOS Version:          iOS 9.1 (13B143)\r\nOS Version:          iOS 9.2 (13C75)\r\nOS Version:          iOS 9.2.1 (13D15)\r\nOS Version:          iOS 9.2.1 (13D20)\r\nOS Version:          iOS 9.3 (13E5214d)\r\nOS Version:          iOS 9.3 (13E5225a)\r\n```\r\n\r\n### dyld versions\r\n\r\n```\r\n~/D/dyld crash » ack -h 'Dyld Version:' **/*.crash | sort | uniq\r\n  Dyld Version: 370.1\r\n  Dyld Version: 370.6\r\n  Dyld Version: 390.7\r\n```\r\n\r\n### Device versions\r\n\r\n```\r\n~/D/dyld crash » ack -h 'Hardware Model:' **/*.crash | sort | uniq\r\nHardware Model:      iPad3,4\r\nHardware Model:      iPad4,4\r\nHardware Model:      iPad4,7\r\nHardware Model:      iPad5,3\r\nHardware Model:      iPad5,4\r\nHardware Model:      iPad6,7\r\nHardware Model:      iPad6,8\r\nHardware Model:      iPhone6,1\r\nHardware Model:      iPhone7,1\r\nHardware Model:      iPhone7,2\r\nHardware Model:      iPhone8,1\r\nHardware Model:      iPhone8,2\r\n```\r\n\r\n## Retracing\r\n\r\n1. The error is raised [here](https://github.com/opensource-apple/dyld/blob/195030646877261f0c8c7ad8b001f52d6a26f514/src/ImageLoaderMachOCompressed.cpp#L1732) in dyld.\r\n2. Only place where `mremap_encrypted` _could_ return `-1` is [here](https://github.com/opensource-apple/xnu/blob/977178fc3e01a280624e2095838cb9530763cda9/bsd/kern/kern_mman.c#L1227).\r\n3. `vn_getpath` [calls](https://github.com/opensource-apple/xnu/blob/977178fc3e01a280624e2095838cb9530763cda9/bsd/vfs/vfs_subr.c#L2632) `build_path`, which in turn has a few possible locations where it _could_ be returning `-1`:\r\n  * https://github.com/opensource-apple/xnu/blob/977178fc3e01a280624e2095838cb9530763cda9/bsd/vfs/vfs_cache.c#L404\r\n  * https://github.com/opensource-apple/xnu/blob/977178fc3e01a280624e2095838cb9530763cda9/bsd/vfs/vfs_cache.c#L446\r\n  * https://github.com/opensource-apple/xnu/blob/977178fc3e01a280624e2095838cb9530763cda9/bsd/vfs/vfs_cache.c#L509\r\n4. But trying to hunt that further down would take a lot more time, suffice to say that:\r\n  * if `mremap_encrypted` returns `-1`, it most likely corresponds to `ERESTART`: https://github.com/opensource-apple/xnu/blob/2fa84067f6cdeb23267f877ca4fd26201316da1b/bsd/sys/errno.h#L270\r\n  * if `errno` is `12`, it most likely corresponds to `ENOMEM`: https://github.com/opensource-apple/xnu/blob/2fa84067f6cdeb23267f877ca4fd26201316da1b/bsd/sys/errno.h#L102\r\n\r\n## Caveats\r\n\r\nAlas, the latest dyld sources to be released (at the time of writing) appear to be for [version 360.18](https://github.com/opensource-apple/dyld/commit/3f928f32597888c5eac6003b9199d972d49857b5).\r\n\r\nTaking a cursory look at the decompiled code for `ImageLoaderMachOCompressed::registerEncryption` from a firmware image for iOS 9.2 for `iPad3,4`, does not appear to have any real changes.\r\n\r\n```\r\n~/t/ios-9.2-iPad3,4 » unzip iPad3,4_9.2_13C75_Restore.ipsw \r\nArchive:  iPad3,4_9.2_13C75_Restore.ipsw\r\n  inflating: 058-25903-078.dmg       \r\n  inflating: 058-26242-078.dmg       \r\n  inflating: 058-26268-078.dmg\r\n[…]\r\n\r\n~/t/ios-9.2-iPad3,4 » ~/Code/ObjectiveC/Debugging/xpwn/dmg/dmg extract 058-26268-078.dmg extracted.dmg -k 7ccd940dfda4b304c87050997e9fd70b458f21b5f3845262d985730b23f41e84aa25dae3\r\nWriting out data..\r\n[…]\r\n\r\n~/t/ios-9.2-iPad3,4 » open extracted.dmg\r\n~/t/ios-9.2-iPad3,4 » hopper -e /Volumes/Castlerock13C75.P101OS/usr/lib/dyld\r\n```\r\n\r\n```c\r\nint ImageLoaderMachOCompressed::registerEncryption(encryption_info_command const*, ImageLoader::LinkContext const&)(void * arg0, void * arg1) {\r\n    r0 = arg0;\r\n    r7 = (sp - 0x4 - 0x4 - 0x4 - 0x4 - 0x4) + 0xc;\r\n    sp = sp - 0x4 - 0x4 - 0x4 - 0x4 - 0x4 - 0x4 - 0x4 - 0x4 - 0x4;\r\n    r6 = arg1;\r\n    r8 = r2;\r\n    r4 = r0;\r\n    if (r6 != 0x0) {\r\n            r5 = 0x0;\r\n            do {\r\n                    r2 = *(*r4 + 0xe8);\r\n                    if ((r2)(r4, r5, r2) != 0x0) goto (null);\r\n                    r5 = r5 + 0x1;\r\n            } while (true);\r\n            r1 = r5;\r\n            r2 = *(*r4 + 0x104);\r\n            r0 = (r2)(r4, r1, r2);\r\n            r10 = r6 + 0x8;\r\n            asm{ ldm.w      sl, {r1, r5, sl} };\r\n            r2 = *(r8 + 0xb2);\r\n            r8 = *(r0 + 0x4);\r\n            r6 = r0 + r1;\r\n            r11 = *(r0 + 0x8);\r\n            if (r2 != 0x0) {\r\n                    dyld::log(\"                      0x%08lX->0x%08lX configured for FairPlay decryption\\n\");\r\n            }\r\n            r0 = _mremap_encrypted();\r\n            r5 = r0;\r\n            if (r5 != 0x0) {\r\n                    r0 = ___error();\r\n                    r3 = *(r4 + 0x4);\r\n                    r2 = *r0;\r\n                    r1 = r5;\r\n                    r0 = dyld::throwf(\"mremap_encrypted() => %d, errno=%d for %s\\n\");\r\n                    // …\r\n            }\r\n    }\r\n    return r0;\r\n}\r\n```\r\n\r\n## Conclusion\r\n\r\nThe crash at launch definitely feels like a very bad experience and I _feel_ like iOS should handle this more gracefully. Maybe show an alert? Will have a think about this and possibly file a radar.\r\n\r\nI’m not sure what/if we can do about this. We could probably improve it somewhat by trimming some framework fat from the bundle, but I don’t think these are large anyways.\r\n\r\nIt would be interesting to know if such failures occurred prior to iOS 9 and if they were labelled differently.\r\n\r\n## Status\r\n\r\n* A couple of radar tickets have already been filed:\r\n  - http://www.openradar.me/radar?id=4952667885404160 (mine)\r\n  - http://www.openradar.me/radar?id=5054516256833536\r\n  - http://www.openradar.me/radar?id=4992459683659776\r\n* A DTS ticket has been filed http://twitter.com/senior/status/707996353964728321\r\n* A guess has been made as to the issue being a jetsam change https://twitter.com/Catfish_Man/status/707955795493687296 ([jetsam](http://newosxbook.com/articles/MemoryPressure.html) is the process in which the kernel basically kills apps to free up memory)"},"comment":{"url":"https://api.github.com/repos/artsy/eigen/issues/comments/215143395","html_url":"https://github.com/artsy/eigen/issues/1246#issuecomment-215143395","issue_url":"https://api.github.com/repos/artsy/eigen/issues/1246","id":215143395,"user":{"login":"perryzheng","id":592011,"avatar_url":"https://avatars.githubusercontent.com/u/592011?v=3","gravatar_id":"","url":"https://api.github.com/users/perryzheng","html_url":"https://github.com/perryzheng","followers_url":"https://api.github.com/users/perryzheng/followers","following_url":"https://api.github.com/users/perryzheng/following{/other_user}","gists_url":"https://api.github.com/users/perryzheng/gists{/gist_id}","starred_url":"https://api.github.com/users/perryzheng/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/perryzheng/subscriptions","organizations_url":"https://api.github.com/users/perryzheng/orgs","repos_url":"https://api.github.com/users/perryzheng/repos","events_url":"https://api.github.com/users/perryzheng/events{/privacy}","received_events_url":"https://api.github.com/users/perryzheng/received_events","type":"User","site_admin":false},"created_at":"2016-04-27T16:43:37Z","updated_at":"2016-04-27T16:43:37Z","body":"Just want to chime in on the conversation here. I ran into this issue this morning - lyft app crashed during startup. I reinstalled the app from app store and still to no avail. And then I found *some* other apps (not *all*) are having the same issue. Restarting my phone 'fixed' it. "}}