{"action":"created","issue":{"url":"https://api.github.com/repos/yarnpkg/rfcs/issues/18","repository_url":"https://api.github.com/repos/yarnpkg/rfcs","labels_url":"https://api.github.com/repos/yarnpkg/rfcs/issues/18/labels{/name}","comments_url":"https://api.github.com/repos/yarnpkg/rfcs/issues/18/comments","events_url":"https://api.github.com/repos/yarnpkg/rfcs/issues/18/events","html_url":"https://github.com/yarnpkg/rfcs/issues/18","id":187237506,"number":18,"title":"Symlink all modules to global install cache to reduce storage & install times by 30x's","user":{"login":"phestermcs","id":4985028,"avatar_url":"https://avatars.githubusercontent.com/u/4985028?v=3","gravatar_id":"","url":"https://api.github.com/users/phestermcs","html_url":"https://github.com/phestermcs","followers_url":"https://api.github.com/users/phestermcs/followers","following_url":"https://api.github.com/users/phestermcs/following{/other_user}","gists_url":"https://api.github.com/users/phestermcs/gists{/gist_id}","starred_url":"https://api.github.com/users/phestermcs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/phestermcs/subscriptions","organizations_url":"https://api.github.com/users/phestermcs/orgs","repos_url":"https://api.github.com/users/phestermcs/repos","events_url":"https://api.github.com/users/phestermcs/events{/privacy}","received_events_url":"https://api.github.com/users/phestermcs/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":13,"created_at":"2016-11-04T02:06:55Z","updated_at":"2016-11-29T23:38:34Z","closed_at":"2016-11-21T04:28:43Z","body":"This feature is primarily meant to improve development time experience when working with many node projects, and storing dozens or more projects on a development machine.\r\n\r\nAnyone who understands node's 'node_modules' resolution algo and issues with versions may at first believe this impossible for all practical purposes, and they would be right. But please read on.\r\n\r\nCopying modules into every project works, but takes up so much space (gigabytes) and slows install times. Preliminary tests (using forked branches of node and npm) indicate storage space across dozes of projects could be reduced by 30 times or more (30GB to 1GB) and install times by 40 times or more (40 seconds to 1 second) if modules were only ever installed once to a global install cache, then symlinked by a package manager when installed for any given project.\r\n\r\nHowever, because package.json allows the versions of a module's dependencies to be ranges, installing the module in multiple projects means the versions of it's dependencies that actually get installed in each project may differ slightly between projects for various reasons.\r\n\r\nThis prevents package managers from being able to symlink modules to single global physical copies and still guarantee versions of an entire dependency tree on a per-project basis, because node only allows for precise dependency resolution by searching for a module's dependencies under a 'node_modules' subfolder of the module.\r\n\r\nAn opt-in, interopable, and simple change to node's search-path-list enables symlinking a module and it's dependencies across projects while still allowing for version guarantees on a per project basis. The change adds an additional search path that rather than being a 'node_modules' folder under a module, is a 'module.node_modules' folder adjacent to the module. To illustrate, this:\r\n\r\n```\r\n// project A's modules where installed 1 month ago, so\r\nprojectA\r\n    /node_modules\r\n        /modA //[v1.0.0]\r\n            /node_modules\r\n                /modB //[v1.1.0] from pkg.deps.modB: \"^1.0.0\"\r\n\r\n// project B's modules were installed today.\r\n// between a month ago and today a new modB was released.\r\nprojectB\r\n    /node_modules\r\n        /modA //[v1.0.0]\r\n            /node_modules\r\n                /modB //[V1.2.0] from pkg.deps.modB: \"^1.0.0\"\r\n```\r\nbecomes this:\r\n```\r\n// project A's modules where installed 1 month ago, so\r\nprojectA\r\n    /node_modules\r\n        /modA //[v1.0.0] symlinked to global /cache/modA/v1.0.0, sans node_modules\r\n        /modA.node_modules\r\n            /modB //[v1.1.0] from pkg.deps.modB: \"^1.0.0\", symlinked to /cache/modB/v1.1.0\r\n\r\n// project B's modules were installed today.\r\n// between a month ago and today a new modB was released.\r\nprojectB\r\n    /node_modules\r\n        /modA //[v1.0.0] symlinked to /cache/modA/v1.0.0\r\n        /modA.node_modules\r\n            /modB //[V1.2.0] from pkg.deps.modB: \"^1.0.0\", symlinked to /cache/modB/v1.2.0\r\n```\r\nIn other words, node's search paths go from this:\r\n```\r\nprojectB/node_modules/modA/node_modules/modB/node_modules\r\nprojectB/node_modules/modA/node_modules\r\nprojectB/node_modules\r\n```\r\nto this:\r\n```\r\nprojectB/node_modules/modA.node_modules/modB/node_modules\r\nprojectB/node_modules/modA.node_modules/modB.node_modules\r\nprojectB/node_modules/modA/node_modules\r\nprojectB/node_modules/modA.node_modules\r\nprojectB/node_modules\r\n```\r\n\r\nA fork/branch (off the v7.0.0 tag) of node implementing this behaviour in about 50 LOC has been created for experimentation at [phestermcs/node/adj-nm](https://github.com/phestermcs/node/tree/adj-nm).\r\n\r\nA switch has been added to node, --adjacent-node-modules, that enables the additional search paths. It still allows for dependency modules to be resolved from a node_modules subfolder, giving precedence to that, so for example bundling would still work as expected.\r\n\r\nThe behavior is also activated by setting the env var NODE_ADJACENT_NODE_MODULES=1. This was done so that lifecycle scripts, like `{\"scripts\": {\"install\": \"node somthing.js\"}}`, that depended on symlinked modules, could be spawned with the env var set, and node would implement the new behavior without requiring the command line switch.\r\n\r\nThe switch also forces node to preserve all symlinks, including the entry module. This does create a potential issue if there are multiple symlinks to the same node add-on (ie an OS loadable dll), but this could easily be resolved by the package manager creating multiple copies in the global install cache, and symlinking as required. A package manager would know to do this by searching a module for any *.node files, and if finding marking the module as an add-on requiring copies per symlink.\r\n\r\nThere's probably some other edge case issues, like that install lifecycle events would only run once when installing into the global cache.\r\n\r\nFundamentally, this issue is a request to the yarn gatekeepers, if there's any interest in enhancing yarn to take advantage of this change to node, on an experimental basis. I would certainly be willing to work on yarn to implement if there is interest. I hope there is. I've been playing with a tweaked version of npm for the past week, and I want this and I bet others would as well. It makes so many things soooo much faster (installing react project for example goes from about a minute to about 2 seconds after the first install), and I get dozens of gigabytes of storage back!"},"comment":{"url":"https://api.github.com/repos/yarnpkg/rfcs/issues/comments/263735961","html_url":"https://github.com/yarnpkg/rfcs/issues/18#issuecomment-263735961","issue_url":"https://api.github.com/repos/yarnpkg/rfcs/issues/18","id":263735961,"user":{"login":"ljharb","id":45469,"avatar_url":"https://avatars.githubusercontent.com/u/45469?v=3","gravatar_id":"","url":"https://api.github.com/users/ljharb","html_url":"https://github.com/ljharb","followers_url":"https://api.github.com/users/ljharb/followers","following_url":"https://api.github.com/users/ljharb/following{/other_user}","gists_url":"https://api.github.com/users/ljharb/gists{/gist_id}","starred_url":"https://api.github.com/users/ljharb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ljharb/subscriptions","organizations_url":"https://api.github.com/users/ljharb/orgs","repos_url":"https://api.github.com/users/ljharb/repos","events_url":"https://api.github.com/users/ljharb/events{/privacy}","received_events_url":"https://api.github.com/users/ljharb/received_events","type":"User","site_admin":false},"created_at":"2016-11-29T23:38:34Z","updated_at":"2016-11-29T23:38:34Z","body":"@phestermcs Just to provide a potential concrete use case: if my npm package shells out to `find` to search `node_modules` (for example), and I haven't included `-L`, then `node` supporting symlinks will be irrelevant - the change to symlinks will break my module."}}
{"actor":{"display_login":"ljharb"}}