{"action":"created","issue":{"url":"https://api.github.com/repos/facebook/buck/issues/939","repository_url":"https://api.github.com/repos/facebook/buck","labels_url":"https://api.github.com/repos/facebook/buck/issues/939/labels{/name}","comments_url":"https://api.github.com/repos/facebook/buck/issues/939/comments","events_url":"https://api.github.com/repos/facebook/buck/issues/939/events","html_url":"https://github.com/facebook/buck/issues/939","id":182467174,"number":939,"title":"Don't require .buckconfig and/or fix nested projects","user":{"login":"Qix-","id":885648,"avatar_url":"https://avatars.githubusercontent.com/u/885648?v=3","gravatar_id":"","url":"https://api.github.com/users/Qix-","html_url":"https://github.com/Qix-","followers_url":"https://api.github.com/users/Qix-/followers","following_url":"https://api.github.com/users/Qix-/following{/other_user}","gists_url":"https://api.github.com/users/Qix-/gists{/gist_id}","starred_url":"https://api.github.com/users/Qix-/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Qix-/subscriptions","organizations_url":"https://api.github.com/users/Qix-/orgs","repos_url":"https://api.github.com/users/Qix-/repos","events_url":"https://api.github.com/users/Qix-/events{/privacy}","received_events_url":"https://api.github.com/users/Qix-/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2016-10-12T08:39:58Z","updated_at":"2016-10-17T16:06:43Z","closed_at":null,"body":"I know there was some [discussion a while back](https://github.com/facebook/buck/issues/130#issuecomment-45268067) about `.buckconfig` files being mandatory.\r\n\r\nThis anti-nesting paradigm is ultimately the reason why I have personally strayed away from build systems like [Bazel](https://github.com/bazelbuild/bazel) and (now, potentially) Buck.\r\n\r\nLet's start with the needless empty `.buckconfig` file:\r\n\r\nI'm still getting the following\r\n\r\n```console\r\n$ buck --version\r\nbuck version de38173cc0b9de07cb129298191280260a6c35e0\r\n\r\n$ buck build //:foo\r\nThis does not appear to be the root of a Buck project. Please 'cd'\r\nto the root of your project before running buck. If this really is\r\nthe root of your project, run\r\n'touch .buckconfig'\r\nand then re-run your buck command.\r\n```\r\n\r\nThis whole concept of \"project roots\" throws a wrench into the whole [modularity thing](https://12factor.net/dependencies) (link is to factor 2).\r\n\r\nWhy? Because you can't nest modules anymore. At least, not in any clear or direct way. **Projects/repositories will (should) never know if they're really the \"project root\".**\r\n\r\n---\r\n\r\nHere's an example. Let's take [this sample repository](https://github.com/Qix-/nested-buck) that has a dependency within a dependency.\r\n\r\nAt a glance, here is the directory structure for each dependency:\r\n\r\n```\r\n/BUCK\r\n/ext/names/BUCK\r\n/ext/names/ext/compiler/BUCK\r\n\r\n<------ downstream ---------\r\n-------- upstream --------->\r\n```\r\n\r\nIn the real world, each of the above would be a dependency of some sort, either via a submodule or some package manager. Each dependency is **completely unaware of their consumers (downstream dependents)** and merely expose a `'PUBLIC'`-ly visible member for the outer codebase (downstream dependent) to consume.\r\n\r\nLet's take a look at the compilation chain here, starting first with the inner-most dependency. All it does is export a single executable script using `genrule()` (since [`export_file()` doesn't support executables](https://github.com/facebook/buck/issues/938)).\r\n\r\nCompiling it by itself is, of course, just fine.\r\n\r\n```console\r\n$ cd ext/names/ext/compiler\r\n$ buck build //:compiler\r\nUsing watchman.\r\n[-] PROCESSING BUCK FILES...FINISHED 0.0s [100%] 🐳  New buck daemon\r\n[-] DOWNLOADING... (0.00 B/S AVG, TOTAL: 0.00 B, 0 Artifacts)\r\n[-] BUILDING...FINISHED 0.6s [100%] (1/1 JOBS, 1 UPDATED, 1 [100.0%] CACHE MISS)\r\n```\r\n\r\nNow let's go up a directory and try the `names` project. This project will \"build\" the compiler if it isn't already built (copied), and will proceed to use it to compile all of the names into a single file.\r\n\r\n```console\r\n$ cd ext/names/\r\n$ buck build //:all\r\nUsing watchman.\r\ncompiling 2 files\r\n\r\n[-] PROCESSING BUCK FILES...FINISHED 0.0s [100%] 🐳  New buck daemon\r\n[-] DOWNLOADING... (0.00 B/S AVG, TOTAL: 0.00 B, 0 Artifacts)\r\n[-] BUILDING...FINISHED 0.6s [100%] (2/2 JOBS, 2 UPDATED, 2 [100.0%] CACHE MISS)\r\n```\r\n\r\nSweet, it appears to have found the compiler and has executed it (the output `compiling 2 files` resulting from our 'compiler' script).\r\n\r\nLet's go back up to the main repository and try to build it. It uses the names dependency to get a list of all names, and then reverses them. We're in the business of reversing things so this is _really_ our intended output. We don't care that it uses a compiler dependency (the product manager has approved, etc.) thus we're just after its main `all.names` file.\r\n\r\n```console\r\n$ buck build //:reversed\r\nUsing watchman.\r\n[-] PROCESSING BUCK FILES...FINISHED 0.0s [100%] 🐳  New buck daemon\r\n[+] DOWNLOADING... (0.00 B/S, TOTAL: 0.00 B, 0 Artifacts)\r\n[+] BUILDING...0.3s\r\nBUILD FAILED: Couldn't get dependency '//ext/compiler:compiler' of target '//ext/names:all':\r\nNo build file at ext/compiler/BUCK when resolving target //ext/compiler:compiler.\r\n```\r\n\r\nWait a second! Buck is looking for the nested compiler dependency in `<repo>/ext`. That's not right!\r\n\r\nMaybe we need relative paths. Let's apply this diff:\r\n\r\n```diff\r\ndiff --git a/ext/names/BUCK b/ext/names/BUCK\r\nindex 6fc3742..e634800 100644\r\n--- a/ext/names/BUCK\r\n+++ b/ext/names/BUCK\r\n@@ -5,5 +5,5 @@ genrule(\r\n     name='all',\r\n     srcs=['a.names', 'b.names'],\r\n     out='all.names',\r\n-    cmd='$(exe //ext/compiler:compiler) $SRCS > $OUT',\r\n+    cmd='$(exe //./ext/compiler:compiler) $SRCS > $OUT',\r\n     visibility=['PUBLIC'])\r\n```\r\n\r\nCool, now we'll look for `//./ext/compiler:compiler` instead of the root `ext` folder.\r\n\r\n```console\r\n$ buck build //:reversed\r\nUsing watchman.\r\n[-] PROCESSING BUCK FILES...FINISHED 0.0s [100%] 🐳  New buck daemon\r\n[+] DOWNLOADING... (0.00 B/S, TOTAL: 0.00 B, 0 Artifacts)\r\n[+] BUILDING...0.3s\r\nBUILD FAILED: Couldn't get dependency '//ext/names:all' of target '//:reversed':\r\n//ext/names:all: Build target path cannot be absolute or contain . or .. (found //./ext/compiler:compiler)\r\n```\r\n\r\nAlright, what if we drop the `//` altogether? Maybe that means 'relative':\r\n\r\n```diff\r\ndiff --git a/ext/names/BUCK b/ext/names/BUCK\r\nindex 6fc3742..34613fe 100644\r\n--- a/ext/names/BUCK\r\n+++ b/ext/names/BUCK\r\n@@ -5,5 +5,5 @@ genrule(\r\n     name='all',\r\n     srcs=['a.names', 'b.names'],\r\n     out='all.names',\r\n-    cmd='$(exe //ext/compiler:compiler) $SRCS > $OUT',\r\n+    cmd='$(exe ext/compiler:compiler) $SRCS > $OUT',\r\n     visibility=['PUBLIC'])\r\n```\r\n\r\nLet's try that.\r\n\r\n```console\r\n$ buck build //:reversed\r\n[-] PROCESSING BUCK FILES...FINISHED 0.0s [100%] 🐌  Environment variable changes: [PATH, MANPATH, NVM_PATH, NVM_BIN]\r\n[+] DOWNLOADING... (0.00 B/S, TOTAL: 0.00 B, 0 Artifacts)\r\n[+] BUILDING...0.2s\r\nBUILD FAILED: Couldn't get dependency '//ext/names:all' of target '//:reversed':\r\n//ext/names:all: Path in ext/compiler:compiler must start with //\r\n```\r\n\r\nWell, what do we do now? Do we really have to re-adjust all of those dependencies to use the root of _our_ repository?\r\n\r\nWhat if I (hypothetically) have the source code for [Awesomium](http://www.awesomium.com/) complete with a vendored `BUCK` file by their team, and it in turn has Chromium inside of it with a vendored `BUCK` file vendored by Google.\r\n\r\nIf I build Awesomium by itself, great! It would work, assuming Google doesn't use any nested dependencies. But, when I include it in my own project and use Awesomium as a nested dependency, the whole thing will fail because it will look for Chromium relative to my \"project root\".\r\n\r\n---\r\n\r\nAnd thus we have a completely unusable nested dependency system in Buck.\r\n\r\nCMake has a very similar problem, one which I [addressed](https://gitlab.kitware.com/cmake/cmake/issues/16255#note_156227) a while back, where `CMAKE_SOURCE_DIR` doesn't refer to the directory that houses `CMakeLists.txt` but instead the _some/dir_ when running `cmake some/dir`.\r\n\r\nThe developers insisted all projects update their CMakeLists.txt to use `CMAKE_CURRENT_SOURCE_DIR` instead, which was pretty poorly documented (completely unocumented?) [until version 3.0.2](https://cmake.org/cmake/help/v3.0/variable/CMAKE_CURRENT_SOURCE_DIR.html). While that isn't a problem _now_, most CMakeLists.txt for projects [older than 2014](https://cmake.org/files/v3.0/) that haven't been updated in a while (there are a lot!) will fall fate to this flaw.\r\n\r\nThis renders `add_subdirectory()` very, very spotty in terms of intended functionality.\r\n\r\nCMake isn't the only culprit. Several build systems fall short in this area, which is troubling since this directly hinders any usable package managers from spawning that can rely on a battle-tested build system instead of using their own and (ab)using strange tricks like symlinks or path variables to do what they do:\r\n\r\n- [Bazel](https://www.bazel.io/) only has [external](https://www.bazel.io/versions/master/docs/external.html) dependencies and suffers from a very similar limitation to Buck.\r\n- [Ninja](https://ninja-build.org/) gets close but [also falls short](https://github.com/ninja-build/ninja/issues/977) with their ambiguous `subninja` function.\r\n- [Tup](http://gittup.org/tup/manual.html) also gets close, but places [too many restrictions](http://stackoverflow.com/questions/14301543/why-does-tup-need-one-tupfile-per-directory) on visibility of the files underneath a directory with a `Tupfile`.\r\n- GNU Make only has [the `-C` option](https://www.gnu.org/software/make/manual/html_node/Options-Summary.html), which is fine I suppose but you lose out on just about every directed graph benefit with an added cost of increased confusion of how dependencies interact.\r\n\r\nThe whole concept of \"project root\" kind of sullies non-standard dependency management (e.g. not Java). Substitute the above submodule hierarchy with some package manager (e.g. `npm@2`, which we know is [still used a lot](https://semaphoreci.com/blog/2015/12/15/nodejs-version-usage-in-commercial-projects-2015-edition.html) since `npm@3` doesn't work with node 0.10 and [`nvm`](https://github.com/creationix/nvm) automatically configures `npm@2` all the way up through node 4.0.0!) and you'll run into these problems very quickly.\r\n\r\n---\r\n\r\nThere are two proposals here - one is arguably more \"philosophically correct\" but more-breaky, and the other doesn't get rid of that pesky empty `.buckconfig` file -- but won't break as much.\r\n\r\n**Proposal 1**\r\n\r\nI propose the requirement for an empty `.buckconfig` be dropped and instead count on the user running `buck` from within a (sub)directory of a `BUCK` file (i.e. traverse upward until Buck finds a `BUCK` file and run from there). Existing `.buckconfig` files are treated normally.\r\n\r\nThis is pretty much [exactly how Git does it](https://github.com/facebook/buck/issues/130#issuecomment-45235719).\r\n\r\nSecondly, I propose `//`-prefixed paths refer to the `BUCK` file they are used in.\r\n\r\nHow do I reference rules above the current `BUCK` file? Simple: you don't. This would _philosophically_ enforce that `BUCK` files are considered an upstream dependency and thus should never depend on a downstream dependency (parent directory, in this case) which would ultimately break the dependency chain.\r\n\r\nThis results in one less messy empty dotfile and makes building up project dependencies _much_ cleaner and easier to maintain.\r\n\r\n**Proposal 2**\r\n\r\nI propose that `.buckconfig` actually marks a directory as a Buck project (officially). This means that all `//...` paths in all recursive `BUCK` files (that do not, in turn, have a `.buckconfig` sibling) refer to the next parent up directory with a `.buckconfig` file.\r\n\r\nThis is almost identical to the current functionality, though any nested projects that also have a `.buckconfig` file will have their target paths resolve to those respective directories instead of the \"project root\".\r\n\r\n---\r\n\r\nI understand either of these would introduce a breaking change, though without them, _it makes Buck very hard to adopt_ by ad-hoc languages or environments, especially in the world of native development.\r\n\r\nPlease consider at least addressing this issue and discussing ways to handle such use cases. 💃 "},"comment":{"url":"https://api.github.com/repos/facebook/buck/issues/comments/254252727","html_url":"https://github.com/facebook/buck/issues/939#issuecomment-254252727","issue_url":"https://api.github.com/repos/facebook/buck/issues/939","id":254252727,"user":{"login":"Coneko","id":1614903,"avatar_url":"https://avatars.githubusercontent.com/u/1614903?v=3","gravatar_id":"","url":"https://api.github.com/users/Coneko","html_url":"https://github.com/Coneko","followers_url":"https://api.github.com/users/Coneko/followers","following_url":"https://api.github.com/users/Coneko/following{/other_user}","gists_url":"https://api.github.com/users/Coneko/gists{/gist_id}","starred_url":"https://api.github.com/users/Coneko/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Coneko/subscriptions","organizations_url":"https://api.github.com/users/Coneko/orgs","repos_url":"https://api.github.com/users/Coneko/repos","events_url":"https://api.github.com/users/Coneko/events{/privacy}","received_events_url":"https://api.github.com/users/Coneko/received_events","type":"User","site_admin":false},"created_at":"2016-10-17T16:06:43Z","updated_at":"2016-10-17T16:06:43Z","body":"Buck was initially developed for use in a monorepo, and is still pretty biased towards it.\r\n\r\nWe currently use cross cell with a flat hierarchy of cells, so I'm not sure how well it works with nested cells, but at least in theory it should as long as each cell only refers to its child cells, and not other descendants or ancestors.\r\n\r\nIn your example repo, I applied this patch:\r\n\r\n```\r\ndiff --git a/.buckconfig b/.buckconfig\r\nindex e69de29..612c65d 100644\r\n--- a/.buckconfig\r\n+++ b/.buckconfig\r\n@@ -0,0 +1,2 @@\r\n+[repositories]\r\n+names=ext/names\r\ndiff --git a/BUCK b/BUCK\r\nindex 9ebe9f4..c372eda 100644\r\n--- a/BUCK\r\n+++ b/BUCK\r\n@@ -1,5 +1,5 @@\r\n genrule(\r\n        name='reversed',\r\n-       srcs=['//ext/names:all'],\r\n+       srcs=['names//:all'],\r\n        out='reversed.names',\r\n        cmd='cat $SRCS | rev > $OUT')\r\ndiff --git a/ext/names/.buckconfig b/ext/names/.buckconfig\r\nindex e69de29..205fa39 100644\r\n--- a/ext/names/.buckconfig\r\n+++ b/ext/names/.buckconfig\r\n@@ -0,0 +1,2 @@\r\n+[repositories]\r\n+compiler=ext/compiler\r\ndiff --git a/ext/names/BUCK b/ext/names/BUCK\r\nindex 6fc3742..068beaa 100644\r\n--- a/ext/names/BUCK\r\n+++ b/ext/names/BUCK\r\n@@ -5,5 +5,5 @@ genrule(\r\n     name='all',\r\n     srcs=['a.names', 'b.names'],\r\n     out='all.names',\r\n-    cmd='$(exe //ext/compiler:compiler) $SRCS > $OUT',\r\n+    cmd='$(exe compiler//:compiler) $SRCS > $OUT',\r\n     visibility=['PUBLIC'])\r\n```\r\n\r\nNow you can build it:\r\n\r\n```\r\nlog-mbp:nested-buck coneko$ NO_BUCKD=1 buck build //:reversed --show-output\r\nNot using buckd because NO_BUCKD is set.\r\n[-] PROCESSING BUCK FILES...FINISHED 0.7s [100%] 🐳  New buck daemon\r\n[+] DOWNLOADING... (0.00 B/S, TOTAL: 0.00 B, 0 Artifacts)\r\n[+] BUILDING...0.4s [0%] (0/1 JOBS, 0 UPDATED, 0 [0.0%] CACHE MISS)\r\n |=> //:reversed...  0.0s (checking local cache)\r\n |=> IDLE\r\n |=> IDLE\r\n |=> IDLE\r\nThe outputs are:\r\n//:reversed buck-out/gen/reversed/reversed.names\r\nlog-mbp:nested-buck coneko$ cat buck-out/gen/reversed/reversed.names\r\nxelA\r\nnitsuA\r\nnairdA\r\nlA\r\nkcuB\r\nylliB\r\nboB\r\npmurB\r\necniboB\r\n```\r\n\r\nDoes this feature fit your needs?"}}
{"actor":{"display_login":"Coneko"}}