{"action":"created","issue":{"url":"https://api.github.com/repos/bitcoin/bitcoin/issues/15277","repository_url":"https://api.github.com/repos/bitcoin/bitcoin","labels_url":"https://api.github.com/repos/bitcoin/bitcoin/issues/15277/labels{/name}","comments_url":"https://api.github.com/repos/bitcoin/bitcoin/issues/15277/comments","events_url":"https://api.github.com/repos/bitcoin/bitcoin/issues/15277/events","html_url":"https://github.com/bitcoin/bitcoin/pull/15277","id":403917276,"node_id":"MDExOlB1bGxSZXF1ZXN0MjQ4MjE5MDUx","number":15277,"title":"[Help Wanted] contrib: Enable building in Guix containers","user":{"login":"dongcarl","id":3445290,"node_id":"MDQ6VXNlcjM0NDUyOTA=","avatar_url":"https://avatars1.githubusercontent.com/u/3445290?v=4","gravatar_id":"","url":"https://api.github.com/users/dongcarl","html_url":"https://github.com/dongcarl","followers_url":"https://api.github.com/users/dongcarl/followers","following_url":"https://api.github.com/users/dongcarl/following{/other_user}","gists_url":"https://api.github.com/users/dongcarl/gists{/gist_id}","starred_url":"https://api.github.com/users/dongcarl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dongcarl/subscriptions","organizations_url":"https://api.github.com/users/dongcarl/orgs","repos_url":"https://api.github.com/users/dongcarl/repos","events_url":"https://api.github.com/users/dongcarl/events{/privacy}","received_events_url":"https://api.github.com/users/dongcarl/received_events","type":"User","site_admin":false},"labels":[{"id":61889416,"node_id":"MDU6TGFiZWw2MTg4OTQxNg==","url":"https://api.github.com/repos/bitcoin/bitcoin/labels/Build%20system","name":"Build system","color":"5319e7","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/bitcoin/bitcoin/milestones/37","html_url":"https://github.com/bitcoin/bitcoin/milestone/37","labels_url":"https://api.github.com/repos/bitcoin/bitcoin/milestones/37/labels","id":3391434,"node_id":"MDk6TWlsZXN0b25lMzM5MTQzNA==","number":37,"title":"0.19.0","description":null,"creator":{"login":"MarcoFalke","id":6399679,"node_id":"MDQ6VXNlcjYzOTk2Nzk=","avatar_url":"https://avatars0.githubusercontent.com/u/6399679?v=4","gravatar_id":"","url":"https://api.github.com/users/MarcoFalke","html_url":"https://github.com/MarcoFalke","followers_url":"https://api.github.com/users/MarcoFalke/followers","following_url":"https://api.github.com/users/MarcoFalke/following{/other_user}","gists_url":"https://api.github.com/users/MarcoFalke/gists{/gist_id}","starred_url":"https://api.github.com/users/MarcoFalke/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MarcoFalke/subscriptions","organizations_url":"https://api.github.com/users/MarcoFalke/orgs","repos_url":"https://api.github.com/users/MarcoFalke/repos","events_url":"https://api.github.com/users/MarcoFalke/events{/privacy}","received_events_url":"https://api.github.com/users/MarcoFalke/received_events","type":"User","site_admin":false},"open_issues":20,"closed_issues":45,"state":"open","created_at":"2018-05-31T19:40:07Z","updated_at":"2019-06-12T10:15:36Z","due_on":null,"closed_at":null},"comments":41,"created_at":"2019-01-28T17:10:45Z","updated_at":"2019-06-12T11:38:15Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/bitcoin/bitcoin/pulls/15277","html_url":"https://github.com/bitcoin/bitcoin/pull/15277","diff_url":"https://github.com/bitcoin/bitcoin/pull/15277.diff","patch_url":"https://github.com/bitcoin/bitcoin/pull/15277.patch"},"body":"**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**\r\n\r\n-----\r\n\r\n### Guix Introduction\r\n\r\nThis PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin.\r\n\r\n### Guix Build Walkthrough\r\n\r\n[Newest instructions](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718)\r\n\r\n[Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011)\r\n\r\n[Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439)\r\n\r\n<details>\r\n<summary>Old instructions no. 1</summary>\r\nIn this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment.\r\n\r\nWe can then invoke\r\n```\r\nguix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes\r\n```\r\nTo have Guix:\r\n1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details).\r\n2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in.\r\n3. Drop you into a shell in that container.\r\n\r\n> Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>`\r\n\r\nTherefore, we can perform a build of bitcoin much like in Gitian by invoking the following:\r\n\r\n```\r\nmake -C depends -j\"$(nproc)\" download && \\\r\n    cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes\r\n```\r\n\r\nWe don't include `make -C depends -j\"$(nproc)\" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing).\r\n</details>\r\n\r\n\r\n\r\n### Rationale\r\n\r\nI believe that this represents a substantial improvement for the \"supply chain security\" of bitcoin because:\r\n\r\n1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system.\r\n2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)).\r\n3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler).\r\n\r\n### Steps to Completion\r\n\r\n- [x] Successfully build bitcoin inside the Guix environment\r\n- [x] Make `check-symbols` pass\r\n- [x] Do the above but without nasty hacks\r\n- [x] Solve some of the more innocuous hacks\r\n- [ ] Make it cross-compile (HELP WANTED HERE)\r\n  - [x] Linux\r\n    - [x] x86_64-linux-gnu\r\n    - [x] i686-linux-gnu\r\n    - [x] aarch64-linux-gnu\r\n    - [x] arm-linux-gnueabihf\r\n    - [x] riscv64-linux-gnu\r\n  - [ ] OS X\r\n    - [ ] x86_64-apple-darwin14\r\n  - [ ] Windows\r\n    - [ ] x86_64-w64-mingw32\r\n- [ ] Maybe make importer for depends syntax\r\n- [ ] Document build process for future releases\r\n- [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html)\r\n\r\n\r\n### Help Wanted\r\n\r\n[Fresh content](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210)\r\n\r\n[Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439)\r\n\r\n<details>\r\n<summary>Old content no. 1</summary>\r\nAs of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be.\r\n\r\nHere's what `ldd src/bitcoind` looks like when built in a Guix container:\r\n```\r\n\tlinux-vdso.so.1 (0x00007ffcc2d90000)\r\n\tlibdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000)\r\n\tlibrt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000)\r\n\tlibstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000)\r\n\tlibpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000)\r\n\tlibm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000)\r\n\tlibgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000)\r\n\tlibc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000)\r\n\t/gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000)\r\n```\r\n\r\nAnd here's what it looks in one of our releases:\r\n```\r\n\tlinux-vdso.so.1 (0x00007ffff52cd000)\r\n\tlibpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000)\r\n\tlibrt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000)\r\n\tlibm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000)\r\n\tlibgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000)\r\n\tlibc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000)\r\n\t/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000)\r\n```\r\n\r\n~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely.\r\n\r\nEdit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14)\r\n</details>\r\n\r\n### How to Help\r\n\r\n1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation)\r\n2. Try out my branch and the command described above!"},"comment":{"url":"https://api.github.com/repos/bitcoin/bitcoin/issues/comments/501235235","html_url":"https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-501235235","issue_url":"https://api.github.com/repos/bitcoin/bitcoin/issues/15277","id":501235235,"node_id":"MDEyOklzc3VlQ29tbWVudDUwMTIzNTIzNQ==","user":{"login":"laanwj","id":126646,"node_id":"MDQ6VXNlcjEyNjY0Ng==","avatar_url":"https://avatars0.githubusercontent.com/u/126646?v=4","gravatar_id":"","url":"https://api.github.com/users/laanwj","html_url":"https://github.com/laanwj","followers_url":"https://api.github.com/users/laanwj/followers","following_url":"https://api.github.com/users/laanwj/following{/other_user}","gists_url":"https://api.github.com/users/laanwj/gists{/gist_id}","starred_url":"https://api.github.com/users/laanwj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/laanwj/subscriptions","organizations_url":"https://api.github.com/users/laanwj/orgs","repos_url":"https://api.github.com/users/laanwj/repos","events_url":"https://api.github.com/users/laanwj/events{/privacy}","received_events_url":"https://api.github.com/users/laanwj/received_events","type":"User","site_admin":false},"created_at":"2019-06-12T11:38:15Z","updated_at":"2019-06-12T11:38:15Z","author_association":"MEMBER","body":">   @laanwj It seems that my intermediary changes have fixed thi\r\n\r\nRight—looks like only `bitcoin-qt` has a dependency on `libdl` now, and the dependency indeed comes through the `QLibrary` loading mechanism:\r\n```\r\n$ objdump -d bitcoin-qt |grep dlopen\r\n00000000001468d0 <dlopen@plt>:\r\n  1468d0:       ff 25 92 89 11 02       jmpq   *0x2118992(%rip)        # 225f268 <dlopen@GLIBC_2.2.5>\r\n\r\n  d69e3a:       e8 91 ca 3d ff          callq  1468d0 <dlopen@plt>\r\n$ addr2line -Cife bitcoin-qt.dbg 0xd69e3a\r\nQLibraryPrivate::load_sys()\r\n```\r\nI checked and this is no different than the gitian-build executables. Maybe there's a way to disable QLibrary's dynamic loading to remove the dependency, but there's no need to do so here.\r\n"}}
{"actor":{"display_login":"laanwj"}}