{"action":"created","issue":{"url":"https://api.github.com/repos/moby/moby/issues/37129","repository_url":"https://api.github.com/repos/moby/moby","labels_url":"https://api.github.com/repos/moby/moby/issues/37129/labels{/name}","comments_url":"https://api.github.com/repos/moby/moby/issues/37129/comments","events_url":"https://api.github.com/repos/moby/moby/issues/37129/events","html_url":"https://github.com/moby/moby/issues/37129","id":325506496,"node_id":"MDU6SXNzdWUzMjU1MDY0OTY=","number":37129,"title":"Proposal: add support for multiple (named) build-contexts","user":{"login":"thaJeztah","id":1804568,"node_id":"MDQ6VXNlcjE4MDQ1Njg=","avatar_url":"https://avatars2.githubusercontent.com/u/1804568?v=4","gravatar_id":"","url":"https://api.github.com/users/thaJeztah","html_url":"https://github.com/thaJeztah","followers_url":"https://api.github.com/users/thaJeztah/followers","following_url":"https://api.github.com/users/thaJeztah/following{/other_user}","gists_url":"https://api.github.com/users/thaJeztah/gists{/gist_id}","starred_url":"https://api.github.com/users/thaJeztah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thaJeztah/subscriptions","organizations_url":"https://api.github.com/users/thaJeztah/orgs","repos_url":"https://api.github.com/users/thaJeztah/repos","events_url":"https://api.github.com/users/thaJeztah/events{/privacy}","received_events_url":"https://api.github.com/users/thaJeztah/received_events","type":"User","site_admin":false},"labels":[{"id":73616520,"node_id":"MDU6TGFiZWw3MzYxNjUyMA==","url":"https://api.github.com/repos/moby/moby/labels/area/builder","name":"area/builder","color":"fbca04","default":false},{"id":179237833,"node_id":"MDU6TGFiZWwxNzkyMzc4MzM=","url":"https://api.github.com/repos/moby/moby/labels/kind/feature","name":"kind/feature","color":"AAB3AB","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":8,"created_at":"2018-05-23T00:18:17Z","updated_at":"2018-10-22T20:44:31Z","closed_at":null,"author_association":"MEMBER","body":"# Add support for multiple (named) build-contexts\r\n\r\n\r\nRelated issues:\r\n\r\n- https://github.com/moby/moby/issues/2745 \"Add with relative path to parent directory fails with \"Forbidden path\"\"\r\n- https://github.com/moby/moby/issues/18789 Dockerfile ADD relative symlinks support\r\n- https://github.com/moby/moby/issues/12886 \"Add support for specifying .dockerignore file with -i/--ignore\"\r\n- probably others\r\n\r\n\r\n# Problem statement\r\n\r\nTake the following directory structure for a project;\r\n\r\n```\r\nproject\r\n ├── .dockerignore\r\n ├── Dockerfile\r\n ├── ginormous\r\n │   ├── big-file-1\r\n │   ├── big-file-2\r\n │   └── big-file-xx\r\n ├── Makefile\r\n ├── service1\r\n │   ├── Dockerfile\r\n │   └── src\r\n │       ├── Makefile\r\n │       ├── source-file-1\r\n │       ├── source-file-2\r\n │       └── source-file-xx\r\n ├── service2\r\n │   ├── Dockerfile\r\n │   └── src\r\n │       ├── Makefile\r\n │       ├── source-file-1\r\n │       ├── source-file-2\r\n │       └── source-file-xx\r\n ├── common\r\n │   └── src\r\n │       ├── 0-various\r\n │       ├── 1-files\r\n │       ├── 2-used-by\r\n │       ├── 3-service-1-and-2\r\n └── common-2\r\n     └── src\r\n         ├── 0-various\r\n         ├── 1-files\r\n         ├── 2-used-by\r\n         ├── 3-service-1-and-2\r\n```\r\n\r\nIn the above;\r\n\r\n- the `Dockerfile` at the root of the project uses `ginormous`, and `shared`\r\n- `service1` has a Dockerfile, and source-files used to build the service in `service1/src`\r\n- `service2` has a Dockerfile, and source-files used to build the service in `service2/src`\r\n- both `service1` and `service2` share some code/resources, located in `common` and `common-2`\r\n- nor `service1`, nor `service2` use `ginormous` (a big directory)\r\n\r\n\r\n## Challenges with this example\r\n\r\nBuilding service1 and 2 is a challenge;\r\n\r\n- When building a Dockerfile, all files used have to be within the build-context. This means that in the project structure above, the only context that can be used is the root `project` directory. Doing so results in the entire project, including `ginormous` to be sent to the daemon (even though it's not used at all). Relative paths outside of the build-context cannot be used (also see https://github.com/moby/moby/issues/2745)\r\n- Similarly; when contructing/sending the build-context, docker won't resolve symlinks, and copy symlinks as-is, so putting symlinks to `common` and `common-2` inside  `project1` and `project2` will not resolve this problem.\r\n- Only a single `.dockerignore` is supported, so it's not possible to \"conditionally\" exclude files (e.g. when building `service1`, exclude the `ginormous` and `service2` directories, and vice-versa)\r\n\r\n\r\n## Proposal: allow multiple (named) build-contexts\r\n\r\nI propose to add support for multiple build contexts, implemented as a `--context` flag on `COPY` and `ADD`, and a `--context <name>=<path>` option on the `docker image build` subcommand.\r\n\r\nFor example, to build `project1`, the Dockerfile could look like this:\r\n\r\n```Dockerfile\r\nFROM baseimage\r\n\r\n# no --context option set: use the default build-context\r\nCOPY . /build/service1/src/\r\n\r\n# use the build-context named \"common\"\r\nCOPY --context=common   . /build/common/src/\r\n\r\n# use the build-context named \"common-2\"\r\nADD --context=common-2 . /build/common-2/src/\r\n\r\nRUN cd /build && make && make install\r\n```\r\n\r\nWhen building the Dockerfile, docker expects two named build-contexts to be provided, in addition to the default (positional) build-context:\r\n\r\nFrom within the `project` directory:\r\n\r\n```bash\r\ndocker build \\\r\n  -f ./service1/Dockerfile \\\r\n  --context common=./common/src \\\r\n  --context common-2=./common-2/src \\\r\n  ./service1\r\n````\r\n\r\nIn the above:\r\n\r\n- `-f ./service1/Dockerfile` is the Dockerfile used to build the image\r\n- the `./common/src` directory is used as build-context \"common\"\r\n- the `./common-2/src` directory is used as build-context \"common-2\"\r\n- the `./service1` directory is used as default build-context\r\n\r\nOnly the `common/src`, `common-2/src` and `service1` directories are uploaded to the daemon. All other directories are not part of the build-context, so won't be uploaded.\r\n\r\n\r\nSimilarly, when building from within the `project/service1` directory:\r\n\r\n```bash\r\ndocker build \\\r\n  --context common=../common/src \\\r\n  --context common-2=../common-2/src \\\r\n  .\r\n````\r\n\r\n- No `-f` is provided, so the Dockerfile in the current directory is used to build the image.\r\n- the `../common/src` directory is used as build-context \"common\"\r\n- the `../common-2/src` directory is used as build-context \"common-2\"\r\n- the current (`.`) directory is used as default build-context\r\n\r\n\r\nBoth relative and absolute paths can be used to specify the location of a build-context, so all of these are valid:\r\n\r\n```\r\n--context foo=~/go/src/github.com/foobar/foo\r\n--context bar=/dev/sdb/share/bar/\r\n--context bar=./some/dir\r\n--context baz=../../../foobar\r\n```\r\n\r\n## Validation\r\n\r\n### Validation: missing build-contexts\r\n\r\nBefore building (and sending the build-context), docker validates if all build-contexts are provided. If a context is missing, an error is produced, and the build is aborted. Trying to build the Dockerfile from the example above without specifying any build-context:\r\n\r\n\r\n```bash\r\ndocker build .\r\nError: missing build-context \"common\"\r\nError: missing build-context \"common-2\"\r\n```\r\n\r\nIn a multi-stage build, only contexts that are required for the stages that are built should be taken into account. For example:\r\n\r\n```Dockerfile\r\nFROM baseimage AS stage-one\r\nCOPY --context=one /subdir/foo /target/dir\r\n\r\nFROM busybox AS stage-two\r\nADD --context=two /foo.tar.gz /target\r\n\r\nFROM scratch AS final\r\nCOPY --from=stage-one /foo /bar\r\nCOPY --from=stage-two /bar /baz\r\nCOPY --context=config /config.ini /config.ini\r\n```\r\n\r\nGiven the Dockerfile above:\r\n\r\nBuilding just `stage-one`:\r\n\r\n```bash\r\ndocker build --target=stage-one .\r\nError: missing build-context \"one\"\r\n```\r\n\r\nBuilding up until `stage-two`:\r\n\r\n```bash\r\ndocker build --target=stage-two .\r\nError: missing build-context \"one\"\r\nError: missing build-context \"two\"\r\n```\r\n\r\nBuilding the whole Dockerfile:\r\n\r\n```bash\r\ndocker build .\r\nError: missing build-context \"one\"\r\nError: missing build-context \"two\"\r\n```\r\n\r\nThe default build context (positional argument) is never optional:\r\n\r\n```bash\r\ndocker build --context one=./one --context two=./two\r\n\"docker build\" requires exactly 1 argument.\r\nSee 'docker build --help'.\r\n\r\nUsage:  docker build [OPTIONS] PATH | URL | - [flags]\r\n\r\nBuild an image from a Dockerfile\r\n```\r\n\r\n\r\n### Validation: unused build-contexts\r\n\r\nSimilar to unused `--build-arg`, specifying a build-context that is not used will produce a warning. When determining which contexts are expected/used in \r\n\r\nGiven the Dockerfile from the previous example:\r\n\r\n```bash\r\ndocker build --target=stage-one --context one=./one --context two=./two .\r\nSending build context to Docker daemon  2.048kB\r\nStep 1/4 : FROM baseimage AS stage-one\r\n.....\r\nRemoving intermediate container edc9bb0f15dc\r\n ---> a77418a9ddad\r\n[Warning] One or more contexts [two] were not consumed\r\nSuccessfully built a77418a9ddad\r\n```\r\n\r\n### Validation: conflicting options\r\n\r\nThe `--context` and `--from` options cannot be combined. Using both will produce an error:\r\n\r\n```Dockerfile\r\nFROM baseimage AS stage-one\r\nRUN echo foo\r\n\r\nFROM busybox\r\nCOPY --from=stage-one --context=foo /foo /bar\r\n```\r\n\r\n```bash\r\ndocker build --context=one .\r\nSending build context to Docker daemon  2.048kB\r\nError response from daemon: Dockerfile error line 5: conflicting options '--from' and '--context'\r\n```\r\n\r\nThe `--context` option _can_ be combined with the `--chown` option.\r\n\r\n\r\n### Validation: context names\r\n\r\nBuild context names follow these rules:\r\n\r\n- lowercase Alphanumeric characters (a-z, 0-9)\r\n- punctuation symbols: dashes and underscores\r\n- must start, and end, with an alphanumberic character\r\n- consecutive punctuation symbols are not allowed\r\n\r\nSpecifying an invalid context-name (either on the command-line, or inside a Dockerfile) produces an error."},"comment":{"url":"https://api.github.com/repos/moby/moby/issues/comments/431978001","html_url":"https://github.com/moby/moby/issues/37129#issuecomment-431978001","issue_url":"https://api.github.com/repos/moby/moby/issues/37129","id":431978001,"node_id":"MDEyOklzc3VlQ29tbWVudDQzMTk3ODAwMQ==","user":{"login":"IMBurbank","id":21133719,"node_id":"MDQ6VXNlcjIxMTMzNzE5","avatar_url":"https://avatars1.githubusercontent.com/u/21133719?v=4","gravatar_id":"","url":"https://api.github.com/users/IMBurbank","html_url":"https://github.com/IMBurbank","followers_url":"https://api.github.com/users/IMBurbank/followers","following_url":"https://api.github.com/users/IMBurbank/following{/other_user}","gists_url":"https://api.github.com/users/IMBurbank/gists{/gist_id}","starred_url":"https://api.github.com/users/IMBurbank/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/IMBurbank/subscriptions","organizations_url":"https://api.github.com/users/IMBurbank/orgs","repos_url":"https://api.github.com/users/IMBurbank/repos","events_url":"https://api.github.com/users/IMBurbank/events{/privacy}","received_events_url":"https://api.github.com/users/IMBurbank/received_events","type":"User","site_admin":false},"created_at":"2018-10-22T20:44:31Z","updated_at":"2018-10-22T20:44:31Z","author_association":"NONE","body":"Hi, I was wondering if there will any movement on this proposal.\r\n\r\nIt would be incredibly useful, but it doesn't currently seem to be conceptually accepted or rejected.\r\n\r\nIs this waiting for additional discussion, a working PR, a future release or is it just dead?"}}
{"actor":{"display_login":"IMBurbank"}}