{"action":"created","issue":{"url":"https://api.github.com/repos/bazelbuild/rules_docker/issues/324","repository_url":"https://api.github.com/repos/bazelbuild/rules_docker","labels_url":"https://api.github.com/repos/bazelbuild/rules_docker/issues/324/labels{/name}","comments_url":"https://api.github.com/repos/bazelbuild/rules_docker/issues/324/comments","events_url":"https://api.github.com/repos/bazelbuild/rules_docker/issues/324/events","html_url":"https://github.com/bazelbuild/rules_docker/pull/324","id":297819354,"node_id":"MDExOlB1bGxSZXF1ZXN0MTY5NjUwMDMx","number":324,"title":"Improve py_image layering","user":{"login":"scele","id":3192632,"node_id":"MDQ6VXNlcjMxOTI2MzI=","avatar_url":"https://avatars2.githubusercontent.com/u/3192632?v=4","gravatar_id":"","url":"https://api.github.com/users/scele","html_url":"https://github.com/scele","followers_url":"https://api.github.com/users/scele/followers","following_url":"https://api.github.com/users/scele/following{/other_user}","gists_url":"https://api.github.com/users/scele/gists{/gist_id}","starred_url":"https://api.github.com/users/scele/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/scele/subscriptions","organizations_url":"https://api.github.com/users/scele/orgs","repos_url":"https://api.github.com/users/scele/repos","events_url":"https://api.github.com/users/scele/events{/privacy}","received_events_url":"https://api.github.com/users/scele/received_events","type":"User","site_admin":false},"labels":[{"id":577265609,"node_id":"MDU6TGFiZWw1NzcyNjU2MDk=","url":"https://api.github.com/repos/bazelbuild/rules_docker/labels/cla:%20yes","name":"cla: yes","color":"0e8a16","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":11,"created_at":"2018-02-16T15:20:06Z","updated_at":"2018-09-29T21:08:33Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/bazelbuild/rules_docker/pulls/324","html_url":"https://github.com/bazelbuild/rules_docker/pull/324","diff_url":"https://github.com/bazelbuild/rules_docker/pull/324.diff","patch_url":"https://github.com/bazelbuild/rules_docker/pull/324.patch"},"body":"Consider the following more complex python image that uses the pip requirement feature of rules_python:\r\n\r\n    load(\"@pip_deps//:requirements.bzl\", \"requirement\")\r\n\r\n    py_library(\r\n        name = \"py_image_library_with_external_deps\",\r\n        srcs = [\"py_image_library.py\"],\r\n        deps = [requirement(\"tensorflow\")],\r\n    )\r\n\r\n    py_library(\r\n        name = \"py_image_library\",\r\n        srcs = [\"py_image_library.py\"],\r\n        deps = [\":py_image_library_with_external_deps\"],\r\n    )\r\n\r\n    py_image(\r\n        name = \"py_image_complex\",\r\n        srcs = [\"py_image.py\"],\r\n        layers = [\r\n            \":py_image_library_with_external_deps\",\r\n            \":py_image_library\",\r\n        ],\r\n        main = \"py_image.py\",\r\n    )\r\n\r\nCurrently, it will produce the following layers:\r\n\r\n    301M bazel-bin/testdata/py_image_complex.0-layer.tar\r\n    301M bazel-bin/testdata/py_image_complex.1-layer.tar\r\n     15M bazel-bin/testdata/py_image_complex-layer.tar\r\n\r\nThis change improves the layering by fixing the following issues:\r\n    \r\n1. The middle layer does not detect that all tensorflow files were already added by the first layer.\r\n    \r\n2. The last layer is 15MB in size due to the huge symlink forest. Create a separate symlink layer for each dependency layer to avoid bloating the top layer, while still keeping the dependency layers image agnostic.\r\n    \r\n3. Getting \"tensorflow\" to its own layer requires the author of the \"py_image\" rule to know that libraries transitively depend on \"tensorflow\".  This is not great, since library authors may addand remove dependencies without remembering to update the \"layers\" list.  To counter this, add a new py_layer rule that can be used to select from transitive dependencies and group them to layers, without explicitly naming the transitive dependencies like \"tensorflow\".\r\n\r\nWith these improvements, the above example can be rewritten as:\r\n\r\n    load(\"@pip_deps//:requirements.bzl\", \"requirement\")\r\n\r\n    py_library(\r\n        name = \"py_image_library_with_external_deps\",\r\n        srcs = [\"py_image_library.py\"],\r\n        deps = [requirement(\"tensorflow\")],\r\n    )\r\n\r\n    py_library(\r\n        name = \"py_image_library\",\r\n        srcs = [\"py_image_library.py\"],\r\n        deps = [\":py_image_library_with_external_deps\"],\r\n    )\r\n\r\n    py_layer(\r\n        name = \"external_deps\",\r\n        deps = [\":py_image_library\"],\r\n        filter = \"@\",\r\n    )\r\n\r\n    py_image(\r\n        name = \"py_image_complex\",\r\n        srcs = [\"py_image.py\"],\r\n        layers = [\r\n            \":external_deps\",\r\n            \":py_image_library\",\r\n        ],\r\n        main = \"py_image.py\",\r\n    )\r\n\r\nThis will yield the following layer structure:\r\n\r\n    310M bazel-bin/testdata/py_image_complex.0-layer.tar           (agnostic)\r\n     15M bazel-bin/testdata/py_image_complex.0-symlinks-layer.tar  (non-agnostic)\r\n     10K bazel-bin/testdata/py_image_complex.1-layer.tar           (agnostic)\r\n     10K bazel-bin/testdata/py_image_complex.1-symlinks-layer.tar  (non-agnostic)\r\n     20K bazel-bin/testdata/py_image_complex-layer.tar             (non-agnostic)\r\n"},"comment":{"url":"https://api.github.com/repos/bazelbuild/rules_docker/issues/comments/425675670","html_url":"https://github.com/bazelbuild/rules_docker/pull/324#issuecomment-425675670","issue_url":"https://api.github.com/repos/bazelbuild/rules_docker/issues/324","id":425675670,"node_id":"MDEyOklzc3VlQ29tbWVudDQyNTY3NTY3MA==","user":{"login":"scele","id":3192632,"node_id":"MDQ6VXNlcjMxOTI2MzI=","avatar_url":"https://avatars2.githubusercontent.com/u/3192632?v=4","gravatar_id":"","url":"https://api.github.com/users/scele","html_url":"https://github.com/scele","followers_url":"https://api.github.com/users/scele/followers","following_url":"https://api.github.com/users/scele/following{/other_user}","gists_url":"https://api.github.com/users/scele/gists{/gist_id}","starred_url":"https://api.github.com/users/scele/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/scele/subscriptions","organizations_url":"https://api.github.com/users/scele/orgs","repos_url":"https://api.github.com/users/scele/repos","events_url":"https://api.github.com/users/scele/events{/privacy}","received_events_url":"https://api.github.com/users/scele/received_events","type":"User","site_admin":false},"created_at":"2018-09-29T21:08:33Z","updated_at":"2018-09-29T21:08:33Z","author_association":"CONTRIBUTOR","body":"Fixed WORKSPACE, typo, and [replied](https://github.com/bazelbuild/rules_docker/pull/324#discussion_r221439623) about layer sharing."}}
{"actor":{"display_login":"scele"}}