{"action":"created","issue":{"url":"https://api.github.com/repos/conda/conda-build/issues/848","repository_url":"https://api.github.com/repos/conda/conda-build","labels_url":"https://api.github.com/repos/conda/conda-build/issues/848/labels{/name}","comments_url":"https://api.github.com/repos/conda/conda-build/issues/848/comments","events_url":"https://api.github.com/repos/conda/conda-build/issues/848/events","html_url":"https://github.com/conda/conda-build/pull/848","id":143543429,"number":848,"title":"WIP: Build customization","user":{"login":"ukoethe","id":897680,"avatar_url":"https://avatars.githubusercontent.com/u/897680?v=3","gravatar_id":"","url":"https://api.github.com/users/ukoethe","html_url":"https://github.com/ukoethe","followers_url":"https://api.github.com/users/ukoethe/followers","following_url":"https://api.github.com/users/ukoethe/following{/other_user}","gists_url":"https://api.github.com/users/ukoethe/gists{/gist_id}","starred_url":"https://api.github.com/users/ukoethe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ukoethe/subscriptions","organizations_url":"https://api.github.com/users/ukoethe/orgs","repos_url":"https://api.github.com/users/ukoethe/repos","events_url":"https://api.github.com/users/ukoethe/events{/privacy}","received_events_url":"https://api.github.com/users/ukoethe/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/conda/conda-build/labels/2%20-%20Needs%20Decision","name":"2 - Needs Decision","color":"CCCCCC"},{"url":"https://api.github.com/repos/conda/conda-build/labels/enhancement","name":"enhancement","color":"84b6eb"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":40,"created_at":"2016-03-25T17:08:01Z","updated_at":"2016-04-07T08:55:23Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/conda/conda-build/pulls/848","html_url":"https://github.com/conda/conda-build/pull/848","diff_url":"https://github.com/conda/conda-build/pull/848.diff","patch_url":"https://github.com/conda/conda-build/pull/848.patch"},"body":"# Conda-build Customization\r\n\r\nThis PR implements a number of enhancements that greatly simplify customization of the build and run requirements of a recipe. The general idea is this:\r\n\r\n* the recipe's build requirements should state source compatibility constraints\r\n* constraints on build dependency *binaries* are added at build time via customization\r\n* constraints on run requirements are computed at build time\r\n\r\nThe PR is not necessarily meant to be merged as-is, but should primarily serve as a solid basis for discussion of customization ideas.\r\n\r\nNote: This PR intentionally omits the problems of features, which will be the subject of another proposal.\r\n\r\n## Option `--build-config`\r\n\r\nThis option allows the user to pass a build config file to conda-build via\r\n```\r\nconda build --build-config /path/to/config.yaml   foo\r\n```\r\n`config.yaml` is similar to `meta.yaml`, but only contains the section `requirements/build` (everything else is currently ignored). The contents of this file are simply concatenated to the recipe's build requirements. If this results in multiple lines referring to the same package, all corresponding constraints are joined and must be valid simultaneously. This behavior is already realized in conda's version resolution algorithm, and there is no need for any additional logic to be implemented. Semantically, recipes state the specific requirements of particular packages, whereas config files state requirements to be shared by a family of builds (e.g. (meta)packages that activate desired `track_features` declarations). In an automated deployment system, config files can be auto-generated from a build matrix.\r\n\r\n## Option `--bootstrap`\r\n\r\nThis option allows the user to pass a bootstrap environment to conda-build via\r\n```\r\nconda build --bootstrap some_env_name   foo\r\n```\r\nThis behaves like `--build-config`, but the config file is automatically constructed from the given environment: All packages currently installed in `some_env_name` are considered as build requirements of the package to be built, in precisely the version found in the environment. This approach leverages the full power of conda's version resolution logic to free the user from the tedious and error-prone task of maintaining config files manually.\r\n\r\n## Jinja2 variable `{{ installed }}`\r\n\r\nThis variable is a dictionary whose keys are the package names found in the current `_build` environment, and the corresponding values contain the complete metadata from `_build/conda-meta/package_name.json`. This information can be used to configure run requirements in `meta.yaml` at build time with up-to-date information from the build itself:\r\n```yaml\r\nrequirements:\r\n  build:\r\n    - zlib\r\n    - jpeg \r\n\r\n  run:\r\n    - zlib  {{ installed['zlib']['version'] }}\r\n    - jpeg  {{ installed['jpeg']['version'] }}\r\n```\r\nThis pins the run requirements to the exact dependency versions present during the build. Of course, strict run requirements like this are a bit too pessimistic to be practical, so it is desirable to suitably relax the constraints. A solution using jinja filters is described in the next section. Another good idea would be to adapt the \"compatible release\" operator `~=` from [PEP 440](https://www.python.org/dev/peps/pep-0440/#compatible-release) to conda-build, so that one could write\r\n```yaml\r\n  run:\r\n    - zlib  ~={{ installed['zlib']['version'] }}\r\n    - jpeg  ~={{ installed['jpeg']['version'] }}\r\n```\r\nto permit all versions that are considered compatible with the build dependency.\r\n\r\n## Jinja2 configuration callback\r\n\r\nIf the recipe directory contains a file `jinja_config.py` that defines a function `jinja_config(jinja_env)`, this function is called by conda-build just before asking jinja to parse `meta.yaml`. This allows the user to add additional variables and filters to the jinja namespace. A useful application of this capability is a filter that approximates the behavior of the compatible release operator `~=` by transforming a version number like `1.2.3` into a version constraint like `1.2*,>=1.2.3`. A simple implementation of the callback function might look like this:\r\n```python\r\ndef jinja_config(jinja_env):\r\n\r\n    def compatible_versions(version):\r\n        v = version.split('.')\r\n        if len(v) > 1:\r\n            return v[0]+'.'+v[1]+'*,>='+version\r\n        else:\r\n            return version\r\n\r\n    jinja_env.filters['compatible'] = compatible_versions\r\n```\r\nand could be used in `meta.yaml` via the pipe operator `|`\r\n```yaml\r\n  run:\r\n    - zlib  {{ installed['zlib']['version'] | compatible }}\r\n    - jpeg   {{ installed['jpeg']['version'] | compatible }}\r\n```\r\nThe callbeck mechanism would provide recipe designers a very useful tool for experimentation with jinja magic. Eventually, the best ideas will be implemented natively within conda-build to make them conveniently accessible to everyone. (Then, the configuration callback will mainly serve as a last resort.)"},"comment":{"url":"https://api.github.com/repos/conda/conda-build/issues/comments/206767774","html_url":"https://github.com/conda/conda-build/pull/848#issuecomment-206767774","issue_url":"https://api.github.com/repos/conda/conda-build/issues/848","id":206767774,"user":{"login":"pelson","id":810663,"avatar_url":"https://avatars.githubusercontent.com/u/810663?v=3","gravatar_id":"","url":"https://api.github.com/users/pelson","html_url":"https://github.com/pelson","followers_url":"https://api.github.com/users/pelson/followers","following_url":"https://api.github.com/users/pelson/following{/other_user}","gists_url":"https://api.github.com/users/pelson/gists{/gist_id}","starred_url":"https://api.github.com/users/pelson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pelson/subscriptions","organizations_url":"https://api.github.com/users/pelson/orgs","repos_url":"https://api.github.com/users/pelson/repos","events_url":"https://api.github.com/users/pelson/events{/privacy}","received_events_url":"https://api.github.com/users/pelson/received_events","type":"User","site_admin":false},"created_at":"2016-04-07T08:55:23Z","updated_at":"2016-04-07T08:55:23Z","body":"> If this is important, we can easily reproduce this behavior by a suitably defined and well-documented jinja variable\r\n\r\nIf we are defining the pinning as Jinja, it is not possible to know what you have pinned as a Jinja variable.\r\n\r\nBy necessity a recipe *has* to implement the build string as Jinja logic unless we add another Jinja pass:\r\n\r\n```\r\npackage:\r\n  name: my_pinned_package\r\n\r\nrequirements:\r\n  build:\r\n     - libnetcdf\r\n  run:\r\n     - libnetcdf {{ installed.libnetcdf | vn_format('x.x') }}*\r\n\r\nbuild:\r\n   number: 2\r\n   string: libnetcdf{{ installed.libnetcdf | vn_format('x.x') }}_{{ build_number }}\r\n```\r\n\r\nBy contrast, if the pinning is determined outside of the Jinja, the build string can also be computed outside of the Jinja:\r\n\r\n```\r\npackage:\r\n  name: my_pinned_package\r\n\r\nrequirements:\r\n  build:\r\n     - libnetcdf\r\n  run:\r\n     - libnetcdf\r\n  pin_from_build:\r\n     - libnetcdf x.x.*\r\n\r\nbuild:\r\n   number: 2\r\n```\r\n\r\nObviously the build string determination gets more and more complex as we add more pinned dependencies, and that is when we see a growing benefit from having conda-build do the logic rather than each recipe's Jinja.\r\n\r\nAs I've mentioned before, I'm not wed to the ```pin_from_build``` section, but I do feel we need to encode this as metadata that is defined *before* the second parsing pass."}}