{"action":"created","issue":{"url":"https://api.github.com/repos/conda/conda-build/issues/911","repository_url":"https://api.github.com/repos/conda/conda-build","labels_url":"https://api.github.com/repos/conda/conda-build/issues/911/labels{/name}","comments_url":"https://api.github.com/repos/conda/conda-build/issues/911/comments","events_url":"https://api.github.com/repos/conda/conda-build/issues/911/events","html_url":"https://github.com/conda/conda-build/issues/911","id":152675429,"number":911,"title":"Using conda to manage multiple native library configurations without Anaconda or Python in the mix","user":{"login":"mwiebe","id":399551,"avatar_url":"https://avatars.githubusercontent.com/u/399551?v=3","gravatar_id":"","url":"https://api.github.com/users/mwiebe","html_url":"https://github.com/mwiebe","followers_url":"https://api.github.com/users/mwiebe/followers","following_url":"https://api.github.com/users/mwiebe/following{/other_user}","gists_url":"https://api.github.com/users/mwiebe/gists{/gist_id}","starred_url":"https://api.github.com/users/mwiebe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mwiebe/subscriptions","organizations_url":"https://api.github.com/users/mwiebe/orgs","repos_url":"https://api.github.com/users/mwiebe/repos","events_url":"https://api.github.com/users/mwiebe/events{/privacy}","received_events_url":"https://api.github.com/users/mwiebe/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":15,"created_at":"2016-05-03T00:27:38Z","updated_at":"2016-05-04T06:09:12Z","closed_at":null,"body":"We're most of the way through setting this up so we can use conda for this internally at Thinkbox. It's much better than managing 50 dependent libraries and SDKs manually, but could be better. I'm posting this now because I saw https://twitter.com/wesmckinn/status/725448295774969857 from @wesm, and thought I would describe some parts of what we're doing. Our solution is passable as an internal tool, but integrating support for what we're doing directly from conda and conda-build would be much nicer.\r\n\r\nInitially we evaluated between conda and rez (https://github.com/nerdvegas/rez). Being a tool from VFX, I thought rez might make integrating with some graphics-related packages easier out of the box, but conda came out as a clear winner in the comparison. The biggest challenge to be able to use conda was to find a way to manage multiple compiler configurations. Conda-build has some hard-coded internal logic that selects a compiler based on platform and python version, so out of the box it doesn't support this idea at all.\r\n\r\nExample compiler configurations one might want:\r\n\r\n* MSVC configured in debug mode. This is a different ABI than release, so all dependent packages must be built this way.\r\n* MSVC 2013 or 2015 with the  `/Gv` flag, so numeric code uses the faster vectorcall convention.\r\n* Clang or gcc with various sanitizer instrumentations enabled. This works best if all dependent packages are built with the same flags.\r\n* Builds with everything configured to link statically, or to link dynamically. To be able to build applications that link into one big executable or as a collection of libraries depending on the chosen configuration.\r\n* Builds with the compiler flags set to target a specific CPU microarchitecture, e.g. the one used for a  machine type in cloud infrastructure, to optimize usage of computing within that cloud.\r\n\r\nOur solution to defining multiple compiler configurations uses a conda feature paired with a conda package of the same name. The feature gets applied to all packages built with that compiler configuration.\r\n\r\nFor example, the package `msvc2012rel` provides the feature `msvc2012rel`, and installs files to configure the compiler for builds with MSVC 2012 in release mode. Its `meta.yaml` file looks like:\r\n\r\n```\r\npackage:\r\n    name: msvc2012rel\r\n    version: 1.0\r\n\t\r\nbuild:\r\n  number: 4\r\n  track_features:\r\n    - msvc2012rel\r\n```\r\n\r\nand its `bld.bat` installs the file `compiler_config.bat` which looks like the following. It's bit messy, admittedly, but provides environment variables to easily tell cmake and other build systems the compiler configuration.\r\n\r\n```\r\n@echo on\r\nset MSVC_VER=11.0\r\nset MSCFULL_VER=17.00.61030\r\nset MSVC_VER_YEAR=2012\r\nset ARCH=@ARCH@\r\n\r\nset CMAKE_BUILD_TYPE=Release\r\nset BUILD_TYPE_LOWER=release\r\nset BUILD_DYNAMIC=OFF\r\n\r\nset CFLAGS=-DUNICODE -D_UNICODE\r\nset CXXFLAGS=%CFLAGS%\r\n\r\nREM ------------------------------------------------------\r\nREM The rest below is generic\r\n\r\nif \"%ARCH%\" == \"64\" set VCVARS_ARCH=amd64\r\nif \"%ARCH%\" == \"32\" set VCVARS_ARCH=x86\r\nif \"%ARCH%\" == \"64\" set CMAKEGEN_ARCH= Win64\r\nif \"%ARCH%\" == \"32\" set CMAKEGEN_ARCH=\r\nif \"%ARCH%\" == \"64\" set MSVC_ARCH=x64\r\nif \"%ARCH%\" == \"32\" set MSVC_ARCH=Win32\r\n\r\nREM Activate the compiler command line\r\ncall \"C:\\Program Files (x86)\\Microsoft Visual Studio %MSVC_VER%\\VC\\vcvarsall.bat\" %VCVARS_ARCH% || exit /b 1\r\n@echo on\r\n\r\nREM Validate version of MSVC\r\ncl 2>&1 | C:\\Gow\\bin\\grep %MSCFULL_VER%\r\nif \"%ERRORLEVEL%\" == \"0\" goto good_msc_ver\r\necho Bad MSVC Version: expected %MSCFULL_VER%\r\ncl 2>&1\r\nexit /b 1\r\n:good_msc_ver\r\n\r\nset CMAKE_GENERATOR=Visual Studio %MSVC_VER:~0,2% %MSVC_VER_YEAR%%CMAKEGEN_ARCH%\r\n```\r\n\r\nWith this system, normal conda-build recipes do not work, they must be told which compiler configuration to use, and then they need to depend on the feature tag for that compiler configuration. This is done through the `COMPILER_CONFIG` environment variable, which gets interpolated into the recipe using conda's jinja2 templating. See the `meta.yaml` file for a zlib recipe following this standard:\r\n\r\n```\r\npackage:\r\n  name: zlib\r\n  version: 1.2.8\r\n\r\nsource:\r\n  fn: zlib-1.2.8.tar.gz\r\n  url: http://zlib.net/zlib-1.2.8.tar.gz\r\n  md5: 44d667c142d7cda120332623eab69f40\r\n\r\nbuild:\r\n  number: 3\r\n  features:\r\n    - {{ environ['COMPILER_CONFIG'] }}\r\n\r\nrequirements:\r\n  build:\r\n    - {{ environ['COMPILER_CONFIG'] }}\r\n\r\nabout:\r\n  home: http://zlib.net/\r\n  license: zlib (http://zlib.net/zlib_license.html)\r\n  summary: \"A Massively Spiffy Yet Delicately Unobtrusive Compression Library\"\r\n```\r\n\r\nEvery recipe's `bld.bat` and `build.sh` first source the `compiler_config.bat` or `compiler_config.sh` file provided by the compiler config package, then use the defined variables and configured compiler to build. The `bld.bat` for zlib is:\r\n\r\n```\r\ncall compiler_config.bat || exit /b 1\r\n@echo on\r\n\r\nmkdir build\r\ncd build\r\n\r\nREM Configure step\r\ncmake -G \"%CMAKE_GENERATOR%\" -DBUILD_SHARED_LIBS=%BUILD_DYNAMIC% -DCMAKE_BUILD_TYPE=%CMAKE_BUILD_TYPE% -DCMAKE_PREFIX_PATH=%LIBRARY_PREFIX% -DCMAKE_INSTALL_PREFIX:PATH=%LIBRARY_PREFIX% %SRC_DIR% || exit /b 1\r\n\r\nREM Build step\r\ncmake --build . --config %CMAKE_BUILD_TYPE% || exit /b 1\r\n\r\nREM Install step\r\ncmake --build . --config %CMAKE_BUILD_TYPE% --target install || exit /b 1\r\n\r\ncopy ..\\contrib\\iostream3\\zfstream.h %LIBRARY_PREFIX%\\include || exit /b 1\r\ncopy ..\\contrib\\iostream3\\zfstream.cc %LIBRARY_PREFIX%\\include || exit /b 1\r\n```\r\n\r\nand the `build.sh` is basically identical to how a typical recipe's would be except for sourcing `compiler_config.sh`:\r\n\r\n```\r\n#!/bin/bash\r\nset -xe\r\n\r\nsource compiler_config.sh\r\n\r\n./configure --prefix=$PREFIX\r\nmake\r\nmake install\r\n\r\ncp $SRC_DIR/contrib/iostream3/zfstream.h $PREFIX/include\r\ncp $SRC_DIR/contrib/iostream3/zfstream.cc $PREFIX/include\r\n```\r\n\r\nTo conclude things, this system is working, but would be a lot better with first-class support from conda. What we're using isn't suitable for dropping directly into conda, but maybe the way we've done things can inspire something that is."},"comment":{"url":"https://api.github.com/repos/conda/conda-build/issues/comments/216753156","html_url":"https://github.com/conda/conda-build/issues/911#issuecomment-216753156","issue_url":"https://api.github.com/repos/conda/conda-build/issues/911","id":216753156,"user":{"login":"mcg1969","id":2797735,"avatar_url":"https://avatars.githubusercontent.com/u/2797735?v=3","gravatar_id":"","url":"https://api.github.com/users/mcg1969","html_url":"https://github.com/mcg1969","followers_url":"https://api.github.com/users/mcg1969/followers","following_url":"https://api.github.com/users/mcg1969/following{/other_user}","gists_url":"https://api.github.com/users/mcg1969/gists{/gist_id}","starred_url":"https://api.github.com/users/mcg1969/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mcg1969/subscriptions","organizations_url":"https://api.github.com/users/mcg1969/orgs","repos_url":"https://api.github.com/users/mcg1969/repos","events_url":"https://api.github.com/users/mcg1969/events{/privacy}","received_events_url":"https://api.github.com/users/mcg1969/received_events","type":"User","site_admin":false},"created_at":"2016-05-04T06:09:12Z","updated_at":"2016-05-04T06:09:12Z","body":"I don't know if there's an option to clear out the `conda-bld` directory, but it can't hurt to just remove it outright before such a build. I also don't know if it relies on anything in the package cache (the `pkgs` subdirectory), but if so that would need to be wiped, too. This is just off the top of my head though."}}