{"action":"created","issue":{"url":"https://api.github.com/repos/giampaolo/psutil/issues/723","labels_url":"https://api.github.com/repos/giampaolo/psutil/issues/723/labels{/name}","comments_url":"https://api.github.com/repos/giampaolo/psutil/issues/723/comments","events_url":"https://api.github.com/repos/giampaolo/psutil/issues/723/events","html_url":"https://github.com/giampaolo/psutil/issues/723","id":125060970,"number":723,"title":"extensions fail to build when dependency of package with extensions, and Cython installed","user":{"login":"johnyf","id":3135273,"avatar_url":"https://avatars.githubusercontent.com/u/3135273?v=3","gravatar_id":"","url":"https://api.github.com/users/johnyf","html_url":"https://github.com/johnyf","followers_url":"https://api.github.com/users/johnyf/followers","following_url":"https://api.github.com/users/johnyf/following{/other_user}","gists_url":"https://api.github.com/users/johnyf/gists{/gist_id}","starred_url":"https://api.github.com/users/johnyf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/johnyf/subscriptions","organizations_url":"https://api.github.com/users/johnyf/orgs","repos_url":"https://api.github.com/users/johnyf/repos","events_url":"https://api.github.com/users/johnyf/events{/privacy}","received_events_url":"https://api.github.com/users/johnyf/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2016-01-05T21:59:59Z","updated_at":"2016-01-06T13:14:24Z","closed_at":null,"body":"The following minimal example, with `foo.c` trivial (and trivial package `hoho` content) fails to install with `python setup.py install`, due to `psutil` failing to compile its extensions.\r\n\r\n(More accurately, `hoho` is installed, but `setuptools` fails in its attempt to install the dependency `psutil` after finishing with `hoho` itself).\r\n\r\n```python\r\nfrom setuptools import setup, Extension\r\n\r\nname = 'hoho'\r\nversion = '0.0.1'\r\next_modules = [Extension('foo', ['hoho/foo.c'])]\r\nsetup(\r\n     name=name,\r\n     version=version,\r\n     packages=[name],\r\n     package_dir={name: name},\r\n     install_requires=['psutil'],\r\n     ext_modules=ext_modules)\r\n```\r\n\r\nThe failure happens only if `cython` is installed. If no `ext_modules` are passed, then no error arises.\r\nThe output is:\r\n\r\n```shell\r\n(bug)hoho:pip install cython\r\nCollecting cython\r\nInstalling collected packages: cython\r\nSuccessfully installed cython-0.23.4\r\n[154839 refs]\r\n(bug)hoho:python setup.py install\r\nrunning install\r\nrunning bdist_egg\r\nrunning egg_info\r\nwriting requirements to hoho.egg-info/requires.txt\r\nwriting hoho.egg-info/PKG-INFO\r\nwriting top-level names to hoho.egg-info/top_level.txt\r\nwriting dependency_links to hoho.egg-info/dependency_links.txt\r\nreading manifest file 'hoho.egg-info/SOURCES.txt'\r\nwriting manifest file 'hoho.egg-info/SOURCES.txt'\r\ninstalling library code to build/bdist.macosx-10.4-x86_64/egg\r\nrunning install_lib\r\nrunning build_py\r\nrunning build_ext\r\ncreating build/bdist.macosx-10.4-x86_64/egg\r\ncopying build/lib.macosx-10.4-x86_64-2.7-pydebug/foo.so -> build/bdist.macosx-10.4-x86_64/egg\r\ncreating build/bdist.macosx-10.4-x86_64/egg/hoho\r\ncopying build/lib.macosx-10.4-x86_64-2.7-pydebug/hoho/__init__.py -> build/bdist.macosx-10.4-x86_64/egg/hoho\r\ncopying build/lib.macosx-10.4-x86_64-2.7-pydebug/hoho/hehe.py -> build/bdist.macosx-10.4-x86_64/egg/hoho\r\nbyte-compiling build/bdist.macosx-10.4-x86_64/egg/hoho/__init__.py to __init__.pyc\r\nbyte-compiling build/bdist.macosx-10.4-x86_64/egg/hoho/hehe.py to hehe.pyc\r\ncreating stub loader for foo.so\r\nbyte-compiling build/bdist.macosx-10.4-x86_64/egg/foo.py to foo.pyc\r\ncreating build/bdist.macosx-10.4-x86_64/egg/EGG-INFO\r\ncopying hoho.egg-info/PKG-INFO -> build/bdist.macosx-10.4-x86_64/egg/EGG-INFO\r\ncopying hoho.egg-info/SOURCES.txt -> build/bdist.macosx-10.4-x86_64/egg/EGG-INFO\r\ncopying hoho.egg-info/dependency_links.txt -> build/bdist.macosx-10.4-x86_64/egg/EGG-INFO\r\ncopying hoho.egg-info/requires.txt -> build/bdist.macosx-10.4-x86_64/egg/EGG-INFO\r\ncopying hoho.egg-info/top_level.txt -> build/bdist.macosx-10.4-x86_64/egg/EGG-INFO\r\nwriting build/bdist.macosx-10.4-x86_64/egg/EGG-INFO/native_libs.txt\r\nzip_safe flag not set; analyzing archive contents...\r\ncreating 'dist/hoho-0.0.1-py2.7-macosx-10.4-x86_64.egg' and adding 'build/bdist.macosx-10.4-x86_64/egg' to it\r\nremoving 'build/bdist.macosx-10.4-x86_64/egg' (and everything under it)\r\nProcessing hoho-0.0.1-py2.7-macosx-10.4-x86_64.egg\r\nCopying hoho-0.0.1-py2.7-macosx-10.4-x86_64.egg to /Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages\r\nAdding hoho 0.0.1 to easy-install.pth file\r\n\r\nInstalled /Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/hoho-0.0.1-py2.7-macosx-10.4-x86_64.egg\r\nProcessing dependencies for hoho==0.0.1\r\nSearching for psutil\r\nReading https://pypi.python.org/simple/psutil/\r\nBest match: psutil 3.3.0\r\nDownloading https://pypi.python.org/packages/source/p/psutil/psutil-3.3.0.tar.gz#md5=0625db01ea46417308db94cc59362b65\r\nProcessing psutil-3.3.0.tar.gz\r\nWriting /var/folders/cy/b25_dhcn0bj1s5j95zf0j9_h0000gp/T/easy_install-paZ0Po/psutil-3.3.0/setup.cfg\r\nRunning psutil-3.3.0/setup.py -q bdist_egg --dist-dir /var/folders/cy/b25_dhcn0bj1s5j95zf0j9_h0000gp/T/easy_install-paZ0Po/psutil-3.3.0/egg-dist-tmp-MmWzk4\r\nTraceback (most recent call last):\r\n  File \"setup.py\", line 12, in <module>\r\n    ext_modules=ext_modules)\r\n  File \"/Users/ifilippi/python/lib/python2.7/distutils/core.py\", line 151, in setup\r\n    dist.run_commands()\r\n  File \"/Users/ifilippi/python/lib/python2.7/distutils/dist.py\", line 953, in run_commands\r\n    self.run_command(cmd)\r\n  File \"/Users/ifilippi/python/lib/python2.7/distutils/dist.py\", line 972, in run_command\r\n    cmd_obj.run()\r\n  File \"/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/command/install.py\", line 67, in run\r\n    self.do_egg_install()\r\n  File \"/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/command/install.py\", line 117, in do_egg_install\r\n    cmd.run()\r\n  File \"/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/command/easy_install.py\", line 390, in run\r\n    self.easy_install(spec, not self.no_deps)\r\n  File \"/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/command/easy_install.py\", line 620, in easy_install\r\n    return self.install_item(None, spec, tmpdir, deps, True)\r\n  File \"/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/command/easy_install.py\", line 671, in install_item\r\n    self.process_distribution(spec, dist, deps)\r\n  File \"/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/command/easy_install.py\", line 719, in process_distribution\r\n    [requirement], self.local_index, self.easy_install\r\n  File \"/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/pkg_resources/__init__.py\", line 846, in resolve\r\n    dist = best[req.key] = env.best_match(req, ws, installer)\r\n  File \"/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/pkg_resources/__init__.py\", line 1091, in best_match\r\n    return self.obtain(req, installer)\r\n  File \"/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/pkg_resources/__init__.py\", line 1103, in obtain\r\n    return installer(requirement)\r\n  File \"/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/command/easy_install.py\", line 639, in easy_install\r\n    return self.install_item(spec, dist.location, tmpdir, deps)\r\n  File \"/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/command/easy_install.py\", line 669, in install_item\r\n    dists = self.install_eggs(spec, download, tmpdir)\r\n  File \"/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/command/easy_install.py\", line 852, in install_eggs\r\n    return self.build_and_install(setup_script, setup_base)\r\n  File \"/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/command/easy_install.py\", line 1080, in build_and_install\r\n    self.run_setup(setup_script, setup_base, args)\r\n  File \"/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/command/easy_install.py\", line 1066, in run_setup\r\n    run_setup(setup_script, args)\r\n  File \"/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/sandbox.py\", line 242, in run_setup\r\n    raise\r\n  File \"/Users/ifilippi/python/lib/python2.7/contextlib.py\", line 35, in __exit__\r\n    self.gen.throw(type, value, traceback)\r\n  File \"/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/sandbox.py\", line 195, in setup_context\r\n    yield\r\n  File \"/Users/ifilippi/python/lib/python2.7/contextlib.py\", line 35, in __exit__\r\n    self.gen.throw(type, value, traceback)\r\n  File \"/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/sandbox.py\", line 166, in save_modules\r\n    saved_exc.resume()\r\n  File \"/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/sandbox.py\", line 141, in resume\r\n    compat.reraise(type, exc, self._tb)\r\n  File \"/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/sandbox.py\", line 154, in save_modules\r\n    yield saved\r\n  File \"/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/sandbox.py\", line 195, in setup_context\r\n    yield\r\n  File \"/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/sandbox.py\", line 239, in run_setup\r\n    DirectorySandbox(setup_dir).run(runner)\r\n  File \"/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/sandbox.py\", line 269, in run\r\n    return func()\r\n  File \"/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/sandbox.py\", line 238, in runner\r\n    _execfile(setup_script, ns)\r\n  File \"/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/sandbox.py\", line 46, in _execfile\r\n    exec(code, globals, locals)\r\n  File \"/var/folders/cy/b25_dhcn0bj1s5j95zf0j9_h0000gp/T/easy_install-paZ0Po/psutil-3.3.0/setup.py\", line 268, in <module>\r\n\r\n  File \"/var/folders/cy/b25_dhcn0bj1s5j95zf0j9_h0000gp/T/easy_install-paZ0Po/psutil-3.3.0/setup.py\", line 265, in main\r\n\r\n  File \"/Users/ifilippi/python/lib/python2.7/distutils/core.py\", line 151, in setup\r\n    dist.run_commands()\r\n  File \"/Users/ifilippi/python/lib/python2.7/distutils/dist.py\", line 953, in run_commands\r\n    self.run_command(cmd)\r\n  File \"/Users/ifilippi/python/lib/python2.7/distutils/dist.py\", line 972, in run_command\r\n    cmd_obj.run()\r\n  File \"/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/command/bdist_egg.py\", line 151, in run\r\n    self.run_command(\"egg_info\")\r\n  File \"/Users/ifilippi/python/lib/python2.7/distutils/cmd.py\", line 326, in run_command\r\n    self.distribution.run_command(command)\r\n  File \"/Users/ifilippi/python/lib/python2.7/distutils/dist.py\", line 972, in run_command\r\n    cmd_obj.run()\r\n  File \"/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/command/egg_info.py\", line 185, in run\r\n    self.find_sources()\r\n  File \"/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/command/egg_info.py\", line 208, in find_sources\r\n    mm.run()\r\n  File \"/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/command/egg_info.py\", line 292, in run\r\n    self.add_defaults()\r\n  File \"/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/command/egg_info.py\", line 321, in add_defaults\r\n    sdist.add_defaults(self)\r\n  File \"/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/command/sdist.py\", line 131, in add_defaults\r\n    build_ext = self.get_finalized_command('build_ext')\r\n  File \"/Users/ifilippi/python/lib/python2.7/distutils/cmd.py\", line 312, in get_finalized_command\r\n    cmd_obj.ensure_finalized()\r\n  File \"/Users/ifilippi/python/lib/python2.7/distutils/cmd.py\", line 109, in ensure_finalized\r\n    self.finalize_options()\r\n  File \"/Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/setuptools/command/build_ext.py\", line 97, in finalize_options\r\n    self.check_extensions_list(self.extensions)\r\n  File \"/Users/ifilippi/python/lib/python2.7/distutils/command/build_ext.py\", line 362, in check_extensions_list\r\n    (\"each element of 'ext_modules' option must be an \"\r\nsetuptools.sandbox.UnpickleableException: DistutilsSetupError(\"each element of 'ext_modules' option must be an Extension instance or 2-tuple\",)\r\n[223031 refs]\r\n```\r\n\r\nIt appears to be an instance of:\r\n\r\n- [issue 309 of `setuptools`](https://bitbucket.org/pypa/setuptools/issues/309/error-each-element-of-ext_modules-option)\r\n- the discussion over here: http://stackoverflow.com/questions/21594925/error-each-element-of-ext-modules-option-must-be-an-extension-instance-or-2-t\r\n\r\nand possibly related to:\r\n\r\n- [issue 275 of `distribute`](https://bitbucket.org/tarek/distribute/issues/275/cannot-specify-cython-under-setup_requires)\r\n- [issue 288 of `setuptools`](https://bitbucket.org/pypa/setuptools/issues/288/cannot-specify-cython-under-setup_requires)\r\n\r\nIt arose when testing `dd` [on Travis](https://travis-ci.org/johnyf/dd/jobs/100388826).\r\nThere is no other workaround that I expect will work for `dd`, other than to either:\r\n\r\n- avoid this from happening in `psutil`\r\n- pursue a fix in `setuptools` or `cython` (not sure whose the culprit -- could be `distribute`, but not expected to make a fix available soon)\r\n- drop `psutil` from `install_requires` of `dd`.\r\n\r\nFor comparison, a successful installation produces as output:\r\n\r\n```shell\r\n(bug)hoho:python setup.py install\r\nrunning install\r\nrunning bdist_egg\r\nrunning egg_info\r\nwriting requirements to hoho.egg-info/requires.txt\r\nwriting hoho.egg-info/PKG-INFO\r\nwriting top-level names to hoho.egg-info/top_level.txt\r\nwriting dependency_links to hoho.egg-info/dependency_links.txt\r\nreading manifest file 'hoho.egg-info/SOURCES.txt'\r\nwriting manifest file 'hoho.egg-info/SOURCES.txt'\r\ninstalling library code to build/bdist.macosx-10.4-x86_64/egg\r\nrunning install_lib\r\nrunning build_py\r\ncreating build/lib\r\ncreating build/lib/hoho\r\ncopying hoho/__init__.py -> build/lib/hoho\r\ncopying hoho/hehe.py -> build/lib/hoho\r\ncreating build/bdist.macosx-10.4-x86_64/egg\r\ncreating build/bdist.macosx-10.4-x86_64/egg/hoho\r\ncopying build/lib/hoho/__init__.py -> build/bdist.macosx-10.4-x86_64/egg/hoho\r\ncopying build/lib/hoho/hehe.py -> build/bdist.macosx-10.4-x86_64/egg/hoho\r\nbyte-compiling build/bdist.macosx-10.4-x86_64/egg/hoho/__init__.py to __init__.pyc\r\nbyte-compiling build/bdist.macosx-10.4-x86_64/egg/hoho/hehe.py to hehe.pyc\r\ncreating build/bdist.macosx-10.4-x86_64/egg/EGG-INFO\r\ncopying hoho.egg-info/PKG-INFO -> build/bdist.macosx-10.4-x86_64/egg/EGG-INFO\r\ncopying hoho.egg-info/SOURCES.txt -> build/bdist.macosx-10.4-x86_64/egg/EGG-INFO\r\ncopying hoho.egg-info/dependency_links.txt -> build/bdist.macosx-10.4-x86_64/egg/EGG-INFO\r\ncopying hoho.egg-info/requires.txt -> build/bdist.macosx-10.4-x86_64/egg/EGG-INFO\r\ncopying hoho.egg-info/top_level.txt -> build/bdist.macosx-10.4-x86_64/egg/EGG-INFO\r\nzip_safe flag not set; analyzing archive contents...\r\ncreating 'dist/hoho-0.0.1-py2.7.egg' and adding 'build/bdist.macosx-10.4-x86_64/egg' to it\r\nremoving 'build/bdist.macosx-10.4-x86_64/egg' (and everything under it)\r\nProcessing hoho-0.0.1-py2.7.egg\r\nCopying hoho-0.0.1-py2.7.egg to /Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages\r\nAdding hoho 0.0.1 to easy-install.pth file\r\n\r\nInstalled /Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/hoho-0.0.1-py2.7.egg\r\nProcessing dependencies for hoho==0.0.1\r\nSearching for psutil\r\nReading https://pypi.python.org/simple/psutil/\r\nBest match: psutil 3.3.0\r\nDownloading https://pypi.python.org/packages/source/p/psutil/psutil-3.3.0.tar.gz#md5=0625db01ea46417308db94cc59362b65\r\nProcessing psutil-3.3.0.tar.gz\r\nWriting /var/folders/cy/b25_dhcn0bj1s5j95zf0j9_h0000gp/T/easy_install-8JI14G/psutil-3.3.0/setup.cfg\r\nRunning psutil-3.3.0/setup.py -q bdist_egg --dist-dir /var/folders/cy/b25_dhcn0bj1s5j95zf0j9_h0000gp/T/easy_install-8JI14G/psutil-3.3.0/egg-dist-tmp-pM1MD7\r\nwarning: no previously-included files matching '*' found under directory 'docs/_build'\r\nwarning: manifest_maker: MANIFEST.in, line 18: 'recursive-include' expects <dir> <pattern1> <pattern2> ...\r\n\r\nzip_safe flag not set; analyzing archive contents...\r\npsutil.__init__: module references __file__\r\ncreating /Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/psutil-3.3.0-py2.7-macosx-10.4-x86_64.egg\r\nExtracting psutil-3.3.0-py2.7-macosx-10.4-x86_64.egg to /Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages\r\nAdding psutil 3.3.0 to easy-install.pth file\r\n\r\nInstalled /Users/ifilippi/.virtualenvs/bug/lib/python2.7/site-packages/psutil-3.3.0-py2.7-macosx-10.4-x86_64.egg\r\nFinished processing dependencies for hoho==0.0.1\r\n[119389 refs]\r\n```"},"comment":{"url":"https://api.github.com/repos/giampaolo/psutil/issues/comments/169320558","html_url":"https://github.com/giampaolo/psutil/issues/723#issuecomment-169320558","issue_url":"https://api.github.com/repos/giampaolo/psutil/issues/723","id":169320558,"user":{"login":"johnyf","id":3135273,"avatar_url":"https://avatars.githubusercontent.com/u/3135273?v=3","gravatar_id":"","url":"https://api.github.com/users/johnyf","html_url":"https://github.com/johnyf","followers_url":"https://api.github.com/users/johnyf/followers","following_url":"https://api.github.com/users/johnyf/following{/other_user}","gists_url":"https://api.github.com/users/johnyf/gists{/gist_id}","starred_url":"https://api.github.com/users/johnyf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/johnyf/subscriptions","organizations_url":"https://api.github.com/users/johnyf/orgs","repos_url":"https://api.github.com/users/johnyf/repos","events_url":"https://api.github.com/users/johnyf/events{/privacy}","received_events_url":"https://api.github.com/users/johnyf/received_events","type":"User","site_admin":false},"created_at":"2016-01-06T13:14:24Z","updated_at":"2016-01-06T13:14:24Z","body":"The main difference between `setuptools` 18.8 and 18.8.1 appears to be the memory blow up reported in [`setuptools` issue 440](https://bitbucket.org/pypa/setuptools/issues/440/unpickleableexceptiondump-blows-up-when). So, the [fix 2461fe779674](https://bitbucket.org/pypa/setuptools/commits/2461fe779674) in `setuptools` to avoid infinite recursion appears to uncover another bug."}}