{"action":"created","issue":{"url":"https://api.github.com/repos/numpy/numpy/issues/7570","repository_url":"https://api.github.com/repos/numpy/numpy","labels_url":"https://api.github.com/repos/numpy/numpy/issues/7570/labels{/name}","comments_url":"https://api.github.com/repos/numpy/numpy/issues/7570/comments","events_url":"https://api.github.com/repos/numpy/numpy/issues/7570/events","html_url":"https://github.com/numpy/numpy/issues/7570","id":150962639,"number":7570,"title":"The Linux wheels broke our build process","user":{"login":"Grk0","id":9025497,"avatar_url":"https://avatars.githubusercontent.com/u/9025497?v=3","gravatar_id":"","url":"https://api.github.com/users/Grk0","html_url":"https://github.com/Grk0","followers_url":"https://api.github.com/users/Grk0/followers","following_url":"https://api.github.com/users/Grk0/following{/other_user}","gists_url":"https://api.github.com/users/Grk0/gists{/gist_id}","starred_url":"https://api.github.com/users/Grk0/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Grk0/subscriptions","organizations_url":"https://api.github.com/users/Grk0/orgs","repos_url":"https://api.github.com/users/Grk0/repos","events_url":"https://api.github.com/users/Grk0/events{/privacy}","received_events_url":"https://api.github.com/users/Grk0/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2016-04-25T19:47:08Z","updated_at":"2016-04-25T22:13:52Z","closed_at":null,"body":"We are running Ubuntu 12.04 and 16.04 on amd64 and we're building several Debian packages that contain NumPy in a Python 2.7 virtualenv. Recently, several of our build processes started failing because of the addition of Linux wheels to NumPy.\r\n\r\nThe wheels contain pre-built libraries (libgfortran-ed201abd.so.3.0.0 and libopenblasp-r0-39a31c03.2.18.so). For whatever reason, the Ubuntu `strip` command (used to remove debugging information) fails when called for these two libraries:\r\n\r\n    strip --remove-section=.comment --remove-section=.note --strip-unneeded libopenblasp-r0-39a31c03.2.18.so\r\n    BFD: stRpMnp0: Not enough room for program headers, try linking with -N\r\n    strip:stRpMnp0[.note.gnu.build-id]: Bad value\r\n    BFD: stRpMnp0: Not enough room for program headers, try linking with -N\r\n    strip:stRpMnp0: Bad value\r\n\r\nSince `strip` is automatically called by `dh_strip` during package build, that lead to the build errors. Even disabling stripping for this binary package lead to errors as `dh_shlibdeps` (which generates Debian package dependencies based on ELF executable/shlib dependencies) also fails for these binaries.\r\n\r\n### Steps to reproduce\r\n\r\nIn a freshly installed Ubuntu 12.04 or Ubuntu 16.04 on amd64, e.g. inside Vagrant:\r\n\r\n    sudo apt-get install python-virtualenv\r\n    python -m virtualenv venv\r\n    source venv/bin/activate\r\n    pip install -U pip          # pip-8.1.1.tar.gz\r\n    pip install -U setuptools   # setuptools-20.10.1\r\n    pip install numpy           # numpy-1.11.0-cp27-cp27mu-manylinux1_x86_64.whl\r\n    strip --remove-section=.comment --remove-section=.note --strip-unneeded venv/lib/python2.7/site-packages/numpy/.libs/libopenblasp-r0-39a31c03.2.18.so\r\n\r\n### Possible solutions\r\n\r\n* We can (and will) try to install numpy with pip's ``--no-binary`` option in the future\r\n* Remove the wheels from older NumPy releases. The wheels were added back to 1.6.0, which was released in 2011. In our package builds we previously pinned a 6-month-old NumPy version (1.9.3), exactly to avoid such surprises. If the wheels were only added starting from 1.11.0 going forward, this wouldn't be an issue.\r\n\r\nFixing the `strip` problem alone is not a viable solution, since `dh_shlibdeps` still fails after that, because the two libraries are not native to Ubuntu.\r\n\r\n### Broader issues\r\n\r\nThe pre-built wheel is great for individual users, but it's the wrong thing to use for anyone building RPM/Debian packages for others. Even if our build continued to work, we still wouldn't want to ship generic pre-built libraries. Silently slipping this change into older releases, is going to surprise a lot of people.\r\n\r\nIt's still going to surprise people if you start doing this in new releases only, but you could communicate this change more vocally via release notes, etc. so distributors/packagers are aware of it. Ultimately, they should all move to installing NumPy with``pip install --no-binary numpy numpy``, but that's not going to happen over night."},"comment":{"url":"https://api.github.com/repos/numpy/numpy/issues/comments/214547144","html_url":"https://github.com/numpy/numpy/issues/7570#issuecomment-214547144","issue_url":"https://api.github.com/repos/numpy/numpy/issues/7570","id":214547144,"user":{"login":"yarikoptic","id":39889,"avatar_url":"https://avatars.githubusercontent.com/u/39889?v=3","gravatar_id":"","url":"https://api.github.com/users/yarikoptic","html_url":"https://github.com/yarikoptic","followers_url":"https://api.github.com/users/yarikoptic/followers","following_url":"https://api.github.com/users/yarikoptic/following{/other_user}","gists_url":"https://api.github.com/users/yarikoptic/gists{/gist_id}","starred_url":"https://api.github.com/users/yarikoptic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yarikoptic/subscriptions","organizations_url":"https://api.github.com/users/yarikoptic/orgs","repos_url":"https://api.github.com/users/yarikoptic/repos","events_url":"https://api.github.com/users/yarikoptic/events{/privacy}","received_events_url":"https://api.github.com/users/yarikoptic/received_events","type":"User","site_admin":false},"created_at":"2016-04-25T22:13:52Z","updated_at":"2016-04-25T22:13:52Z","body":"\nOn Mon, 25 Apr 2016, Christian Aichinger wrote:\n>    The virtualenv-inside-deb packaging setup is reasonably common, given the\n>    popularity of dh_virtualenv. It's especially appealing when working on\n>    older distributions (Ubuntu 12.04), where many interesting Python packages\n>    are just not available or available only in ancient versions. In our case,\n>    @yarikoptic's approach would be rather painful, as we'd have to backport\n>    tons of Python packages to 12.04.\n\nyeah -- 12.04 is quite aged by now... still, just out of interest, what\nis the project/packages?\n\nI saw dh_virtualenv lightning talk at pycon but never used it myself and\nforgot detail... in your case I would have considered then generating\nsome separate  PROJECT-virtualenv  package which then would have\nused/depend/build-depend-on on by leaf packages to use the environment\nof.  Not sure if it is available from stock dh_virtualenv functionality,\nbut it would have provided assurance that you have at least some\nspecific, versioned (packages demand versions) set of such base\nenvironments instead of generating a new one on the fly each time and\nhoping that all (network connection, etc) would be swell ;)\n\n>    It would be easier if there was some reasonable way to communicate this\n>    change to users. Having to find this out by digging through your CI\n>    failure logs after pushing a harmless change doesn't brighten anyone's\n>    day.\n\nhm... interesting point.  I saw tweets about wheels for numpy on linux\nbeing added (admiring the effort, but not carrying personally since not\nusing myself), but I wonder if there is a\npython-announcements@python.org mailing list of some kind, which\nis low-volume enough for big mass subscription and which would provide\nannouncements on big infrastructural innovations etc?  if not -- we\nshould initiate one IMHO and we both would subscribe ;)\n\n>    Fixing the stripping issue (no clue) and the shlib issue (maybe adding an\n>    appropriate relative rpath would help) would also go a long way toward\n>    solving the issue. If things continue to work, few people will complain,\n>    even if they don't get exactly what they used to get. As @matthew-brett\n>    said, we can't update the already-published wheels on PyPI, though :-(\n\n>    For reference / Google, here are the errors that dh_shlibdeps generates:\n\n>  dh_shlibdeps -ldebian/myproject//usr/lib\n>  dpkg-shlibdeps: warning: Can't extract name and version from library name `libopenblasp-r0-ea933816.2.18.so'\n\nindeed, version was completely screwed up by joining with the git\ntreeish or some other checksum lead... how was it generated, do you know\n@matthew-brett?\n\n-- \nYaroslav O. Halchenko\nCenter for Open Neuroscience     http://centerforopenneuroscience.org\nDartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755\nPhone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419\nWWW:   http://www.linkedin.com/in/yarik        \n"}}