{"action":"created","issue":{"url":"https://api.github.com/repos/avocado-framework/avocado/issues/1290","repository_url":"https://api.github.com/repos/avocado-framework/avocado","labels_url":"https://api.github.com/repos/avocado-framework/avocado/issues/1290/labels{/name}","comments_url":"https://api.github.com/repos/avocado-framework/avocado/issues/1290/comments","events_url":"https://api.github.com/repos/avocado-framework/avocado/issues/1290/events","html_url":"https://github.com/avocado-framework/avocado/issues/1290","id":163836737,"number":1290,"title":"avocado use broken lzma,tarfile implemntation from python 2.7","user":{"login":"dmonakhov","id":466925,"avatar_url":"https://avatars.githubusercontent.com/u/466925?v=3","gravatar_id":"","url":"https://api.github.com/users/dmonakhov","html_url":"https://github.com/dmonakhov","followers_url":"https://api.github.com/users/dmonakhov/followers","following_url":"https://api.github.com/users/dmonakhov/following{/other_user}","gists_url":"https://api.github.com/users/dmonakhov/gists{/gist_id}","starred_url":"https://api.github.com/users/dmonakhov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dmonakhov/subscriptions","organizations_url":"https://api.github.com/users/dmonakhov/orgs","repos_url":"https://api.github.com/users/dmonakhov/repos","events_url":"https://api.github.com/users/dmonakhov/events{/privacy}","received_events_url":"https://api.github.com/users/dmonakhov/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-07-05T12:15:45Z","updated_at":"2016-07-11T08:19:25Z","closed_at":null,"body":"lzma  library is known to being broken for python 2.7 (at least python 3.4 is OK)\r\ntarfile implementation is also broken http://stackoverflow.com/questions/30302204/tar-extractall-does-not-recognize-unexpected-eof\r\n\r\nLet's temproraly switch to '/bin/xzcat' helpers as we do it in 'autotest'\r\nExample: \r\n##Usage: python2.7 ./extract.py \r\nimport lzma\r\nimport tarfile\r\n##File can be found here: https://www.kernel.org/pub/linux/kernel/v4.x/linux-4.0.tar.xz\r\n## python2.7 lzma has bug which result in in silent truncation of stream ~200Mb\r\n## so uncompressed tar stream will be also corrupted\r\nlz = lzma.LZMAFile('linux-4.0.tar.xz')\r\nt = tarfile.open(fileobj=lz)\r\n## Silently ignores corrupted tarstream and extract incomplete archive ('fs','net' directories will absent)\r\nt.extractall('.')"},"comment":{"url":"https://api.github.com/repos/avocado-framework/avocado/issues/comments/231670149","html_url":"https://github.com/avocado-framework/avocado/issues/1290#issuecomment-231670149","issue_url":"https://api.github.com/repos/avocado-framework/avocado/issues/1290","id":231670149,"user":{"login":"ldoktor","id":1074021,"avatar_url":"https://avatars.githubusercontent.com/u/1074021?v=3","gravatar_id":"","url":"https://api.github.com/users/ldoktor","html_url":"https://github.com/ldoktor","followers_url":"https://api.github.com/users/ldoktor/followers","following_url":"https://api.github.com/users/ldoktor/following{/other_user}","gists_url":"https://api.github.com/users/ldoktor/gists{/gist_id}","starred_url":"https://api.github.com/users/ldoktor/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ldoktor/subscriptions","organizations_url":"https://api.github.com/users/ldoktor/orgs","repos_url":"https://api.github.com/users/ldoktor/repos","events_url":"https://api.github.com/users/ldoktor/events{/privacy}","received_events_url":"https://api.github.com/users/ldoktor/received_events","type":"User","site_admin":false},"created_at":"2016-07-11T08:19:25Z","updated_at":"2016-07-11T08:19:25Z","body":"It seems like the `silent` part had been fixed:\r\n\r\n```\r\n>>> import lzma\r\n>>> import tarfile\r\n>>> lz = lzma.LZMAFile('linux-4.0.tar.xz')\r\n>>> t = tarfile.open(fileobj=lz)\r\n>>> t.extractall('.')\r\nTraceback (most recent call last):\r\n  File \"<stdin>\", line 1, in <module>\r\n  File \"/usr/lib64/python2.7/tarfile.py\", line 2078, in extractall\r\n    self.extract(tarinfo, path)\r\n  File \"/usr/lib64/python2.7/tarfile.py\", line 2115, in extract\r\n    self._extract_member(tarinfo, os.path.join(path, tarinfo.name))\r\n  File \"/usr/lib64/python2.7/tarfile.py\", line 2191, in _extract_member\r\n    self.makefile(tarinfo, targetpath)\r\n  File \"/usr/lib64/python2.7/tarfile.py\", line 2232, in makefile\r\n    copyfileobj(source, target)\r\n  File \"/usr/lib64/python2.7/tarfile.py\", line 266, in copyfileobj\r\n    shutil.copyfileobj(src, dst)\r\n  File \"/usr/lib64/python2.7/shutil.py\", line 49, in copyfileobj\r\n    buf = fsrc.read(length)\r\n  File \"/usr/lib64/python2.7/tarfile.py\", line 831, in read\r\n    buf += self.fileobj.read(size - len(buf))\r\n  File \"/usr/lib64/python2.7/tarfile.py\", line 743, in read\r\n    return self.readnormal(size)\r\n  File \"/usr/lib64/python2.7/tarfile.py\", line 758, in readnormal\r\n    return self.__read(size)\r\n  File \"/usr/lib64/python2.7/tarfile.py\", line 750, in __read\r\n    raise ReadError(\"unexpected end of data\")\r\ntarfile.ReadError: unexpected end of data\r\n\r\n$ pip list |grep lzma\r\npyliblzma (0.5.3)\r\n```\r\n\r\nAnyway I see your point and having `archive` library which is only able to uncompress small files is not really useful. I think we should keep the code there and add optional dependency on tar. When it's available we should use it, but when it's not available we should attempt to extract the file using python library and re-raise the possible exceptions along with information that they should install `tar` for advanced features.\r\n\r\nThere is a similar problem with `zip` as well, currently we handle symlinks/file permissions manually. So we could re-use this code and first try using `unzip` utility and when not available fall-back to standard library.\r\n\r\nI'll mention this on the avocado meeting and get back to you. Thank you for this notice!"}}
{"actor":{"display_login":"ldoktor"}}