{"action":"created","issue":{"url":"https://api.github.com/repos/dask/dask/issues/3345","repository_url":"https://api.github.com/repos/dask/dask","labels_url":"https://api.github.com/repos/dask/dask/issues/3345/labels{/name}","comments_url":"https://api.github.com/repos/dask/dask/issues/3345/comments","events_url":"https://api.github.com/repos/dask/dask/issues/3345/events","html_url":"https://github.com/dask/dask/issues/3345","id":309260562,"number":3345,"title":"CI test failures with pyarrow and hdfs","user":{"login":"jakirkham","id":3019665,"avatar_url":"https://avatars1.githubusercontent.com/u/3019665?v=4","gravatar_id":"","url":"https://api.github.com/users/jakirkham","html_url":"https://github.com/jakirkham","followers_url":"https://api.github.com/users/jakirkham/followers","following_url":"https://api.github.com/users/jakirkham/following{/other_user}","gists_url":"https://api.github.com/users/jakirkham/gists{/gist_id}","starred_url":"https://api.github.com/users/jakirkham/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jakirkham/subscriptions","organizations_url":"https://api.github.com/users/jakirkham/orgs","repos_url":"https://api.github.com/users/jakirkham/repos","events_url":"https://api.github.com/users/jakirkham/events{/privacy}","received_events_url":"https://api.github.com/users/jakirkham/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":14,"created_at":"2018-03-28T08:04:57Z","updated_at":"2018-05-08T14:44:25Z","closed_at":null,"author_association":"OWNER","body":"Seeing this on CIs. Looks like a packaging issue actually as it is missing `boost`. Maybe related to PR ( https://github.com/conda-forge/parquet-cpp-feedstock/pull/43 ).\r\n\r\n<details>\r\n\r\n```python\r\n=================================== FAILURES ===================================\r\n_________________________ test_parquet_pyarrow[hdfs3] __________________________\r\nmod_name = 'pyarrow', error_msg = '`pyarrow` not installed'\r\n    def import_required(mod_name, error_msg):\r\n        \"\"\"Attempt to import a required dependency.\r\n    \r\n        Raises a RuntimeError if the requested module is not available.\r\n        \"\"\"\r\n        try:\r\n>           return import_module(mod_name)\r\ndask/utils.py:92: \r\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \r\nname = 'pyarrow', package = None\r\n    def import_module(name, package=None):\r\n        \"\"\"Import a module.\r\n    \r\n        The 'package' argument is required when performing a relative import. It\r\n        specifies the package to use as the anchor point from which to resolve the\r\n        relative import to an absolute import.\r\n    \r\n        \"\"\"\r\n        level = 0\r\n        if name.startswith('.'):\r\n            if not package:\r\n                msg = (\"the 'package' argument is required to perform a relative \"\r\n                       \"import for {!r}\")\r\n                raise TypeError(msg.format(name))\r\n            for character in name:\r\n                if character != '.':\r\n                    break\r\n                level += 1\r\n>       return _bootstrap._gcd_import(name[level:], package, level)\r\n/opt/conda/lib/python3.6/importlib/__init__.py:126: \r\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \r\nname = 'pyarrow', package = None, level = 0\r\n>   ???\r\n<frozen importlib._bootstrap>:994: \r\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \r\nname = 'pyarrow', import_ = <function _gcd_import at 0x7f105ba66e18>\r\n>   ???\r\n<frozen importlib._bootstrap>:971: \r\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \r\nname = 'pyarrow', import_ = <function _gcd_import at 0x7f105ba66e18>\r\n>   ???\r\n<frozen importlib._bootstrap>:955: \r\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \r\nspec = ModuleSpec(name='pyarrow', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7f102ff02c50>, origin='/opt...n3.6/site-packages/pyarrow/__init__.py', submodule_search_locations=['/opt/conda/lib/python3.6/site-packages/pyarrow'])\r\n>   ???\r\n<frozen importlib._bootstrap>:665: \r\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \r\nself = <_frozen_importlib_external.SourceFileLoader object at 0x7f102ff02c50>\r\nmodule = <module 'pyarrow' from '/opt/conda/lib/python3.6/site-packages/pyarrow/__init__.py'>\r\n>   ???\r\n<frozen importlib._bootstrap_external>:678: \r\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \r\nf = <built-in function exec>\r\nargs = (<code object <module> at 0x7f102fef2c00, file \"/opt/conda/lib/python3.6/site-packages/pyarrow/__init__.py\", line 20>,...cached__': '/opt/conda/lib/python3.6/site-packages/pyarrow/__pycache__/__init__.cpython-36.pyc', '__doc__': None, ...})\r\nkwds = {}\r\n>   ???\r\n<frozen importlib._bootstrap>:219: \r\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \r\n    from pkg_resources import get_distribution, DistributionNotFound\r\n    try:\r\n        __version__ = get_distribution(__name__).version\r\n    except DistributionNotFound:\r\n       # package is not installed\r\n        try:\r\n            # This code is duplicated from setup.py to avoid a dependency on each\r\n            # other.\r\n            def parse_version(root):\r\n                from setuptools_scm import version_from_scm\r\n                import setuptools_scm.git\r\n                describe = (setuptools_scm.git.DEFAULT_DESCRIBE +\r\n                            \" --match 'apache-arrow-[0-9]*'\")\r\n                # Strip catchall from the commandline\r\n                describe = describe.replace(\"--match *.*\", \"\")\r\n                version = setuptools_scm.git.parse(root, describe)\r\n                if not version:\r\n                    return version_from_scm(root)\r\n                else:\r\n                    return version\r\n    \r\n            import setuptools_scm\r\n            __version__ = setuptools_scm.get_version('../', parse=parse_version)\r\n        except (ImportError, LookupError):\r\n            __version__ = None\r\n    \r\n    \r\n>   from pyarrow.lib import cpu_count, set_cpu_count\r\nE   ImportError: libboost_regex.so.1.66.0: cannot open shared object file: No such file or directory\r\n/opt/conda/lib/python3.6/site-packages/pyarrow/__init__.py:47: ImportError\r\nDuring handling of the above exception, another exception occurred:\r\nhdfs = hdfs://localhost:8020, Connected\r\n    @require_pyarrow\r\n    def test_parquet_pyarrow(hdfs):\r\n        dd = pytest.importorskip('dask.dataframe')\r\n        import pandas as pd\r\n        import numpy as np\r\n    \r\n        fn = '%s/test.parquet' % basedir\r\n        hdfs_fn = 'hdfs://%s' % fn\r\n        df = pd.DataFrame(np.random.normal(size=(1000, 4)),\r\n                          columns=list('abcd'))\r\n        ddf = dd.from_pandas(df, npartitions=4)\r\n    \r\n>       ddf.to_parquet(hdfs_fn, engine='pyarrow')\r\ndask/bytes/tests/test_hdfs.py:245: \r\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \r\ndask/dataframe/core.py:1045: in to_parquet\r\n    return to_parquet(self, path, *args, **kwargs)\r\ndask/dataframe/io/parquet.py:892: in to_parquet\r\n    write = get_engine(engine)['write']\r\ndask/dataframe/io/parquet.py:749: in get_engine\r\n    import_required('pyarrow', \"`pyarrow` not installed\")\r\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \r\nmod_name = 'pyarrow', error_msg = '`pyarrow` not installed'\r\n    def import_required(mod_name, error_msg):\r\n        \"\"\"Attempt to import a required dependency.\r\n    \r\n        Raises a RuntimeError if the requested module is not available.\r\n        \"\"\"\r\n        try:\r\n            return import_module(mod_name)\r\n        except ImportError:\r\n>           raise RuntimeError(error_msg)\r\nE           RuntimeError: `pyarrow` not installed\r\ndask/utils.py:94: RuntimeError\r\n------------------------------ Captured log setup ------------------------------\r\ncore.py                    137 DEBUG    Connect to handle 94361929179216\r\n=============== 1 failed, 10 passed, 13 skipped in 11.03 seconds ===============\r\nThe command \"if [[ $TEST_HDFS == 'true' ]]; then source continuous_integration/hdfs/run_tests.sh; fi\" exited with 1.\r\n```\r\n\r\n</details>\r\n\r\n<br>\r\n\r\nref: https://travis-ci.org/dask/dask/jobs/359059325"},"comment":{"url":"https://api.github.com/repos/dask/dask/issues/comments/387427095","html_url":"https://github.com/dask/dask/issues/3345#issuecomment-387427095","issue_url":"https://api.github.com/repos/dask/dask/issues/3345","id":387427095,"user":{"login":"cpcloud","id":417981,"avatar_url":"https://avatars1.githubusercontent.com/u/417981?v=4","gravatar_id":"","url":"https://api.github.com/users/cpcloud","html_url":"https://github.com/cpcloud","followers_url":"https://api.github.com/users/cpcloud/followers","following_url":"https://api.github.com/users/cpcloud/following{/other_user}","gists_url":"https://api.github.com/users/cpcloud/gists{/gist_id}","starred_url":"https://api.github.com/users/cpcloud/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cpcloud/subscriptions","organizations_url":"https://api.github.com/users/cpcloud/orgs","repos_url":"https://api.github.com/users/cpcloud/repos","events_url":"https://api.github.com/users/cpcloud/events{/privacy}","received_events_url":"https://api.github.com/users/cpcloud/received_events","type":"User","site_admin":false},"created_at":"2018-05-08T14:44:25Z","updated_at":"2018-05-08T14:44:25Z","author_association":"CONTRIBUTOR","body":"That looks screwy, and on my local machine I get matching versions."}}
{"actor":{"display_login":"cpcloud"}}